Well, I don't completely agree. Wysz from Google's Search Quality team repeats that spam sites linking to you doesn't hurt. (Hat tip: Barry Schwartz).
"I wouldn't really worry about spam sites hurting your ranking by linking to you, as we understand that you can't (for the most part) control who links to your sites."
Uhuh. Of course. It would be way too easy otherwise to mess up Google's ranking system big time.
Spam links count for zero. Nothing gets added, nothing gets subtracted. So just as spam links are not supposed to help, they can't hurt either.
Or can they?
A much better look inside the kitchen can be seen when Barry Schwartz quotes Adam Lasnik:
"Also, be assured that we're not looking to penalize folks for a "bad" link here and there. Rather, our algorithms are tuned to look for patterns of "egregious" linking behavior... both on individual sites and in the aggregate."
And that is the real deal. "Here and there", "patterns".
As a recurrent neural network Google is able to learn about common link patterns.
There are a few sites like Adobe or Amazon whom everybody and the whole world links to, including spammers, while those sites themselves hardly link out.
The pattern is similar to that of a a link hoarder except for the quantity.
Link hoarding is not always a deliberate and manipulative SEO tactic. Most e-commerce sites are link hoarders, for example.
By the way, what is easy to forget when you work with popular sites or work within SEO or SEM is that the above two patterns are actually quite rare when you look at the web at large. For the majority of web sites out there the pattern looks more like this:
Of course there are overlap patterns. Some hugely popular sites receive a lot of inbound links while the site itself links out a lot too.
Now, given the above link patterns it is easy to spot a site which is trying to "make it" the wrong way.
The site stands out because the ratio of normal/good links to spammy links is out of balance.
Although the spam links themselves may not hurt (they represent a zero value in the larger equation), the pattern that emerges may most definitely hurt.
Your own actions in that pattern count just as much -- if not more so. Link out to, or worse link back to spammy sites and you've just closed a loop that confirms Google's suspicions.
So, in the end, I would rather say it like this:
"In general I wouldn't really worry about spam sites hurting your ranking by linking to you, as long as you fit the pattern of a normal, healthy site."