Bad bots are used by competitors, hackers and fraudsters and are the key culprits behind web scraping, brute force attacks, competitive data mining, online fraud, account hijacking, data theft, spam, digital ad fraud and downtime. In 2017, bad bots accounted for 21.8% of all website traffic, a 9.5% increase over the previous year. Good bots increased by 8.7% to make up 20.4% of all website traffic.
According to Distil Networks’ fifth annual Bad Bot Report, which details the analysis of hundreds of billions of bad bot requests at the application layer, gambling companies and airlines suffer from higher proportions of bad bot traffic than other industries, with 53.1% and 43.9% of traffic coming from bad bots, respectively. E-commerce, healthcare and ticketing websites meanwhile suffer from highly sophisticated bots, which are difficult to detect.
A full 83.2% of bad bots report their user agent as web browsers Chrome, Firefox, Safari or Internet Explorer; 10.4% claim to come from mobile browsers such as Safari Mobile, Android or Opera.
Additionally, 82.7% of bad bot traffic emanated from data centers in 2017, compared to 60.1% in 2016. The availability and low cost of cloud computing explains the dominance of data center use.
“This year bots took over public conversation, as the FBI continues its investigation into Russia’s involvement in the 2016 US presidential election and new legislation made way for stricter regulations,” said Tiffany Olson Jones, CEO of Distil Networks. “Yet as awareness grows, bot traffic and sophistication continue to escalate at an alarming rate. Despite bad bot awareness being at an all-time high, this year’s Bad Bot Report illustrates that no industry is immune to automated threats and constant vigilance is required in order to thwart attacks of this kind.”
For the first time, Russia became the most blocked country, with one in five companies (20.7%) implementing country-specific IP block requests. Last year's leader, China, dropped down to sixth place with 8.3%.
In terms of tactics, the analysis found that account takeover attacks occur two to three times per month on the average website, but immediately following a breach, they are three times more frequent, as bot operators know that people reuse the same credentials across multiple websites.
About 74% of bad bot traffic is made up of moderate or sophisticated bots, which evade detection by distributing their attacks over multiple IP addresses or simulating human behavior such as mouse movements and mobile swipes.
Also, bots can be distributed on multiple hosts to perform automated distributed denial of service (DDoS) but can also be "low and slow," use browser automation or other evasion techniques to bypass existing web application security controls, such as IP blacklisting and rate limiting.