Over one third of web traffic in 2018 was driven by bots, according to a new report from Distil Networks, a Virginia-based bot detection and API security specialist.
Over 20 percent of that figure was “bad bots”, Distil said (as opposed to benevolent bots like search engine spiders); a figure that’s down 6.4 percent on 2017.
The company cited Amazon as the source of most global bad bot traffic at 18 percent; up from 10/6 percent in 2017. The vast majority of bad bot traffic came from the US, with the Netherlands in second place; figures backed up by recent honeypot findings.
Read this: A Tale of Two Honeypots: From Telnet to the Cloud
An alarming 76 percent of bad bots were classified by Distil Networks as Advanced Persistent Bots (APBs), which can cycle through random IP addresses, enter through anonymous proxies, change their identities and mimic human behavior.
(A bot is, crudely, an automated software script/application used to perform repetitive tasks online. These can run from botnets comprising a vast network of compromised online devices; ranging from desktops to “smart” fridges or cameras.)
The Open Web Application Security Project (OWASP) recognises 21 specific automated threat events that bots can deploy, ranging from card cracking (identifying missing security codes for stolen payment cards by trying different values) to denial of inventory, via scraping, scalping, skewing and sniping.
Recognition of the problem led Forrester in Q3 2018 to run its first ever review of bot management providers, singling out Akamai Technologies, Alibaba Cloud, Cloudflare, DataDome, Distil Networks, Oracle Dyn, PerimeterX, Reblaze, ShieldSquare, Stealth Security, Unbotify, and White Ops as the top 12.
Content from our partners
“Bot operators and bot defenders are playing an incessant game of cat and mouse, and techniques used today, such as mimicking mouse movements, are more human-like than ever before,” said Tiffany Olson Kleemann, CEO of Distil Networks.
“As sophistication strengthens, so too does the breadth of industries impacted by bad bots. While bot activity on industries like airlines and ticketing are well-documented, no organization – large or small, public or private – is immune.”
Bad Bots: Financial Services Most Attacked
Bad bots are used by competitors, hackers and fraudsters and are the key culprits behind account takeovers or hijacking, web scraping, brute-force attacks, competitive data mining, transaction fraud, data theft, spam, digital ad fraud and downtime.
Financial services were most targeted by such traffic, Distil found.
Distil recognises four categories of bad bots, ranging from “simple” bots that connect to sites using automated scripts rather than browsers; “moderate” bots that use headless browser software that simulates browser technology— including the ability to execute JavaScript—and “sophisticated” bots capable of reproducing mouse movements and clicks that fool even sophisticated detection methods, using malware installed within real browsers, to connect to sites. (Fourth was APBs).
Some bad bot problems run across all industries while others are industry-specific, Distil notes. Websites with login screens are hit by bot-driven account takeover attacks two to three times per month. Content and price scraping is rampant and is undertaken by bots. Meanwhile, nefarious competitors use bots to undercut prices on ecommerce sites, hoard seats on airline flights, and scalp the best concert tickets.
The weaponisation of the data centre continues, with almost three-quarters of bad bot traffic coming from data centres, but the figure fell from 2017. The number of bad bots running from residential devices meanwhile grew sharply from 14.8 percent to 22.7 percent, amid a proliferation of smart, but not secure, devices.
