Internet Performance Delivered right to your inbox

Effectively Combatting Malicious Bots in 2017

2017 marks the 10-year anniversary of Cisco’s declaration that “botnets are the primary security threat on the Internet today.” At the time, it was consumers’ access to broadband connections that gave botnets the ability to launch distributed denial of service (DDoS) attacks; today, unsecured Internet of Things (IoT) devices on even higher speed networks are (for now) the culprit. Despite advances in preventive and detective controls, bots remain a formidable and ever increasing threat to the integrity of business applications and information.

Bots come in multiple varieties. It’s important to distinguish between the infected hosts that make up botnets to host malware payloads (e.g. Necurs: 6M hosts), launch distributed denial of service and other attacks, from the automated programs developed to mimic human activity (i.e. to conduct fraud) or scrape content. (Granted, the same infected host can do both). Apparent fans of Mr. Robot have even made the former available as a DIY model (still in beta):

As 2016 came to a close, we learned from WhiteOps of the financial losses suffered by advertising agencies at the hands of ‘Methbot’, a sophisticated bot farm developed to “masquerade as engaged human consumers” using a custom browser to generate impressions on ad content. While the extent of losses has been disputed (Nolet, O’Kelley), click fraud remains a profitable (albeit, unethical) enterprise.

Not only were the Mirai (September) and Leet (December) botnets stark reminders of the work ahead to properly secure IoT devices, they established a whole new scale for botnet-driven DDoS attacks. The highly publicized attack on Dyn was perpetrated by Mirai-infected hosts that led to downtime or lag for many popular sites, such as Netflix, CNN, and Twitter. In 2017, we expect similar size or larger disruptive attacks until concerted efforts are made to secure devices out of the box or automatically manage such traffic further upstream. With the Mirai source code available in the wild, it is likely a matter of time before more bad actors make use of it to make new legions out of the thousands of IoT devices shipped daily. Businesses cannot afford to wait for the problem to be fixed by others: In 2017, we are likely to see the accelerated industrialization of crimeware-as-a-service, to include all manners of bots.

Why are bots a serious concern for your web applications?

Bots continue to exploit normal business logic functions, developed to allow legitimate users complete authorized actions, such as request resources. Sites that offer information that might be considered sensitive in totality, such as pricing information or proprietary directories, are especially susceptible to bot traffic. A competitor might wield a bot farm to scrape prices from an airline to dynamically price similar products a few dollars cheaper to gain a market advantage. A malicious actor might use a botnet to seek out vulnerabilities in website technology stacks. Bots were a key focus of the 2016 election as the spread of fake news and comments on social media and news sites was attributed to bot activity. These are classic bot use cases, and ones we expect to see more of in 2017.

Without any added technology, there are many measures available to developers and site owners to counter the threat of bots. Maintaining current versions of CMS, plug-ins and other site components is chief among them. Depending on how the back end is built, you can prevent responses to requests with anomalous headers. Enable authentication whenever possible and leverage trusted open source code like Google’s reCAPTCHA as an added security measure. For those supported by a MSSP or with outsourced website management, it’s important to ask how bots – especially those responsible for Layer 7 DDoS attacks – are singled out and blocked.

Oracle Dyn Bot Management and Mitigation

Bot management is a key component of Oracle Dyn’s Web Application Firewall. We’ve built out a number of response handlers to mitigate scraping and various Layer 7 DDoS attacks. Today customers can leverage CAPTCHA challenges, user agent rate limiting, JavaScript challenges, and access rules to manage good and bad bots. And we’ve recently expanded our capability set to include new challenges given the increasing sophistication of botnets today. Our human interaction challenge and device fingerprinting address the bots that might not meet traditional hard-coded request thresholds and are JS aware, to include toolkits like Selenium or PhantomJS (which have legitimate purposes) and custom plugins. In 2017, Oracle Dyn is leveraging machine learning to take on bots that are capable of bypassing traditional rate limiting and responding to JavaScript challenge queries.

Deploying countermeasures against click-fraud on a scale similar to Methbot is probably futile. One can deploy more and more sophisticated countermeasures, but it’s just a matter of time until the bad guys collect enough data and are able to replicate real user behavior with enough accuracy that your human hand-written detection algorithm won’t be able to tell the difference. This is one of many reasons why Oracle Dyn has leveraged artificial intelligence and machine learning. ML models can detect patterns and anomalies, by inspecting far more data than any human can and developing a much more sophisticated model, that will be virtually impossible for an attacker to overcome.

In Conclusion

Bot mitigation services are only one component of holistic website security. The WAF as a platform provides great insight into the requests made to protected web applications; in the past, we’ve been able to trace bot communication back to source servers to determine what was being collected (airline fares) and how. With a WAF in place, you’ll have much greater visibility of the kind of traffic reaching your web properties as well as the ability to block malicious activity – from bots and humans.

Share Now

Whois: Nick Deshpande

Nick Deshpande is a Principal Product Strategist at Oracle Dyn Global Business Unit, a pioneer in managed DNS and a leader in cloud-based infrastructure that connects users with digital content and experiences across a global internet.