Internet Performance Delivered right to your inbox

What is Bot Management?

Bot management is a feature of application security software that identifies whether a traffic request comes from a human or from a machine and then controls or blocks non-human and other suspicious requests.

Roughly half of the traffic on the internet comes from bots, which are bits of code created to perform automated tasks. There are good bots, such as those that index content for search engines, making it easier for people to find websites. But there are also malicious bots that exploit vulnerabilities, expose data, shut down entire websites, and steal intellectual property. As such, bot management is a crucial part of modern website security.

Benefits of bot management software

Bot managers enable organizations to reap the benefits of good bots while protecting themselves against malicious bots.

Some machine traffic, such as that from Google’s bots, is fine. Organizations obviously want Google bots to access every one of their pages and index them in search results. But they don’t want bad bots or unknown bots trying to access their sites, which are supposed to be for humans.

Bot management software classifies incoming requests into two buckets: humans or good bots, and unknown or malicious bots. It treats unknown bots the same as malicious bots, because organizations can’t afford to assume something unknown has good intentions. The software directs malicious and suspect bot traffic away from the site, protecting data and preventing detrimental effects on the end-user experience.

This video explains the different types of malicious bots.

Additionally, just because a request comes from a good bot or a human, that doesn’t mean it contains no attack payload. A web application firewall would look for that. For that reason, bot management must be part of a bigger web application security platform that vets every traffic request.

How bot management works: Detection and mitigation

There are a few different bot management features for detection and mitigation, and they have different levels of sophistication.

Basic capabilities include IP rate limiting, which prevents too many requests from the same address from overloading the server, and CAPTCHA challenges, which present a puzzle the user must solve before accessing the application.

Some bot detection techniques are based simply on the features and capabilities of the machine that is making the request. The simplest, a JavaScript challenge, is linked to the end user’s browser. When a person uses a browser such as Apple Safari, Google Chrome, or Microsoft Edge, we know that browser has a JavaScript engine and other specific features that are necessary for almost any application today. If the browser doesn’t have a JavaScript engine and the like, then it is an almost certainty that the end user is a bot.

bot mitigation
This Oracle Dyn Web Application Security screenshot shows an example of blocked malicious traffic and the methods used to identify the requests.

This bot mitigation technique is very efficient, because it’s expensive for hackers to simulate a full internet browser. This is especially true when they launch a DDoS attack, because it’s even more expensive to run full internet browser simulations inside of a botnet. It’s not impossible, however, so there needs to be other bot management features at organizations’ disposal.

Second-generation bot management features are much more sophisticated. They analyze the behavior of the user to identify whether it is human or not. Malicious bots are getting better at replicating human behavior, however.

The next generation uses supervised machine learning techniques and artificial intelligence engines to classify whether requests are from humans or bots on a per-application basis. Early adopters of this technology are typically large enterprises with strict security and compliance requirements.

bot management options
This Oracle Dyn Web Application Security screenshot shows how organizations can customize their bot management software’s detection, alerting, and blocking capabilities.

Bot management use cases

The main use case of bot management software is to identify and protect websites and applications from malicious bot traffic. The four main types of bot attacks are:

  • vulnerability scans and exploits;
  • network-based DDoS attacks;
  • application layer DDoS attacks; and
  • fraudulent purchases, content scraping, and other malicious site activity.

But even when the bot traffic is not malicious, there are times when a site might want to prioritize human traffic over bot traffic. Bot management helps with that as well.

Oracle Dyn has a customer, a rental car company, that has good bot traffic coming to its site from travel agencies that want to check prices and show them to their customers. The rental car company is fine with this traffic but uses our bot management technology to prioritize human traffic over bot traffic, because a human coming to the site to rent a car has a direct impact on business.

blocked bot traffic
This Oracle Dyn Web Application Security screenshot shows how bot management software can block or delay bot traffic when its requests exceed a certain threshold.

The bot management software identifies which traffic is from bots, then takes action based on what the traffic is trying to do and what the company wants to allow it to do.

Getting started with bot management

Proof-of-concept deployments allow organizations to see all the bots on their networks. A lot of the time, organizations are not aware of what is happening to their applications, and so this bot detection is an eye-opener.

Share Now

Laurent Gil
Whois: Laurent Gil

Laurent Gil is a security product strategy architect at Oracle Cloud Infrastructure. Previously, Laurent was the cofounder of Zenedge (acquired by Oracle in March 2018) and CEO and cofounder of Ukraine-based Viewdle, which focused on machine learning and computer vision (acquired by Google in 2012).