Geekflare is supported by our audience. We may earn affiliate commissions from buying links on this site.
In Security Last updated: June 29, 2019
Share on:
Invicti Web Application Security Scanner – the only solution that delivers automatic verification of vulnerabilities with Proof-Based Scanning™.

Getting lots of requests from crawler and bots, which doesn’t add value to your business?

There are thousands of crawlers/bots visit your site every day, and very few are helpful. Some of them are considered bad bots or spam.

How do you know what all bots are visiting your websites?

Well, there is no easy answer. To find out, you need to review your web server access.log file and look for User-Agent column. Let’s say you want to list all the bots except Googlebot then you may run the following command on your web server where the access.log file exists.

grep bot access.log |grep -v Googlebot

Are you surprised to see so many?

I was too when I checked my logs.

root@gf-prod:nginx# grep bot access.log |grep -v Googlebot | wc -l

Before you block anything, you need to review them to ensure you are not blocking, which may be required for your business. And, there are many ways to get this done, but I always prefer to block them at the edge. 

The reason is simple – why let the requests hit the web server when you don’t need them at all. 

Like the headline, let’s find out how to block the bots you don’t need using Cloudflare firewall.

Note: if you are running a large to enterprise business then you may be interested in Cloudflare bot management service.

  • Login to Cloudflare
  • Go to the Firewall tab and then firewall rules and create a firewall rule

  • Enter the rule name
  • Select Field as User Agent, Operator as contains and value as the bot name you want to block
  • Use the or condition to add multiple bots in a single rule

Note: the bots mentioned in the above image is for illustration only. Not necessary, they are bad bots.

  • and then select action as block and deploy
  • If you know the expression then you may also write it by clicking edit expression instead of following the GUI. Once deployed, you should see the newly created rule in the list, and status should be ON to be effective.

Easy, isn’t it?

What else can you do with the Firewall rules?

Well, a lot more for better security. Let’s see the following blocking conditions.

  • If a request is coming from a particular ASN, IP address
  • Matching cookie, referrer, X-Forwarded-for keywords
  • Restrict serving requests to the entire country
  • Disable unwanted HTTP method like PUT, DELETE, OPTIONS, PURGE, etc.

And more options…

All you can do it either through GUI builder or write your expression. Applying changes are nearly instant.


Cloudflare firewall rules is a great way to add protection for your web applications at the network edge without downtime. If not already, you may also want to consider using Cloud WAF for better application security and protection from DDoS and other online vulnerabilities.

  • Chandan Kumar
    Chandan Kumar is the founder of Geekflare. He’s helped millions to excel in the digital realm. Passionate about technology, He’s on a mission to explore the world and amplify growth for professionals and businesses.
Thanks to our Sponsors
More great readings on Security
Power Your Business
Some of the tools and services to help your business grow.
  • Invicti uses the Proof-Based Scanning™ to automatically verify the identified vulnerabilities and generate actionable results within just hours.
    Try Invicti
  • Web scraping, residential proxy, proxy manager, web unlocker, search engine crawler, and all you need to collect web data.
    Try Brightdata
  • is an all-in-one work OS to help you manage projects, tasks, work, sales, CRM, operations, workflows, and more.
    Try Monday
  • Intruder is an online vulnerability scanner that finds cyber security weaknesses in your infrastructure, to avoid costly data breaches.
    Try Intruder