Getting lots of requests from crawler and bots, which doesn’t add value to your business?
There are thousands of crawlers/bots visit your site every day, and very few are helpful. Some of them are considered bad bots or spam.
How do you know what all bots are visiting your websites?
Well, there is no easy answer. To find out, you need to review your web server access.log file and look for User-Agent column. Let’s say you want to list all the bots except Googlebot then you may run the following command on your web server where the access.log file exists.
Before you block anything, you need to review them to ensure you are not blocking, which may be required for your business. And, there are many ways to get this done, but I always prefer to block them at the edge.
The reason is simple – why let the requests hit the web server when you don’t need them at all.
Like the headline, let’s find out how to block the bots you don’t need using Cloudflare firewall.
Go to the Firewall tab and then firewall rules and create a firewall rule
Enter the rule name
Select Field as User Agent, Operator as contains and value as the bot name you want to block
Use the or condition to add multiple bots in a single rule
Note: the bots mentioned in the above image is for illustration only. Not necessary, they are bad bots.
and then select action as block and deploy
If you know the expression then you may also write it by clicking edit expression instead of following the GUI. Once deployed, you should see the newly created rule in the list, and status should be ON to be effective.
Easy, isn’t it?
What else can you do with the Firewall rules?
Well, a lot more for better security. Let’s see the following blocking conditions.
If a request is coming from a particular ASN, IP address
Disable unwanted HTTP method like PUT, DELETE, OPTIONS, PURGE, etc.
And more options…
All you can do it either through GUI builder or write your expression. Applying changes are nearly instant.
Cloudflare firewall rules is a great way to add protection for your web applications at the network edge without downtime. If not already, you may also want to consider using Cloud WAF for better application security and protection from DDoS and other online vulnerabilities.
Chandan Kumar is the founder of Geekflare. He’s helped millions to excel in the digital realm. Passionate about technology, He’s on a mission to explore the world and amplify growth for professionals and businesses.