Getting lots of requests from crawler and bots, which doesn’t add value to your business?
There are thousands of crawlers/bots visit your site every day, and very few are helpful. Some of them are considered bad bots or spam.
How do you know what all bots are visiting your websites?
Well, there is no easy answer. To find out, you need to review your web server access.log file and look for User-Agent column. Let’s say you want to list all the bots except Googlebot then you may run the following command on your web server where the access.log file exists.
grep bot access.log |grep -v GooglebotAre you surprised to see so many?
I was too when I checked my logs.
root@gf-prod:nginx# grep bot access.log |grep -v Googlebot | wc -l
616834
root@gf-prod:nginx#Before you block anything, you need to review them to ensure you are not blocking, which may be required for your business. And, there are many ways to get this done, but I always prefer to block them at the edge.
The reason is simple – why let the requests hit the web server when you don’t need them at all.
Like the headline, let’s find out how to block the bots you don’t need using Cloudflare firewall.
Note: if you are running a large to enterprise business then you may be interested in Cloudflare bot management service.
- Login to Cloudflare
- Go to the Firewall tab and then firewall rules and create a firewall rule
- Enter the rule name
- Select Field as User Agent, Operator as contains and value as the bot name you want to block
- Use the or condition to add multiple bots in a single rule
Note: the bots mentioned in the above image is for illustration only. Not necessary, they are bad bots.
- and then select action as block and deploy
- If you know the expression then you may also write it by clicking edit expression instead of following the GUI. Once deployed, you should see the newly created rule in the list, and status should be ON to be effective.
Easy, isn’t it?
What else can you do with the Firewall rules?
Well, a lot more for better security. Let’s see the following blocking conditions.
- If a request is coming from a particular ASN, IP address
- Matching cookie, referrer, X-Forwarded-for keywords
- Restrict serving requests to the entire country
- Disable unwanted HTTP method like PUT, DELETE, OPTIONS, PURGE, etc.
And more options…
All you can do it either through GUI builder or write your expression. Applying changes are nearly instant.
Conclusion
Cloudflare firewall rules is a great way to add protection for your web applications at the network edge without downtime. If not already, you may also want to consider using Cloud WAF for better application security and protection from DDoS and other online vulnerabilities.
