How to Block Bots using Robots.txt File in cPanel?

A robots.txt is a simple text file created in the root directory of your website, which allows search engine bots to crawl your website to index it. The robots.txt file also controls how and which files the bots should access and configuring it properly is an essential part. You can block bots using robots.txt file from indexing your website in search engines.

This article will show you how to block bots by editing your robots.txt file. Follow the below steps to get started.

1. Login to your cPanel account.
2. Open File Manager.
3. Browse File Manager to the website directory (e.g. public_html ) and Click on File >> Type in robots.txt >> Click on ‘Create New File’.
4. This code will block the bot from the Baidu search engine.
5. That’s it. Now, Baidu will not be able to bot your website anymore.

Below are more examples and syntax.

• Block all crawlers (user-agents) from accessing the logs and ssl directories.
  User-agent:*
  Disallow: /logs/
  Disallow: /ssl/
• Block all crawlers to index the whole site.
  User-agent: *
  Disallow: /
• Allow all user agents to access the entire site.
  User-agent: *
  Allow: /
• Block indexation for the whole site from a specific crawler.
  User-agent: Bot1
  Disallow: /
• Allow index to a specific web crawler and prevents indexation from others.
  User-agent: Googlebot
  Disallow:
  User-agent: *
  Disallow: /