When the search engine looks at your website, the first thing they view is the robot.txt file, this will indicate the search engine in which information needs to be made available for Web Host users and which should not be. They also help to give the view of XML sitemap locations and indicate the robots.txt.file as “Bot” or “Robot” or “Spider” to recognize.
Web Host With A Robot
The Google bot is referred to as Googlebot, Microsoft bing bot as Bingbot and other search engines also will have their own bot names. The robots.txt acts like a gateway and decides which can be accessed and blocks other accesses. The robots.txt files will be available in your own root folder, when you name an empty file as robots.txt it will reduce the website error that is occurred and rank the website on their own.
When you want to block the access of other users to your website and do not want to get ranked then you can use the code. In this code “*” blocks all the robots and “/” block the access for entering the website by search engines.
#code not to allow any search engine!
Some parts of your website are allowed to crawl, while other parts can also be blocked. For example, when you want to block the access for the cgi-bin folder, temp folder, and junk folder, you can use the below code on your website.
#Blocks Robots from specific folders/directories
To allow only one file from many others. For example, if you want to exclude stuff.
If you want your website to get listed in the search engines, then it is important to remove the”*” and “/” which allows search engines to enter your website and get ranked. The below steps help to edit the robots.txt file.
- Login the cPanel
- Enter into the file manager and go to your website root directory
- The robots.txt must be present in the same location as mentioned in your index file of the website
- Go to EDITOR that will be present in the menu
- Do the changes and saves changes in the right corner