Disabling Automatic Creation of the robots.txt File
On shared and reseller servers, if a robots.txt file is missing in the document root (public_html), the server automatically generates one at midnight. This guide explains how to disable this behavior.
Steps to Disable Automatic robots.txt Generation
-
Log in to cPanel
If you are unsure how to log in, refer to the cPanel login documentation. -
Open File Manager
In the FILES section of the cPanel home screen, click File Manager. -
Navigate to the Document Root
In the left sidebar, select thepublic_htmlfolder. -
Create a Special File
-
Click + File on the top menu bar.
-
In the New File Name text box, type:
robots.txt.ignore -
Ensure that the file will be created in
/public_html. -
Click Create New File.
-
Once this file exists, the server will no longer automatically generate a robots.txt file in the public_html directory.
For more information about robots.txt, visit http://www.robotstxt.org.
