In the 21st century, WordPress has established itself as one of the most beginner friendly content management system, mainly because it allows people to create and manage their website free of cost with very little technical expertise. One other reason why WordPress has become so popular is because it provides important features such as adding WordPress robots.txt file, enabling individuals to have much better control over their website.
WordPress offers its users many such important features which make their life much easier. However, the WordPress robots.txt file is one such aspect which you have to pay close attention to while making a website.
This is what we shall discuss in this article. We will explain what a robots.txt file is, what it does, where you can find it, and various ways in which you can add robots.txt file in WordPress and edit it.
What is a use of WordPress robots.txt file?
Whenever you create a website it is very important to keep its SEO ranking in the SERPs in mind. You always want your website to be ranked higher for your targeted keywords. In order to rank a website a search engine has to first index the website by crawling through all of the data present in it. These crawlers are programmed bots or minions which are developed by a search engine to scan through the contents of a website in order to display the most relevant results whenever a keyword search is made.
The search engine bots by default will go through the whole website in order to present it n the search result. However, this is not something you want all of the time, as your website may contain some irrelevant or sensitive information such as log in information, which you would not want the bot to scan through.
Where does robots.txt go on a site?
A robots.txt file guides the crawlers to avoid going through the unwanted or private pieces of information on your website. This enables it to scan it faster and ranking it higher for the targeted keywords. It also has increases the loading speed of your website as a slow loading website is very unfavourable.
Although, it should be noted that by disallowing a bot to access certain section of your website, you are merely requesting the bot to not waste its resources on that particular section. It doesn’t guarantee that it won’t go looking even after disallowing it. Instead if restricting access is your sole purpose, then you should be using the Meta no-index tag or a different direct approach.
Locating a WordPress robots.txt file
WordPress automatically provides a robots.txt file with default guidelines which can be found in the root directory of your website. You can locate this file by typing /robots.txt after the name of your website and it should take you to the file.
Although, the WordPress robots.txt file created is a virtual file and hence you can edit it. You can however create a physical file on your server, and add it to your website, so that you can influence it as per your needs.
How we can create and add robots.txt file?
You can create a robots.txt file either by FTP or with the help of a plugin. There most popular techniques for creating and maintaining WordPress robots.txt file have been explained below:
- Creating and editing a robot.txt file through FTP
This is can be termed as the traditional or primitive way of creating and editing a physical robots.txt file on the server. Open any text editor of your choice, for instance Notepad, and create and empty file and save it as a ‘.txt’ document named “robots”. Then connect your website through SFTP and upload the file in your root directory. You can edit the file from this location later on, or simply upload a new file.
- Using Yoast SEO Plugin
If you have created your website using WordPress, you might have probably heard about the Yoast SEO plugin. It is one of the most popular and efficient plugin for all sorts of SEO tasks on WordPress. It is very easy to use and provided different capabilities, such as optimizing the content, targeting the keywords as well as managing the robots.txt file.
You can create the robots.txt file in Yoast SEO plugin, by clicking on File Editor in the SEO> Tools tab. There you will find a ‘Create a robots.txt file’ button. Once the file is created, you can edit it from the same window and save the changes.
- Using All in One SEO Plugin
This is another popular SEO plugin for WordPress which brings a lot of features and accessibilities to your fingertips. All you have to do is, install and open the plugin, and in its main interface you will find the ‘Feature Manager’ tab where you can activate the robots.txt feature. Then you can manage, edit, and create the WordPress robots.txt file right from the plugin interface.
Things to remember while creating a WordPress robots.txt file
While most websites keep their WordPress robots.txt file settings to default, you can take advantage of it by making the required changes and give your website the competitive edge over the others.
- User- Agent
The first thing to keep in mind while editing a robots.txt file is to know which bot you want to target. You can do so through the user agent command. The User-Agent commands dictates which bot the following rules or restrictions are for.
By default, WordPress robots.txt file puts a ‘*’ which implies that the rules or guidelines are standard for all bots. As each search engines has a different ranking algorithm, website owners use different guidelines for each bot in order to maintain the highest SEO rankings.
- Disallow
This is the command that tells the bots to not spend its resources and time on crawling through a particular directory. There is a corresponding Allow command but it is not much used since a bot is allowed to go anywhere by default unless told otherwise. After specifying which bot the rules apply to via the User Agent command, you are then required to add the needed rules through the disallow command.
To disallow a bot to crawl through a particular section, you have to type in the path or directory of that location in front of the Disallow command. A ‘/’ in disallow command with no directory implies a bot is not supposed to crawl any parts or pages on your website.
Limitations of the robots.txt file
Robots.txt does have some of its drawbacks and shortcomings which you must understand while creating your website, in order to make informed decisions regarding your privacy and making amends in your website. The various limitations in a WordPress robots.txt file are:
- It is not a means to hide a webpage from google search results
As mentioned earlier, by creating a robots.txt file you’re simply requesting a search engine crawler to avoid looking into it. It may still appear in google search results and the bots may still crawl through the web page, as there is no rule against it.
- Maximum size for a robots.txt file is 500Kb
As per current guidelines by google, the maximum size for a robots.txt file allowed by google is 500KB.
- Each web crawler bot is different
Different crawler bots are programmed to interpret the robots.txt syntax differently, hence the file used for google crawlers may not be as efficient for a bot by Bing! Try to learn the syntax followed by the bot you want to target in your robots.txt file in order to make sure readability of the directives.
- A disallowed page can still be indexed in google if it is linked to from other websites
Even if you have disallowed a URL and the bot follows the directives, it can still appear in search results and be indexed if the webpage has been linked to from different sites on the internet. This is way the contents of the webpage will be exposed. To avoid that, use a password or a noindex meta tag.
In conclusion, a WordPress robots.txt file is a very important file for your website as it can not only protect your private information from being revealed, it can also achieve better SEO results by keeping your website easy to scan through for the search engine bots.
