Skip to Main Content

WordPress has announced a change in the way it will block search engines from indexing website location. This change abandons the traditional Robots.txt solution in favor Robots Meta Tag access. The reason for this is the desire that blocked pages don't really show up Googlesearch results.

This is Robots Meta Tag  which will the new version WordPress-a use:

It has long been a common practice to Robots.txt the file is used to block indexing website locations.

The word indexing means search website locations by GoogleBota. Based on this search, Google understands the content of the site and indexes the pages. Using the lock function Robots.txt you could have stopped Google to download the said page and, presumably, to Google will not be able to display your pages in search results. But a directive robots.txt it just prevented Google to search a page. Google was still free to add it to his index if he could find out URL.

WordPress has announced a change in version 5.3 

Starting with version 5.3, WordPress has announced an important change in the way it will block search engines from indexing websites. This change abandons the traditional Robots.txt solution in favor of the Robots Meta Tag approach. a more reliable approach will be adopted with Robots Meta Tagwhich prevents indexing website locations. This change is a great improvement and help for everyone who does not want certain parts of their site to be indexed. Users WordPress-and they can be safer when they know they are blocked website pages will not be displayed in Googlesearch results.

WordPress relied on Robots.txt as a standard way to block indexing website locations that everyone used. Yet, as explained, it was an unreliable approach.

The most effective method to exclude development sites from browser indexing is to include HTTP Header X-Robots-tag: noindex, nofollow when uploading images, PDF files, videos and other content.

The release of the new version has been announced for November 2019. See the official announcement of WordPress here.

Made by Nebojsa Radovanovic - SEO Expert @Digitizer