Owners of Russian-language Internet resources and webmasters should already be aware of news from Yandex regarding the robots.txt file, which has to do with indexing the site.
Previously, it was necessary to correctly write the host’s directive in the robots.txt file, which we recommended to our customers in the absence or incorrect specification. But since it does not matter to Google, and since 20.03.2018, Yandex himself on his blog announced that he refuses this directive, while the main important condition remains that there should be 301 paged redirects on all existing NOT the main mirrors of your site.
If you decide for some reason to leave the host in robots.txt in the old fashioned way or just for the time being afraid to take such a step, this will not be a problem, the point is that the robot will only ignore it, in this case it simply no role will not play. Therefore, we just recommend that you properly prescribe what should be and do not add anything extra.
Despite this, some webmasters manage even in such a simple but necessary robots file to show their creativity and make it really unique. A look in the file of every well-known company Nike…
Not that we put this as an example for imitation, just be inspired and be the best in your business.