In this short but very important article, we want to tell you the difference between Robots metatag and Robots.txt file so that you can choose the best one in different situations. But before reading this article, it is recommended that you read the robots.txt article and the robots metatag well. Be with Seo Taching
The difference between a Robots metatag and a Robots.txt file
If a robot wants to index a page from a website, it first sees robots.txt and follows its instructions. Naturally, the robots.txt file takes precedence over the robots meta tag.
The robots.txt file is a real file at www.domain.com/robots.txt. But there are also metatag robots in the form of tags before the head, as in the example below.
<META NAME = “ROBOTS” CONTENT = “NOINDEX, NOFOLLOW”>
With a robots.txt command you can block many pages for bots and search engines, but if you want to do this with a robots meta tag, you must also include a meta tag on each of those pages.
In the robots.txt file, you can introduce the sitemap site to bots and search engines, but not with meta tags.
The robots metatag commands are also more flexible than the robots.txt file.
Important: In the robots.txt file, we tell the robot not to crawl, but sometimes it is indexed, while in order not to be indexed, it is better to use the robots metatag.
[gap] [icon type = ”vector” icon = ”momizat-icon-question” align = ”right” size = ”32 ″ icon_color =” # ffbd23 ″ hover_animation = ”border_increase”]
Question: Is there an SEO problem if Use both methods, the Robots metatag with the robots.txt file to prevent a single page from being indexed from the site?
Absolutely not. Google has also written a text on this subject, noting that:
The Google crawler does not crawl or index content blocked by robots.txt, but we may find links to that content elsewhere on the web, making the content available to the public and appearing in Google search results. Be. You can use other methods to block URLs to prevent content from appearing in Google results. For example, encrypt that address through your server or use the robotx meta tag with the noindex command.