Everything about robots

MetaTag is the best way to prevent your site from being indexed

What is a robots meta tag and how can it be set up? How to noindex a page or site with a robot meta tag? In other words, how to remove a page from our site from Google results (Google index)? In this very useful  SEO teaching article, we intend to provide you with important and useful explanations along with 4 important examples of robot metatags. Be sure to read the following article carefully.

Note: The meta-tags “robots”, “robot”, “robot” and “robot” all have the same meaning, but it is suggested to remember in the form of robots that this type of writing is more principled.

Also, if we use the term “Noindex metatag” or “noindex metatag”, we mean the same robots metatag that has noindex value and we do not have a Novindex metatag in practice, but due to the habit of some users, it may be used.

 

The importance of robots meta tags in SEO

Every day in the world of SEO, various methods are used to optimize and increase the site’s ranking in search engines, especially Google, but one of the old methods that have a good effect on search engine visibility is the optimal use of meta tags on site pages. One of the most important meta tags in SEO is the robots meta tag.

Using robots meta tags, you can explain a series of important commands to search engines, and sometimes you have to enter a specific meta tag for each search engine to refer to these commands. The robots meta tag is considered as an internal SEO of the site because its changes and actions on the pages of the site are completely in the hands of the administrator or SEO consultant of that site.

What is a meta tag for robots? With the robots meta tag you can tell search engine robots not to index this page or site or follow the links of this page or site, and vice versa, even if you use the command <META NAME = “ROBOTS” CONTENT = “INDEX , FOLLOW “> Do not use By default, Google will index the page and follow the links.

What does indexing or following mean?

Index means that Mr. Robot, you have the right to check this page and search in the results, and now the reverse index is noindex, which indicates otherwise.

Follow means that Mr. Robot has the right to check the links on this page and follow it if it is not a nofalo link. And vice versa means nofollow.

There are several ways to prevent the site from being indexed, but the best and safest way to prevent and close the site pages from Google results is the html meta tag.

Of course, we will explain more in the following, and on the other hand, more commands can be used through this metatag, which we will discuss in detail with an example in the continuation of the SEO teaching article.

Example of a robots metatag

<head>
<META NAME = “ROBOTS” CONTENT = “NOINDEX, NOFOLLOW”>
</head>

In this type of metatag, we basically have two important types of attributes.
The first attribute is called name, which most often takes the value of robots, except for a specific robot, which we will explain in more detail below. The second Content attribute also takes different values ​​to give commands to search engine robots, which are as follows.

 

Types of robots metatag commands

Noindex command

With the noindex command (basically) we tell search engine bots not to crawl or index this page. Note that if you are fully aware that your pages are not indexed, use the noindex command, for example, if you have duplicate content and so on. Otherwise it is better not to use.

Index command

The index command means access permissions to index radar. The noindex command works the opposite.

Nofollow command

The nofollow command is similar to the rel = “nofollow” attribute in the a tags, which are called nofollow links. Now the nofollow command in a one-page meta tag means that none of the links are followed by search engine bots, but the nofollow link is only used for one link. If you want to read about nouveau riche links, we suggest you read the interesting nouveau riche article.

Follow command

Informs search engines to follow the links on the page, as opposed to the nofollow command.

Command none

This command is equivalent to the noindex and nofollow commands, which means do not index the site and its links. You can use either the noindex, nofollow commands or the none command to do this. Although generally not recommended, use these commands with full awareness.

Noimageindex command

Search engine spiders (the same as robots) will not be allowed to access page images with this command. Of course, if a photo is linked elsewhere, the search engine can index it.

  • Notranslate – prevents the page being translated in the Google SERPs

Notranslate command

It is for the Google bot and does not allow the Google bot to translate in the SERP.

Noarchive command

The noarchive command is to prevent search engines from caching a copy of your page

Nocache command

It does the same as the noarchive command, but the nocache command only applies to the MSN / Live search engine.

Nosnippet command

If you do not want to display the description or the Description tag in the search results, you can use the nosnippet command for that page.

Noodp command

The function of this command is not to display the description of our site that exists in directory sites such as dmoz in search results. This means that the dmoz site collects a series of information from each site, and search engines may use that information in the results below our site links. However, the dmoz site is no longer active. Directory sites are basically called ODP.

No order

It is the same as the noodp command, but it refers to the Yahoo site directory. This command says that no search engine should use my site description in the Yahoo directory in the results. Although the Yahoo directory has been down for a long time, it is obsolete.

Which commands are supported by search engines?
We have specified the commands supported by the following search engine bots with the word “yes” along with the source.

Google Microsoft Yahoo Ask
NOINDEX Yes Yes Yes Yes Yes
NOFOLLOW Yes Yes Yes Yes
NOARCHIVE Yes Yes Yes Yes
NOODP No Yes Yes Yes
NOYDIR neither no nor yes
NOSNIPPET neither yes nor no
Robot name TEOMA GOOGLEBOT MSNBOT SLURP

 

A few examples of the robots metatag

To better understand, we have implemented 4 different scenarios that may be used more than the rest of the commands, and then you can use the commands on your pages correctly by looking at these examples and understanding them.

Example one – index, follow

If all pages have the index and follow commands. The photo is quite clear.

<meta name = ”robots” content = ”index, follow” />

Example Two – noindex, follow

If all pages have noindex and follow commands. The robot, for example, does not index page B, but crawls that page and looks for its links.

<meta name = ”robots” content = ”noindex, follow” />

Example three – index, nofollow

If all pages have the index and nofollow commands. Here, page B is indexed and appears in search engine results, but the links on page B are not followed, but as you can see, page E is indexed because the robot may have access to it through another site or through a page from your site. .

<meta name = ”robots” content = ”index, nofollow” />

Example four – noindex, nofollow

If all pages have noindex and nofollow commands. The photo is quite clear.

<meta name = ”robots” content = ”noindex, nofollow” />

 

Some important points about Meta tag robots

  1. If you do not use the noindex command, it is indexed by search engines by default, or if you do not use the nofollow command, it is followed by default. This applies to all commands, and if not used, bots have access to all items.
  2. Uppercase and lowercase letters do not matter in the commands, meaning that all three commands below have the same meaning for the robots. And Google has stated that the following meta tags are no different for Googlebot.

    <meta name = ”ROBOTS” content = ”NOFOLLOW”>
    <meta name = ”robots” content = ”nofolloW”>
    <meta name = ”rObOtS” content = ”nOfOlloW”>

     

  3. Crawlers or some search engine bots may ignore the meta tags listed on the pages. Especially spam or malware robots that scan sites for vulnerabilities regardless of meta tags. And are looking for users’ emails or other items.

     

  4. Suppose link A is on page A and the nofollow command is listed on page A. In this case, link A is not followed by standard bots and search engines. If page B was without the nofollow command and also had link A, then the bots would link to A have access and can follow it. Sometimes bots may inadvertently follow links through other pages.

     

  5. You can use these commands for a specific search engine. This means that if you only want to execute commands for the Google search engine, it is enough to enter the phrase googlebot instead of the phrase robots in the meta tag. A Google meta tag was created. The names of the other search engine robots are listed in the table below.

    GOOGLEBOT Google
    SLURP Yahoo!
    MSNBOT MSN / Live
    TEOMA Ask
    for example:

    <meta name = “googlebot” content = “noindex”>

     

  6. If you plan to use multiple commands for a page, you do not need to enter a robots meta tag for each command. Just separate the commands with a comma. Like the following example:

    <META NAME = “ROBOTS” CONTENT = “NOINDEX, NOFOLLOW”>

     

  7. You must have several landing pages on your site (what is a landing page). Be sure to check the robots meta tag for these pages and allow crawlers to crawl and index if you want landing pages to appear in search results.

 

Metatag settings in WordPress with the help of USB SEO plugin

With the help of Yoast SEO plugin in WordPress, you can change the robots meta tag settings for the whole site or any page. If you noticed the bottom of the USB plug-in when inserting the post, you can make the necessary changes in the settings.

The above settings only apply to one page of the site. To change the general settings of the robots meta tag, go to the titles & meta section of this plugin, and in the tabs of the submission types, categories, archives, etc., you can index or noindex the robot meta settings.

 

For information:

  • In the future, we will give a complete tutorial on the USB SEO plugin.
  • Be sure to read the article The difference between a robots meta tag and a robots.txt file so that you can use different methods to set up bots and how they deal with your site pages.
  • Information about the robots meta tag on the Google site is available at this link.
  • If you are looking for up-to-date and comprehensive SEO training, do not miss the mentioned link

I hope the MetaTag Robots article you used was in practice, keep in mind that this type of MetaTag is the best way to prevent your site pages from being indexed, although there are other days that we have in the removals report article. As for how to remove the site from Google, we mentioned it.

A few frequently asked questions

  1. What is the best way to prevent Google from displaying pages?
    Use the same noindex meta tag and we explained in the article how to use this meta tag.
  2. What is the difference between a robots meta tag and a robots file?
    The robots metatag is primarily used for page tags, but the robots file is for a group of pages. However, there are many other categories as well, such as the power of their work, which is further explained in the article.
  3. Is Yoast plugin capable of adding Navindex meta tags?
    Yes, in the advanced section of each page you visit, you can index or rewind that page according to your needs.

Any comments, suggestions, experiences and criticisms you have in the comments section.

Leave a Comment