Effectively Using Robots Meta Tags

Home
>
Web Development
>
Effectively Using Robots Meta Tags

Effectively Using Robots Meta Tags

The "robots" meta tag, when used properly, will tell the search engine spiders whether or not to index and follow a particular page. For the purposes of this article, we will be using the "( )" symbols to represent the "< >" in html coding.

Some examples of robot usage are as follows:

(meta name="robots" content="index,follow")

(meta name="robots" content="noindex,follow")

(meta name="robots" content="index,nofollow")

(meta name="robots" content="noindex,nofollow")

Let us first examine what these terms mean before we explain the usage for each one:

"index"- This directive tells the search engine robots (or spiders) that it is okay to index the page. Another words, you are allowing the search engine to include your page within their search directory.

"noindex"- Using this tag, you are informing the robots that this page should not be indexed. Simply put, this page will not appear in their search directory.

"follow"- When you use this tag, you are telling the search engines that you want their robot to follow any links that are found on that page.

"nofollow"- The opposite of the above definition, this directive will tell the robots not to follow any links on your page.

Putting It All Together:

With the robots tags explained, let's examine the usage for each one.

1. (meta name="robots" content="index,follow")

This tag will be used when you want the search engine spiders to index the page and follow the links to other pages. Most search engines use this setting as a "default" setting. It is possible that you may not even need to use this tag if you want the search engines to follow and index the page. However, an article at Search Engine World (searchengineworld.com/metatag/robots.htm) suggests that Inktomi does not use this as their default setting. Instead, they use the "index, nofollow" tag.

Better safe than sorry!

There has been much debate over whether or not it is necessary to use this tag. If there is even a slight possibility that some search engines do not use this as the default setting, then it would only make sense to include this tag if you want your page included in their search directory AND your links to be followed. Do the research and decide for yourself.

2. (meta name="robots" content="noindex,follow")

This tag can be used to tell the search engines that you do not want the page included in their directory, but you DO want them to follow the links that lead to other pages. A good example of its usage would be your disclaimer or privacy policy pages. You may not want these pages to show up in the search engines if they are only important to your actual visitors. However, if the links on these pages point to other pages that you want the search engines to find, then you would still want the spiders to "follow" those links.

3. (meta name="robots" content="index,nofollow")

This tag will allow your page to be indexed in the search engines, but any links on that page will not be followed.

4. (meta name="robots" content="noindex,nofollow")

When using this tag, the search engine spiders will not include this page in their directory and will not follow any links on the page either.

Where does the "robots" tag belong?

The "robots" meta tag should be used within the (head) and (/head) tags of your page. These tags are located at the top of the html coding. It will look something like this:

(html)
(head)
(title)Title of your page goes here(/title)
(meta name="keywords" content="word1,word2,word3,word4")
(meta name="description" content="A brief description of the content of this page.")
(meta name="robots" content="index,follow")
(/head)
(body)
Your webpage information here.
(/body)
(/html)

More Robots Tags

Google automatically archives a page as it crawls it. This is called a "cached" version of the page. Visitors can retrieve the archived version of the page by clicking on the "cached" link within Google's search results. If you do not want your content to be archived, you can use the following tag:

(meta name="robots" content="noarchive")

*This will only prevent your page from being "cached". If you do not want your page to be indexed at all, you will still need to include the "noindex" tag.

Another alternative to the above tag is the tag that specifically addresses Google only. If you want other search engine robots to archive your site, but you would like to prevent Google from doing so, then you can use the following tag:

(meta name="googlebot" content="noarchive")

The Misuse of Robots Tags

Something that has been popping up on websites everywhere is the Google indexing tag. This is a silly little tag that is not necessary. Some people think this tag helps Google to spider your site, but this simply isn't true. The tag looks like this: (meta name="googlebot" content="index,follow"). Some website owners believe that by specifying "googlebot" that their site has the advantage of being spidered faster and listed by Google. According to Google's web crawler information, you may use the noindex, nofollow, or noarchive tags when you DO NOT want Google to cache, index, or follow that page. Google's default setting is to index and follow the links on the page, so this "so called" googlebot index/follow tag that some site owner made up one day is completely unnecessary.

Another silly little tag--- The "Revisit-After" Tag

(meta name="revisit-after" content="90 days")
(meta name="revisit-after" content="15 days")

I'm not sure where this myth was started. Today, you will find this tag all over the Internet. Webmasters have even promoted it, claiming that it actually works. Are we so naive to believe the search engine spiders need to know when to come back? I have never used this tag, and my site has no problem with being crawled on a regular basis. Even some SEO (search engine optimization) sites are claiming its value. This comes back to the importance of researching the topic thoroughly first. My research came from some interesting information on WebmasterWorld.

It is important to examine the correct usage of the "robots" tag before applying it to your website. Incorrect usage of tags could result in spidering errors that cause robots to completely ignore the page all together. If you are interested in learning more about web robots, this great little site will provide you with the information you need to use them effectively: www.robotstxt.org/wc/robots.html

Stephani Richardson is a work at home mother of 4 who has been actively involved with affiliate marketing and home business opportunities since December 1999. She owns and operates several business related websites including 1 Work At Home Dot Com.

In The News:

Top Stories - Google News
Updated : Sun, 21 Mar 2010 18:52:09 GMT+00:00

Iceland prepares for second, more devastating volcanic eruption - Times Online

The Hindu

Iceland prepares for second, more devastating volcanic eruption
Times Online
Iceland is preparing for an even more powerful and potentially destructive volcano after a small eruption at the weekend shot red-hot molten lava high into the sky. About 500 people were safely evacuated from the land close to the Eyjafjallajokull ...
Volcano erupts in Iceland, hundreds fleeWashington Post
Icelands fears 2nd even larger volcanic eruptionHouston Chronicle
Iceland fears 2nd, even larger volcanic eruptionAustin American-Statesman
Atlanta Journal Constitution -BusinessWeek -BBC News
all 824 news articles »

Publ.Date : Sun, 21 Mar 2010 17:05:46 GMT+00:00

Tea Party Leader Condemns Racial Slurs Hurled at Black Lawmakers - FOXNews

FOXNews

Tea Party Leader Condemns Racial Slurs Hurled at Black Lawmakers
FOXNews
Republican National Chairman Michael Steele and one of the organizers of Saturday's Tea Party rally strongly condemned the racial slurs that some black lawmakers alleged were yelled at them by some health care protesters as they headed for a procedural ...
'Tea party' protesters accused of spitting on lawmaker, using slursWashington Post
GOP, Tea Party try to distance themselves from racial tauntsBelleville News Democrat
Tea Party Members Hurl Gay Insults at Barney Frank Over Health Care ReformSheWired
Voice of America -Politico -NBC Washington
all 362 news articles »

Publ.Date : Sun, 21 Mar 2010 13:32:19 GMT+00:00

Tensions rise as Israeli soldiers kill 2 Palestinians - Los Angeles Times

Ha'aretz

Tensions rise as Israeli soldiers kill 2 Palestinians
Los Angeles Times
The military has killed four Palestinians over the weekend. Israeli Prime Minister Benjamin Netanyahu is soon to leave for Washington after a recent diplomatic fracas over Israeli housing settlements. By Edmund Sanders As Israeli Prime Minister ...
UN Chief Blames Israeli Blockade for Suffering in GazaVoice of America
Netanyahu to Discuss Core Issues, Is Invited to White HouseBloomberg
Ban to Gazans: We stand with youJerusalem Post
Reuters -Ynetnews -Telegraph.co.uk
all 1,733 news articles »

Publ.Date : Sun, 21 Mar 2010 17:11:43 GMT+00:00

Lazio Picks Up State Conservative Nod - NY1

Boston Globe

Lazio Picks Up State Conservative Nod
NY1
Republican candidate for governor Rick Lazio has officially picked up the backing of the New York State Conservative Party. Chairman Michael Long says Lazio won the vote of the party by a margin of 5-to-1 Saturday. No Republican has won statewide ...
Lazio Endorsed by Conservative PartyNew York Times
Lazio Wins Shaky Conservative Party Endorsement; Levy Heats UpGothamist
Editorial: Steve steals the showNew York Daily News
New York Post -Press & Sun-Bulletin -SILive.com
all 135 news articles »

Publ.Date : Sun, 21 Mar 2010 18:17:46 GMT+00:00

Immigration reform rally draws tens of thousands to Washington - Los Angeles Times

Los Angeles Times

Immigration reform rally draws tens of thousands to Washington
Los Angeles Times
Organizers of the 'March for America' are trying to cast the issue as an element of economic recovery. They say hundreds of groups from almost every state are taking part. Juana Castizo, 59, of Mulberry, Florida, raises her hands in prayer as ...
Immigrants, supporters gather for major rally on The MallWashington Post
Thousands Expected to Rally for Immigration Overhaul in WashingtonFOXNews
Luis Gutierrez's remarks on immigration reformPolitico (blog)
New York Times -Huffington Post (blog) -ABC7Chicago.com
all 132 news articles »

Publ.Date : Sun, 21 Mar 2010 18:11:05 GMT+00:00

Democrats Need 216 Votes to Pass Health Care Reform Bill - ABC News

ABC News

Democrats Need 216 Votes to Pass Health Care Reform Bill
ABC News
After months of negotiations, closed-door meetings and persuasion by President Obama, members of the House of Representatives today will vote on the controversial $940 billion health care bill. President Obama makes final pitch to pass ...
Health care latest: Dems still chasing votesCNN
House leaders express confidence they will secure enough votes to pass health billWashington Post
House Democrats Expect to Secure Health VotesWall Street Journal
USA Today -Los Angeles Times -CBS News
all 14,516 news articles »

Publ.Date : Sun, 21 Mar 2010 17:20:48 GMT+00:00

Fargo dikes hold as Red River nears crest - Reuters

CBC.ca

Fargo dikes hold as Red River nears crest
Reuters
WINNIPEG, Manitoba (Reuters) - The flooded Red River moved within inches of its crest at Fargo, North Dakota on Sunday, but dikes of earth and sandbags kept the water from causing major damage, officials said. The city had no reports of damage to homes ...
Fargo on brink of declaring victory in flood fightDanbury News Times
Fargo's fight against flooding nears its climaxDickinson Press
Fargo on brink of declaring victory in flood fightphillyBurbs.com
Merced Sun-Star -Minneapolis Star Tribune -New York Times
all 4,768 news articles »

Publ.Date : Sun, 21 Mar 2010 16:52:22 GMT+00:00

RSS Software
Increase Online Traffic

Effectively Using Robots Meta Tags

Web Development