Normally, It takes between 4 days and 4 weeks for your brand new website to be crawled and indexed by Google. This range, however, is fairly broad and has been challenged by those who claim to have indexed sites in less than 4 days. Even though Google’s inimitable search engine works on an algorithm, the eternal math that’s happening behind the scenes can’t produce a single, solid answer for us. Still, a guideline of 4 days to 1 month gives most webmasters a small amount of comfort while they wait to see where their page will appear in the search results pages.
However, there are ways you need to do to ensure that your site is indexable:
Your Site is Indexed Under a www- or Non-www Domain
Technically www is a subdomain. Thus, http://example.com is not the same as http://www.example.com. Make sure you add both sites to your GWT account to ensure they are both indexed. Be sure to set your preferred domain, but verify ownership of both.
Google Hasn’t Found Your Site Yet
This is usually a problem with new sites. Give it a few days (at least), but if Google still hasn’t indexed your site, make sure your sitemap is uploaded and working properly. If you haven’t created or submitted a sitemap, this could be your problem. You should also request Google crawl and fetch your site. Here is Google’s instructions on how to do that.
The Site or Page(s) are Blocked With robots.txt
Another problem is your developer or editor has blocked the site using robots.txt. This is an easy fix. Just remove the entry from the robots.txt, and your site will be reappear in the index. Read more about robots.txt here.
You Don’t Have a sitemap.xml
Every website should have a sitemap.xml, which is a simple list of directions that Google should follow to index your site. You can read about Google’s Sitemap policy, and create one pretty easily.
If you are experiencing indexation issues on any portion of your site, I recommend that you revise and resubmit your sitemap.xml just to make sure.
You Have Crawl Errors
In some cases, Google will not index some pages on your site because it can’t crawl them. Even though it can’t crawl them, it can still see them.
To identify these crawl errors, go to Google Webmaster Tools → Select your site, → Click on “Crawl” → Click on “Crawl Errors”. If you have any errors, i.e., unindexed pages, you will see them in the list of “Top 1,000 pages with errors.”
You Have Lots of Duplicate Content
Too much duplicate content on a site can confuse search engines and make them give up on indexing your site. If multiple URLs on your site are returning the exact same content, then you have a duplicate content issue on your site. To correct this problem, pick the page you want to keep and 301 the rest.
It sometimes makes sense to canonicalize pages, but be careful. Some sites have reported that a confused canonicalization issue has prevented indexation.
You’ve Turned On Your Privacy Settings
You may have accidentally kept the privacy settings on.
The Site is Blocked by .htaccess
Your .htaccess file is part of your website’s existence on the server, which allows it to be available on the world-wide web. The .htaccess file is written in Apache. Although .htacess is handy and useful, it can be used to block crawlers and prevent indexation.
The Site Has NOINDEX in the Meta Tag
Another way of saying “no” to the robots, and thus not having any indexation, is to have noindex meta tags. It often looks like this:
<META NAME=”ROBOTS” CONTENT=”NOINDEX, NOFOLLOW”>
You Have AJAX/JavaScript Issues
Google does index JavaScript and AJAX. But these languages are not as easily indexable as HTML. So, if you are incorrectly configuring your AJAX pages and JavaScript execution, Google will not index the page.
Your Site is Slow to Load
Google doesn’t like it if your site takes an eternity to load. If the crawler encounters interminable load times, it will likely not index the site at all.
You Have Hosting Down Times
If the crawlers can’t access your site, they won’t index it. This is obvious enough, but why does it happen? Check your connectivity. If your host has frequent outage, it could be that the site isn’t getting crawled. Time to go shopping for a new host.
You Got Deinedexed
This one is really bad.
If you got hit with a manual penalty and removed from the index, you probably already know about it. If you have a site with a shady history (that you don’t know about) it could be that a lurking manual penalty is preventing indexation.
If your site has dropped from the index, you’re going to have to work very hard to get it back in.
Additional Notes:
