A lot of SEO folks inquire about why there is so much difference in the figures of URLs submitted and URLs indexed in google webmaster. So, I thought, it’d be a good idea to write a post about it. That way, anyone can find the gist of the problem easily rather than crunching a lot of web pages to find a solution. If anyone wants to read this topic in detail the best source would be: https://support.google.com/webmasters/answer/156184?hl=en
There are couple of reasons for why you might find a difference in the urls submitted in webmaster and the urls indexed. Some of them are listed below.
- Firstly, the primary reason for this is Google or any other search engine, has indexed other variation of urls submitted. E.g. if indexed url is http://example.com/abc/ and sitemap contains its variant http://example.com/abc. In this case, the webmaster doesn’t count the link indexed out of submitted urls. Therefore, an important piece of advice is to carefully match the url submitted through sitemap and url indexed in SERP.
- In some cases, we have seen that the urls submitted in sitemap have crawl errors like 404/410/500 response codes. Therefore, google hasn’t indexed them. So, the best way is to check your crawl errors in webmaster and see if it shows the urls listed on the sitemap.
- Unknowingly, if these urls have <META NAME=”ROBOTS” CONTENT=”NOINDEX, NOFOLLOW”>, search engines aren’t indexing them. Besides, if a url is blocked by robots.txt, it won’t be indexed in that case as well. Therefore, double check your URLs’ headers and robots.txt file before submitting a sitemap.
- Google doesn’t guarantee that all the web pages listed in your sitemap can be crawled or indexed. In general, this is usually the case for some rare urls. So, the difference in indexed urls as compared to submitted urls would be very small. Therefore, you can use an option called “Fetch as Google“. Once you fetch the url, you get an option for “Submit to Index“. This option speeds up the indexing of the url.
- Sitemap hasn’t been updated for a long time and websites have redesigned. Old urls are being 301 redirected to new urls. So, the search engine index new urls and remove old urls out of SERPs.
If you don’t find any of the above cases, there might be something you’re missing from the search engine point of view. In that case, it would be a great idea to consult with a good SEO agency to troubleshoot the problem.
