How Google Crawls and Indexes Web Pages

Have you ever wondered how Google crawls and indexes web pages? If you haven’t, you should. Why? Because it affects how well your site ranks on Google.

In order to help you understand Google and improve your rankings, I’ve created the following infographic. It not only breaks down the way Google indexes your website, but it also covers the 7 most common reasons why your website isn’t ranking well.

Click on the image below to see a larger view:

How Google Crawls and Indexes Web Pages

Click here to view an enlarged version of this infographic.


If Google can’t efficiently crawl your website, your rankings will tank. By making things easier for the search engine, you’ll increase the likelihood of your pages being crawled and, more importantly, indexed.

Make sure you avoid the 7 mistakes above and check htaccess to robots.txt files. It’s the little things that can affect your rankings in a negative way.

So, how else can you increase your rankings?

P.S. If you need help with your Google rankings click here.

Embed This Image On Your Site (copy code below):


  1. Great post Neil! Correct me if I’m wrong here, but you can also have Google crawl your website by requesting it through Webmaster tools. Isn’t that right?

  2. Hey Neil

    A very useful infographic and Blog post


    Sean Durrant

  3. Very basic, BUT very useful information for ALL website owners. It doesn’t matter how educated you are in the SEO world, if you don’t have these bare necessities in place, you’re setting your website up for failure. Great stuff as usual, Neil.

  4. Thanks Nail, Great information. Comparison is very interesting info.

  5. Love it Neil!
    Thanks for compiling these useful information. I will forward that graphic to my clients when they ask about getting their site indexed at Google. In Germany small-business owners still think the web designers turns a magic switch and their site is #1 in Google…

    Time to educate them 😉

  6. Melissa Samaroo :

    Great post that highlights the importance of the little details. Small things like titles tags can have a big impact!

  7. Hey Neil

    Thanks for explaining the world of Crawl and Index..!!

    I hope this infographic will help lots of people..!!

  8. Thanks Neil.
    And I’d add a coupe of notes to your post.

    1. Google’s John Mueller said they are probably not going to update Google Page rank at least in the Toolbar PageRank (maybe they will one day, but it is what they said), so I guess that indexing should not depend on Page rank as much as it did before. Anyway, PageRanks has not been updated for a year.

    2. As regards improving indexing pages, I believe that social shares help well (especially Google Plus) and links from other indexed pages (including pages from both other websites and your own website).

    • True @Michael, i noticed using Google+ sharing on you website works really well. Before i didn’t really consider G+ as a channel to get more people to read my blog post, but i realise that its doing well than FB, TW and the rest.

      • Michael/Albert:
        Thanks for the additional tips. They will definitely prove helpful to a lot of marketers who read this blog. Looking forward to hearing much more from you.

  9. Neil, You always awesome… While reading this infographic, it takes me back to the days when I start my career.

    Thanks for sharing this infographic…

  10. great article. Good knowledge. thanks for this post

  11. I`am very thank full because I`ve subscribe to your newsletter. Really Owesome knowledge I found in your site. Thank you Neil. I wrote this because I do not know what to say about SEO. and infact I`am still learning right now.

  12. Niel,
    I generally enjoy your posts, but am unable to view this infographic on my mobile device?

    I must say, I’m a bit surprised.


  13. Awesome Info. i have not created robot.txt file..i am going to Google it and will create it.Thank you

  14. Hey neil,
    Another awesome article… I knew a bit about crawling but your infographics are worth reading.

  15. Nice explanation Neil! Just wondering how about body content of webpage? Is that too important in indexing pages?

  16. Adarsh Sojitra :

    I read your each and every article and this infographic is also a great piece of art. I have one question that what will be the price of this type of infographics? (if we want to develop).
    Very much informative and liked it!

    • Adarsh, it really depends. I have a designer that I have been working with for years so my price point is a lot different than others.

  17. Hi Niel,

    Thanks for this info. I have to change my site a lot in the description as I used lengthy one in may pages.

  18. Hi Neil !

    So basically you have decided to post something basic for newbies. Glad to see at least someone on this planet is taking such innovative start to train newbie SEO enthusiasts. And more importantly, I think you have reacted to my mail through this post.

    Thanks for these sharing basics. Hope you’ll go from basic to advance step by step.

    • Rahul, glad I could help. The basics are the foundation upon which everything is built — which is why I wanted to go back to them.

  19. Glad!

    Very informative but could be more detailed!

  20. Thanks for the primer. I am not a SEO or Google expert by any stretch, but #3, ‘Badly written title, meta tags, and author tags’ Aren’t author tags no longer used as of August 2014?

    • For Google, that’s technically correct. Google doesn’t display authorship results anymore. However, I would still add author data since it’s part of the rich text that is shown on social (Pinterest, Facebook, Twitter).

      • Ah thanks for the info, Kyler!

        • Alexandra/Kyle,

          Glad you guys could figure this one out. While the results aren’t displayed it’s still important to focus on making sure you still have everything set up for other channels as Kyle mentioned.

  21. Fascinating about the “hidden web”. I have a few unindexed pages (on purpose) but I had no idea there was that much content hidden from Google.

  22. Thanks for this basic info. Basics are very useful for every business.

  23. Good basic getting started info. But there is much more behind the scene. This is just the tip of iceberg 🙂

  24. Regularly update your content- helps a lot in crawling and indexing. What’s the proper way to update a post’s content – rewrite or add extra content without change the original content or edit some part of the post
    what’s the proper place of updated content ? No one other than Neil can reply this!!

  25. Neil Sir,

    Thanks for this post, I have been reading your blog since I have joined this industry..each post is truly helpful and gives you better understanding of latest updates on search engines.

    Will surely look our website and apply your tactics to improve site visibilty.


  26. Hey Neil,

    This is a good overview infographic for newbies. I have a couple of comments that aren’t meant to be critical, just some observations and information extension.

    First: “How Google Crawls and Indexes Web Pages?” on the infographic is not a properly structured english question. If you take the question mark off, it works fine:
    “How Google Crawls and Indexes Web Pages”
    otherwise, if you want to leave it as a question, it should say:
    “How Does Google Crawl and Index Web Pages?”

    Second: Search engine bots do not actually move from page to page and link to link in a single visit. Search bots grab the source code of the page, strip out all the links and insert them into a database to come back and visit later. So if you submit a single page to Google for indexing, it will visit that page, strip out all the links, and then go back and visit all the linked pages later. Then they will do the same for those pages. Finally, they will analyze all the links with their algorithm to determine the purpose and relevancy of each linked page.

    Thanks for all you do, Neil. I was telling my wife about your $100,000 watch experiment last night at dinner. You are an amazingly creative person!

  27. Farcas Gelu Danut :

    Thank you, Neil. Very useful infographic.

  28. Great article Neil…I found that to be super interesting.

    Really enjoying receiving your regular articles.

  29. Very Useful but Basic Level information….. and described in a good understandable manner.. so 2 kudos to Neil for this post.

  30. I made my own google-like search Engine by reverse engineering on few main SEO factors. It was an awesome experience.

    After creating it, i realized the complexity of Google search !!! Thousand times more than my project.

  31. This questions pinched in my mind from the day I started blogging and entered in to this search engine field.

    Neil, you nailed it today, just wonderful!

  32. Hey Neil. Nice Infographic about google crawl and indexing pages by google awesome and informative infographic. hopes it get huge exposure

  33. Anirudh Pulikonda :

    This Info-graphics is really awesome Neil. Gained very good knowledge about google crawling and indexing. Keep rocking

  34. Nice work Neil 🙂

  35. gaurav khurana :

    Again a very nice article the best part is the infographics used…

  36. It’s complicated.

  37. Neil Thanks for this post. I learned a lot from this. But one thing did not cleared to me.

    “The 90% of the web thats not hidden from google, is called “The Invisible Web”

    ……..That’s around 450 billion web pages”

    Is it correct??

  38. Nice infographic Neil, thanks for sharing.

  39. Thanks Neil, we will implement your tips and tricks in our blog.


  40. Hi Neil !

    My comment is not about this post but the forum. Where is it ? Did you delete it ?

    I was going to ask a question on it and I couldn’t find it. I launched a crowdfunding campaign on Indiegogo and I didn’t get any fund yet.

    I thought some people on the forum could help me getting traffic or anything else but there wasn’t any forum :(.

    Do you want to learn a new language ? Help iSpeakSpokeSpoken coming the best platform to learn a language ever !–2/x/8634546

  41. Hi Neil

    As always a great post. Reminds us all that you’ve got to get the basics right first!

    Thanks again

  42. Hi Neil Bro,
    Nice infographic about Google Crawls and indexes.Keep rocking
    Thanks For It.

  43. Thank you for the heads up. I will monitor my site to make sure it is up par. I have followed your advice and so far everything is happening as you said it would.

  44. Neil,

    Litlte question on google crawling.

    What if the site is failrly new site and wanted to index them sooner. Can we bump up the crawl rate in the google?


  45. sklep z zabawkami :

    Superb blog! Do you have any hints for aspiring writers? I’m planning to start
    my own website soon but I’m a little lost on everything.

    Would you propose starting with a free platform like WordPress or go
    for a paid option? There are so many options out there that I’m
    totally confused .. Any tips? Thanks!

    • Write something every single day no matter what, and in 30 days you’ll be 10x better that day 1

  46. webcolegios nuestra señora de belen :

    También se debería procurar que los alumnos reflexionaran sobre sus necesidades comunicativas como un aspecto más de su propia reflexión y de su independencia.

  47. ksi?gowo?? internetowa :

    Do you mind if I quote a few of your posts as long as I provide credit and
    sources back to your website? My blog site is in the very same
    niche as yours and my visitors would genuinely benefit
    from some of the information you present here.
    Please let me know if this okay with you.

  48. Hello Neil, I have a problems with my some urls. after modified some urls i always get 404 urls did not found. How i solve this issue?Thanks!

  49. Mount Damavand :

    We are a group off volunteers and opening a brand new scheme iin our community.
    Your website provided us with valuable info to work on. You
    havee done a formidable activit and our entire group shall bee thankful to you.

  50. Great article.

  51. campervan rental :

    My brother recommended I may like this website. He was
    totally right. This put up actually made my day. You cann’t imagine just how much time I had spent for this
    information! Thanks!

  52. Excellent, what a web site it is! This blog gives helpful
    information to us, keep it up.

  53. Pokemon Go Cheat :

    It’s actually very complicated in this busy life to listen news on Television, so I only use world wide web for that purpose, and obtain the most recent news.

  54. I quite like reading through an article that can make people think.
    Also, many thanks for allowing me to comment!

  55. Maijntain the remarkable wwork !! Lovin’ it!

  56. Hi Neil, thanks for great post. i thing want to know that ” what are the benefits of google crawler log data file” what all inferences i can get.

Speak Your Mind