Indexation and Accessibility Guide

In this post you’re going to learn some advanced techniques for evaluation and optimization of your website for indexation and accessibility.

This doesn’t just mean accessibility for the search engines but accessibility for humans too. That’s why this post covers best practices for both engines and users – with this like installing Google translate to making AJAX crawlable.

After applying to techniques in this post to your website where most applicable, you should have an exceptionally crawlable and accessible website.

Browse Your Site Like a Search Engine

When optimizing your site for SEO, wouldn’t it make sense to put yourself in the shoes of the search engine? And “see” your site like a search engine does? Now you all know you can simply “view source” to see the HTML source code from any browser. But I’ve got a killer method that will really put you in their shoes and reveal possible holes in your technical SEO you can easily plug up.

Install the Plugins

You’re going to want to use FireFox to do this. Here’s the plugins:

Web Developer

https://addons.mozilla.org/en-US/firefox/addon/web-developer/

Disable JavaScript in FireFox

Go to “preferences” and “content” and uncheck “Enable JavaScript”.

We do this because items like menus, links and drop downs need to be available to Googlebot without JavaScript. If they are buried within your JS than Google can not crawl them!

Disable CSS with the Web Developer Plugin

Why disable CSS? Googlebot crawls the content in the order of the HTML. The CSS styling can sometimes obscure the order of your content.

Set User-Agent to Googlebot

Fire up your website and browse!

What does QuickSprout look like to Googlebot?

This is just the top (because otherwise the whole page is really LONG). But you can see that the menu appears as clickable links, and none of the other text and links are hidden from Googlebot.

View your site this way and you may be surprised what you find!

Some things to check for:

Can you see all of your menu links (drop downs too!)?
Do all of the menu items and links appear as plain text?
Are all the links clickable?
Does this reveal any text that was previously hidden? (Hidden text can send a red flag to Googlebot. It might not always be there maliciously, but it shouldn’t be there.)
Is your sidebar or widgets content all the way at the top? Remember, your most important links and content should be at the top of the HTML. This is more important the bigger the site is.

Finally. Here’s an example of a site with an issue.

The problem with this site is that the menu text is not real text, its images. What does this mean? There is no anchor text signal given to Googlebot. You all know how important anchor text is for back-links, and its just as important for internal links. On the above website, none of the inner pages are receive the full value of link juice flowing into them from the homepage.

Once doing an on-site inspection from the point of view of a search engine, you may be ready to crawl the site with a web spider.

Crawl Your Site With Screaming Frog

What Is Screaming Frog?

Screaming Frog SEO Spider allows you to crawl your website and gain valuable insight into how its put together much more easily and faster than you ever could just by looking at it. As you’ll see, you will certainly gain a new perspective on how your site looks in the next few minutes!

This being a hands on technical guide, if you’d like more details on the benefits of using your own web spider you can read Screaming Frog’s documentation at;

http://www.screamingfrog.co.uk/seo-spider/

Note: Screaming Frog is free for you to crawl up to 500 pages at a time. For bigger sites you will need to purchase an annual license. BUT do you want a secret workaround for this? I thought so! You can enter subdirectories to crawl – like https://www.quicksprout.com/2012/ will give me only the posts from 2012. Do this for more multiple subdirectories and you can still crawl the entire site, just in pieces.

Crawl The Site

Fire up Screaming Frog and run the first crawl on your site.

Depending upon how large your site is, this can take anywhere from 2 to 20 minutes.

Save Your Crawl

You should save the crawl of your site in the .seospider file format. This way you won’t have to run the crawl again in case you close the program or want to review it again later. Remember though, if you make major changes to the site you should crawl it again anyway. But this way you’ll have a crawl record of your site from this date.

Check Page Levels

Remember, this is a technical guide so that you can make real changes to your website that will bring you definite SEO improvement. So we’re going to focus on getting information out of Screaming Frog that you can then go and apply to your site.

If you have pages that are buried too deep within your website, this is not good for users or your SEO. We’re going to find those pages easily with Screaming Frog and get them into a list for you to take action on.

After your crawl, you’ll be on the main “internal” page showing you all the data gathered internal to your site.

select HTML
scroll right (all the way)
sort pages by level
scroll back
export to CSV

Even QuickSprout has some older blog posts that have ended up 4-7 levels deep.

Side Note: You might want to consider a plugin for WordPress like crosslinker https://wordpress.org/plugins/cross-linker/ which can help you always internally link and cross link your posts.

Scroll back to the left, and you’ve got a nice prioritized list of pages to be sure to link back to when writing new posts.

Now you’ve got a great, actionable list of URLs to link back to from newer posts, right in Excel.

And of course when back-linking, be sure the references are relevant, useful and use descriptive keyword-rich anchor text.

Check for Crawl Errors

Now we’ll begin exploring some of the different menus at the top. There are a lot of hidden gems in Screaming Frog, but you have to know how to find them – which is what I’m showing you now!

Google Webmaster Tools will of course give you crawl errors, but they may be incomplete or old. PLUS, this will give you all external links you’re linking to that are broken. A fresh crawl of your site with your own tool is a fantastic way to get a current accurate list.

Click on “Response Codes”
Select “Client Error 4xx” from the filter drop down menu
Export as CSV

This gives you a list of only the pages which returned some sort of 400 level error (usually 404s).

Find & Fix Long Titles

You all may know that title tags and meta descriptions have a recommended length. And again, Webmaster Tools does give you some of this data.

What’s great about Screaming Frog, is you know this data is complete and you can sort and filter it.

Click on over to “Page Titles” in the top menu.
Select “Over 70 Characters” from the menu.
Export as CSV

Tip: Immediately “save as” an Excel document. You may lose some formatting changes otherwise.

In your Excel document, you’ll want to create a new column for your new title. Also, create a column for length.

What a simple way for Excel to automatically count the characters as you compose your new title tags? Add this simple formula to the “new length” column; =LEN(E3)

Of course be sure to reference the cell you have your new title in.

And then

Select the formula cell
Hover over the bottom right corner of the formula cell.
Wait for the cursor to change to a cross shape.
Drag the sguare down the entire column.

Find & Fix Long Descriptions

Finding and fixing long descriptions is similar.

We’re going to navigate over to the descriptions menu.

Select “Over 156 Characters” from the filter drop down menu
Export to CSV
You can work on your new descriptions just like how we worked with the new title tags in excel. Make new columns and use the formula =LEN(E2) to automatically count the length of your new description tags.

Look at Indexation Settings

You should also head on over to the “meta and canonical” menu to check your indexation settings. You should look for things like;

Missing canonical tags
Incorrect Canonical tags (pointing to a different page etc)
Pages that should be indexed but have a “noindex” tag on them.
Pages that should NOT be indexed but have no meta tag or have “index”.

How To Find All Pages With Any HTML

Let’s get a little more technical. Suppose you wanted to find ALL the pages within a website that had the presence of certain HTML. Let’s say in this instance we want to find all pages within QuickSprout that have links that open in new tabs or windows.

Select “Custom” from the “Configuration” Menu
Enter the HTML you want it to look for in “Filter 1” Note: You can also have it find pages that do NOT contain the HTML entered. You can enter up to five filters.
Crawl the site again
Select “Custom” in the menu
Select “Filter One” from the filter drop down menu.

Now, we have all the pages with links that open in a new tab or window!

This is great for an existing site if its not going to change at all. But what do you do when you are getting set to redesign your site? You’ll want to do a self audit in prep for a redesign.

Self Audit for a Site Redesign

Next is a complete step by step process for auditing your own site if you do a redesign. This can be an important step to take in your web evolution and promoting your authority online, but let’s not let you lose out on any traffic in the process!

This portion of the guide assumes you have followed best practices while making your new site, such as;

Being sure its crawlable
Submitting a new XML sitemap
Putting 301 redirects in place

Create a Spreadsheet to Monitor Progress

The following is a mock up spreadsheet to show you how to monitor these metrics as the new site goes live.

Monitor Number of Pages Indexed

You’re simply going to use the Google site: search to do this;

Monitor Cache Date

Again, we’re just going to do a Google search for cache:sitename.com

The cache date is the largest factor to telling you what version of your site Google is using in their algorithm.

Monitor PageRank

Although it is known that PR is a very loosely used metric, it still gives a rough indication to the value of the site.

Use the SEOquake toolbar as one fast way of checking your pagerank, which you can install at http://www.seoquake.com/

Monitor SEOmoz Domain Authority

This metric will have a delay, depending upon when SEOmoz updates their linkscape index. But still good to monitor – and you can use their toolbar to do so as well. You can install it here: https://moz.com/products/pro/seo-toolbar

DA authority is best to look at.

Monitor “Not Found” Errors

Use Webmaster Tools to watch your not found errors and get your numbers from there;

With these tools and steps you’ll ensure the smoothest transition possible while moving the site over.

Testing a New Site Before It Goes Live

In this tutorial, we will setup your computer so that when you type in the URL, it will go to your test website so you can try it out before it goes live using the real URL.

Get the IP address of your new website.

The instructions will differ greatly depending on where you host your website but in general, they will be listed somewhere in your admin panel. If you can’t find it, call the hosting company up and simply ask.

Edit your hosts file to point to your IP address

On a Mac
1. Open up Application folder > Utilities > Terminal
2. Inside the terminal app, type “sudo nano /etc/hosts” Type in your user’s password if that’s required.
3. At the end of the file, type in the following line: Hit Control-O to and press enter.
4. Hit Control-X to exit the editor.
5. Close the terminal window.
On a PC
1. Click on Start > type in “notepad” into the search box, and look for notepad to show up in your start menu.
2. Right click on it and left click on “run as administrator”. Say yes to Windows if it asks for permission.
3. Click on File > open
4. In the filename box type “\windows\system32\driver\etc” press enter.
5. Change the file type pull-down from “text file” to “all files”.
6. Double click on “hosts”
7. At the end of the file, type in the following line:

Test your website

Open up a browser and test your website to see if it looks like you expect it. Type in the URL to your new website Your hosts file edits will forward you to the test website.

Undo Changes

Once tested, you can undo your changes you made in step 2. Simply go back to the file and remove the line you just created.

Migrating to a New Site Without Downtime

Warning: This might bring down your website if something goes wrong. Be careful!

When migrating to a new website you can use the following guidelines to ensure a safe transition without any downtime. It takes about a day for all the servers around the world to get updated IP and domain information so plan to have both servers running for at least the first week of your new site.

Set the TTL of your new domain to 5 minutes

The instructions to do this is different for every hosting company or domain registrar. Usually, you can find that setting in your control panel for your domain, but if you can’t find it, call up your domain registrar and ask for tech support.

If you’re using GoDaddy:

Login to GoDaddy’s website
Click on My Account. Go down to Domains and click on Launch
Click on one of your domains
Scroll down to DNS Manager and click Launch
Look for “@” under host and click on the pencil icon under “TTL”
Pull down the menu and select the shortest time available (1/2 hour)

Locate the DNS settings for your domain

Now that you’ve tested your site and changed your TTL, you will want to change the DNS settings for your domain name. First, go to your current domain registrar, and locate the current DNS settings. Then go to your new hosting company and write down the new DNS settings you need to enter into your current domain registrar. The instructions is always different from one hosting company to another and from one domain registrar to another.

Usually, you can find that setting in your control panel for your domain, but if you can’t find it, call up your domain registrar and ask for tech support.

Change the DNS settings on your current domain.

Once you have them both registrar to be the DNS address you wrote down from your new hosting company. You should know where to change them because you already found it in step #3.

Unset your hosts file by removing the lines you added in step #1.

Do so by following step #1 except remove the lines you originally added.

Wait 5 minutes, then try to go to your new website.

You may need to clear your browser’s cache and cookies. If you’re new website is up, you done! If not, reverse what you did in step #4 in order to revert back to your old website.

Crawlable AJAX (using jQuery GET)

In this example, we will show you how to use jQuery’s post method to create crawlable AJAX. For this tutorial, we will use the “XMLHttpRequest POST” method.

For additional details about this best practice, go here: https://webmasters.googleblog.com/2011/11/get-post-and-safely-surfacing-more-of.html

Create your HTML template

Add jQuery to your site

In this example, line 4 was added.

Add a <DIV> tag with a unique ID in the body where your dynamic content will go

In this example, line 8 was added.

Add JavaScript to the site that will load content into the <DIV> tag

In this example, lines 10-15 were added.

Create your PHP script

The example code outputs an example blog post.

Test your script on your web server

It should look like this:

Check view-source

It should look like the HTML template.

Check Inspect Element

You should see the dynamic content loaded in the DOM. It should look like this:

Crawlable AJAX (without hashes)

This tutorial is for websites that load dynamic content on the page without refreshing but changes the URL. Google recommends routing “_escaped_fragment_” in your query string to an HTML snapshot. The principal idea is that you want to render HTML for GoogleBot and JavaScript for users.

There are many ways of achieving the same result. The implementation will depend on your particular website’s setup. In this example, we will use PHP to decide what to render.

If the URL looks like this: ”

For additional details about this best practice, see: https://developers.google.com/search/docs/ajax-crawling/docs/getting-started

Start by creating a PHP script that will add the proper <meta> tag in the header

This will tell search spiders that the page is crawlable using the “_escaped_fragment_” query string. In this example, we created a function that creates the entire <head> tag for us.

Note: Line 10 contains the meta tag that tells spiders to crawl using escaped fragment.

Next, we will create a function that renders the page

In this example, the render_post has an argument “$render_snapshot.” By default, the page will render a regular page for users. If render_snapshot is true, then it will render a regular HTML page for GoogleBot with the same content.

Notes:

Line 25 of the PHP decides if the page should be HTML or dynamic.
Lines 26-29 gets the content and returns the HTML inside the DIV tag
Lines 31-37 gets the content using jQuery to dynamically adds HTML inside the DIV tag

Next, we will add code to handle the escaped fragment query string

In this example, if _escaped_fragment_ is found, it will render the post using HTML.

Next, we will create a content.php file

In this example, the code will convert JSON into HTML.

Finally, we will create ajax_crawling.json

Although this is only a demonstration, the principles still apply in a complex website setup. The content usually comes from a database. In this demonstration, it is just a single file.

Test the page as rendered by a user

The page should look similar to this:

Look at the view source

You should not see your content because it’s dynamically added using your JavaScript.

Look at the Inspect Elements view

You should see your content because the Inspect Element view is what the HTML looks like after JavaScript has run.

Check the bot’s view by adding “?_escaped_fragment_” to the end of the URL

It should look the same as the dynamic page:

Check view source of the bot’s view

It should look like regular HTML without any JavaScript.

Cross Domain Rel=canonical

When to Use the Cross Domain Canonical

This is where many people get a little confused, so before the exact technical implementation, let’s just cover when the tag should be used.

Use it when the content from the old site needs to be moved or duplicated to a new site – and the old host does not provide server side redirects.
Use it only after you’ve reduced on-site duplication on the old site as much as possible.
If you can do a 301 redirect instead and this provides a preferred user experience, use the 301 redirect.
Also, you should not put a noindex on a page with a rel = canonical. The page should be crawlable in order to pick up the redirect.
Use it when content is the same on both pages, or with only slight differences.

How To Implement

This is not so different than implementing a regular canonical tag.

Get your two pages ready
Be able to edit the source code on the first (original) page.
Add the tag to the <head> section of the old page, pointing to the new page; where “example-page-name” is the URL of the NEW page pointing to the new page;

A Quick Example

Let’s say I wanted to move a infographic post from QuickSprout to KISSmetrics. Perhaps I wouldn’t do this in real life, but let’s suppose so, just for the example. We’re going to move it to:

http://blog.kissmetrics.com/5-ways-to-get-your-infographic-to-go-viral

Infographic on QuickSprout https://www.quicksprout.com/
2012/06/11/5-ways-to-get-your-infographic-to-go-viral/
</head>
Same infographic on KISSmetrics http://blog.kissmetrics.com/
5-ways-to-get-your-infographic-to-go-viral
Canonical points to new page

Last few hints

Make the links absolute not relative (include the full http:// etc)
Just like with 301s avoid chains of canonicals
Ultimately this is a hint for Google, not an absolute direction, so you will want to check Google’s index and webmaster tools to see if it has been followed.

Fixing Https Duplicate Content Errors

As I’m sure most of you know, https is the protocol by which secure pages are transferred over the world wide web. Pages such as your shopping cart, login pages and other secure areas should be at an https address. However, this can potentially cause duplicate content, because of the added ‘s’ in the URL.

Typically, https pages should not be in the index. They are usually private pages and don’t have a use for being returned in search results.

If your crawl report or site audit comes back as having https URLs found on your site as duplicates, there’s three steps you should take to resolve this;

Identify which pages are indexed
Diagnose why they are indexed
301 Redirect pages that should not be there
Remove them from the index if they should not be there

Finding https pages that have been indexed

Use this special Google search to find pages from your site that have been indexed with https;

site:yourdomain.com inurl:https

You can see the crazyegg website looks totally clean! Except for this flash loader, no https pages have gotten into the index.

KISSmetrics. on the other hand is a good example of a site that has some https in the index.

This second page here is just a normal blog post and should not be in the index (third result down).

And these other pages indicated, are supposed to be https pages but should not be in the index;

So what do you do when you find pages in the index that should not be there? Like any old page you don’t want in the index, you need to find out why it got there to begin with!

Diagnose Why They Ended Up In The Index

Let’s use the blog post from above as an example, and look at the page.

You can see Google Chrome indicating https is present in the URL but the page is not secure. This in fact confirms the page should not be in the index this way.

Its probably in the index because it got linked to from somewhere, either internally or external, so we’re going to use a few tools to try and find the source of the link.

Let’s use Screaming Frog first, since we know it will be a complete crawl of the site.

Enter the root domain of the website in Screaming Frog (because some sites, like KISSmetrics. operate across different subdomains, www / blog etc – we want to be sure we get a complete crawl of the whole site.)

While crawling the site, you can do a search for the URL of that page and wait for it to show up.

Then, wait until the crawl is complete and review its “In Links”.

Look in the “to” column to see if any links use the “https://”

In this case, there are no internal links pointing to the https:// version of the page.

If an internal link is found, you need to change your internal links and 301 redirect the https version to the http version.

If no internal link is found, you may be able to find the external link, but may not have control over changing it. So you have to 301 redirect it to the http version. This will redirect the user and eventually remove/replace the https version from the index.

Pagination with rel=next

Pagination has always been one of the trickiest elements of on-page SEO and architecture to handle. But now Google allows you to use rel=”next” and rel=”prev” to help show them you have pages that belong in a series.

When using a CMS like WordPress there are many plugins that handle this for you, including Yoast SEO. But if you have a custom made site, or a site hand coded in pure HTML, this section will show you have to correct handle pagination with these new tags. It’s quite simple actually! But you may not find the best source on the internet. Here I’ve made sure it’s bulletproof.

Identify Your Pages In The Series

Let’s use Zappos as an example. Here is their Men’s Sneakers page 1.

We’ve identified this page being the first in a paginated series, as you can see the menu for pages 2, 3, 4 etc.

Here is the page 1 URL http://www.zappos.com/mens-sneakers-athletic-shoes~dA

and page 2, 3 etc http://www.zappos.com/mens-sneakers-athletic-shoes~dB

http://www.zappos.com/mens-sneakers-athletic-shoes~dC

Note: they are using letters (a, b, c) to change the page.

Add rel=”next” To Page One

That’s right, once you’ve identified the pages in the series, page one only gets the “next” tag since its the first page in the series. Therefore to page one, in the <head> section we would add;

Change the DNS settings on your current domain

Every page but the first and the last should have a “next” and “prev” tag. This makes sense since there are pages before and after. Page two (mens-sneakers-athletic-shoes~dB) would have this;

Add rel=”prev” to the last page

The last page in the sequence only needs to refer to the page before it, so we would add;

Assuming Z is the last page.

Final Notes

You can include a canonical tag in conjunction with rel next/prev
You can use absolute or relative URLs, but I always recommend absolute whenever possible.

Redirecting error pages with .htaccess

This will follow a few step process;

Create an error page – this page will have a special script on it.
Configure Your .htaccess file to redirect to the error page

Create Your Error Page

Create a page in which the errors will return from – you can call it anything – error.php works.

On that page, add the following code to the top;
<?php switch($_SERVER["REDIRECT_STATUS"]){ case 400: $title = "400 Bad Request"; $description = "The request can not be processed due to bad syntax"; break; case 401: $title = "401 Unauthorized"; $description = "The request has failed authentication"; break; case 403: $title = "403 Forbidden"; $description = "The server refuses to response to the request"; break; case 404: $title = "404 Not Found"; $description = "The resource requested can not be found."; break; case 500: $title = "500 Internal Server Error"; $description = "There was an error which doesn't fit any other error message"; break; case 502: $title = "502 Bad Gateway"; $description = "The server was acting as a proxy and received a bad request."; break; case 504: $title = "504 Gateway Timeout"; $description = "The server was acting as a proxy and the request timed out."; break; } ?>
This PHP code creates a different title for each type of error. This way you don’t need a ton of different files. We’re doing it all within one file.

In this example we’re just creating a unique title and description for each error page. But you can add extra variables and create any unique content you want.

Configure .htaccess

You’ll need to redirect a bunch of error codes to the error page. You should add the following lines to .htaccess
ErrorDocument 400 /error.php ErrorDocument 401 /error.php ErrorDocument 403 /error.php ErrorDocument 404 /error.php ErrorDocument 500 /error.php ErrorDocument 502 /error.php ErrorDocument 504 /error.php

Optimizing RSS Feeds

RSS feeds are such a huge part of blogging. Yet we sometimes overlook how powerful it can be to optimize these feeds! The following hands on tips should help you get the most out of your RSS feed.

This assumes you’re using feedburner.

Replace Default RSS Feed In Your Header

Assuming you’re using FeedBurner, do all the links on your website point to the correct feed? The header section of the Quick Sprout site points to the feedburner feed.

If you’re site does not do this, you need to change the feed URL in the header.php file (if using WordPress) or where ever your CMS allows.

Find the RSS link in your header.php file

Replace it with your feedburner feed URL

Quick Wins In Feedburner

There’s a few simple features you can easily activate in FeedBurner. Let’s make sure you’ve got them covered!

Activate SmartFeed

SmartFeed helps to make your feed compatible in any reader.

Click Optimize->smartfeed

And activate it!

Activate FeedFlare

Feedflare is a way to put link at the bottom of your feed to ask users to do things like share on Facebook, email, bookmark on delicious etc.

This is an essential must for any RSS feed.

In the Optimize tab, click FeedFlare.

Select the links you want to appear. Feed means they will appear in the RSS feed. Site means they will appear in the website, if you send the feed out to a website.

The activate button is easy to miss, its down below.

Next, we’re going to add some “personal” flares. These are simple flares people have created that do not exist in the default set of flares.

Click ‘Browse the Catalog’.

Browse through the available flares. When you find one you like, and want to select it, click ‘Link’.

A tab will open with the flare. Copy the URL.

Go back to the original screen. Paste the flare URL. Click ‘Add New Flare’

you will see your flare appear above. Select where you want it to show (Feed, Site or both).

Below, you can view a preview of your flares. Reorder them by dragging and dropping the items.

Don’t forget to click ‘Save’. It’s easy to miss down there.

Activate PingShot

PingShot will notify the reading services when there’s updates. This speeds up the delivery of your feed.

Go to Publicize > PingShot and click ‘Activate’.

Link to the Original Source of the Feed

Have you ever had your RSS feed scraped and duplicated onto another site without your permission? It happens all the time, especially as you get more popular (with the help of this guide!). Googlebot or users might have a hard time telling which article was the original source.

This is why we’re going to add a link the bottom of your RSS feed citing you as the original source of your content. Not only will this help users and engines make this determination, it will get you some extra back-links as well.

Adding RSS Source Link in Blogger
1. Go to Settings > Site Feed
2. Add the following code:
  <hr /> <a href="http://www.myblog.com">My Blog Name</a>
Adding RSS Source Link in WordPress
1. Go to Appearance > Editor > functions.php
2. Add the following code:
  function embed_rss($content) { if(is_feed()) $content .= "<p><a href='". get_permalink() ."'>'"; $content .= get_the_title() ."</a></p>"; return $content; } add_filter('the_content', 'embed_rss');

You now have reference to the original source (you!) of the content in your RSS feeds. As always, check your work to make sure its been done correctly.

Create Thank You’s

Some personalization and a thank you to your readers can go a long way. Here’s how to set up a simple message in your feed.

Go to Optimize > BrowserFriendly > Content Options

Click ‘enable’ and enter a personal message!

Time Your RSS Emails

Encourage more opens from your RSS email list by controlling send time.

Go to Publicize > Email Subscriptions > Delivery Options

Select your timezone and the best time for your audience. 9am-11am is often a good choice.

Redirect WordPress RSS to Feedburner

You may have a standard RSS feed built into WordPress. You may even have subscribers to it! With the help of a Plugin called “Feedburner Redirect” we’re going to make sure everything is going through feedburner.

You can find the plugin here – https://wordpress.org/plugins/tentbloggers-feedburner-rss-redirect-plugin/

Install it in your WordPress setup.
Activate it.

Enter your feedburner URL in both fields. And you’re done!

Video Sitemaps

If you have any sort of video on your website or blog, and especially if it’s marked up with metadata, you need to have a video sitemap. This gets your video content noticed, processed and indexed much faster by Google and Bing.

OPTION A – MANUAL GENERATION

If you have a small site with only a few videos, and you’re not adding them all of the time, you can easily generate your video XML sitemap manually.

First, I want to provide you with the skeleton template of the XML structure. You can cut and paste the template and then add your own data.

This is just the most basic template with the required fields.

Create an Empty XML File

Create a file, the name doesn’t matter, but I like to use: sitemap_video.xml

Then save it in your root directory, like: https://www.quicksprout.com/sitemap_video.xml

As mentioned, it doesn’t matter so much what you name it or even where you put it, but you’ll need to know this later for when you submit the sitemap to Webmaster Tools.

Paste This Into the XML File

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1">


	<url>

		<loc></loc>

		<video:video>

			<video:thumbnail_loc></video:thumbnail_loc>

			<video:title></video:title>

			<video:description></video:description>

			<video:content_loc></video:content_loc>

			<video:player_loc allow_embed="yes" autoplay="ap=1">

			</video:player_loc>

		</video:video>

	</url>

</urlset>
Explanation of above code;

Many of the properties in the template are optional, but I wanted you to have the whole thing 🙂

Required Fields

URL of Page
Video File URL or Player URL
Title
Description
Thumbnail

So let’s fill in our example template. I’ve stripped out all the other properties for now, so you can clearly see it with only the required elements;

Basic Code for One Video in XML Video Sitemap

<url> <loc>https://www.quicksprout.com/videos/neil-patel-video-1.html</loc> <video:video> <video:thumbnail_loc> https://www.quicksprout.com/thumbs/thumbnail.jpg </video:thumbnail_loc> <video:title>Advanced SEO for Bloggers</video:title> <video:description> An exclusive video with SEO expert Neil Patel. Drive ridiculous amounts of leads to your blog and learn the 7 secrets of conversion rate optimization. </video:description> <video:content_loc>https://www.quicksprout.com/video.flv</video:content_loc> </video:video> </url>

Adding Additional Properties

There are many additional properties you can add to your video sitemap, such as;
<video:duration> <video:expiration_date> <video:rating> <video:view_count> <video:publication_date> <video:tag> <video:tag> <video:category> <video:restriction> <video:restriction> <video:restriction> <video:gallery_loc> <video:gallery_loc> <video:price> <video:requires_subscription> <video:uploader> <video:uploader> <video:platform> <video:platform> <video:platform> <video:live>
Let’s add a few of these back into our example, so you can see them in action!
<url> <loc>https://www.quicksprout.com/videos/neil-patel-video-1.html</loc> <video:video> <video:thumbnail_loc> https://www.quicksprout.com/thumbs/thumbnail.jpg </video:thumbnail_loc> <video:title>Advanced SEO for Bloggers</video:title> <video:description> An exclusive video with SEO expert Neil Patel. Drive ridiculous amounts of leads to your blog and learn the 7 secrets of conversion rate optimization. </video:description> <video:content_loc>https://www.quicksprout.com/video.flv</video:content_loc>  <video:duration>750</video:duration> <video:rating>4.1</video:rating> <video:view_count>54321</video:view_count> <video:publication_date>2012-04-01T19:20:30+08:00</video:publication_date> <video:family_friendly>yes</video:family_friendly> <video:restriction relationship="allow">IE GB US CA</video:restriction> <video:requires_subscription>no</video:requires_subscription> <video:live>no</video:live> </video:video> </url>
Many of these should be self explanatory. You can always check Google’s documentation for further explanation of all the individual fields allowed.

Submit Your Sitemap to Google Webmaster Tools

Option A Submit it to Webmaster Tools Directly

This is the preferred method to submitting any xml sitemap to Google.

Sign into Webmaster Tools
View the profile for your website
Navigate to Site Configuration -> Sitemaps
Click “Add/Test a Sitemap” in the right hand corner
Enter the name of your sitemap and hit submit

Option B Add The Following Line To Your Robots.txt File

Sitemap: http://www.example.com/sitemap_video.xml

Like with any xml sitemap, if your robots.txt file set up correctly, Google with find and process your video xml sitemaps through having found it in robots.txt

.htaccess Hacks

These tips only work if your client is using Apache. If your client is using Windows IIS, then look at IIS hacks.

Locate your .htaccess file on your server.(See “How to locate .htaccess on your server”)
Once you’ve located it, edit the file using a text editor.If you are using Windows, I recommend Notepad. If you’re using Mac, download a free text editor like TextWrangler.
In your htaccess file decide what you want to do, then add a line of code:
- To create a custom 404 page
  
  Use “ErrorDocument” and put the URL to your custom 404 page at the end. Example:
  
  ErrorDocument 404 http://www.example.com/my-custom-404-page.html
- To password protect a folder
  1. First, you will need to first need to create an .htpasswd file. The easy way is to use this online tool to create it: http://www.tools.dynamicdrive.com/password/
  2. Enter your desired username on the left, and the password you want that person to have on the right.
  3. In the “path to .htpasswd” box, put it in a folder that isn’t accessible to the public. Usually, it’s safe to put in your home directory like “/home/myusername”
  4. Click submit and put the downloaded .htpasswd file into “/home/myusername”
  5. Now that you’re done that, type this into your .htaccess file
    AuthUserFile /home/myusername/.htpasswd AuthName EnterPassword AuthType Basic require user some_users_name
    Replace “some_users_name” with the username that will be allowed into this folder.
- To block users by IP address
  
  Type these four lines into your .htaccess file:
  Order allow, deny Deny from 111.222.333.444 Deny from 555.666.777.888 Allow from all
  In the lines that say “deny from,” replace the example IP addresses “111.222.333.444” with the real IP address you want to block.
- To block users by referrer
  
  Add these three lines to your .htaccess file:
  RewriteEngine On RewriteCond %{HTTP_REFERER} somedomain\.com [NC] RewriteRule .* - [F]
  If you want to block more than one referrer, add more RewriteCond lines like this:
  RewriteEngine On RewriteCond %{HTTP_REFERER} somedomain\.com [NC,OR] RewriteCond %{HTTP_REFERER} anotherdomain\.com [NC,OR] RewriteCond %{HTTP_REFERER} 3rdDomain\.com [NC] RewriteRule .* - [F]
  Notice that every line except for the last one should end in “[NC,OR]“
- To make something other than index.html be the default page.
  
  Let’s say you want “home.html” to be your default page. Use this line in the htaccess file:
  DirectoryIndex. home.html
- To 301 redirect an old domain to a new domain
  
  Add these lines to your htaccess file
  RewriteEngine on RewriteCond %{HTTP_HOST} ^olddomain.com [NC,OR] RewriteCond %{HTTP_HOST} ^www.olddomain.com [NC] RewriteRule ^(.*)$ http://www.newdomain.com/$1 [R=301,NC]
  Replace “olddomain.com” with your old domain name. This will 301 redirect links form your old domain to your new one with the WWW in front of it.
- To prevent someone hotlinking resources on your website
  
  Add these lines to your htaccess file
  RewriteEngine on RewriteCond %{HTTP_REFERER} !^$ RewriteCond %{HTTP_REFERER} !^http://(www\.)?mydomain.com/.*$ [NC] RewriteRule \.(gif|jpg|js|css)$ - [F]
  Replace mydomain.com with your domain name. These lines will prevent someone from hot linking GIF, JPG, JS, and CSS files.
- To redirect all pages from HTTPS:// to HTTP://
  
  Add these lines to your htaccess file
  RewriteEngine on RewriteCond %{SERVER_PORT} !^80$ RewriteRule ^(.*)$ https://www.domain.com/$1 [NC,R=301,L]
  Replace domain.com with your own domain.
- To redirect all pages from HTTP:// to HTTPS://
  
  Add these lines to your htaccess file
  RewriteEngine on RewriteCond %{SERVER_PORT} !^443$ RewriteRule ^(.*)$ http://www.domain.com/$1 [NC,R=301,L]
  Replace domain.com with your own domain.
- To redirect one URL from HTTPS:// to HTTP://
  
  Assume the URL is http://www.domain.com/mypage.html
  RewriteEngine on RewriteCond %{HTTP_HOST} !^80$ RewriteCond %{HTTP_HOST} ^www.domain.com/mypage.html [NC] RewriteRule ^(.*)$ http://www.domain.com/mypage.html [NC,R=301,L]

Detecting Googlebot

There may be many reasons why you’d want to detect Googlebot as the User Agent. These can be left to your imagination 🙂

Cut and paste the following code anywhere in the <body> of your document:
if(strstr(strtolower($_SERVER['HTTP_USER_AGENT']), "googlebot")) { // what to do }
Replace with your contentReplace “// what to do” with whatever you want to happen.Tip: To make it HTMLDo this;
if(strstr(strtolower($_SERVER['HTTP_USER_AGENT']), "googlebot")) {?> <h1>Put your HTML here</h1> <p>anything you'd normally do</p> <?php }
Let’s break apart each piece of the PHP.

if(condition){} – this is just a simple function that says “if x is true, do y”.

Now we’ll work from the inside of the nested statement out.
- ‘HTTP_USER_AGENT' – this extracts a browser specific ID string
- $_SERVER – this is an array with information such as headers, paths, and script locations, which is created by the web server
- strtolower – returns string with all alphabetic characters converted to lowercase.
- strstr – returns part of haystack string starting from and including the first occurrence of needle to the end of haystack
{ // what to do }
forward slashes // are just used to make a comment. All we’re saying here, is to put whatever you want to happen between the curly brackets.

If you like visuals better – this is a good explanation of the pieces of code;

Add Custom Search Engine to Your Site

A custom Google search on your site can be a powerful feature, yet still many people are not using it. I’m going to walk you through the steps of installing it on your site.

Go to – https://cse.google.com/cse/

Compose a Title and Description

Add Sites To Be Included in Search

This is where a little technical know how applies.

You can’t just simply add the site URL – you must include an asterisk (*) after the URL for it to search the entire site, like this;

https://www.quicksprout.com/*

Select Edition and Confirm

After we have done all this, we’re given the opportunity to try out the custom search engine before installing it on our site.

Let’s Check It Out

A search for [twitter tips] in our custom engine shows good results and variety among the sites in the engine. (Note the ads will be present unless you pay for a premium version).

Now it’s time to install in your site!

Installing Custom Search In Your Site

This type of installation is going to either be a new page or a new post. Let’s use a new page in this example, however a new post will work in the same way.

Navigate to a “new page”
Edit in HTML ModeWe’re going to edit in HTML mode since we’ll be pasting JavaScript code into the page.
Paste Code
Let’s preview it

Block Potentially Malicious or Harmful Links to Your Site

Occasionally a hacker or even just someone inexperienced without malicious intent will send a back link over to your site with a query parameter attached to the end of it. It might look something like the following:

Grow your business — Quick Sprout

(Please don’t link to me or anyone else like that)

And the malicious query string can end up on different pages:

https://www.quicksprout.com/page/2/?neilpatelscam
https://www.quicksprout.com/page/3/?neilpatelscam

These pages can get indexed as such, and could potentially replace the real pages in their index. This is unlikely, but in either case, you may need a way to fix this if it happens. Here’s a bit of .htaccess code to do so:
# FIX BAD LINKS <ifModule mod_rewrite.c> RewriteCond %{QUERY_STRING} querystring [NC] RewriteRule .* http://example.com/$1? [R=301,L] </ifModule>
Simply do the following:

Make sure the .htaccess is in your root directory.
Place this bit of code at the bottom on the .htaccess file.
Replace “querystring” with the malicious querystring being used.
Replace example.com with your site’s URL
To add multiple querystrings use the “pipes” ( | ) as an “or” expressions: (neilpatelscam|quicksproutripoff|badblogger) for the querystring.
Finally, run a site: query in Google a week or two later, such as: site:quicksprout.com/?neilpatelscam to see if they’ve been removed from the index.

Browser Plugins for On-site Analysis

Browser plugins can greatly speed up your workflow and efficiency. I’m going to show you some plugins for Google Chrome, and a little bit about how to use them in more advanced ways.

This section of browser plugins revolves around the ones that help optimize your sites accessibility and indexation.

First, here’s the list.

Broken Link Checker https://chrome.google.com/webstore/detail/check-my-links/ojkcdipcgfaekbeaelaapakgnjflfglf
Web Developer https://chrispederick.com/work/web-developer/
Redirect Path Checker https://chrome.google.com/webstore/detail/redirect-path/aomidfkchockcldhbkggjokdkkebmdll
Google Analytics Debugger https://chrome.google.com/webstore/detail/jnkmfdileelhofjcijamephohjechhna
Microformats for Chrome https://chrome.google.com/webstore/detail/oalbifknmclbnmjlljdemhjjlkmppjjl

I’m going to show you how to use some of these in an advanced way.

Broken Links Checker

Not only is the broken links checker a great plugin to find broken links quickly on your site, but you can use it in creative ways on other people’s sites to get ideas for linkbuilding and prospecting.

For example, try running it on the sitemap of a competitor’s website. Here’s how:

Find a competitor with an HTML sitemap. For this example I’m going to randomly use www.bizchair.com and their sitemap is https://www.bizchair.com/site-map
1. Run the Link Checker
2. Click the icon for the extension
3. Wait for it to find the broken links – in this case there are quite a few.

Chrome Sniffer

This plugin automatically shows you the CMS or script library a website uses. Extremely handy if you are looking to reach out to only WordPress site owners, for example.

As you browse the web, the icon to the far right of the URL will change to match which CMS or library is being used.

For example, you can see that my site is built on WordPress and here is a site built with Drupal.

Redirect Path Checker

This plugin will automatically alert you if you were taken to a page via any kind of redirect. Can be very useful when browsing your site, in the case that you are internally linking to outdated URLs (or externally for that matter)

For example, I just found on my site this link to Gizmodo 302 redirects:

How did I know? Because the plugin alerted me to the 302.

And then you can click on the icon and it will show you the redirect (or series of redirects) that the browser took to get to a page.

The SEOmoz Toolbar & Plugin

You can do many things with the Moz plugin. A few of the more advanced things you might use it to look for are:

Quickly finding followed vs nofollowed links
Or finding the country and IP address for the website

Conclusion

Well done!

You made it through the post!

Use all of these tips to ensure optimal indexation and accessibility and you will see vast improvements quickly.

Additional menu