How to Get an AJAX Website Indexed - A SEO Test and Working Example

October 14, 2011

This site was built to test and document the agreement between crawler and server that Google published for Making AJAX Applications Crawlable by the search bots. By making your AJAX application accessible to the search bots (Googlebot and supposedly BingBot) you can apply SEO techniques to your content as you would for a non-AJAX site. Technical details for building an AJAX website are provided.

AJAX is great for updating portions of your webpage dynamically. However, since the content is produced dynamically, AJAX sites have not been SEO friendly because the content is not visible to crawlers and thus difficult for search engines to index.

If you are using AJAX to populate content on your page, there are four features that you need to support using server and client side code:

  1. Making the AJAX content crawlable by the search bots
  2. Updating the <title> tag dynamically after the AJAX content has loaded
  3. Making AJAX Crawlable URLs for direct access to the AJAX content
  4. Supporting the browser back button for browser history

This site provides code examples and documentation for how these issues were addressed for this website.

In addition, technical details are provided for the following SEO elements:

  1. Testing Googlebot access to AJAX content
  2. Using Google Analytics to track AJAX content
  3. Preparing a sitemap for AJAX pages

Observations and findings are catalogued for the following topics:

  1. Differences in searchbot indexing for Googlebot, Bingbot and Yahoo Slurp

How This Site Works

This site intentionally does not use progressive enhancement. Other than the homepage (this page), all pages on the site are dynamically loaded and only accessible via an AJAX call that loads <div id="content"></div> with content.

You can test this by turning off all JavaScript on your browser then refresh the page and click the links in the left navigation pane. Notice that nothing happens. (Be sure to re-enable JavaScript!)

This homepage has no AJAX content. You can return to this page at any time by clicking the ‘Overview’ link in the left navigation. Whenever this page loads, the entire page is transferred from the server. This homepage is an HTML template that contains JavaScript code, the page header, the left navigation panel, and the content which you are reading now.

All of the other links below the ‘Overview’ link use AJAX to replace the contents of the <div id="content"> without reloading any other part of the page.

Technical details for this site are explained in the pages of this website. All explanations are based on how the feature or technique was implemented specifically for this site.

NOTE: When you click on Click for ugly Menu the menu will serve snapshot pages. Notice the ugly URLs in the address bar.