Crawlability: Just How Well Can Google Crawl JavaScript

Google has crawled JavaScript since 2008. The idea that Google can crawl these types of pages is nothing new. The problem? Google wasn’t able to gather a whole lot of amount of information from JavaScript pages, limiting their ability to be rendered and found over HTML websites.

Now, Google has evolved, or so it seems.

“Blogs that use JavaScript have started to see more pages crawled and indexed over the past year, which clearly means Google’s evolving its support of this language.”
-- Matt Banner, OnBlastBlog.

Which Google’s Bot Does The Crawling?

Some industry experts have speculated whether Google uses separate bots to crawl JavaScript websites versus regular HTML sites.

The quick answer is no, according to John Mueller, Webmaster Trends Analyst at Google. Still, Mueller was quick to follow up and say that JavaScript won’t always render the first time Google crawls the page.

“The timing of when Google crawls the code on the page and when it renders is an important one when testing and analyzing SEO tactics with JavaScript. With this in mind, you can have more realistic expectations of when you should see a difference in your page’s performance.”
-- Scott Lazerson, Connector.

With this in mind, what can you do to make your JavaScript sites more search engine friendly? Here are a few expert tips.

JavaScript Redirects

Some tests have been done to see if a JavaScript redirect can have the same impact as a 301 redirect (the type of redirect that shares the link love from the original page with the page it’s being directed to). Some show that Google treats these redirects the same, while others show otherwise. There doesn’t seem to be any rhyme or reason to it.

The key is to test your redirects and keep a close eye on them. Sometimes, it might take a little longer for Google to start sending traffic to the new page because it takes longer for the bots to crawl JavaScript.

DOM Elements

Another test done by Search Engine Land contributor, Adam Audette, looked at whether Google is able to read changes to the Document Object Model (DOM) made by JavaScript.

The DOM is an essential part of how a webpage is accessed, read, and built by a browser.

Although the DOM isn’t tied to a specific programming language, like JavaScript, it is the connection point between the search engines and the web pages and programming language. This is important because it’s how search engines are able to read what’s happening on a webpage when ranking it.

Adam's conclusion:

"Dynamically inserted content, and even meta signals such as rel canonical annotations, are treated in an equivalent manner whether in the HTML source, or fired after the initial HTML is parsed with JavaScript in the DOM"
-- Adam Audette

Nix All Ajax Crawling Schemes

If you’re creating a new website, steer clear of using Google's 2009 AJAX crawling scheme. If your current website uses this now, consider making a switch.

In 2009, Google recommended using the AJAX Crawling Scheme. They have since changed their tune.

Google reportedly discontinued their support for the AJAX Crawling scheme in 2015. Google still crawls sites that use it but recommends not using it on new sites.

Progressive enhancement is the preferred method of site building.

Review Your URLs

The symbol # creeps into many JavaScript-driven website URLs. If your URLs have # Google won’t index the page won't crawl and index the part after the #.

The exception to this rule is any URLs with “#!” in the URL. URLs with “#!” are still considered acceptable by Google. (See the above crawling scheme, now deprecated)

Use The Search Console Fetch And Render Tool

As mentioned earlier, testing JavaScript pages is important. Google doesn’t seem to have a method to its madness when it comes to these pages.

To see how your pages appear to Google, use the Search Console’s Fetch and Render tool. Although this tool still won’t see any URLs with “#!” or “#” in the URL, it will still give you a good glimpse into what the powerhouse search engine is seeing on your website.

Limit Your JavaScript Files

We’ve determined that Google can crawl JavaScript pages. Still, too many JavaScript files could be doing you more harm than good.

To maximize how well Google is crawling your pages, it’s a good idea to limit how many JavaScript files you have on your website. If you have a good amount of these files on your website, it can timeout without the files being loaded or rendered.

Upload The Correct Sitemap

This might seem like a small tweak but it’s an important one.

When uploading your sitemap, be sure to use the right “lastmod” dates. This sends Google a signal of what’s been changed and updated on your website so their bots know where to crawl and what to render.

One Last Consideration: Google Is Only One Search Engine

Remember, as you’re optimizing your website and incorporating JavaScript, Google is only one of the search engines you need to please. Although Google is the most widely used search engine, others, such as Bing and Yahoo, are also popular. These other search engines might not always support JavaScript, which could mean your pages are getting ignored by the bots that rank them.