Building Search Engine Friendly Sites: Common Questions

Recently, I spoke at SES San Jose on “building a search engine friendly website” with emphasis focused on the incorporation and embedding of multimedia. Before narrowing my speech down to multimedia I had researched some common questions beginner SEOs or Web developers diving into SEO have when learning to build search engine friendly.

Below are a few common questions I came across and are addressed specifically:

What are common errors that beginners make when designing their "search friendly" site?

1) Using images, Flash, JavaScript, iFrames for to display navigation or textual content. Bots can't crawl content or links embedded with these technologies though Flash capabilities have been significantly improved lately for Google.

2) Separating words in images/page filenames with underscores instead of dashes. With underscores, bots view your words in a filename as all ran together. With dashes, bots can separate the words in your file name as separate words. It's important to note that Google is close to separating words with underscores but isn't quite there yet.

For Example:

"search_engine_optimization.jpg" is viewed by bots as "searchengineoptimization" as the keyphrase in the image.

As opposed to:

"search-engine-optimization.jpg" is viewed by bots as "search engine optimization" as the keyphrase in the image.

3) Duplicating Titles and/or Meta-Data across the whole Website. Each page should have unique meta tags, and most importantly, unique keyword-rich title tags for each individual page. Duplicate titles and meta-data can cause pages to be put in Google's supplemental index which will kill your search traffic.

How does using frames impact how search engine finds your site?

It used to be that search engine bots couldn't crawl frames and therefore couldn't find your content. A lot of people still believe that and while certain search engine bots might not be able to the major ones can.

The issue isn't a matter of crawlability but a matter of linking. You cannot link to individual pages because they are locked in a frameset. People cannot bookmark pages correctly and cannot link to pages correctly so this creates a huge accessibility issue. People not being able to correctly link to your site drastically effects search rankings. If there's any sort of algorithmic quality score assigned to a site by search engines then it's certainly a ding in the score for the accessibility issues.

What are good ways or methods for a beginner to test how search friendly their site is?

1) Simulate the hierarchical order of how search engine bots are viewing your site with Lynx Viewer. There's a good Mozilla add-on called Yellow Pipe Lynx Viewer that shows you how bots are accessing your content and which order. Ideally, you want to make sure bots are seeing all your content, accessing the most important links first, etc.

2) Web Developer Toolbar allows you to disable JavaScript, Cookies, and CSS to make sure your content is still accessible to bots and people who have disabled or don't have those capabilities.

3) Register and verify your site with Google's Webmaster Tools. Google will provide feedback on any crawling errors it may be having with your site.

What are some of the key issues that affect search engine robots?

Duplicate Content – Bots don't like to have multiple versions of content in their index. If you have multiple versions of a page bots will pick one to index and ignore the rest. Google will put duplicate versions in its supplemental index where they aren't accessible under regular search queries.

The real problem with duplicate content is that your links are diluted across multiple versions. It is wise to consolidate your duplicate pages by 301 redirecting duplicates to just one centralized page.

Some fear a penalty with duplicate content but only in rare cases will you be penalized for it. If you site is heavily duplicating content from other domain names without any of your own unique content then that may be cause for a penalty.

Dynamic URLs – Search engine bots have gotten a lot better at crawling parameters in dynamic URLs now but if you have an excessive amount of parameters most bots will have trouble crawling them. You also lose how on the value of adding keywords to your URLs. It is recommended that you use mod_rewrite to make your URLs easily accessible to spiders and have keyword rich value.

HTTP Headers – It's important to know if bots are accessing your content in the correct manner. Essentially, you want to return an HTTP 200 status OK header to bots for your pages. If you are doing a redirect you want to make sure they're receiving an HTTP 301 header (or 302 in rare occasions). I recommend that people create a custom HTTP 404 header error page so bots can continue to crawl your site if they hit a dead internal link.

Server Load – The lower the file size of your page the better. The less calls to the server for external JavaScript or CSS files the better. If you site is too content heavy you can render bots from fully crawling the deeper levels of your site. Sometimes bots have an allocated spend where they'll only crawl X kilobytes of your site until they move on. You want to make sure your pages are small and fast loading so bots can crawl efficiently.

Following Links – Bots can have trouble crawling links in instances where your links are in JavaScript, Flash (even though there's been vastly improved capabilities), or Frames. Make sure your links are easily accessible to search engines.

Robot Directives – You can use Robots.txt directives or Meta-Tag directives to tell bots whether or not to crawl, index, or follow links on your pages. This comes in very handy when trying to block spiders from duplicate content.

Are there any specific development/optimization tools for beginners to make use of when optimizing their site for search?

I could go on all day. I won't detail each tool but here are a few great resources:

WebCEO
RankSense
Back Link Analyzer v2.0
WebConfs SEO Tools
SEO Chat SEO tools
SEOmoz SEO tools
SEO Oregon
Portland SEO
iWebTool
Complete List of the Best SEO Tools
SelfSEO
136 SEO Tools
We Build Pages SEO Tools
XML Sitemap Generator
SEObook Tools
Search Engine Marketing Company

Mozilla add-ons:

• Web Developer Toolbar
• Tails Export (for microformats)
• User Agent Switcher
• XHTML Mobile Profile
• Yellowpipe Lynx Viewer
• Modify Headers
• Search Status
• SEO for Firefox
• Live HTTP Headers
• Link Diagnosis
• IE View
• Greasemonkey
• DOM Inspector
• Firebug

If there are any further questions anyone has about any of the items addressed please feel free to contact me at my SEO blog or through my SEO firm.