http-https-crawl

After switching to HTTPS it's a good idea to run a scan, a crawl. You want to make sure your internal resources and internal links don't use HTTP anymore.

We'll use the https://webhelpforums.net domain as an example. It recently changed to HTTPS.

In this case you can still reach the domain via HTTP. The site deals with that by adding a canonical link to the HTTPS counterpart.

Configure The Website Scan

Required:

  • In "Scan website

Paths" enter the "https" domain root to scan - in his example it is https://webhelpforums.net

  • In "Scan website
  • Crawler options" switch off all options starting with "Fix internal URLs with"... We want all these unchecked when auditing for "http" versus "https" usage in a website.

    Optional:

    • If your website uses Javascript for its navigation or similar:
      In "Scan website

    Crawler options" enable option: "Try search inside Javascript"

  • If your website uses AJAX to load vital content on page load:
    In "Scan website

  • Crawler engine" switch to "HTTP using WinInet/IE - uses Internet Explorer for setings and AJAX content"

    When done, click the "Start scan" button.

    Analyze Results After The Scan Has Ended

    Switch to the "Analyze website data" view. Then click the "External" tab.

    Here you can see all URLs encountered during the website scan that do not belong to the domain https://webhelpforums.net

    In the quick filter enter https://webhelpforums.net: a1wa-quick-filters

    You can now see all URLs from the "https://webhelpforums.net" domain that are linked, used or redirected to by any page crawled at "https://webhelpforums.net".

    Tip: You may also want to check found URLs from other known domains you use to serve content.

    To see from where an URL was found, used or redirected to from, click on it. Then on the right side select "Exended data". You can then check the sub tabs "Linked by", "Used by" and "Redirected from".

    Depending on how the URL was found, you can see additional information such as on which line a link was found or what the anchor text was.

    Include All Data In The Data Columns Area & Export Results

    If you do not want to look under "Extended data" for each URL, you can also have the information visible in the data columns at the left side - also ideal when exporting the data for outside review and auditing.

    You can achieve this by enabling visibility of the following data columns in menu a1wa-data-columns "View

    Data columns

    URL references":

    • Linked to by list
    • Used by list
    • Redirected to by list

    a1wa-extended-data

    Tip: There are more data columns with additional information.

    You can now:

    1. See the new visible columns and data while inside and using the A1 Website Analyzer program. Remember you may need to resize and widen the left side or use the horizontal scrollbar.
    2. Export all the data in the data columns area - simply use the menu "File

    Export selected data as file..." and pick which format you want, e.g. comma separated values .CSV file.

    About the Author: Ruud Hein

    I love helping to make web sites make it. From the ground up if needed. CSS challenges, server-side scripting, user and device friendly JavaScript tricks search engines have no problems with. Tracking how the sites perform and then figuring out how to make that performance and the tracking better. I'm passionate about information. No matter how often I trim my feeds in my feed readers (yes, I use more than one), I always have a couple of hundred in there covering topics ranging from design to usability, from SEO to SEM, from life hacks to productivity blogs, from.... Well, you get the idea, I guess. Knowledge and information management is close to my heart. Has to be with the amount of information I track. My "trusted system" is usually in flux but always at hand and fully searchable. My paid passion job at Search Engine People sees me applying my passions and knowledge to a wide array of problems, ones I usually experience as challenges. It's good to have you here: pleased to meet you!