RankNest

Link Mapping: Crawling Your Site

Run the site crawler wizard: configure the crawl, organize topic clusters, let AI categorize pages, and import them onto the link map.

The link map only shows what RankNest has crawled, so every map starts with a crawl. The site crawler is a guided wizard: it discovers the client's pages, lets you set up topic categories and clusters, offers an AI first pass at sorting and typing every page, and then imports the pages you approve onto the canvas.

You'll run a full crawl once when you set the client up, and then re-crawl whenever the site changes. This page covers both, plus the two smaller ways pages get onto a map: adding a single URL by hand, and approving pages RankNest detects on its own.

Where to find it

Open a client, click Link Mapping in the sidebar, then click Crawl at the top of the Topic Clusters tab. The Crawl Manager opens with two choices.

  • Quick Rescan updates links and status for the pages already on the map. It does not discover new pages. Use it after you've added internal links and want the map (and your Linking Plan) to catch up. It's the faster of the two.
  • Full Discovery crawls the website from scratch to find new pages, broken links, and structure changes. This opens the four-step wizard.

If the map is empty you'll see a notice telling you to use Full Discovery first, since Quick Rescan has nothing to rescan.

Note: If Google Search Console isn't connected for this client, a prompt appears above the crawl options. Crawled pages still import without it, but anything that depends on search data stays empty. See Google Search Console.

How to run a full discovery crawl

The wizard has four numbered steps, shown across the top: Configure, Organize, AI Categorize, Assign Pages. Back and Cancel are always available in the footer.

1. Configure

  1. Check the Website URL. It's locked to the domain assigned to this client in Client Settings, so you can't point a client's map at someone else's site.
  2. Set Crawl Depth, which is how many link levels deep to follow, from 1 to 4 levels.
  3. Set Max Pages, the ceiling for this run (25, 50, 100, or 200 pages).
  4. Under Filter Options, tick Skip category pages and Skip tag pages if the site generates a lot of low-value archive URLs you don't want cluttering the map.
  5. Click Start Crawl.

The status line updates as pages come in ("Crawling… found 42 pages so far"). Large crawls run in stages, so the count climbs in waves rather than all at once. When it finishes you'll see how many new pages were found, and the wizard moves you on.

Tip: If the crawl reports that the site is large and was cut short, run it again. Each run picks up where coverage left off, so a second pass captures additional pages.

2. Organize

Set up the structure you want the pages sorted into. Click New Topic Category, name it, pick a color, and click Create Topic Category. Then use + Add Topic Cluster on any category to add clusters underneath it.

You can skip this step entirely, since the AI step can propose a whole structure from scratch. Build it by hand when you already know how the site should be organized and want the AI to fit pages into your buckets rather than invent its own. See Page Types & Clusters for what categories and clusters mean.

Click Continue to AI Categorize when you're done.

3. AI Categorize

This step reads the newly-crawled pages and proposes where each one belongs and what type it should be. Before you run it you'll see counts for new pages, excluded pages, and your existing categories and clusters, plus a note that it uses one AI credit from your plan. Click Show excluded pages to see which URLs were left out and why.

Click Let AI Categorize Pages, or take the Skip option to assign pages manually and go straight to the next step.

The result is fully editable before you accept it:

  • Rename any category or cluster it invented. Items marked Existing are ones already on your map and will be reused; items marked New get created on import.
  • Change any page's type with the dropdown on its row.
  • Drag a page by its handle onto a different cluster, or drag a cluster's handle onto a different category.
  • Use Add Category and + Add Cluster to add your own, and the trash and ✕ buttons to drop anything you don't want.
  • Collapse all / Expand all make a big proposal easier to skim.
  • Anything the AI left out appears under Unassigned pages with an Assign button.
  • Regenerate throws the proposal away and asks for a fresh one.

Click Apply to accept the proposal. Apply also starts the import immediately. You don't need to visit the next step unless you want to fine-tune individual pages first.

4. Assign Pages

This is the manual control panel: every crawled page is one row, with a checkbox that decides whether it gets imported. The counter on the right reads "N of M pages will be imported".

  • Tick or untick individual rows, or use the checkbox at the top left to mark every visible row.
  • Filter with the search box and the All Types dropdown. The ? icon next to it explains what each page type means.
  • Set a row's type with its type dropdown, and its cluster with the cluster dropdown next to it. Both dropdowns can create a New category or New cluster inline.
  • Drag a row by its grip handle onto a cluster in the Drop on a cluster rail on the right.
  • Click rows to select them for bulk edits. Shift-click for a range, ctrl-click (cmd-click on Mac) to add one at a time. A floating bar appears with bulk type and cluster actions.

Click Import N Pages to finish.

Two confirmations can appear at this point:

  • URL Changes Detected lists pages already on the map that now live at a new URL. Tick the ones to update and click Update N URLs, or Keep Original URLs. Rows flagged Conflict can't be updated because another page on the map already uses that URL.
  • Pages Without Topic Group means some pages have no cluster or category. Choose Go back to assign them, or Import anyway and sort them out on the canvas later.

Adding a single page

You don't need a full crawl to add one URL. In the link map sidebar, click the + button next to the search box.

  1. Paste the URL into Page URL and click Fetch. RankNest scrapes that one page for its title, description, and links.
  2. Review the preview: title, URL, meta description, first H1, how many links it contains, and a suggested page type.
  3. Adjust Page type and Topic category / cluster.
  4. Click Add to map.

Any links from the new page to pages already on the map are created straight away. Links to the new page from existing pages only appear after the next full crawl, because RankNest has to re-read those pages to see them.

Reviewing detected pages

RankNest also picks up pages on its own between crawls, from Search Console data and from the Site Connector. Those pages land on the canvas marked Pending and grouped where the AI thinks they belong, but nothing is committed until you say so.

When any are waiting, a N pages pending review button appears at the top right of the canvas. Click it to open Review detected pages:

  • Every row starts ticked. Untick a row to reject its suggested placement. The page stays on the map, just ungrouped.
  • Change the page type on any row before approving.
  • Approve N pages applies the suggestions; Dismiss all rejects every one.

Re-crawling

A progress card appears at the top of the canvas showing the current phase and a percentage, and the canvas refreshes when it's done.

Re-crawling does more than refresh metadata:

  • New pages and new links appear on the map.
  • Links you added because the Linking Plan told you to are detected automatically and their recommendations flip to implemented, so you don't have to tick them off by hand.
  • Pages whose URL changed are offered as URL updates rather than duplicated.

Use Quick Rescan for a routine refresh of existing pages, and Full Discovery when the site has genuinely new pages.

Keeping a page off the map

Some URLs will never belong on a link map: staging URLs, thank-you pages, paginated duplicates. Deleting the page isn't always enough, because the next crawl or auto-import can bring it back.

When you delete a page, tick Never re-add this page in the confirmation dialog. That blocks it from auto-import and from future crawls. To reverse it, open the sidebar's settings icon (Link Map Settings), find the URL under Ignored pages, and click Un-ignore. It comes back on the next sync rather than immediately.

Tips

  • Crawl first, organize second. It's much easier to build categories once you can see the real page list than to guess them up front.
  • Run the AI categorization even if you plan to rearrange it. Correcting a proposal is faster than sorting a few hundred pages from an empty structure.
  • On a big site, import in passes. Bring in the pages that matter (Money, Pillar, Service, Location) first, get them typed and clustered, then add the long tail of blog posts.
  • After any import, re-crawl before you generate a Linking Plan so link data for the new pages is complete.

Troubleshooting

The crawl found far fewer pages than the site has. Raise Max Pages and Crawl Depth on the Configure step and run it again. Pages that aren't in the sitemap and aren't linked from anywhere the crawler reached can't be discovered. Add those manually.

A link I can see on the live page is missing from the map. Only editorial, in-content links are counted. Navigation menus, headers, footers, and sidebar widgets are deliberately excluded because they appear on every page. Check whether the link lives in one of those on the live site.

"Let AI Categorize Pages" is disabled. Either there are no new pages to categorize, or your plan's AI credits are used up. The step shows which, with a link to your subscription.

A page keeps reappearing after I delete it. It's being re-imported from Search Console, the Site Connector, or a crawl. Delete it again with Never re-add this page ticked.

Pages show up on the canvas that I never imported. They were auto-detected and are waiting for approval. Look for the pages pending review button at the top right of the canvas.

Last updated 2026-07-25