SEO Glossary: Technical SEO

Before Google can rank a page, it has to find it. Crawling is that first step, and on many small business websites it is quietly broken.
What crawling means
Crawling is the process search engines use to discover pages on the web. Google sends an automated program, called Googlebot, to request URLs and read what comes back. Each page it reads gives it new links to follow, so it moves through your site one link at a time, a bit like a visitor clicking through every menu item and button.
Crawling is not the same as indexing or ranking. Crawling is discovery. Indexing is Google deciding to store the page. Ranking is Google choosing where to show it. A page that is never crawled cannot be indexed, and a page that is not indexed cannot rank, no matter how well it is written.
Why it matters for South African businesses
Most local business sites are small, so Google is rarely short of time to crawl them. The real problem is usually that important pages are hard to reach. A new service page that is not in the menu, a location page only linked from an old blog post, or a quote form page hidden behind a script can sit unseen for weeks. When the page that is meant to win enquiries in Bloemfontein or Durban is not being crawled, you lose leads to competitors whose pages are easier to find.
Google also crawls mainly as a smartphone. If your mobile layout hides links or loads key content only after a tap, Googlebot may never see it.
How Google crawls a website
1. Discovery
Google learns about URLs from links on pages it already knows, from links on other websites and from your XML sitemap.
2. Fetching
Googlebot requests the URL. Your server answers with a status code: 200 for a working page, 301 for a permanent redirect, 404 for a missing page or 5xx when the server fails.
3. Reading
Google reads the HTML, renders the page where needed, checks robots rules and collects every link it finds.
4. Queueing
Working pages are passed on for indexing, and the new links join the list of URLs to visit next.

Common crawling mistakes
- Blocking important folders or the whole site in robots.txt, often left over from a development build.
- Orphan pages with no internal links pointing to them.
- Broken internal links that send Googlebot to 404 pages.
- Redirect chains, where one URL redirects to another that redirects again.
- A sitemap full of redirected, deleted or noindexed URLs.
- Key content and links that only appear after JavaScript runs or a button is tapped.
How to check and improve crawling
Start in Google Search Console. The Pages report shows which URLs Google has found and why some are not indexed. The URL Inspection tool tells you when a page was last crawled and whether robots.txt allowed it. The Crawl Stats report, under Settings, shows how often Googlebot visits and how many of its requests fail.
Then fix the basics. Link every service and location page from your menu or from relevant pages. Keep robots.txt short and only block areas that genuinely should stay private, such as admin folders. Submit a clean XML sitemap that lists only live pages you want found. Replace internal links that point to redirects with links to the final URL, and repair or redirect any 404s that still get links.
How LSA helps
As part of our technical SEO services, LSA a Local SEO Agency crawls your site the way Google does, maps every page and flags blocked, orphaned and broken URLs. Our SEO audit service turns those findings into a prioritised fix list, starting with the pages that bring in enquiries. If you trade in Gauteng, our Johannesburg SEO agency page shows how we apply this locally.
Want the short version? Read the crawling definition in our SEO Glossary, alongside related terms such as indexing, robots.txt and XML sitemaps.
Not sure if Google can find your most important pages? Call LSA a Local SEO Agency on 068 786 7331 or contact us for a crawl check.