Technical SEO is the work that lets search engines and AI crawlers reach your pages, read them and store the right version. It covers status codes, redirects, robots.txt, sitemaps, canonical tags, caching headers and page weight, not keywords or content. When it is broken, good content can sit unseen.
On 24 September 2026 we ran a technical SEO audit on the homepages of 208 Australian small business websites taken from Google Maps results, and 193 loaded. The most common problem was a missing caching header that barely matters for a small site. The most surprising one was something few owners know about: one server in eight turned away OpenAI’s GPTBot crawler, while only five sites had asked for that in their robots.txt file.
Key findings
- 24 of 191 sites that loaded normally for a browser (12.6%) refused GPTBot, and 27 (14.1%) refused ClaudeBot. 23 of the 24 had a robots.txt that allowed GPTBot, so the block was not a choice written down anywhere.
- Only 2 sites refused OAI-SearchBot, the crawler ChatGPT search uses, so most of these blocks hit training crawlers rather than live AI answers.
- 3 homepages (2%) were set to noindex and 12 (6%) returned a normal page for a made-up address instead of a 404.
- 72 homepages (37%) sent neither an ETag nor a Last-Modified header, the most common problem we found and one of the least harmful.
- The median homepage was 225 KB of HTML before any image loaded, and 28 (15%) were over 500 KB.
What technical SEO covers, and what it does not
Think of it as the plumbing. Crawlability is whether a crawler can reach a page. Indexability is whether it is allowed to store it. Then come the signals that tell it which version is the real one (redirects and canonicals), how to find everything (sitemaps and internal links), and how heavy the page is to load. Keywords, service pages and reviews sit on top of that plumbing, which is why our SEO agency team starts every campaign with a technical audit before writing anything.
For the on-page and local side of the same 193 sites, such as titles, phone numbers and area pages, see our companion study on SEO for small business. This post sticks to the technical layer.
The technical SEO problems we found, ranked
Here are the 12 problems by how often they appeared.
How often a problem appears tells you little about how much it hurts, so the table puts the same 12 in order of damage, with a way to check each one yourself. We rate damage from what Google documents, not from guesswork about ranking factors.
| Problem | Sites affected | How much it matters | How to check yourself |
|---|---|---|---|
| Homepage set to noindex | 3 of 193 (2%) | Severe. The page is kept out of Google results. | Search Console URL Inspection, or view the page source and search for “noindex” |
| Fake URL returns a normal page (soft 404) | 12 (6%) | Medium. Wastes crawling and lets junk addresses look like real pages. | Visit yoursite.com.au/made-up-page-123 and check the status code in a header checker |
| www and non-www both live | 13 (7%) | Medium. Two copies of every page compete with each other. | Type both versions of your address and see whether one redirects to the other |
| http not redirected to https | 9 (5%) | Medium. Another duplicate copy, and browsers warn visitors. | Type http:// in front of your address |
| AI crawler refused by the server | 31 of 191 (16%) | No effect on Google rankings. Can keep you out of some AI tools. | curl -I -A “GPTBot/1.2” https://yoursite.com.au/ and compare with a browser |
| No canonical tag | 16 (8%) | Low to medium. Google picks the canonical itself. | View source and search for rel=”canonical” |
| Canonical points somewhere else | 7 (4%) | Medium. Mixed signals about which address is real. | Check the canonical matches the address in your browser bar exactly |
| Structured data fails to parse | 9 (5%) | Low to medium. Broken markup is ignored. | Google’s Rich Results Test |
| Two redirects from the http address | 18 (9%) | Low. Google follows up to 10 hops, but chains slow crawling. | Any online redirect checker, starting from http://www. |
| Homepage HTML over 500 KB | 28 (15%) | Low to medium. Heavier pages tend to load slower on phones. | PageSpeed Insights, then look at the document size |
| No working XML sitemap | 15 (8%) | Low for small, well-linked sites. | Open yoursite.com.au/robots.txt and follow the Sitemap line |
| No ETag or Last-Modified header | 72 (37%) | Low for small sites. Matters more as sites grow. | curl -I https://yoursite.com.au/ and look for etag or last-modified |
AI crawlers turned away at the server
We did not expect this finding to be so large. We requested each homepage as a normal browser and then as four AI crawlers, and repeated the test on every site that refused one, to rule out a one-off. A site counts as refusing only if it served a browser normally and returned an error to the crawler both times.
24 sites refused GPTBot and 27 refused ClaudeBot, usually with a 403, 429 or an odd 418 status. Only 5 sites blocked GPTBot in robots.txt, the file where a business normally states which crawlers it welcomes, and only one of those overlapped with the server blocks. In other words, 23 of the 24 GPTBot refusals came from a server rule while robots.txt said the crawler was welcome.
The fingerprints point to hosting and security settings, not owner decisions. 17 of the 24 GPTBot refusals came from LiteSpeed servers, 13 of them with that unusual 418 status, which looks like a bot block list applied at the hosting level. Six came through Cloudflare, which announced in July 2025 that new domains would block AI crawlers by default unless the owner chose otherwise. This matches what we saw in August 2026, when our llms.txt study of 589 Australian sites found 13% of the sites that loaded returning an error to an AI crawler.
Before you panic, look at which crawlers were refused. OpenAI’s crawler documentation says GPTBot collects content that may be used to train its models, while OAI-SearchBot is used to surface websites in ChatGPT’s search features. Anthropic’s crawler help page draws the same line between ClaudeBot (training) and Claude-SearchBot and Claude-User (search and live answers). Only 2 of 191 sites refused OAI-SearchBot and 2 refused PerplexityBot. So most of these blocks affect whether your pages feed future model training, not whether ChatGPT can read you when a customer asks.
It is still worth knowing, because the owner never chose it, and a host list that blocks training crawlers today can add search crawlers tomorrow without anyone noticing. One honest limit: we sent the crawler names from our own address, not from OpenAI’s or Anthropic’s published IP ranges. A firewall that checks those ranges could let the real crawler through. What our test proves is that a rule matching the crawler’s name exists on that server.
To check your own site, run curl -I -A “GPTBot/1.2” followed by your homepage address in a terminal, then the same with “OAI-SearchBot/1.0”, and compare with what a browser gets. If you see 403 or 418, ask your host or check your Cloudflare bot settings, then decide on purpose which crawlers you allow. Our AI SEO audit covers the rest of the AI side, and our guide to getting recommended by ChatGPT covers what to do once crawlers can reach you.
Noindex and soft 404s: pages that tell Google the wrong thing
Three homepages carried a noindex instruction, two in a robots meta tag and one in an X-Robots-Tag server header. Google’s noindex documentation explains that this keeps a page out of search results. All three businesses were showing in Google Maps, so the listing was doing the work while the homepage asked to be ignored. In our experience it is usually a setting left on after a site build.
The soft 404 test was simple: we asked each site for a made-up address. A healthy site answers “404 Not Found”. 179 homepages (93%) did. Six returned a normal page with a 200 status, and six more redirected the made-up address to a real page such as the homepage. Google’s crawling errors guide defines a soft 404 as a page that tells the user it does not exist while returning a 200 success code, and says to return 404 or 410 for pages that are gone.
Four of the six 200 responses came from sites built as JavaScript apps, which send the same empty page shell for every address and fill in the content in the browser. Their homepage HTML held fewer than 10 words of text before JavaScript ran. Google covers this in its guide to avoiding soft 404s in single-page apps. If your site was built with an AI site builder or a JavaScript framework, test a made-up address first.
Redirect chains, duplicate hosts and canonical tags
Every site should have one real address and send everything else there in one step. Most did: 184 of 193 redirected http to https. But 9 served the http version without redirecting, 13 served both the www and non-www version as live pages, and 18 needed two redirects from the plain http address to the live homepage, usually http://www to https://www to https:// without www.
A two-step chain is not a disaster. Google’s documentation on HTTP status codes says its crawlers follow up to 10 redirect hops, and the crawling errors guide says to watch out for long redirect chains, which have a negative effect on crawling. Fixing it is usually one rule on the server that sends every variation straight to the final address.
Canonical tags were mostly fine: 170 homepages pointed the canonical at themselves, as Google’s duplicate URL guide recommends. 16 had none, which Google says is survivable because it will choose the best version itself. The seven that pointed elsewhere were more interesting. Five pointed to the other host version (www or non-www) of their own address. On four of them both versions were live, so the canonical was the only thing tying two copies together. On the fifth, the canonical address redirected straight back to the page we started on, so the redirect says one address is real and the canonical says the other is.
Sitemaps and robots.txt
178 sites (92%) had a working XML sitemap and 168 declared it in robots.txt. The median sitemap listed 91 addresses. Of the 15 with no working sitemap, 5 had a robots.txt pointing to a sitemap that returned a 404. Another 5 sites had a sitemap that listed no pages at all.
Google’s sitemap overview says you might not need one if your site has about 500 pages or fewer and is well linked internally, but that in most cases a site benefits from one. For a small business, a broken sitemap is not an emergency, but a robots.txt pointing at a missing file is sloppy, and it is a two-minute fix.
robots.txt itself was in good shape. 182 sites served one, none blocked all crawlers, and the 8 that returned a 404 lose nothing, because Google treats a missing robots.txt as having no crawl restrictions.
Caching headers: the most common problem and one of the least urgent
When Googlebot comes back to a page, it can ask “has this changed since last time?” if the server gave it an ETag or a Last-Modified date. If nothing changed, the server replies 304 Not Modified and sends no page. Google’s documentation on its crawlers says it supports both and recommends ETag, because it has no date formatting issues.
121 homepages (63%) sent at least one of these, and 110 of them answered our repeat request with a proper 304. 72 (37%) sent neither, so every visit means downloading the whole page again. That sounds bad, but Google’s crawl budget guide is written for sites with over 10,000 pages that change daily or over a million pages, and says smaller sites mostly need an up-to-date sitemap. For a 30-page business site, add caching headers when you next touch the server settings. Do not pay anyone to prioritise it.
Page weight and Core Web Vitals
The median homepage was 225 KB of HTML, before a single image, font or script file loaded. The heaviest 10% were over 645 KB and five were over 1 MB. On a typical homepage, 44% of the HTML was inline code (scripts, styles and embedded icons) rather than words, rising to 71% on pages over 500 KB. Sites built with the Elementor page builder had a median of 290 KB against 182 KB for the rest, and the 11 Wix sites had a median of 715 KB.
Heavier pages tend to load slower on phones, but be careful with the claims you hear about Core Web Vitals. Google says Core Web Vitals are used by its ranking systems, and in the same guide says Google Search always seeks to show the most relevant content, even if the page experience is sub-par, and that site owners should not focus on only one or two aspects of page experience. Its Core Web Vitals page sets the targets: Largest Contentful Paint within 2.5 seconds, Interaction to Next Paint under 200 milliseconds and Cumulative Layout Shift under 0.1.
We did not measure Core Web Vitals, which need real visitor data. Check your own in PageSpeed Insights or the Core Web Vitals report in Search Console. A faster site is worth having because customers stay, as our look at website conversion rates explains. Just do not expect speed alone to beat a competitor with more relevant pages.
Smaller problems worth a look
Nine homepages (5%) had structured data that failed to parse: a comment pasted inside the code, a missing comma, a line break inside a text value, extra characters after the closing bracket, or an empty block. In four of them the broken block was the one describing the business itself, such as Plumber, Dentist or HousePainter, so the markup meant to help search engines may be doing nothing. Run your homepage through Google’s Rich Results Test after any change, and see our schema markup guide for working examples. One Brisbane builder’s website address served a bare server folder listing (“Index of /”) instead of a homepage, a reminder to click your own Google listing link now and then.
What we did not find is useful too. HTTPS was near universal, no robots.txt blocked Googlebot, and every homepage had a mobile viewport tag. If an audit report leads with those, it is padding.
A technical SEO checklist you can run in 30 minutes
- In Search Console, run URL Inspection on your homepage and your top service page. Both should say the page is on Google.
- Type your address four ways (http, https, with and without www). All four should land on one address in one step.
- Visit a made-up page on your site. It should show a not found page with a 404 status.
- View source on your homepage and check the canonical tag matches the address bar exactly.
- Open yoursite.com.au/robots.txt. Check nothing important is disallowed and the Sitemap line opens a real sitemap.
- Run the GPTBot and OAI-SearchBot test above and decide which AI crawlers you want.
- Paste your homepage into the Rich Results Test and fix any errors.
- Run PageSpeed Insights on mobile and note the Core Web Vitals and total page size.
- Submit your sitemap in Bing Webmaster Tools as well as Google.
This video from Google Search Central explains how Googlebot discovers and crawls pages, which makes the checklist easier to follow.
How we did this research
On 24 September 2026 we collected websites from 26 Google Maps searches in Australia across 16 kinds of business (trades, legal, accounting, health and real estate) in Sydney, Melbourne, Brisbane, Perth, Adelaide, the Gold Coast and Canberra. We took the first eight businesses per search with a website, removed directories, duplicates and obvious franchise sites, and ended with 208 websites. 193 homepages loaded for a normal browser.
For each site we followed redirects from the http, https and other host versions, recorded headers, repeated the request with the ETag and Last-Modified values to test for a 304, and requested the homepage as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, retesting any site that refused one. We also fetched robots.txt and declared sitemaps, requested a made-up address, and parsed the homepage for the canonical tag, robots meta tag, structured data and page size. Requests ran six sites at a time with a 10 second timeout.
Limits: we tested homepages, not whole sites. Requests came from outside Australia, so we do not report response times, and a few sites may have refused us for our location. AI crawler names were sent from our own address, not the crawlers’ official IP ranges. HTML size is uncompressed and excludes images and external files. Everything was measured once, on one day.
Frequently asked questions
What is technical SEO?
Technical SEO is making sure search engines and AI crawlers can reach, read and index your pages correctly. It covers status codes, redirects, robots.txt, sitemaps, canonical tags, structured data, caching and page speed, rather than the words on the page.
What are some examples of technical SEO?
Redirecting every version of your address to one https address, returning a real 404 for pages that do not exist, keeping a working XML sitemap, using self-referencing canonical tags and fixing broken structured data. The table above lists the 12 problems we found most often on Australian small business sites.
What is the difference between SEO and technical SEO?
SEO is the whole job of earning visibility in search, including content, local listings, reviews and links. The technical side is the part that makes the site itself readable and indexable. Good content on a site with a noindex tag or a broken redirect will not show.
What should a technical SEO audit include?
At minimum: indexability, redirects and duplicate hosts, status codes including soft 404s, canonical tags, robots.txt and sitemaps, structured data validity, mobile rendering and page speed. In 2026 it should also test how the server answers AI crawlers, because in our audit 16% of sites refused at least one of them.
Do Core Web Vitals affect rankings?
Google says Core Web Vitals are used by its ranking systems, but also that it always tries to show the most relevant content even when page experience is weak. Treat speed as a real advantage for visitors and a smaller factor for rankings, not a shortcut past better content.
Can ChatGPT see my website?
Usually, but test it. In our audit 12.6% of sites refused OpenAI’s GPTBot at the server, though only 1% refused OAI-SearchBot, which ChatGPT search uses. Request your homepage with both crawler names and check you get the same page a browser gets.
If you would like someone to run this audit on your whole site rather than just the homepage, our SEO services start with exactly that, and we will tell you which fixes are worth doing now and which can wait.
