Last Updated on by Azib Yaqoob
A technical SEO audit checks whether search engines can crawl, render and index your site, and whether anything on the page or server is holding it back. You can run a full one for $0 with Google Search Console, PageSpeed Insights, the free version of Screaming Frog and a few open-source tools, then use an AI assistant to make sense of the exports.
This guide gives you the checklist, what to check by hand, and copy-paste prompts for ChatGPT, Claude or Gemini. Tool limits and free tiers below are as of November 2026.
Quick answer: To do a technical SEO audit for free, (1) review Search Console for indexing, Core Web Vitals and manual actions, (2) crawl the site with Screaming Frog (free up to 500 URLs), (3) check robots.txt, sitemaps, canonicals and structured data, (4) test speed in PageSpeed Insights, then (5) paste your crawl exports into an AI assistant to prioritise issues and write fix tickets. AI only sees what you give it, so always verify its claims in the tools.
Table of Contents
What a technical SEO audit covers
A technical audit looks at how your site is built, not what the content says. The main areas are:
- Crawlability: can bots reach your pages (robots.txt, internal links, server errors)?
- Indexability: are the right pages in Google’s index (noindex tags, canonicals, duplicates)?
- Site structure: are important pages a few clicks from the homepage, with clean URLs and redirects?
- Performance: do pages pass Core Web Vitals on real devices?
- Markup: is structured data valid, and are titles and meta descriptions present and unique?
- Access for AI crawlers: are you deliberately allowing or blocking AI bots in robots.txt?
Run an audit when traffic drops suddenly, after a migration or redesign, when reviving an old site, or once or twice a year as maintenance.
The free toolkit
| Tool | What it checks | Free limit (Nov 2026) |
|---|---|---|
| Google Search Console | Indexing, sitemaps, Core Web Vitals, manual actions, URL inspection | Free for verified sites |
| PageSpeed Insights | Lab and real-user speed data, Core Web Vitals | Free |
| Rich Results Test | Structured data that can produce rich results | Free |
| Screaming Frog SEO Spider | Full crawl: titles, status codes, canonicals, redirects, directives | Free Lite version crawls up to 500 URLs |
| Unlighthouse (open source) | Runs Lighthouse across many pages of a site | MIT licence, free |
| advertools (open source, Python) | Crawl a site, download robots.txt and sitemaps into DataFrames | MIT licence, free |
| ChatGPT, Claude, Gemini | Prioritising, explaining and writing up findings from your exports | Free tiers exist, with usage caps |
Paid suites such as Ahrefs Site Audit and Semrush Site Audit do the same crawling with scheduled reports. They are optional. Everything below works without them.

Step 1: Check Google Search Console
Search Console shows what Google actually sees, so start here before crawling anything.
Page indexing report
Open Indexing > Pages and scroll to “Why pages aren’t indexed”. Look for reasons you did not intend, such as “Excluded by noindex tag”, “Blocked by robots.txt”, “Duplicate without user-selected canonical” or “Soft 404”. Some exclusions are normal (old redirects, paginated pages). The question is whether any page that should earn traffic is on the list. If it is, see fixing the “indexed but blocked by robots.txt” error or why new content is slow to index.
URL Inspection
Paste any URL into the search bar at the top. It shows whether the page is indexed, which canonical Google chose, the last crawl date, and a rendered view of the page. Use Test live URL after you fix something.
Core Web Vitals and Manual actions
Under Experience > Core Web Vitals you get mobile and desktop groups of URLs marked Good, Needs improvement or Poor for LCP, INP and CLS. Under Security and Manual Actions you should see no issues. If there is a manual action, Search Console explains what to fix and lets you request a review.
One change from older guides: Google has retired the Mobile Usability report and the standalone Mobile-Friendly Test, so check mobile problems through Core Web Vitals, URL Inspection’s rendered screenshot and Lighthouse instead.

Step 2: Crawl the site with Screaming Frog (free)
Search Console only reports what Google has already tried. A crawler walks your site the way a bot would and finds problems before Google does. The free Screaming Frog Lite version crawls up to 500 URLs per crawl, which covers most small sites. It does not include saved crawls or full configuration.
After the crawl, check these tabs and filters:
- Response Codes: 4xx and 5xx errors, and redirect chains or loops.
- Page Titles and Meta Description: missing, duplicate, too long or too short.
- Directives: noindex, nofollow and canonical tags that point somewhere unexpected.
- Canonicals: pages whose canonical differs from their own URL.
- H1: missing or multiple H1s.
- Images: missing alt text and oversized files.
- Internal links: orphan-like pages with very few links pointing to them.
Export each issue list as CSV. You will feed these to AI in Step 6. If your site is bigger than 500 URLs, crawl one section at a time, or use the open-source options below. For duplicate pages specifically, see how to find and fix duplicate content.
Open-source crawlers and auditors on GitHub
If you are comfortable with a command line, two maintained open-source projects fill the gaps left by the free Screaming Frog limit:
- Unlighthouse scans your whole site with Google Lighthouse and shows performance, accessibility, best practice and SEO scores per page. MIT licence.
- advertools is a Python package with a Scrapy-based
crawlfunction, plus helpers that load robots.txt and XML sitemaps into pandas DataFrames. MIT licence. PyPI lists version 0.18.0 released June 17, 2026.
Both need some Node.js or Python knowledge. If that is not you, Screaming Frog’s free version is enough for most audits. Always crawl only sites you own or have permission to test, and keep the crawl rate gentle.

Step 3: Check robots.txt, sitemaps and canonicals by hand
These three files control most crawl and index problems, and each takes a minute to check manually.
robots.txt
Open yourdomain.com/robots.txt in a browser. Look for a stray Disallow: /, blocked CSS or JavaScript, blocked sections you want indexed, and a Sitemap: line. A typical WordPress file looks like this:
User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Sitemap: https://example.com/sitemap_index.xmlRemember that robots.txt stops crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To keep a page out of the index, use a noindex tag and leave it crawlable. More background in do you really need a robots.txt file.
XML sitemap
Open the sitemap URL and confirm that it lists only indexable, 200-status, canonical URLs. Then submit it under Indexing > Sitemaps in Search Console. Sitemaps full of redirected, noindexed or 404 URLs waste crawl attention. If you run WordPress, see the best XML sitemap plugins.
Canonical tags
View source on a few key pages and search for rel="canonical". Each indexable page should point to itself, with the same protocol and host (https, one version of www or non-www). Check that http and www variants redirect to your preferred version with a single 301.
Step 4: Test speed and Core Web Vitals
Paste a URL into PageSpeed Insights. It shows field data from real Chrome users when enough traffic exists, and lab data from a Lighthouse run. The three Core Web Vitals are LCP (loading), INP (responsiveness) and CLS (visual stability). Field data decides what Search Console reports, so trust it over the lab score when they differ.
Test your homepage, one blog post and one key landing page rather than every URL. For a whole-site view, use Unlighthouse. Common fixes are covered in WordPress cache plugins and image compression.

Step 5: Validate structured data
Use the Rich Results Test on a few templates (a post, a product, a contact page) and fix errors first, warnings second. Also check the Enhancements section in Search Console for sitewide structured data reports. Only add markup that matches visible content on the page.

Step 6: Use AI to prioritise and write up the audit
This is where AI saves the most time. Crawlers give you hundreds of rows. An assistant can group them, rank them and turn them into tickets for a developer.
Two limits matter. First, AI cannot crawl your site or see your Search Console, so it only knows what you paste. Second, it can invent numbers or confidently misread a rule. Treat every answer as a draft and verify it against the tool. Do not paste private client data into a tool you are not allowed to use it with.
Prompt 1: Prioritise crawl issues
What it does: ranks problems by likely SEO impact. Paste: the Response Codes, Titles and Directives exports (trim to the first 200 rows if long).
You are a technical SEO auditor. Below are CSV exports from a Screaming Frog crawl of my site.
1. Group the issues by type and count them.
2. Rank the groups by likely SEO impact (high, medium, low) and explain why in one sentence each.
3. Flag anything that looks like it could block indexing.
4. Only use the data I pasted. If you need more data to judge something, say so instead of guessing.
CSV data:
[paste here]Prompt 2: Write fix tickets
What it does: turns findings into developer-ready tickets. Paste: the issue summary from Prompt 1.
Turn each issue below into a ticket for a web developer. For each ticket give: title, affected URL pattern, what is wrong, how to fix it, how to verify the fix, and priority. Keep each ticket under 100 words.
Issues:
[paste here]Prompt 3: Build a redirect map
What it does: matches old 404 URLs to the closest live pages. Paste: a list of broken URLs and a list of live URLs from your sitemap.
Match each broken URL to the single most relevant live URL based on the words in the path. Output a table with columns: old_url, new_url, confidence (high, medium, low). If no good match exists, write NONE instead of guessing. I will review every row before implementing 301 redirects.
Broken URLs:
[paste here]
Live URLs:
[paste here]Always spot-check the matches. Redirecting to a loosely related page can create soft 404s.
Prompt 4: Review robots.txt
What it does: explains each rule and flags risky ones. Paste: your robots.txt plus 5 to 10 URLs you want indexed.
Explain each line of this robots.txt in plain English. Then, for each URL in my list, say whether Googlebot would be allowed to crawl it under these rules and which rule decides it. Flag any rule that could block important pages, CSS or JavaScript.
robots.txt:
[paste here]
URLs:
[paste here]Prompt 5: Write and test a regex
What it does: builds a pattern for bulk filters in your crawler or Search Console. Paste: a few example URLs you want to match and a few you do not.
Write a regular expression that matches these URLs and does not match the others. Explain each part, and show how each of my example URLs does or does not match.
Should match:
[paste here]
Should NOT match:
[paste here]Search Console’s regex filter uses RE2 syntax, so tell the assistant that if the pattern is for Search Console. Run the regex against your examples before using it.
Step 7: Check AI crawler access in robots.txt
As of November 2026, robots.txt also controls which AI companies’ bots may fetch your pages. Decide on purpose rather than by accident. The main user agents, from each company’s own documentation:
| User agent | Owner | What it does |
|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI’s models |
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT’s search features |
| ChatGPT-User | OpenAI | Fetches pages when a user asks in ChatGPT |
| ClaudeBot | Anthropic | Collects public content that may be used for training |
| Claude-SearchBot | Anthropic | Crawls to improve search result quality |
| Claude-User | Anthropic | Fetches pages when a person asks Claude a question |
| Google-Extended | robots.txt token controlling use of content for Gemini training and grounding; not a ranking factor and does not affect Search inclusion |
OpenAI states that each setting is independent, so you can allow OAI-SearchBot while disallowing GPTBot. Blocking search-related bots may reduce your visibility in those AI answers. Here is an example that opts out of training but stays open to AI search:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /Whether to block is a business decision, not a technical one. Also check that a firewall or security plugin is not silently blocking these bots, since robots.txt only works when the request reaches your server. Vendors can change user agents, so re-check the official pages from time to time: OpenAI crawlers, Anthropic crawlers and Google common crawlers.
Step 8: Review site structure and internal links
In your crawl, check click depth: important pages should be reachable within about three clicks of the homepage. Look for pages with no internal links and for generic anchors like “click here”. A tidy structure also helps new content get found faster. See how to create a logical website structure.
Backlinks are the one area where free tools are thin. Search Console’s Links report shows who links to you. AI cannot see a live backlink index, so do not ask it for your link counts. Paid tools are the option here if you need competitor link data.
Manual audit checklist
If you prefer to check by hand, or want a quick sanity pass, work through this list:
- Open robots.txt and the sitemap in a browser.
- Search
site:yourdomain.comin Google and compare the rough count with your page count. Treat it as an estimate only. - Inspect your homepage, a post and a category page in URL Inspection.
- View source on key pages and check title, meta description, canonical and robots meta tag.
- Visit the http, www and non-www versions of your homepage and confirm each redirects once to the preferred URL.
- Click through 10 internal links and look for 404s or redirects.
- Load the site on a phone and check for layout shifts and tap targets.
- Open Security and Manual Actions in Search Console.
Frequently asked questions
How do I do a technical SEO audit for free?
Use Search Console for indexing and Core Web Vitals, Screaming Frog’s free version for a crawl of up to 500 URLs, PageSpeed Insights for speed and the Rich Results Test for structured data. Then paste the exports into an AI assistant to prioritise the fixes.
Can AI do a technical SEO audit for me?
Not on its own. AI cannot crawl your site or log in to Search Console. It can analyse exports you give it, rank issues, explain errors and write tickets. Verify its output against the original tools.
What should a manual technical SEO audit include?
Check robots.txt, the sitemap, canonical tags, redirects, internal links, mobile rendering, page speed, structured data and Search Console’s indexing report. The manual checklist above covers each one.
How often should I audit my site?
Once or twice a year for most sites, plus after a migration, redesign or sudden traffic drop. Check Search Console monthly in between.
Should I block GPTBot and other AI crawlers?
It depends on whether you want your content used for AI training or shown in AI answers. Training bots and search bots are separate user agents, so you can treat them differently. Google-Extended does not affect Google Search rankings.
Is Screaming Frog free?
As of November 2026, the Lite version is free and limited to 500 URLs per crawl. The paid licence removes that limit and adds saved crawls, full configuration and support.
Conclusion
A good technical audit is a repeatable routine: Search Console first, a crawl second, the three key files third, then speed and markup. AI is most useful at the end, when you need to turn raw exports into a short, ordered to-do list. If you want that done for you and your site also needs accurate GA4 and Meta tracking, get in touch here.
