Crawlability

What Crawlability Measures

Crawlability is the foundation of your AURA score. It evaluates whether AI crawlers can physically reach and index your pages. If AI systems can't crawl your site, nothing else matters — they can't recommend content they can't see.

This dimension runs on every audit, including free ones, and checks a series of technical signals that determine your site's accessibility to AI bots.

Technical Checks

AURA runs the following technical checks to evaluate whether AI systems can access your site.

• policy Robots.txt Configuration — Does your robots.txt file allow AI crawlers like GPTBot, ClaudeBot, and Google-Extended? Sites that block these bots are invisible to the AI systems that could recommend them.
• smart_toy AI Bot Directives — Beyond robots.txt, we check whether your site explicitly allows or blocks specific AI training and search bots. This includes directives for GPTBot, ClaudeBot, Anthropic, Google-Extended, and others.
• code Meta Robots Tags — Are your pages using noindex or nofollow meta tags that prevent AI crawlers from indexing or following links? These tags can accidentally hide important content.
• map XML Sitemap — Does your site have a valid, discoverable XML sitemap? Sitemaps help AI crawlers find all your important pages efficiently rather than relying on link discovery alone.
• data_object Schema.org Presence — Does your site include structured data markup? Even basic Schema.org (Organization, WebSite) helps AI systems understand your site's identity.
• speed Server Response — How quickly does your server respond? Slow response times can cause AI crawlers to time out or deprioritize your site.
• title Page Structure — Does your homepage have a proper H1 heading and a clear page title? These help AI systems understand what your site is about at first glance.
• web Content Rendering — Can AI crawlers read your content from the raw HTML, or does it require JavaScript execution? Content that only appears after JS rendering may be invisible to some AI crawlers.
• description llms.txt — Does your site include an llms.txt file? This emerging standard helps AI systems understand your site's purpose, key pages, and how to interact with your content.

Common Issues

These are the most frequent reasons for a low Crawlability score.

• warning AI bots blocked in robots.txt — The most common issue. Many sites inherited robots.txt rules that block GPTBot or ClaudeBot without realizing it.
• shield WAF or bot protection blocking crawlers — Cloudflare, AWS WAF, and other security tools may block our crawler even when robots.txt allows AI bots. The report distinguishes between intentional blocking (robots.txt) and infrastructure blocking (WAF/firewall).
• link_off Missing XML sitemap — Without a sitemap, AI crawlers rely on link discovery, which may miss important pages buried deep in your site structure.
• hourglass_top Slow server response — Response times over 3 seconds can cause crawler timeouts and reduce how frequently AI systems revisit your site.

How to Improve

Follow these steps to improve your Crawlability score, ordered by typical impact.

• edit_note Audit your robots.txt — Ensure GPTBot, ClaudeBot, Google-Extended, and Anthropic-AI are not disallowed. If in doubt, remove blanket User-agent: * Disallow rules and whitelist specific bots instead.
• security Check your WAF settings — If your report shows crawler blocking despite correct robots.txt, your security infrastructure is likely the cause. Whitelist known AI bot user agents in your WAF/CDN settings.
• add_circle Submit an XML sitemap — Create a sitemap at /sitemap.xml listing all important pages. Submit it to Google Search Console — AI systems often discover sitemaps through the same channels.
• note_add Add an llms.txt file — Create a /llms.txt file describing your site's purpose, key pages, and content organization. This is a simple text file that takes minutes to create.
• dns Ensure server-side rendering — If your site uses a JavaScript framework (React, Vue, Angular), configure server-side rendering so the HTML contains your content without requiring JS execution.

Frequently Asked Questions

• Why does my Crawlability score differ from my SEO crawl score? expand_more
  Traditional SEO tools check whether Googlebot can crawl your site. AURA's Crawlability specifically checks AI-related bots like GPTBot, ClaudeBot, and Google-Extended, which may have different access rules in your robots.txt.
• My robots.txt allows all bots but I still got a low score. Why? expand_more
  Your security infrastructure (Cloudflare, WAF, or cookie walls) may be blocking crawlers at the network level before they reach your robots.txt. The report's "crawler blocked" indicator shows when this happens.
• Is Crawlability included in the free plan? expand_more
  Yes. Crawlability and Authority are the two dimensions that run on every audit, including free Starter plan audits.