The reading room · 15 guides
Guides, written answer-first
Everything we know about making sites visible to AI search — each guide practices the structure it preaches and links to the matching tools.
AI Crawling & Access / 7
- What is llms.txt? Spec, examples, and how to create one
- List of AI crawlers and their user agents
- Should you block AI bots? Trade-offs explained
- Does ChatGPT read robots.txt? And other AI crawler questions, answered
- Technical SEO for AI crawlers: sitemaps, redirects, and hreflang
- llms.txt vs robots.txt vs sitemap.xml: what each file does
- How to check if AI bots are crawling your site