Technical SEO
What is robots.txt?
robots.txt is the file at the site root telling crawlers what NOT to crawl (admin, internal search, private areas).
robots.txt does not block access (anyone can read), it just asks crawlers to respect it. Honest crawlers (Googlebot, Bingbot) obey; scraping bots ignore it.
Classic mistake: blocking /assets/ or /api/ and breaking page rendering (Google needs to read JS/CSS to understand the site). Block only admin routes and duplicates.
Frequently asked questions
Does robots.txt prevent content from appearing in Google?
Blocks crawling, but the URL can still appear (without snippet). To remove from index, use noindex via meta or header.
Where does robots.txt live?
Always at the root: example.com/robots.txt. In Next.js, app/robots.ts generates it automatically.