No — and this is a critical misconception. Robots.txt tells crawlers not to crawl a page, but it does not prevent indexing. If other websites link to a blocked page, Google may still index it (and show it in results) based on those links, even without crawling the content. To fully prevent indexing, use a noindex meta tag or X-Robots-Tag HTTP header instead. Robots.txt is for managing crawl budget and access, not for hiding pages from search results.

Without a robots.txt file, all compliant crawlers assume they are allowed to crawl everything. Your entire site is open to all bots. This is actually fine for most small websites — the absence of a robots.txt is not penalised by Google. However, having a properly configured robots.txt improves crawl efficiency (especially for large sites), prevents admin page exposure, and lets you point crawlers to your sitemap via the Sitemap: directive.

Yes. While robots.txt is most commonly used to block directories (e.g. Disallow: /admin/), you can also block specific files: Disallow: /private-page.html. You can also use wildcards with * and $ to match URL patterns: Disallow: /*?sort= blocks any URL with a sort= parameter. Note that Google supports wildcards but not all crawlers do — check the documentation for the specific bots you're targeting.

Absolutely not — this was old advice from the days when crawlers couldn't render pages. Google now renders JavaScript-heavy pages and needs access to your CSS and JS files to understand your content accurately. Blocking these files with robots.txt prevents Google from rendering your pages correctly and can harm your rankings. The only things to block are truly non-public resources like admin panels, staging environments, and user-generated duplicate content.

Crawl-delay tells a bot to wait N seconds between requests — useful for protecting server resources from aggressive crawlers. However, Google does not support Crawl-delay in robots.txt; you need to set crawl rate in Google Search Console instead. Bing, Yandex, and some other crawlers do honour it. For most websites on modern hosting, crawl delay is unnecessary. Only use it if your server logs show bot traffic causing performance issues.

Google typically re-crawls robots.txt files every 24 hours, though it can vary. Changes you make today may not be reflected in crawler behavior until tomorrow. If you need immediate enforcement (e.g. urgent blocking of a page), submit a URL removal request in Google Search Console — this takes effect within hours. For less urgent changes, updating robots.txt and waiting is sufficient.