What Is robots.txt?

🤖 What Is Robots.txt?

Let’s go back to our LEGO city. You already gave Google a map (sitemap.xml), but now you want to say: "Hey, don’t go into my secret lab!"

That’s what robots.txt does — it tells robots where they can and cannot go.

🪧 What Does Robots.txt Do?

It’s a set of instructions for search engines. It doesn’t block people, only robots that follow the rules (like Google).

🔍 How Robots.txt and Sitemap.xml Work Together

sitemap.xml = map
robots.txt = rules

User-agent: *
Disallow: /private/
Allow: /

Sitemap: https://example.com/sitemap.xml

🧾 Sample robots.txt Explained

• User-agent: * — this applies to all bots
• Disallow: /private/ — bots should avoid that folder
• Allow: / — everything else is okay
• Sitemap: ... — link to your sitemap

🛠️ How to Make One

1. Open a text editor
2. Add your rules
3. Save it as robots.txt
4. Place it at https://yourdomain.com/robots.txt

🤔 Should You Always Block Things?

No. Block pages like admin panels, search results, or staging sites — but not your homepage or blog!

🚫 Blocking ≠ Hiding

Google may still index a blocked page if it sees a link to it. Use <meta name="robots" content="noindex"> for true hiding.

✅ Summary

robots.txt = front gate rules

sitemap.xml = map for Google

They work together to help Google understand and respect your website.