User-agent: * Allow: / # Disallow admin and sensitive areas Disallow: /admin/ Disallow: /api/ Disallow: /_next/ Disallow: /dashboard/ Disallow: /test-indexnow/ Disallow: /*.json$ Disallow: /*.txt$ Disallow: /curl-* Disallow: /fix-* Disallow: /improve-* Disallow: /generate-* # Allow important directories Allow: /blog/ Allow: /product/ Allow: /Training/ Allow: /Industries/ Allow: /languages/ Allow: /integrations/ Allow: /roles/ Allow: /personas/ Allow: /use-cases/ Allow: /case-studies/ Allow: /competitors/ Allow: /pricing/ Allow: /about/ Allow: /contact/ Allow: /privacy-policy/ Allow: /terms-of-service/ # Allow static assets Allow: /assets/ Allow: /public/ Allow: /*.css Allow: /*.js Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.svg Allow: /*.webp Allow: /*.ico # Sitemap Sitemap: https://salesroleplay.app/sitemap.xml ```robots.txt User-agent: * # Allow crawling of main pages Allow: / Allow: /home Allow: /about Allow: /pricing Allow: /contact # Allow crawling of product pages Allow: /product/* # Allow crawling of training pages Allow: /training/* # Allow crawling of industry-specific pages Allow: /industry/* # Allow crawling of language-specific pages Allow: /language/* # Allow crawling of integration pages Allow: /integrations/* # Allow crawling of role-specific pages Allow: /roles/* # Allow crawling of blog pages Allow: /blog/* # Allow crawling of resource pages Allow: /resources/* # Allow crawling of persona pages Allow: /personas/* # Allow crawling of competitor comparison pages Allow: /competitors/* # Allow crawling of use case pages Allow: /use-cases/* # Disallow crawling of sensitive areas Disallow: /admin/ Disallow: /api/ Disallow: /wp-admin/ # WordPress specific Disallow: /private/ Disallow: /staging/ # Sitemap location Sitemap: https://salesroleplay.app/sitemap.xml # Add any other specific disallows as needed. Example below: # Disallow: /sensitive-data/ # Note: Wildcards (*) are used liberally for broad allowances. # If you have very specific needs to block certain pages within these categories, # you will need to add more specific Disallow directives. ``` **Explanation:** * **`User-agent: *`**: This line applies the following rules to all user-agents (including search engine crawlers). * **`Allow: /`**: This allows crawling of the root directory and all pages within it unless specifically disallowed. It's a good practice to start with a broad `Allow` statement. * **`Allow: /home`, `Allow: /about`, etc.**: These explicitly allow crawling of the main navigation pages. While covered by `/`, it's good practice to be explicit for important pages. * **`Allow: /product/*`, `Allow: /training/*`, etc.:** These use wildcards to allow crawling of all pages within the specified directories. This is efficient and less prone to errors than listing every single page. * **`Disallow: /admin/`, `Disallow: /api/`, etc.:** These disallow access to administrative, API, and other sensitive areas. The `wp-admin` is specifically for WordPress installations. * **`Sitemap: https://salesroleplay.app/sitemap.xml`**: This tells search engines where to find your sitemap, which contains a comprehensive list of all the pages you want indexed. **Make sure you have a properly generated sitemap.** * **Comments:** The comments explain the purpose of each section, making the `robots.txt` file easier to understand and maintain. **Important Considerations:** * **Sitemap:** A well-structured sitemap is crucial. Ensure your sitemap includes all the pages you want indexed. * **Dynamic Content:** If your site generates dynamic URLs (e.g., based on user actions or parameters), you may need to adjust your `robots.txt` or use other techniques (like canonical URLs) to manage how search engines crawl these. * **Regular Review:** Periodically review and update your `robots.txt` as your website structure changes. This `robots.txt` provides a strong foundation. You may need to refine it further based on the specific structure and content of `salesroleplay.app`. Remember to test your `robots.txt` using tools like Google's robots.txt Tester to ensure it works as intended.