Working With Your Robots.txt File

Your website automatically includes a robots.txt file that you can use to control search engine indexing for specific pages or your entire site. You can view your robots file by going to www.yourdomain.com/robots.txt or yoursitehere.weebly.com/robots.txt (using your unique site address in place of the examples, of course).

The default setting is to allow search engines to index your entire site. If you want to prevent your entire website from being indexed by search engines, go to the Settings tab in the editor and click on the SEO section. Scroll down to the "Hide site from search engines" toggle, switch it to the On position, and re-publish your site.

hide-seo-toggle.jpg

What if you only want to protect some of your pages from being indexed? Not a problem! You can do this for each page individually from the SEO Settings menu. First, make sure that the "Hide site from search engines" toggle shown above is set to Off. Then go to the Pages tab, click on the page you want to hide, and then on the SEO Settings button.

robots-seo-settings.jpg

On the following section, click the checkbox to hide the page from search engines, and then click on the back arrow at the top to save your changes and return to the main settings for the page.

hide-page-search.jpg

Follow the same steps for any additional pages you want to hide from search, and then publish your site when you're done.

That's really all there is to it, but there are a couple of important things to keep in mind:

* If you're using Weebly's site search feature, any pages you hide from search engine indexing will also be hidden from the local site search.

* You can go back and disable (or enable) search engine blocking if you change your mind later, but it will take some time for search engines to catch up. In other words, pages that were previously blocked will need to be indexed before you may see them show up in search results, and pages that were previously indexed may still show in search results until they're re-crawled by search engines.

* There are some things that are automatically blocked for all Weebly sites no matter how you've configured your settings. This includes protecting the directory where uploaded files for Digital Products are stored (even if you don't have a store or aren't a Business subscriber) and certain user-agents that send an exorbitant amount of crawl requests. These blocks will not have any negative effect on your site or its search engine ranking, and are actually beneficial since they help to secure your site.

Was this article helpful? 17 out of 25 found this helpful