- Aug 8, 2020
I have a quick question. I recently launched a classifieds website and the software is somewhat of a mess. It's not optimized for search engines at all and I think there are many pages that aren't necessary for the crawlers to crawl. So my question is, how do I know what to block in my robots.txt file? Which pages should I block? What will happen if I block them? I am pretty new at this.