3 years ago

Harnessing the Power of Robots.txt

Sometimes, we may want search-engines to not index certain areas of the site, and on occasion even prohibit other SE in the site altogether.

This really is the place where a simple, little 2 line text file called robots.txt is avail read more...