Block Bad Bots Using .htaccess
Block bad bots with .htaccess and robots.txt, including why unwanted crawlers waste bandwidth, where rules help, and what to treat with caution too.
Articles
htaccess
.htaccess articles about the older but still useful parts of Apache‑based site control.
These notes show how a small configuration file can still affect crawler access, search behaviour, and the safety of an older site.
Block Bad Bots Using .htaccess. Block Bad Bots Using
Automatically Generate urllist.txt from sitemap.xml. Automatically Generate
urllist.txtfromsitemap.xmlUsing PHP it is quick and easy to automatically generate your urllist.txt sitemap from your sitemap.xml file (for example, using gatsby‑plugin‑sitemap).