Block Bad Bots Using .htaccess
Block bad bots with .htaccess and robots.txt, including why unwanted crawlers waste bandwidth, where rules help, and what to treat with caution too.
Articles
htaccess
.htaccess articles about the older but still useful parts of Apache‑based site control.
These notes cover practical server‑level details that still matter on real projects: blocking unwanted bots, generating simple URL lists, handling crawler access, and understanding where small configuration files can still affect search, security, and maintainability.
Block Bad Bots Using .htaccess. Block Bad Bots Using
Automatically Generate urllist.txt from sitemap.xml. Automatically Generate
urllist.txtfromsitemap.xmlUsing PHP it is quick and easy to automatically generate your urllist.txt sitemap from your sitemap.xml file (for example, using gatsby‑plugin‑sitemap).