[DOCS] Document how to deal with bots on live sites about documentation

[joshdentremont]

@kayakr that would be awesome if we could get that patched into the facets module. I really like the checkboxes for facets but am having the same issue with bots.

[mjordan]

bots are getting stuck looping over facets

We've experienced this as well and it's brought out site to its knees.

[ajstanley]

Same. Tiktok ignores robots.txt. We have one sight that was getting several hits per second before we stuck a user agent filter in.

[Natkeeran]

Drupal specific info here: https://dev.acquia.com/blog/automated-bot-traffic-strategies-handle-and-manage-it

[joshdentremont]

Suggestions from tech call below:

Blocking bots by user agent:

• add user agents as an env variable
• mount drupal.defaults.conf as a volume so changes persist
  • can you add a secondary file to drupal.defaults.conf and mount that instead?

Stopping legit bots from crawling facets:

• ignore query params in robots.txt
• block collections and search pages in robots.txt, and instead use a sitemap (simple sitemap was a suggested drupal module)

Remaining questions:

• How do we block by IP in Docker
• How do we update robots.txt? Should we supply a default or just document how to change it?

[ajstanley]

Nginx allows for multiple conf files. We could add an include in nginx.conf to point to a file in /var/www/drupal which would eliminate the need for a separate mount.

[kayakr]

fwiw, I've found the patch for facets at https://www.drupal.org/node/2937191 useful; it converts the facets into actual checkboxes instead of the default that renders them as links (followable by bots) that get converted to checkboxes by js.