The "robot-exception" setting:

The value of this setting should be a regular expression. If it matches the REQUEST_URI without the SCRIPT_NAME prefix matches this regular expression, then the request is an exception to anti-robot defenses and should be allowed through. For example, to allow robots to download tarballs or ZIP archives for named versions and releases, you could use an expression like this:

^/tarball/(version-[0-9.]+|release)/

This setting can hold multiple regular expressions, one regular expression per line. The input URL is exempted from anti-robot defenses if any of the multiple regular expressions matches.