robots RFC

robots.txt is standardised in RFC 9309: a crawler obeys the group matching its product token, else the '*' group; the most specific (longest) path match wins; a 4xx robots.txt lets the crawler access anything, while a 5xx or network error means it must assume complete disallow, and after a reasonably long period (for example 30 days) it may treat the file as unavailable or keep a cached copy. Google does not support crawl-delay, treats 4xx except 429 as no restrictions, and on 5xx stops crawling for 12 hours, then uses the last good copy for 30 days; after that it acts as if there is no robots.txt if the site is otherwise available.