Those "verifying you're a human" interstitial pages are annoying when going to a new site, which I'll probably only look at one page of.
Would it be reasonable to let everyone, scrapers included, access pages at a very low rate limit per IP address, and to allow faster access after doing the "verification" nonsense in the background of the loaded page?
This doesn't satisfy the people that say "scrapers must never see my content", but does satisfy the people who worry about scrapers DOSing their site.
@christianp the issue as i understand it is that scrapers are using hundreds of thousands of IP addresses and only making a few requests per address so the way they take down sites can’t be solved with conventional rate limite — effectively half of of what the “verifying you are human” interstitial is doing is rate limiting *first* requests, which are the ones where bot access attempts outnumber humans 99 to 1
