does the fedi search respect robots.txt or is it being completely facetious about caring about instance policy regarding scrapers
i'm just saying as far as opting out goes, there is an extremely widely-deployed, recognized solution
Follow
(i'm wondering if we should be blocking them on principle or because they're an active threat to our users)
