Our crawler

When someone asks FindAIValue to scan a company, our crawler reads a small number of that company’s public web pages — 3 to 10: home, about, services, careers, news and similar. It never logs in, never submits forms and never reads pages about individual people.

It respects robots.txt. To block it, add:

User-agent: FindAIValueBot
Disallow: /

It is gentle. At most one request per second to a site, pages up to 2 MB and a 20-second limit per page.

Questions or a problem with the crawler: hello@findaivalue.com.