About our crawler
What it is
IntelBrief is a competitive intelligence service. Our customers add the companies they follow (competitors, customers, prospects), and our crawler reads those companies' public web pages — homepage, pricing, blog, careers, press pages — so we can tell our customers when something meaningful changes.
What it does — and doesn't do
- It only requests publicly accessible pages. It never logs in, never submits forms and never tries to get past paywalls or access controls.
- It visits each tracked page about once a week, plus one initial visit when a customer first adds the company. It fetches a handful of pages per site, not the whole site.
- Some pages are fetched through a page-reading service (Jina Reader) acting on our behalf, so requests may come from that service's infrastructure.
- It identifies itself with the user agent above on every direct request.
robots.txt
Before fetching a page on a tracked company's website, we check that site's robots.txt for the IntelBrief user agent and skip any disallowed URL. To block our crawler from your entire site, add:
User-agent: IntelBrief Disallow: /
robots.txt files are cached for a few hours, so a change can take up to that long to apply.
Opt-out & contact
If you'd rather we stop visiting your site, or if you believe our crawler is misbehaving, email hello@intelbrief.xyz with your domain. We'll respond and, on request, exclude the domain from direct crawling.