OmniscienteBot

The web crawler of Omnisciente, an independent search engine.

If you see requests from OmniscienteBot, they come from us. OmniscienteBot fetches public web pages so they can be found through Omnisciente's search. This page explains what the bot does, how it decides what to fetch and how fast, how to verify that a request really comes from us, and how to reach us.

No generative AI or public models. We do not use crawled content to train large language models or any other generative AI model. It is not used for any model that is published, shared or offered as a service. We do not sell, license or hand the content over to anyone for such purposes. Content is used only to build and rank our search index (details below).

How it identifies itself

Every request carries this User-Agent:

OmniscienteBot/0.1 (+https://omnisciente.dev/bot; crawler@omnisciente.dev)

It also sends a From: crawler@omnisciente.dev header. The robots.txt product token is OmniscienteBot, matched case-insensitively.

robots.txt (RFC 9309)

OmniscienteBot follows the Robots Exclusion Protocol as standardised in RFC 9309:

• It reads /robots.txt on each host before crawling it, and checks it again right before each fetch, using a cached copy at most 24 hours old.
• Rules in a group naming OmniscienteBot apply to us. Otherwise the * group applies. The most specific (longest) matching rule wins, and Allow wins a tie.
• If robots.txt returns a 4xx status, we treat it as absent: there are no restrictions.
• If it returns a 5xx status or cannot be reached, we treat the whole site as disallowed and don't crawl it until robots.txt answers again.
• Up to five redirects are followed, and at least the first 500 KiB of the file is parsed.

To keep OmniscienteBot off your whole site:

User-agent: OmniscienteBot
Disallow: /

Changes take effect within 24 hours. For anything faster, write to crawler@omnisciente.dev.

Crawl-delay

OmniscienteBot honours the Crawl-delay directive, although it is not part of RFC 9309:

• The value is the minimum number of seconds between two requests. Decimals are accepted, e.g. Crawl-delay: 2.5.
• We read it from the group that applies to us (OmniscienteBot, otherwise *). A Crawl-delay written for another crawler does not apply to us, and neither does one outside any group.
• When it is longer than our own default spacing, it replaces it.

User-agent: OmniscienteBot
Crawl-delay: 10

How fast it crawls

Limits apply per site, which means a registrable domain together with all its hostnames. For example, all of *.example.com counts as one site. Across all our machines, we make:

• at most one request every 5 seconds per site, with no bursts;
• at most two concurrent requests per site.

When a site asks us to slow down, we stop:

• An HTTP 429 or 503, or a Retry-After header, pauses the whole site. We wait at least as long as Retry-After says; otherwise the pause starts at 5 minutes and doubles each time, up to 24 hours.
• Repeated 403 responses pause the site for 24 hours.

What it fetches

• Only GET requests for public HTML pages, plus /robots.txt and /.well-known/tdmrep.json.
• No form submissions, no logins and no attempts to get past access controls.
• Responses are capped at 32 MB, and requests time out after 15 seconds.

How crawled pages are used

Pages are stored and processed only to run Omnisciente's search: to index them, rank them and show links and short excerpts in results.

As part of that, we train small internal classification models on crawled pages. These models recognise:

• the kind of a page, for example a blog post, a news article, a tutorial or a product page;
• the context of a link, for example a citation, a navigation link or an advertisement.

These models output labels, not text. They are used only inside our own search system.

We do not:

• train generative models (models that write text, images or code) on crawled content;
• publish, share, sell or license any model trained on it, or offer one as a service;
• give crawled content or derived datasets to third parties to train their own models.

To keep your pages out of all of this, including the internal classifiers, block OmniscienteBot in robots.txt (see above).

Verifying that a request comes from us

Anyone can copy a User-Agent. You can check genuine OmniscienteBot traffic in three ways:

1. IP addresses. We crawl only from the addresses in /bot/ips.json, also listed below.
2. Reverse DNS. Each of these addresses has a PTR record under crawler.omnisciente.dev, and that name resolves back to the same address. Check both directions:

   $ host 62.210.192.81
   81.192.210.62.in-addr.arpa domain name pointer crawl-1.crawler.omnisciente.dev.
   $ host crawl-1.crawler.omnisciente.dev
   crawl-1.crawler.omnisciente.dev has address 62.210.192.81

3. Web Bot Auth. Requests are signed with HTTP Message Signatures (RFC 9421), following the IETF Web Bot Auth draft. They carry Signature-Agent: sig1="https://omnisciente.dev", plus Signature-Input and Signature headers with tag="web-bot-auth". The public keys are at /.well-known/http-message-signatures-directory.

Crawler IP addresses

Contact

Security contacts are also published in /.well-known/security.txt.