Skip to content

Wasl AI · crawler policy

You found WaslAI-Research in your logs.

Here is exactly what it did, and how to make it stop.

Opt out

Open an issue with your domain. A dedicated opt-out mailbox is not live yet, and printing an address that discards your request would be worse than printing none. Removal is applied within 24 hours, is permanent, and also removes any published score. You do not need to give a reason and we will not ask for one.

Or block us in robots.txt — honoured on the next crawl:

User-agent: WaslAI-Research
Disallow: /

Limits

These are constants in services/api/wasl/crawler/policy.py, not configuration. There is no setting, environment variable or API parameter that raises them.

method
GET only
No POST, PUT, PATCH or DELETE. Ever.
rate
0.5 req/s
One request every two seconds, per domain.
volume
12 / 40 pages
12 per interactive scan, 40 per batch crawl.
scheme
https only
Plain http is refused.
assets
blocked
Images, fonts and media are never downloaded.
auth
never
No login, no credentials, no session.

What it never does

  • Authenticate, or attempt to.
  • Submit a form. POST forms are recorded as markup and left alone.
  • Fetch /checkout, /cart, /login, /signin, /register, /account, /payment or /admin — regardless of what robots.txt permits.
  • Bypass a paywall, CAPTCHA, bot wall or rate limit. If you block us, we record that you blocked us and move on.
  • Probe for rate limits. Headers are read only when a server volunteers them during an ordinary crawl.
  • Collect personal data. If any is encountered incidentally, it is not stored.

robots.txt

robots.txt is authoritative. Disallowed paths are not fetched.

A disallow does not lower your score. Wasl scores whether a site has made a legible decision about agent access. An explicit User-agent: GPTBot / Disallow: / stanza scores exactly the same as one that allows it — both are clear. Silence scores nothing. You are never penalised for telling automated clients to go away.

Which sites are crawled

Only two categories: domains on a reviewed list committed in the repository at seeds/seed_urls.yaml, and a domain a user submits through the web interface for a site they are checking themselves. There is no open crawl and no off-domain link following.

An exclusion registry is checked before the allowlist, so an opt-out cannot be overridden by a later seed-list entry.

Published results

  • Government and public-sector entities are anonymised by default.
  • Commercial entities are named.
  • Any entity is removed on request, within 24 hours, without argument.
  • We publish scores, findings and short evidence snippets. We do not republish substantial content from any site.