Our crawler

If you found this address in your logs, a visit came from BitsExplorer, an index of work people have built with AI and published.

How to recognise it

It identifies itself with this user agent:

BitsExplorerBot/1.0 (+https://bitsexplorer.com/crawler)

What it visits, and what it does there

• Pages that have already been published and linked publicly: an app's store listing, a repository's homepage, a game's page, a project announced on a public forum.
• It loads the page once in a real browser, takes a picture of it, and reads the text that is visible, so the listing can describe what the page is.
• It looks at what a visit already shows, such as whether the connection is secure. Nothing is submitted, nothing is signed into, and nothing is probed for weaknesses.
• It may come back later to see whether the page still answers.

How it behaves

• It reads your site's robots.txt before taking a picture of a page, reading it, or checking it, and stays away from what it asks crawlers to avoid. It follows the rules written for bitsexplorer if there are any, and otherwise the rules for all crawlers. It keeps what it read for up to a day.
• If your robots.txt cannot be reached at all, it does not visit, and tries again later.
• At most one request a second to any one host, and no more than two pages open in its browser at a time across everything it visits.
• It never visits private or internal network addresses.

Asking it to stay away

Add this to your site's robots.txt:

User-agent: bitsexplorer
Disallow: /

That stops new visits. To have what BitsExplorer already lists from your site removed as well, send the form on the removal and reports page with your site's address.