Our crawler

If you found this address in your logs, a visit came from BitsExplorer, an index of work people have built with AI and published.

How to recognise it

It identifies itself with this user agent:

bitsexplorer/0.1 (+https://bitsexplorer.com/crawler)

What it visits, and what it does there

• Pages that have already been published and linked publicly: an app's store listing, a repository's homepage, a game's page, a project announced on a public forum.
• It loads the page once in a real browser, takes a picture of it, and reads the text that is visible, so the listing can describe what the page is.
• It looks at what a visit already shows, such as whether the connection is secure. Nothing is submitted, nothing is signed into, and nothing is probed for weaknesses.
• It may come back later to see whether the page still answers.

How it behaves

• At most one request a second to any one host, and no more than two pages open in its browser at a time across everything it visits.
• It never visits private or internal network addresses.
• It does not yet read robots.txt on every visit. If you want it to stay away, say so below.

Asking it to stay away

Send the form on the removal and reports page with your site's address. What BitsExplorer lists from it is removed and not added again.