InstaClaw's agent

This page is for the people who run websites. If you've found our agent in your logs, here's who runs it, who it works for, and how to turn it away if you'd rather it didn't visit.

Turning it away

Name it in your robots.txt. These two lines keep it away from your whole site, and you can narrow the second line to a folder if you only want to keep it out of part of it.

User-agent: InstaClaw
Disallow: /

You can also block the Agent/InstaClaw token at your firewall. If you'd rather just tell us, write to help@instaclaw.io and we'll take care of it.

Who runs it

InstaClaw's agent is operated by Cooper Wrenn, doing business as InstaClaw, a sole proprietorship. You can reach us at help@instaclaw.io with a question, a problem, or a request to stop, and a person reads every message. Please write from an address at the site you're asking about, so we can act on it quickly.

Who it works for

Every InstaClaw customer has their own AI agent, running on its own computer. The agent visits a site only when the person it works for has asked it to do something, like checking a price or reading a page they care about. It doesn't crawl, it doesn't build a search index, and it doesn't browse on its own initiative. Each visit is one person's errand, started by that person.

It never trains on your site

What the agent reads is used to finish the errand it was given, and for nothing else. We never use it to train or fine-tune an AI model, and we never use it to build an index.

Recognising it

The agent's browser, its page reader and the curl and wget tools it uses send a user agent containing Agent/InstaClaw. The page reader and the command line tools send this full string. A script the agent writes itself sends its own language's default name, so on a site whose terms ask for the name, the agent uses these tools instead.

Mozilla/5.0 (compatible; InstaClaw/1.0; +https://instaclaw.io/agent) Agent/InstaClaw

Requests from the agent's browser and its page reader carry a signature made with our published key, so you can check a request really comes from us.

Our keys are published at https://instaclaw.io/.well-known/http-message-signatures-directory.

One thing we'd rather you read here than find yourself. Code the agent writes for itself, in a language like Python, still sends that language's own default name, so a request from that code does not carry the name above.

The public agent, InstaClawPublic

InstaClaw agents that nobody has paid for yet will visit sites under a separate name, InstaClawPublic, with a separate key, so that a free agent can never borrow the reputation of a customer's agent. It is not running yet.

Its user agent ends with

Agent/InstaClawPublic (+https://instaclaw.io/agent)

Its key is published, apart from ours, at https://public.instaclaw.io/.well-known/http-message-signatures-directory.

When it runs, it will sign its requests with that key, follow any rules a site writes for it by name in robots.txt, and never train on what it reads.

robots.txt

Instaclaw's agent acts for one person, when that person asks, and it is not a crawler. It reads robots.txt and follows any rules a site writes for InstaClaw by name. Rules a site writes for every agent are written for crawlers, and the agent does not treat them as addressed to it. It stops when a site answers 403 or 429, or asks in writing. At a page that asks for a person, it stops and hands that page to the person it works for. If a site's robots.txt cannot be read, the agent does not treat that as a no.

When a page is closed, the agent looks for an allowed way to the same thing, such as the site's own search results or a connected app, and says which way it went.

All of this describes agents on our new version. Our customers' agents on the older version don't do it yet, and they will when they move. Until then they don't run this robots.txt check, their tools send their own default names, and the lines above won't reach them. If one of them is on your site and you want it gone, write to help@instaclaw.io and we'll take care of it.

A group names InstaClaw if its user agent line contains the word InstaClaw in any case, including Agent/InstaClaw.

When it meets a robot check

The agent doesn't solve a robot check, get around one, or pay anyone else to solve it. It stops and hands the step to the person it's working for, who finishes the check themselves.

How fast it goes

One errand at a time, at the pace of a person reading and clicking.