InstaClaw's agent
This page is for the people who run websites. If you've found our agent in your logs, here's who runs it, who it works for, and how to turn it away if you'd rather it didn't visit.
Turning it away
Name it in your robots.txt. These two lines ask it to stay away from your whole site, and you can narrow the second line to a folder if you only want to keep it out of part of it.
User-agent: InstaClaw Disallow: /
You'll also be able to block the Agent/InstaClaw token at your firewall. Both work as soon as the name reaches our customers' agents, which is rolling out now and is described below. If you'd rather just tell us, write to help@instaclaw.io and we'll take care of it.
Who runs it
InstaClaw's agent is operated by Cooper Wrenn, doing business as InstaClaw, a sole proprietorship. You can reach us at help@instaclaw.io with a question, a problem, or a request to stop, and a person reads every message. Please write from an address at the site you're asking about, so we can act on it quickly.
Who it works for
Every InstaClaw customer has their own AI agent, running on its own computer. The agent visits a site only when the person it works for has asked it to do something, like checking a price or reading a page they care about. It doesn't crawl, it doesn't build a search index, and it doesn't browse on its own initiative. Each visit is one person's errand, started by that person.
It never trains on your site
What the agent reads is used to finish the errand it was given, and for nothing else. We never use it to train or fine-tune an AI model, and we never use it to build an index.
How you'll recognise it, and how it will treat robots.txt
We're rolling out one name across the ways our agent reaches your site. When it reaches our customers' agents, the agent's browser, its page reader and the curl and wget tools it uses will all send a user agent containing Agent/InstaClaw, and its browser and page reader will honour every robots.txt group that names InstaClaw. The name and the check travel together.
The page reader and the command line tools will send this full string.
Mozilla/5.0 (compatible; InstaClaw/1.0; +https://instaclaw.io/agent) Agent/InstaClaw
A group names InstaClaw if its user agent line contains the word InstaClaw in any case, including Agent/InstaClaw. A page you disallow will be refused before the request leaves our customer's agent, and if your robots.txt can't be read, because your server errors or doesn't answer, the agent will treat that as a no. The agent won't apply the * group, because it isn't crawling your site, it's running one person's errand.
Its browser will also sign each request with our published key, so you can check the request really comes from us. The keys are already published at https://instaclaw.io/.well-known/http-message-signatures-directory.
One thing we'd rather you read here than find yourself. Code the agent writes for itself, in a language like Python, still sends that language's own default name, and nothing on our side can change that yet. When it can, this page will say so.
When it meets a robot check
The agent doesn't solve a robot check, get around one, or pay anyone else to solve it. It stops and hands the step to the person it's working for, who finishes the check themselves.
How fast it goes
One errand at a time, at the pace of a person reading and clicking.