ResearchAgent Discovery

Making the agentic web
discoverable.

Darwin Crawler explores how public websites, documentation, and protocol metadata become useful, source-backed capability records.

How it works ↓Explore source records ↓

Public sources

WebsitesDocumentationProtocol metadata

One understandable capability record.

Find what an agent can actually do

The web describes capabilities in many forms. A useful index needs to connect those descriptions to the right provider, interface, and operation while retaining the evidence behind the connection.

Our crawler research studies public-source discovery and representation. Indexing a capability does not authorize a provider action and does not, by itself, establish that the capability completes a task.

Observe

Read public capability descriptions and record where they came from.

Resolve

Reconcile provider identities, interfaces, operations, and duplicates.

Represent

Keep structured observations that discovery and verification can inspect.

A conceptual indexing pipeline. Provider execution belongs to a separately authorized path.

Keep the source attached

A capability description should remain traceable to an observation. We study which fields preserve enough meaning for retrieval without turning source claims into verified facts.

JSON

1{2  "capability": "Clean structured spreadsheet rows",3  "source": {4    "kind": "public_documentation",5    "observed_at": "<observation timestamp>",6    "reference": "<original public source>"7  },8  "evidence": "Provider description",9  "outcome_verification": "Not established by indexing"10}

Illustrative research record. This is not a public ingestion API or evidence of a completed provider task.

The web does not stand still

Capabilities change. Endpoints move, documentation is revised, access conditions shift, and the same provider may appear under several identities. A record needs an observation window and a way to reconcile change.

Research question	What to preserve
Has the source changed?	Observation timestamps, source revisions, and the fields affected by the change.
Is this the same agent?	Provider identity and interface evidence, with duplicate resolution kept explicit.
Is a claim still current?	The original claim, its source, and a fresh observation rather than an inferred guarantee.

Freshness and identity are distinct from verified task completion.

Measure what the index misses

Coverage includes missing agents, unsupported interfaces, stale records, identity errors, and descriptions that fail to preserve a capability’s constraints. Evaluation should compare records with their original sources and report uncertainty.

We connect this work to Darwin Rank’s discovery evaluation and Darwin Verifier’s outcome evidence. The crawler supplies observations; those downstream studies ask different questions.

Discovery starts with good evidence.

Explore how source-backed capability records become a task-specific ranking.

Explore Darwin Rank

Find what an agent can actually do

The web describes capabilities in many forms. A useful index needs to connect those descriptions to the right provider, interface, and operation while retaining the evidence behind the connection.