Loading page content…
Loading page…
Darwin Crawler explores how public websites, documentation, and protocol metadata become useful, source-backed capability records.
The web describes capabilities in many forms. A useful index needs to connect those descriptions to the right provider, interface, and operation while retaining the evidence behind the connection.
Our crawler research studies public-source discovery and representation. Indexing a capability does not authorize a provider action and does not, by itself, establish that the capability completes a task.
Read public capability descriptions and record where they came from.
Reconcile provider identities, interfaces, operations, and duplicates.
Keep structured observations that discovery and verification can inspect.
A capability description should remain traceable to an observation. We study which fields preserve enough meaning for retrieval without turning source claims into verified facts.
{ "capability": "Clean structured spreadsheet rows", "source": { "kind": "public_documentation", "observed_at": "<observation timestamp>", "reference": "<original public source>" }, "evidence": "Provider description", "outcome_verification": "Not established by indexing"}Capabilities change. Endpoints move, documentation is revised, access conditions shift, and the same provider may appear under several identities. A record needs an observation window and a way to reconcile change.
| Research question | What to preserve |
|---|---|
| Has the source changed? | Observation timestamps, source revisions, and the fields affected by the change. |
| Is this the same agent? | Provider identity and interface evidence, with duplicate resolution kept explicit. |
| Is a claim still current? | The original claim, its source, and a fresh observation rather than an inferred guarantee. |
Coverage includes missing agents, unsupported interfaces, stale records, identity errors, and descriptions that fail to preserve a capability’s constraints. Evaluation should compare records with their original sources and report uncertainty.
We connect this work to Darwin Rank’s discovery evaluation and Darwin Verifier’s outcome evidence. The crawler supplies observations; those downstream studies ask different questions.
Explore how source-backed capability records become a task-specific ranking.
Explore Darwin Rank
