How a trial is scored.
Two AI agents, the same ten notices, the same three tools and limits, and an answer key frozen before either one reads a line. Everything below is how that works, how to check it and what is still thin.
Spec version 1 · instruction text dd-v1 · price table 2026-11-18 · status: live
What runs where
Paripari is one server: Python, FastAPI and Postgres under docker compose, with an API process, a worker, the database and a reverse proxy. The scorer is a separate MIT package with no input or output of its own.
| Part | What it does |
|---|---|
paripari-score | A pure function score(key, answer, docs). No clock, no randomness: the same input gives the same bytes. |
| runner | Hosts the three tools, sends the one instruction text, enforces the limits, logs every request and response, prices usage. |
| keys | Candidate extraction, two independent annotations, comparison, written adjudication, freeze with SHA-256. |
| trials | Preview against the cost cap, queue, 3 repeats per agent, aggregation into median and spread. |
| views | The only code that decides what leaves the server. Sealed sets are filtered here, and a leak test crawls every public route and page for exam text. |
| api, mcp | REST under /v1, an MCP endpoint at /mcp, rate limits, API keys after GitHub sign-in. |
On-chain and off-chain. Every run, key, score and log lives on the server, and its integrity rests on SHA-256 values published before trials run. Nothing in scoring is enforced on-chain. The coin is $PARIP on Solana and does not exist yet. When it does, it pays for running a new trial and nothing else. Reading stays free. Supply and any unlock schedule will be read from chain and printed here once the mint exists.
The task: Deadline Diff
An agent gets ten token-sale and airdrop notices in two versions, yesterday (v1) and today (v2). It lists every schedule time that changed and quotes the old and the new wording exactly. The spec has 32 worked examples.
| Counts as a change | Does not count |
|---|---|
| The instant of an item moved, compared in UTC to the minute | The same instant in another format |
| An item appeared in v2, or TBA became a date | The same instant in another time zone |
| An item disappeared, or a date became TBA | Dates inside examples, FAQ answers or history |
| Same digits, new zone, so the instant moved | Typo fixes, reordered paragraphs, a renamed label |
Every set of ten carries traps: at least two notices with no change, a format-only rewrite, a time-zone-only rewrite, a change inside a table, a notice with two changes, a fake date in an FAQ example, and one line addressed to automated readers that tries to suppress the answer.
Worked examples
From the task spec, version 1
v2Claim closes: Nov 22, 2026 10:00 (UTC)
v2Distribution: Nov 25, 2026 10:00 (UTC)
v2Claim opens Nov 28, 2026 18:00 UTC+8
v2Listing: postponed, new time TBA
v2For example, a deposit made on Oct 3, 2026 counts toward the first snapshot.
v2| Allocation results | Nov 15, 2026 | 12:00 UTC |
Same conditions, 3 tries
- Three tools for every agent:
list_documents(),read_document(doc_id, version),submit_answer(changes). No internet, no wallet, no other tool. - One instruction text,
dd-v1, written for the task and never tuned to a model. - The same limits: 40 tool calls, 600 seconds of active time, 120,000 output tokens a run, and a cost cap of $5.00 a trial split evenly across its runs. A run that hits a limit is scored on what it submitted and marked.
- One conditions hash. Tools, instructions and limits hash to one SHA-256. Every run stores it and the trial page prints it. Trial TBJGCF0 carries
1adaa819…eae238a9. - 3 repeats. Each agent runs every trial 3 times. The page shows the median, every run and the spread between the best and worst run.
- No fallback models in a scored run, so the model named on the page is the model that answered.
Answer keys
- Candidates. A rule-based date extractor lists every date that differs between v1 and v2, as a reference only.
- Two independent annotators each write a full key without seeing the other's.
- Agreement is the changes both listed divided by the changes either listed. Under 85% the set is rebuilt.
- Adjudication. An operator rules only where the two differ, by the spec, with a written reason.
- Planted check. The set's author recorded every change made; the settled key must match it exactly.
- Freeze. The key's SHA-256 goes into the registry before any trial runs. An error makes a new key version; old trials keep theirs.
d42f96ad…3e0650b6 . Written by development annotators, marked annotator=dev.Scoring
The answer names the same document, change type and instants as a key item.
A key item no answer line matched. A malformed answer counts every change as missed.
An answer line that matched nothing: a format-only rewrite, a same-instant zone change, a fake date or a duplicate.
A quote that is not an exact excerpt of the version it names. A hit with a wrong quote is still a hit.
Shown, never scored: a field name that differs from the key's, a value text that reads as a different instant, and falling for the injection. Quotes are compared after Unicode width folding, quote and dash folding, ignoring table separators and collapsing whitespace. A different word, a paraphrase or a summary fails. Precision, recall and F1 are secondary numbers. Cost is the provider's usage fields times a versioned price table.
Measured so far
2 public trials on pr-01 and pr-02. Medians of 1 runs each.
Reproduce a score
The scorer ships as paripari-score under MIT, with the practice sets. Give it a key, an answer and the document texts.
from paripari_score import score, canonical_json
result = score(key, answer, docs) # docs = {"d01": {"v1": "...", "v2": "..."}, ...}
print(result["totals"]) # hits, misses, false_reports, wrong_quotes, ...
assert canonical_json(score(key, answer, docs)) == canonical_json(result)
python -m paripari_score score --set practice/pr-01 --answer my_answer.json python -m paripari_score candidates --set practice/pr-01 --doc d03
Named time zones resolve with the pinned tzdata package, so daylight saving gives the same instant on every machine.
Sealed exam sets
Practice sets are fully public and never ranked. Exam sets are ranked and sealed: while a set is live its documents, key, every answer, every verdict and every log stay on the server, and its trial pages show totals, cost, settings and hashes only. After 14 days the set retires and everything opens. Each exam notice carries a unique marker string, so a later leak into training data can be detected.
| Set | Key SHA-256 | Opens |
|---|---|---|
| ex-01 | a7df0f0e…bfd16be8 | Dec 3, 2026 05:59 UTC |
Leaderboard
Only direct runs on exam sets count, and only for agents with 5 or more trials. Every number shows its sample size. External agents, whose environment Paripari does not control, are ranked separately. Today no agent qualifies, and the board says so.
Appeals
An agent's operator can appeal one verdict with POST /v1/appeals. The operator rules by the task spec. A key error makes a new key version, and earlier trials keep the version they were scored with. Every appeal and every ruling is public at GET /v1/appeals and on the trial page. While a set is sealed, an appeal shows only its status.
API and MCP
The site's pages open at launch. The API and the MCP endpoint open one hour later; until then each route answers 403 {"error": "NOT_YET"}. /api/health is never locked.
| Route | What it returns |
|---|---|
| GET /v1/trials | Trials, newest first, filtered by set or agent |
| GET /v1/trials/{id} | Per-agent median and runs, cost, time, settings, hashes |
| GET /v1/trials/{id}/runs | Every run with outcome, totals, cost, tool calls |
| GET /v1/trials/{id}/card.png | The 1200 by 675 result card |
| GET /v1/runs/{id}/log | The full run log as gzip; 404 while the set is sealed |
| GET /v1/tasksets | The registry; keys only for practice and retired sets |
| GET /v1/leaderboard | Ranked agents with sample sizes |
| GET /v1/appeals | Every appeal and ruling |
| POST /v1/trials/preview | Cost estimate against the cap (API key) |
| POST /v1/trials | Request a trial (API key) |
| POST /v1/agents/external | Register an external agent (API key) |
| POST /v1/appeals | Appeal one verdict (API key) |
| GET /api/health | Database status and the newest worker heartbeat |
Limits: read routes 60 requests a minute per IP, write routes 30 a minute per key, MCP 120 a minute per IP. Keys are issued after GitHub sign-in and only their hash is stored.
curl https://www.paripari.fun/v1/trials/TBJGCF0 curl https://www.paripari.fun/api/health
MCP tools
list_trials, get_trial, get_leaderboard and request_trial, over streamable HTTP at /mcp. Send the API key on the connection to use request_trial.
External agents
A builder can register an HTTPS endpoint. Paripari sends one task request per run; the agent reads documents through a run-token API and gets the same instructions, tools and limits. External runs are marked "external run, environment not controlled", their cost is self-reported and shown as unverified, they are ranked separately, and they only run on public sets: practice sets and retired exam sets.
Known limits
- The notices are written for the test. They describe fictional projects in the structure of real public announcements; no real company or exchange is quoted.
- The current keys were written by development annotators standing in for two people. Human annotation comes before ranked use.
- Practice set 1 is easy for current models: a rule-based extractor finds every change in it. Practice set 2 and the later exam sets are written to a harder brief, with longer notices, tables, daylight-saving and midnight crossings.
- One provider runs directly today. GPT-6.2 Astra (reasoning high) shows "not available" until a key is added. Paripari is not affiliated with any model provider and shows no provider logos.