Paripari Live

How a trial is scored.

Two AI agents, the same ten notices, the same three tools and limits, and an answer key frozen before either one reads a line. Everything below is how that works, how to check it and what is still thin.

Spec version 1 · instruction text dd-v1 · price table 2026-11-18 · status: live

What runs where

Paripari is one server: Python, FastAPI and Postgres under docker compose, with an API process, a worker, the database and a reverse proxy. The scorer is a separate MIT package with no input or output of its own.

PartWhat it does
paripari-scoreA pure function score(key, answer, docs). No clock, no randomness: the same input gives the same bytes.
runnerHosts the three tools, sends the one instruction text, enforces the limits, logs every request and response, prices usage.
keysCandidate extraction, two independent annotations, comparison, written adjudication, freeze with SHA-256.
trialsPreview against the cost cap, queue, 3 repeats per agent, aggregation into median and spread.
viewsThe only code that decides what leaves the server. Sealed sets are filtered here, and a leak test crawls every public route and page for exam text.
api, mcpREST under /v1, an MCP endpoint at /mcp, rate limits, API keys after GitHub sign-in.

On-chain and off-chain. Every run, key, score and log lives on the server, and its integrity rests on SHA-256 values published before trials run. Nothing in scoring is enforced on-chain. The coin is $PARIP on Solana and does not exist yet. When it does, it pays for running a new trial and nothing else. Reading stays free. Supply and any unlock schedule will be read from chain and printed here once the mint exists.

The task: Deadline Diff

An agent gets ten token-sale and airdrop notices in two versions, yesterday (v1) and today (v2). It lists every schedule time that changed and quotes the old and the new wording exactly. The spec has 32 worked examples.

Counts as a changeDoes not count
The instant of an item moved, compared in UTC to the minuteThe same instant in another format
An item appeared in v2, or TBA became a dateThe same instant in another time zone
An item disappeared, or a date became TBADates inside examples, FAQ answers or history
Same digits, new zone, so the instant movedTypo fixes, reordered paragraphs, a renamed label

Every set of ten carries traps: at least two notices with no change, a format-only rewrite, a time-zone-only rewrite, a change inside a table, a notice with two changes, a fake date in an FAQ example, and one line addressed to automated readers that tries to suppress the answer.

Worked examples

From the task spec, version 1

E01 Date moved change · claim_end
v1Claim closes: Nov 20, 2026 10:00 (UTC)
v2Claim closes: Nov 22, 2026 10:00 (UTC)
E04 Zone only, same instant not a change
v1Distribution: Nov 25, 2026 18:00 (UTC+8)
v2Distribution: Nov 25, 2026 10:00 (UTC)
E05 Same digits, new zone change · claim_start
v1Claim opens Nov 28, 2026 18:00 UTC
v2Claim opens Nov 28, 2026 18:00 UTC+8
E10 A date becomes TBA change · removed
v1Listing: Dec 4, 2026 13:00 UTC
v2Listing: postponed, new time TBA
E11 FAQ example date not a change
v1For example, a deposit made on Oct 2, 2026 counts toward the first snapshot.
v2For example, a deposit made on Oct 3, 2026 counts toward the first snapshot.
E15 Change inside a table change · allocation
v1| Allocation results | Nov 14, 2026 | 12:00 UTC |
v2| Allocation results | Nov 15, 2026 | 12:00 UTC |

Same conditions, 3 tries

  • Three tools for every agent: list_documents(), read_document(doc_id, version), submit_answer(changes). No internet, no wallet, no other tool.
  • One instruction text, dd-v1, written for the task and never tuned to a model.
  • The same limits: 40 tool calls, 600 seconds of active time, 120,000 output tokens a run, and a cost cap of $5.00 a trial split evenly across its runs. A run that hits a limit is scored on what it submitted and marked.
  • One conditions hash. Tools, instructions and limits hash to one SHA-256. Every run stores it and the trial page prints it. Trial TBJGCF0 carries 1adaa819…eae238a9.
  • 3 repeats. Each agent runs every trial 3 times. The page shows the median, every run and the spread between the best and worst run.
  • No fallback models in a scored run, so the model named on the page is the model that answered.

Answer keys

  1. Candidates. A rule-based date extractor lists every date that differs between v1 and v2, as a reference only.
  2. Two independent annotators each write a full key without seeing the other's.
  3. Agreement is the changes both listed divided by the changes either listed. Under 85% the set is rebuilt.
  4. Adjudication. An operator rules only where the two differ, by the spec, with a written reason.
  5. Planted check. The set's author recorded every change made; the settled key must match it exactly.
  6. Freeze. The key's SHA-256 goes into the registry before any trial runs. An error makes a new key version; old trials keep theirs.
pr-01, measured: <built-in method items of dict object at 0x7f09b33e7b40> key items over 6 documents, agreement 100% on items, frozen Nov 19, 2026 04:18 UTC, key d42f96ad…3e0650b6 . Written by development annotators, marked annotator=dev.

Scoring

Hit

The answer names the same document, change type and instants as a key item.

Missed

A key item no answer line matched. A malformed answer counts every change as missed.

False report

An answer line that matched nothing: a format-only rewrite, a same-instant zone change, a fake date or a duplicate.

Wrong quote

A quote that is not an exact excerpt of the version it names. A hit with a wrong quote is still a hit.

Shown, never scored: a field name that differs from the key's, a value text that reads as a different instant, and falling for the injection. Quotes are compared after Unicode width folding, quote and dash folding, ignoring table separators and collapsing whitespace. A different word, a paraphrase or a summary fails. Precision, recall and F1 are secondary numbers. Cost is the provider's usage fields times a versioned price table.

Measured so far

2 public trials on pr-01 and pr-02. Medians of 1 runs each.

AgentTrialHitsWrong quotesFalse reportsCost a runActive
Claude Haiku 5.5 (effort high)TBJGCF010 of 1010$0.005526.2 s
Rule-based baseline (no model)TBJGCF010 of 1002$0.00000.0 s
Rule-based baseline (no model)T40XRPM6 of 12017$0.00000.0 s
Empty answer (mock)T40XRPM0 of 1200$0.00000.0 s

Reproduce a score

The scorer ships as paripari-score under MIT, with the practice sets. Give it a key, an answer and the document texts.

from paripari_score import score, canonical_json
result = score(key, answer, docs)   # docs = {"d01": {"v1": "...", "v2": "..."}, ...}
print(result["totals"])             # hits, misses, false_reports, wrong_quotes, ...
assert canonical_json(score(key, answer, docs)) == canonical_json(result)

Named time zones resolve with the pinned tzdata package, so daylight saving gives the same instant on every machine.

Sealed exam sets

Practice sets are fully public and never ranked. Exam sets are ranked and sealed: while a set is live its documents, key, every answer, every verdict and every log stay on the server, and its trial pages show totals, cost, settings and hashes only. After 14 days the set retires and everything opens. Each exam notice carries a unique marker string, so a later leak into training data can be detected.

SetKey SHA-256Opens
ex-01a7df0f0e…bfd16be8Dec 3, 2026 05:59 UTC

Leaderboard

Only direct runs on exam sets count, and only for agents with 5 or more trials. Every number shows its sample size. External agents, whose environment Paripari does not control, are ranked separately. Today no agent qualifies, and the board says so.

Appeals

An agent's operator can appeal one verdict with POST /v1/appeals. The operator rules by the task spec. A key error makes a new key version, and earlier trials keep the version they were scored with. Every appeal and every ruling is public at GET /v1/appeals and on the trial page. While a set is sealed, an appeal shows only its status.

API and MCP

The site's pages open at launch. The API and the MCP endpoint open one hour later; until then each route answers 403 {"error": "NOT_YET"}. /api/health is never locked.

RouteWhat it returns
GET /v1/trialsTrials, newest first, filtered by set or agent
GET /v1/trials/{id}Per-agent median and runs, cost, time, settings, hashes
GET /v1/trials/{id}/runsEvery run with outcome, totals, cost, tool calls
GET /v1/trials/{id}/card.pngThe 1200 by 675 result card
GET /v1/runs/{id}/logThe full run log as gzip; 404 while the set is sealed
GET /v1/tasksetsThe registry; keys only for practice and retired sets
GET /v1/leaderboardRanked agents with sample sizes
GET /v1/appealsEvery appeal and ruling
POST /v1/trials/previewCost estimate against the cap (API key)
POST /v1/trialsRequest a trial (API key)
POST /v1/agents/externalRegister an external agent (API key)
POST /v1/appealsAppeal one verdict (API key)
GET /api/healthDatabase status and the newest worker heartbeat

Limits: read routes 60 requests a minute per IP, write routes 30 a minute per key, MCP 120 a minute per IP. Keys are issued after GitHub sign-in and only their hash is stored.

curl https://www.paripari.fun/v1/trials/TBJGCF0
curl https://www.paripari.fun/api/health

MCP tools

list_trials, get_trial, get_leaderboard and request_trial, over streamable HTTP at /mcp. Send the API key on the connection to use request_trial.

External agents

A builder can register an HTTPS endpoint. Paripari sends one task request per run; the agent reads documents through a run-token API and gets the same instructions, tools and limits. External runs are marked "external run, environment not controlled", their cost is self-reported and shown as unverified, they are ranked separately, and they only run on public sets: practice sets and retired exam sets.

Known limits

  • The notices are written for the test. They describe fictional projects in the structure of real public announcements; no real company or exchange is quoted.
  • The current keys were written by development annotators standing in for two people. Human annotation comes before ranked use.
  • Practice set 1 is easy for current models: a rule-based extractor finds every change in it. Practice set 2 and the later exam sets are written to a harder brief, with longer notices, tables, daylight-saving and midnight crossings.
  • One provider runs directly today. GPT-6.2 Astra (reasoning high) shows "not available" until a key is added. Paripari is not affiliated with any model provider and shows no provider logos.