Is a chimpanzee or an LLM more like a human? Humans pass for LLMs; LLMs don't pass for humans. An experiment in subject-swapped, blind-rated sentences.
---
format: typebulb/v1
name: Chimp vs Chatbot
---
**code.tsx**
```tsx
import React, { useEffect, useMemo, useRef, useState } from "react"
import { createRoot } from "react-dom/client"
import pluralize from "pluralize"
/* Two views: Run writes a corpus and rates every substituted sentence 0-100; Results reads it.
* The three objects on the Results page carry three different numbers — the grid is DIRECTED
* (the column word in the row word's sentences), the ranking is POOLED (both directions), and
* the chart is PAIRED (both candidates in the anchor's own sentences). The design record, the
* results and the limits are in `notes.md`; what is down here is why the CODE is shaped as it
* is. Saved files key the model `judge` — that is what `refresh-data.mjs` reads. */
const WORDS = ["human", "LLM", "chimpanzee", "frog", "virus"]
const CONC = 8 // polite enough not to 429, fast enough to be worth it
/** An unordered pair. `x` is whichever word comes first in WORDS, purely so the key is stable. */
type Pair = { key: string; x: string; y: string }
/** One dot: a sentence written about `from`, scored with the other word as its subject. */
type Dot = { from: string; score: number | null }
type Cell = { anchor: string; subject: string; scores: (number | null)[] }
type Run = {
timestamp: string; judge: string; provider?: string
n: number; words: string[]
corpus: Record<string, string[]>
results: Cell[]
}
type TbModel = { provider: string; name: string; friendlyName: string; providerName: string }
// ── shape ────────────────────────────────────────────────────────────────────
/** Anything without a populated `results` array is "no run yet", which is a state, not an
* error. Cells naming a word this bulb no longer scores are hidden and logged, so a run file
* stays the archive of what was measured even after the word list changes. */
function normalize(raw: unknown): Run | null {
const d = raw as Partial<Run>
if (!d || !Array.isArray(d.results) || !d.results.length || !d.corpus) return null
const words = d.words?.length ? d.words : WORDS
const live = d.results.filter((c) => words.includes(c.anchor) && words.includes(c.subject))
const dropped = d.results.length - live.length
if (dropped) tb.log(`[data] hiding ${dropped} cell(s) naming a word no longer scored`)
if (!live.length) return null
return {
timestamp: d.timestamp ?? "", judge: d.judge ?? "an unrecorded model", provider: d.provider,
n: d.n ?? Math.max(...live.map((c) => c.scores?.length ?? 0)),
words, corpus: d.corpus, results: live,
}
}
function baked(): Run | null {
try { return normalize(tb.json<unknown>(0)) } catch { return null }
}
// ── words ────────────────────────────────────────────────────────────────────
/* Plurals are not cosmetic here: the subject of every rated sentence is one, so a wrong plural
* means a whole word's worth of ungrammatical sentences and a row that cannot be quoted. That
* has already happened once on this project — a `/s$/` guard turned `virus` into `Virus` and
* invalidated the entire n=25 virus row.
*
* Hand-rolled rules cannot survive words being typed in: `person` wants `people`, `sheep` wants
* `sheep`, and an already-plural `humans` must stay put rather than becoming `humanses`. So the
* library does it — with one exception it gets wrong for us. `pluralize` restores the case of
* the input, so an all-caps initialism comes back all-caps: `LLM` → `LLMS`. Those just take an
* `s`. */
/* And one where the library is right about Latin and wrong about English: it gives `virus` →
* `viri`. Nobody says that. Its `-us → -i` family is the place to look first if another word
* comes out strange — which is what the preview chips in the Run view are for. */
pluralize.addIrregularRule("virus", "viruses")
const plural = (w: string) =>
w.length > 1 && w === w.toUpperCase() ? w + "s" : pluralize(w)
const cap = (w: string) => w.charAt(0).toUpperCase() + w.slice(1)
const subject = (w: string) => cap(plural(w))
/** The invariant every rated string obeys: plural subject, space, predicate. */
const say = (w: string, predicate: string) => `${subject(w)} ${predicate}`
// no article helper here: "what's more LLM-like" needs no a/an, which is one small thing the
// adjective phrasing buys over "in an LLM's place"
// ── stats ────────────────────────────────────────────────────────────────────
/** The `anchor|subject` key every cell lookup shares — one spelling of the format. */
const key = (a: string, b: string) => `${a}|${b}`
/** Order two words the way the word list does — which decides who is `a`, blue, the circle. */
const byList = (words: string[]) => (m: string, n: string) => words.indexOf(m) - words.indexOf(n)
const mean = (xs: number[]) => (xs.length ? xs.reduce((s, v) => s + v, 0) / xs.length : 0)
/* the ends of the scale — the two warning paragraphs and notes.md quote these numbers */
const CEIL = 90, FLOOR = 10
type Stat = {
pair: Pair; ok: Dot[]
score: number // the row's number: mean substituted rating, 0-100
xy: number; yx: number // the same, split by direction: y in x's sentences, and back
asym: number // yx − xy. Off the headline on purpose; it is the dot's shape
saturated: boolean // interchangeable, and the scale has run out of room to say so
floored: boolean // the same failure upside down
}
const pairsOf = (words: string[]): Pair[] => {
const out: Pair[] = []
for (let i = 0; i < words.length; i++)
for (let j = i + 1; j < words.length; j++)
out.push({ key: key(words[i], words[j]), x: words[i], y: words[j] })
return out
}
const cellMap = (run: Run) => new Map(run.results.map((c) => [key(c.anchor, c.subject), c]))
/** Both directions of a pair, concatenated. A sentence appears once, under the word it was
* written about, with the OTHER word swapped in — never twice, and never against itself. */
function dotsFor(run: Run, cells: Map<string, Cell>, p: Pair): Dot[] {
const half = (from: string, into: string) =>
(run.corpus[from] ?? []).map((_s, i) => ({
from, score: cells.get(key(from, into))?.scores[i] ?? null,
}))
return [...half(p.x, p.y), ...half(p.y, p.x)]
}
function stat(pair: Pair, dots: Dot[]): Stat {
const ok = dots.filter((d) => d.score != null)
const avg = (ds: Dot[]) => mean(ds.map((d) => d.score as number))
const score = avg(ok)
const xy = avg(ok.filter((d) => d.from === pair.x))
const yx = avg(ok.filter((d) => d.from === pair.y))
return {
pair, ok, score, xy, yx, asym: yx - xy,
saturated: ok.length > 0 && score >= CEIL,
floored: ok.length > 0 && score <= FLOOR,
}
}
/** Every pair, best first — the ranking IS the finding, so it is also the row order. */
function pairStats(run: Run): Stat[] {
const cells = cellMap(run)
return pairsOf(run.words)
.map((p) => stat(p, dotsFor(run, cells, p)))
.filter((s) => s.ok.length)
.sort((a, b) => b.score - a.score)
}
// ── the triad ────────────────────────────────────────────────────────────────
/* An anchor and two candidates. The anchor names the SHARED SENTENCES, and that is what makes
* this the clean comparison: both candidates are read into the same corpus, so it is paired
* sentence by sentence. It costs precision — n sentences, where a pooled ranking row has 2n. */
type Tri = { anchor: string; a: string; b: string }
type TriRow = { sentence: string; a: number | null; b: number | null; was: number | null }
type TriStat = {
tri: Tri; rows: TriRow[]; ok: TriRow[]
meanA: number; meanB: number
saturated: boolean // both candidates fit, and the scale has no room to say more
floored: boolean // and the same failure upside down
}
/** Two distinct pairs share at most one word, so the anchor is never ambiguous. Candidates are
* ordered by the word list, which is the strip's own left-to-right. */
function triOf(words: string[], p: Pair, q: Pair): Tri | null {
const anchor = [p.x, p.y].find((w) => w === q.x || w === q.y)
if (!anchor) return null
const ca = p.x === anchor ? p.y : p.x
const cb = q.x === anchor ? q.y : q.x
if (ca === cb) return null
const [a, b] = [ca, cb].sort(byList(words))
return { anchor, a, b }
}
function triStat(run: Run, cells: Map<string, Cell>, tri: Tri): TriStat {
const at = (w: string, i: number) => cells.get(key(tri.anchor, w))?.scores[i] ?? null
const rows = (run.corpus[tri.anchor] ?? []).map((sentence, i) => ({
sentence, a: at(tri.a, i), b: at(tri.b, i), was: at(tri.anchor, i),
}))
const ok = rows.filter((r) => r.a != null && r.b != null)
const meanA = mean(ok.map((r) => r.a as number))
const meanB = mean(ok.map((r) => r.b as number))
return {
tri, rows, ok, meanA, meanB,
saturated: ok.length > 0 && meanA >= CEIL && meanB >= CEIL,
floored: ok.length > 0 && meanA <= FLOOR && meanB <= FLOOR,
}
}
// ── generation ───────────────────────────────────────────────────────────────
async function pool<T>(jobs: (() => Promise<T>)[], width: number,
tick: () => void, stop: () => boolean): Promise<void> {
let next = 0
await Promise.all(Array.from({ length: Math.min(width, jobs.length) }, async () => {
for (;;) {
const i = next++
if (i >= jobs.length || stop()) return
await jobs[i]()
tick()
}
}))
}
/** What gets stored is the PREDICATE, not the sentence: "seek meaning in their experiences."
* Substitution is then a prepend — `Chimpanzees` + that string — so both candidates share the
* continuation verbatim and differ only in the subject noun. That is what makes the pair
* comparable, and what makes it safe for one model to both write and rate.
*
* Asked for N in one call a model returns fewer than N, repeats itself, or both, and the
* shortfall used to be silent — you simply got a smaller corpus than you asked for and
* nothing said so. So: dedupe, top up in further calls naming what it already wrote, and log
* the yield either way. Near-duplicates that differ in wording still get through; the fix for
* that, and for the corpus being model-written at all, is real source text. */
const norm = (s: string) => s.toLowerCase().replace(/[^a-z0-9 ]/g, "").replace(/\s+/g, " ").trim()
/** `JSON.stringify(x, null, 1)` gives every score a line of its own, so a 5-word n=20 run is
* 500 lines holding one two-digit number each. Arrays of plain numbers are folded back onto a
* single line; arrays of STRINGS are left expanded, because one sentence per line is exactly
* what you want when reading a corpus. */
const toJson = (o: unknown) =>
JSON.stringify(o, null, 1).replace(
/\[\s*((?:-?\d+(?:\.\d+)?|null)(?:\s*,\s*(?:-?\d+(?:\.\d+)?|null))*)\s*\]/g,
(_m, body: string) => `[${body.split(/\s*,\s*/).join(", ")}]`)
async function generate(m: TbModel, anchor: string, n: number): Promise<string[]> {
const subj = subject(anchor)
const out: string[] = []
const seen = new Set<string>()
let calls = 0, dupes = 0
for (let pass = 0; pass < 4 && out.length < n; pass++) {
const want = n - out.length
const avoid = out.length
? ` Do not repeat any of these, or restate them in other words:\n` +
out.map((s) => say(anchor, s)).join("\n")
: ""
const { text } = await tb.ai({
provider: m.provider, model: m.name, webSearch: false,
system: "You write plain declarative sentences. No preamble, no numbering, no commentary.",
messages: [{
role: "user",
content:
`Write ${Math.max(want, 6)} different general statements about ${plural(anchor)}, ` +
`each beginning with the exact word "${subj}". Vary the subject matter widely. ` +
`One per line, nothing else.${avoid}`,
}],
})
calls++
for (const line of text.split("\n")) {
const l = line.trim().replace(/^[-*\d.)\s]+/, "")
if (!l.toLowerCase().startsWith(subj.toLowerCase())) continue
const body = l.slice(subj.length).trim()
if (body.length <= 12) continue
const k = norm(body)
if (seen.has(k)) { dupes++; continue }
seen.add(k)
out.push(body)
if (out.length >= n) break
}
}
tb.log(`[corpus] ${anchor}: ${out.length}/${n} in ${calls} call(s)` +
`${dupes ? `, ${dupes} duplicate(s) dropped` : ""}` +
`${out.length < n ? " — SHORT, the anchor starved" : ""}`)
return out
}
/** 0-100: how much sense does this sentence make as a claim about the world?
*
* Blind by construction — one sentence, no candidates named, no topic stated, no way to infer
* which answer is wanted. That is the whole defence against the model playing along. */
async function rate(m: TbModel, sentence: string): Promise<number | null> {
const { text } = await tb.ai({
provider: m.provider, model: m.name, webSearch: false,
system:
"You rate how much sense a statement makes as a claim about the real world. " +
"100 = obviously true and unremarkable. 50 = arguable or strained. " +
"0 = nonsense, or plainly false of the thing named. " +
"Reply with a single integer from 0 to 100 and nothing else.",
messages: [{ role: "user", content: sentence }],
})
const mm = text.match(/\d+/)
if (!mm) return null
const v = parseInt(mm[0], 10)
return v >= 0 && v <= 100 ? v : null
}
// ── chart primitives ─────────────────────────────────────────────────────────
/* WHAT COLOUR MEANS, AND WHY IT IS NOT THE SAME IN BOTH OBJECTS.
*
* The rule is: colour carries whatever the geometry cannot.
*
* matrix — position is identity (which row, which column), so colour must carry the VALUE.
* It takes WEIGHT, not hue: the page's own ink at an opacity set by the score
* strip — position IS the value, so colour is free to carry IDENTITY: blue for the first
* candidate, orange for the second
*
* Ramping the dots as well was tried and reverted. It looked tidier — one scale everywhere —
* but it spent colour on what the x-axis already said, and paid for it by taking away the only
* thing that told the two clouds apart at a glance where they overlap. Shape alone is a weaker
* channel than hue, and the ramp's grey low end made a 0 recede when a 0 is as much a data point
* as a 95.
*
* Shape says which candidate too — picked square, dot, quoted sentence. That is redundant with
* hue in the strip on purpose: it is what keeps the pair legible without relying on colour
* vision.
*
* THE MATRIX HAS NO HUE OF ITS OWN, and that is the point. A vivid ramp was tried — blue, cyan,
* green, yellow — and it competed: it contains a blue that reads as the first candidate and a
* warm yellow that reads as the second, so the grid kept suggesting an identity it was not
* encoding. Two colour systems on one page must not share hues. Hue is the candidates'; the
* matrix gets weight. */
/** The score as ink. Floored just above nothing so a low cell is still visibly a cell, capped
* short of solid so the mark on a picked square still reads. It inverts with the theme at no
* cost, because `--ink` already does. */
const shade = (v: number) =>
`rgba(var(--ink-rgb), ${(0.05 + (Math.max(0, Math.min(100, v)) / 100) * 0.8).toFixed(3)})`
/** The one geometry both marks wear — defined once so they cannot drift apart. */
const Shape = ({ tri }: { tri: boolean }) =>
tri ? <path d="M12 6 L18.5 17.5 L5.5 17.5 Z" /> : <circle cx="12" cy="12" r="6.2" />
/** The candidate glyph at text size, for the title and the quoted sentences. */
const Glyph = ({ tri }: { tri: boolean }) => (
<svg className={"glyph " + (tri ? "b" : "a")} viewBox="0 0 24 24" aria-hidden="true">
<Shape tri={tri} />
</svg>
)
/** The mark a selected square wears — the same circle and triangle its dots wear in the strip,
* so the two objects are one system and the marks need no legend.
*
* Filled with the page's surface colour and edged in the candidate's, so it reads on any cell
* the ramp can produce: a mark on a cell is a POINTER and must contrast with whatever colour
* that square happens to be, where a dot in the strip is a DATUM and can be filled outright. */
const CellMark = ({ tri }: { tri: boolean }) => (
<svg className={"cmark " + (tri ? "b" : "a")} viewBox="0 0 24 24" aria-hidden="true">
<Shape tri={tri} />
</svg>
)
/** The matrix, and the reason this shape fits this question.
*
* A pair's score is SYMMETRIC — pooled over both directions — so `cell(i,j) = cell(j,i)` and the
* square mirrors itself. That redundancy is what earns it: **row `w` holds every pair `w`
* belongs to, in a straight line.** A row is therefore the "given a human" clause of the
* question, and two cells in it are the two candidates. Everything selected is a cell, and the
* triad falls out of where they sit.
*
* THE CELL IS DIRECTED: it is the COLUMN word read in the ROW word's sentences, so the grid is
* deliberately not symmetric and a square against its mirror is the direction gap — 42 points
* between an LLM in human sentences and a human in LLM ones, reproduced across two runs.
*
* Pooling them was tried, on the theory that symmetry is what lets a row mean "every pair this
* word belongs to". That confused the cell's COLOUR with the row's MEANING: selection is
* positional — row is the anchor, two columns are the candidates — and no encoding of the fill
* touches it. (The first matrix failed because it forced the anchor onto the DIAGONAL, which is
* a different defect entirely.) Nothing is lost by directing them either, because the pooled
* score now has a home of its own in the ranking beside the grid.
*
* And it makes the page agree with itself: the two picked squares are exactly the two averages
* the chart below draws. Pooled, they were not — the squares read 83 and 64 while the chart
* said 78 and 50, because the squares were folding in a second corpus the chart never touches.
*
* Weight carries the whole ranking at once — the thing a list can only give you in one order at
* a time. The diagonal is drawn at the top of the scale, because a word always stands in for
* itself; it is not selectable, being a stipulation rather than a pair anybody measured. (The
* measured near-diagonal — a word read into its own sentences — is 97.6-99.0, which at this end
* of the ramp is the same colour anyway.)
*
* Cells carry colour only. A grid of two-digit numbers is a table pretending to be a picture,
* and the number is a hover away. */
function Matrix({ words, val, dir, row, pair, onCell, onRow }: {
words: string[]; val: (a: string, b: string) => number | null
dir: (sentencesAbout: string, substituted: string) => number | null
row: string; pair: string[] // ordered by the word list, so pair[0] is the circle
onCell: (r: string, c: string) => void; onRow: (r: string) => void
}) {
return (
<div className="mx-wrap">
<div className="mx"
style={{ gridTemplateColumns: `auto repeat(${words.length}, var(--cell))` }}>
<div className="corner" />
{words.map((c) => <div className="colLabel" key={c}>{plural(c)}</div>)}
{words.map((r) => (
<React.Fragment key={r}>
<button className={"rowLabel" + (row === r ? " on" : "")} aria-pressed={row === r}
aria-label={`which substitutes for ${plural(r)}`} onClick={() => onRow(r)}>
{plural(r)}
</button>
{words.map((c) => {
/* the diagonal is now a real measurement — a word read in its own sentences, which
came out 97.6-99.0 and so is a control that could have failed. It is drawn but
not selectable: a word is not a candidate to replace itself. */
const v = dir(r, c)
if (v == null) return <div className="cell none" key={c} aria-label="no data" />
if (r === c) return (
<div className="cell diag" key={c} role="img" style={{ background: shade(v) }}
aria-label={`${plural(r)} in their own sentences: ${v.toFixed(0)} of 100`} />
)
const on = row === r && pair.includes(c)
const back = dir(c, r), pooled = val(r, c)
// attributive singular — "in human sentences", not "in humans sentences"
const more =
(back == null ? "" : `\nthe other way round, ${plural(r)} in ${c} sentences ` +
back.toFixed(1)) +
(pooled == null ? "" : `\nthe pair, both ways pooled ${pooled.toFixed(1)}`)
return (
<button key={c} aria-pressed={on}
className={"cell" + (on ? " sel" : "")}
style={{ background: shade(v) }}
title={`${plural(c)} in ${r} sentences — ${v.toFixed(1)} of 100${more}`}
aria-label={
`${plural(c)} in ${r} sentences: ${v.toFixed(0)} of 100`}
onClick={() => onCell(r, c)}>
{on && <CellMark tri={c !== pair[0]} />}
</button>
)
})}
</React.Fragment>
))}
</div>
<div className="legend">
{/* the same svg rect the strip draws for its axis — one renderer, so the two bars
cannot differ. It carries its own gradient id: ids resolve document-wide, and the
legend must not depend on the strip being mounted. */}
<svg className="lgBar" viewBox="0 0 302 12" preserveAspectRatio="none" aria-hidden="true">
<defs>
<linearGradient id="lgramp">
<stop offset="0" stopColor="currentColor" stopOpacity=".05" />
<stop offset="1" stopColor="currentColor" stopOpacity=".85" />
</linearGradient>
</defs>
<rect className="axisbar" fill="url(#lgramp)" x="1" y="1" width="300" height="10" />
</svg>
<span className="lgEnds">
<span>nonsense</span>
<span className="mid">← substitution →</span>
<span>makes sense</span>
</span>
</div>
</div>
)
}
// ── views ────────────────────────────────────────────────────────────────────
/** An SVG scales everything with its viewBox, so simply letting the chart shrink would render
* its labels at 9px and its dots at 3px — illegible and untappable. The fix is a second
* geometry rather than a smaller drawing: a narrower viewBox, so the same marks take up more of
* it and land at real sizes once scaled to the screen. */
const narrowQ = window.matchMedia("(max-width: 640px)")
function useNarrow(): boolean {
const [narrow, setNarrow] = useState(narrowQ.matches)
useEffect(() => {
const on = () => setNarrow(narrowQ.matches)
narrowQ.addEventListener("change", on)
return () => narrowQ.removeEventListener("change", on)
}, [])
return narrow
}
/* The two geometries, one column each — what the narrow layout is should be readable here,
* not diffed out of nine ternaries. LAB and TICK are in viewBox units, so they track W rather
* than sit in CSS; `clear` is what the first dot lane needs to duck the mean labels. */
const GEO = {
wide: { W: 900, H: 251, PAD: 54, R: 8, base: 210, TOP_A: 26, TOP_B: 56, LAB: 22, TICK: 13,
clear: 16, ticks: [0, 25, 50, 75, 100] },
narrow: { W: 380, H: 273, PAD: 22, R: 7, base: 232, TOP_A: 24, TOP_B: 52, LAB: 15, TICK: 12,
clear: 12, ticks: [0, 50, 100] },
}
/** The triad, on the anchor's own sentences: two dots per sentence, one per candidate, on the
* same 0-100 axis the pair strip uses. Here the SHAPE means which candidate — the two clouds'
* separation is the qua-judgment, and it is fully paired because both candidates were read
* into the very same sentences. */
function TriStrip({ t, sel, onSel, narrow }: {
t: TriStat; sel: number; onSel: (i: number) => void; narrow: boolean
}) {
/* The two averages sit ABOVE the cloud, each over its own place on the scale, with a line
* dropping to the axis. That is the summary of the page, so it is read before the dots rather
* than after them. `base` is set outright rather than derived from H — deriving it once put a
* label at y=270 in a 262-high canvas, where the viewBox silently clipped it. */
const { W, H, PAD, R, base, TOP_A, TOP_B, LAB, TICK, ticks, clear } =
narrow ? GEO.narrow : GEO.wide
const BIN = R * 2 + 2
const x = (v: number) => PAD + (v / 100) * (W - PAD * 2)
const pts: { i: number; v: number; tri: boolean }[] = []
t.rows.forEach((r, i) => {
if (r.a != null) pts.push({ i, v: r.a, tri: false })
if (r.b != null) pts.push({ i, v: r.b, tri: true })
})
if (!pts.length) return <p className="note">Nothing scored for this triad.</p>
/* Stacks spill SIDEWAYS rather than upward once they reach the top of the plot. A tall column
* either grows through the labels above it or gets clipped by the viewBox, and both are worse
* lies than a dot sitting one bin off its exact value: the nudge is a few percent of a scale
* whose own margin is ±20, while a clipped dot is a sentence that silently stopped existing.
*
* The two candidate bins either side are tried in order of how full they already are, so a
* pile-up spreads evenly instead of drifting left. */
const laneH = R * 2 + 3
const top = TOP_B + clear // the first lane must clear the labels
const lanes = Math.max(1, Math.floor((base - top) / laneH) + 1)
const binMax = Math.round((W - PAD * 2) / BIN)
const fill: Record<number, number> = {}
const seat = (want: number) => {
for (let d = 0; d <= binMax; d++) {
const side = d === 0 ? [want] : [want - d, want + d]
const open = side
.filter((b) => b >= 0 && b <= binMax && (fill[b] ?? 0) < lanes)
.sort((a, b) => (fill[a] ?? 0) - (fill[b] ?? 0))
if (open.length) {
const b = open[0]
const lane = fill[b] ?? 0
fill[b] = lane + 1
return { b, lane }
}
}
return { b: want, lane: 0 } // every bin full: only with absurd counts
}
const placed = pts.sort((p, q) => p.v - q.v).map((p) => {
const { b, lane } = seat(Math.round((x(p.v) - PAD) / BIN))
return { ...p, cx: PAD + b * BIN, lane }
})
const mark = (cx: number, cy: number, tri: boolean) =>
tri ? `M${cx},${cy - R - 1} L${cx + R + 1},${cy + R} L${cx - R - 1},${cy + R} Z`
: `M${cx - R},${cy} a${R},${R} 0 1 0 ${R * 2},0 a${R},${R} 0 1 0 ${-R * 2},0`
return (
<div className="strip-wrap">
{/* the question sits over the chart that answers it, INSIDE the strip block, so a
screenshot of the chart carries its own framing: the averages name the candidates,
this names the anchor. Not "which is more LLM-like": likeness reads as symmetric, and
this measure is not — what is asked is whether a candidate can take the anchor's
place, one direction only. The plural also sidesteps a/an, which was the source of
two earlier bugs. */}
<p className="tagline">Which substitutes for <b>{plural(t.tri.anchor)}</b> better?</p>
<svg viewBox={`0 0 ${W} ${H}`} className="strip" role="img"
aria-label={`${plural(t.tri.a)} and ${plural(t.tri.b)} read into sentences about ` +
plural(t.tri.anchor)}>
{/* the axis IS the matrix legend's ramp — same bar, same key, one system. The gradient
is currentColor at shade()'s endpoint opacities, so it inverts with the theme. */}
<defs>
<linearGradient id="ramp">
<stop offset="0" stopColor="currentColor" stopOpacity=".05" />
<stop offset="1" stopColor="currentColor" stopOpacity=".85" />
</linearGradient>
</defs>
<rect className="axisbar" fill="url(#ramp)" x={PAD} y={base + 11}
width={W - PAD * 2} height={10} />
{ticks.map((v) => (
<g key={v}>
<line x1={x(v)} y1={base + 23} x2={x(v)} y2={base + 27} className="axis" />
<text x={x(v)} y={base + 37} textAnchor="middle" className="tick"
style={{ fontSize: TICK }}>{v}</text>
</g>
))}
{/* drawn before the dots so the cloud sits over its own average marker */}
{([[t.meanA, t.tri.a, false], [t.meanB, t.tri.b, true]] as const).map(([m, w, isB]) => (
<g key={w}>
<text x={x(m)} y={isB ? TOP_B : TOP_A} textAnchor="middle"
className={"meanlab" + (isB ? " b" : "")} style={{ fontSize: LAB }}>
{plural(w)} {Math.round(m)}%
</text>
<line x1={x(m)} y1={(isB ? TOP_B : TOP_A) + 10} x2={x(m)} y2={base + 11}
className={"meanline" + (isB ? " b" : "")} />
</g>
))}
{placed.map((p, k) => (
<path key={k} d={mark(p.cx, base - p.lane * laneH, p.tri)}
className={"dot " + (p.tri ? "b" : "a") + (p.i === sel ? " on" : "")}
onClick={() => onSel(p.i)}>
<title>
{`${say(p.tri ? t.tri.b : t.tri.a, t.rows[p.i].sentence)}\n${p.v} of 100`}
</title>
</path>
))}
</svg>
{/* the trio is the axis's LABEL, under the bar exactly as it sits under the matrix
legend. HTML rather than SVG text, so it holds its size at any width and wraps on a
narrow screen instead of colliding; the inline padding aligns it with the bar's ends. */}
<div className="ends" style={{ padding: `0 ${((PAD / W) * 100).toFixed(1)}%` }}>
<span>nonsense</span>
<span className="mid">← substitution →</span>
<span>makes sense</span>
</div>
</div>
)
}
/** Every pair in order, which is the one thing the grid cannot show: a matrix says which cells
* are strong but not how they rank, because the eye cannot sort ten shades of one colour.
*
* A row here pools both directions into one symmetric number, where a cell is directed and
* measures one way only. The header names what is ranked, not the arithmetic — "most
* substitutable" reads as an ordering, keeps the page's one verb, and the pooling is a
* tooltip away in the grid.
*
* The
* two pairs currently in play are marked in the candidates' own colours, so the list also says
* where this comparison sits among all of them. Clicking a row is the same move as clicking its
* square — the anchor is kept if the pair contains it, and otherwise taken from the pair. */
function Ranks({ stats, row, a, b, onPick }: {
stats: Stat[]; row: string; a: string; b: string; onPick: (p: Pair) => void
}) {
const side = (p: Pair) => {
const has = (w: string) => p.x === w || p.y === w
if (!has(row)) return ""
return has(a) ? " a" : has(b) ? " b" : ""
}
return (
<div className="ranks" role="group" aria-label="most substitutable pairs">
<div className="ranks-h">Most substitutable pairs</div>
{stats.map((s, i) => (
<button key={s.pair.key} className={"rank" + side(s.pair)}
aria-label={
`rank ${i + 1}, ${plural(s.pair.x)} and ${plural(s.pair.y)}: ` +
`${s.score.toFixed(0)} of 100`}
onClick={() => onPick(s.pair)}>
<span className="rank-n">{s.pair.x} · {s.pair.y}</span>
<span className="rank-v">{s.score.toFixed(1)}</span>
</button>
))}
</div>
)
}
function Read({ run }: { run: Run }) {
const stats = useMemo(() => pairStats(run), [run])
const cells = useMemo(() => cellMap(run), [run])
const narrow = useNarrow()
const score = useMemo(() => {
const m = new Map(stats.map((s) => [s.pair.key, s.score]))
return (a: string, b: string) => {
const [x, y] = [a, b].sort(byList(run.words))
return m.get(key(x, y)) ?? null
}
}, [stats, run])
/** One direction: how the SUBSTITUTED word read in sentences written about the other. This is
* exactly what the pooled cell averages away, kept so the tooltip can hand it back. */
const dir = useMemo(() => {
const means = new Map<string, number>()
for (const c of run.results) {
const ok = c.scores.filter((v): v is number => v != null)
if (ok.length) means.set(key(c.anchor, c.subject), mean(ok))
}
return (about: string, sub: string) => means.get(key(about, sub)) ?? null
}, [run])
/* Two cells in a row: the row is who is being stood in for, the two columns are the candidates.
* The default is the top pair plus the best pair sharing a word with it, which on this data is
* `human · chimpanzee` and `human · LLM` — the question the whole exercise is about. */
const start = useMemo(() => {
const first = stats[0]
if (!first) return null
for (const s of stats.slice(1)) {
const t = triOf(run.words, first.pair, s.pair)
if (t) return t
}
return { anchor: first.pair.x, a: first.pair.y, b: first.pair.y }
}, [stats, run])
const [row, setRow] = useState<string>(() => start?.anchor ?? run.words[0])
const [pair, setPair] = useState<string[]>(() =>
start ? [start.a, start.b] : run.words.slice(1, 3))
/* The selection is a SENTENCE, not an index, so it survives every move whose corpus still
* holds it — moving along one row keeps the anchor, so the sentence stays put. */
const [pin, setPin] = useState<string | null>(null)
/* A candidate can never be the row itself — a word does not stand in for itself. So a move that
* would leave fewer than two candidates tops up with the word this row is most similar to,
* which keeps the landing spot meaningful rather than arbitrary. */
const fill = (r: string, want: string[]) => {
const out = want.filter((w) => w !== r)
const rest = run.words
.filter((w) => w !== r && !out.includes(w))
.sort((m, n) => (score(r, n) ?? 0) - (score(r, m) ?? 0))
while (out.length < 2 && rest.length) out.unshift(rest.shift() as string)
return out.slice(-2)
}
const pickCell = (r: string, c: string) => {
setRow(r)
setPair((p) => fill(r, [...p.filter((w) => w !== c), c]))
}
const pickRow = (r: string) => { setRow(r); setPair((p) => fill(r, p)) }
// a ranked row names a pair, not a triad: keep the anchor if the pair contains it
const pickPair = (p: Pair) => {
const anchor = row === p.x || row === p.y ? row : p.x
pickCell(anchor, anchor === p.x ? p.y : p.x)
}
// candidates ordered by the word list, which is the strip's own left to right
const [a, b] = [...pair].sort(byList(run.words))
const tri: Tri = { anchor: row, a, b }
const t = useMemo(() => triStat(run, cells, tri), [run, cells, row, a, b])
if (!stats.length) return <p className="note">No run to show yet.</p>
let idx = pin ? t.rows.findIndex((r) => r.sentence === pin) : -1
if (idx < 0) { // the sentence the two candidates most disagree about
let wv = -Infinity
t.rows.forEach((r, i) => {
if (r.a == null || r.b == null) return
if (Math.abs(r.b - r.a) > wv) { wv = Math.abs(r.b - r.a); idx = i }
})
if (idx < 0) idx = 0
}
const r = t.rows[idx]
const onSel = (i: number) => setPin(t.rows[i].sentence)
return (
<>
{/* singular in the title — the plurals belong in the sentences, where they are what the
model was actually shown. The question itself now lives above the chart, which is the
object that answers it. */}
<h1>
<span className="t-a">{cap(a)}</span> <span className="vs">vs.</span>{" "}
<span className="t-b">{cap(b)}</span>
</h1>
{/* 2×2: the two pickers on top, each with its own text beneath, so no block spans the
width and the prose is never squeezed to the width of the object above it */}
<div className="head">
<Matrix words={run.words} val={score} dir={dir} row={row} pair={[a, b]}
onCell={pickCell} onRow={pickRow} />
<Ranks stats={stats} row={row} a={a} b={b} onPick={pickPair} />
<p className="pickhint">
Each square is <b>the column word read in the row word’s sentences</b>, so it does
not read the same both ways round. <b>Pick two in a row:</b> the row is who is being
substituted for, the two columns are the candidates. Click a square, or a row’s name.
</p>
<p className="sub">
Both candidates are read into the same {t.ok.length} sentences written about
{" "}{plural(row)}, so the two averages are directly comparable.
</p>
</div>
<TriStrip t={t} sel={idx} onSel={onSel} narrow={narrow} />
{/* one sentence, written twice — the substitution shown rather than described */}
<div className="quote">
{([["a", a, r.a], ["b", b, r.b]] as const).map(([side, w, v]) => (
<div className="q-line" key={side}>
<div className="q-text">
<Glyph tri={side === "b"} />{" "}
<s className="sw-was">{subject(row)}</s>{" "}
<span className={"sw " + side}>{subject(w)}</span>{" "}
{r.sentence}
</div>
<span className={"qs-n " + side}>{v ?? "—"}</span>
</div>
))}
<div className="q-cap">
Written about {plural(row)}, and rated <b>{r.was ?? "—"}</b> as written.
</div>
</div>
{t.saturated && (
<p className="note warn">
Both candidates average above 90, so this comparison is at the ceiling of the scale:
everything sensible reads about the same up there and the two cannot be separated. Read
it as “both fit”, not as a measured tie. Resolving it would need a forced
choice between the two sentences rather than a rating of each.
</p>
)}
{t.floored && (
<p className="note warn">
Both candidates average below 10, so this comparison is at the floor of the scale,
the same failure as the ceiling upside down. Nothing that makes no sense can be
told apart from anything else that makes no sense. Read it as “both
nonsense”, not as a measured tie.
</p>
)}
<details className="tableview">
<summary>All {t.rows.length} sentences</summary>
<table>
<thead>
<tr>
<th>sentence about {plural(row)}</th>
<th>{a}</th><th>{b}</th><th>as written</th>
</tr>
</thead>
<tbody>
{t.rows.map((w, i) => (
<tr key={i} className={i === idx ? "on" : ""} onClick={() => onSel(i)}>
<td>{w.sentence}</td>
<td className="num">{w.a ?? "—"}</td>
<td className="num">{w.b ?? "—"}</td>
<td className="num dim">{w.was ?? "—"}</td>
</tr>
))}
</tbody>
</table>
</details>
</>
)
}
type Prog = { done: number; total: number; label: string }
/** Kept short on purpose. A reader wants the claim, the recipe, what the three views show, and
* which numbers to believe — in that order, and nothing else. */
function Method({ run }: { run: Run | null }) {
const when = run?.timestamp
? new Date(run.timestamp).toLocaleDateString(undefined,
{ day: "numeric", month: "long", year: "numeric" })
: null
return (
<div className="method">
<p>
Henry Shevlin{" "}
<a href="https://x.com/dioscuri/status/2083852207758221782" target="_blank"
rel="noopener">contends</a>:
</p>
<blockquote>
Everything is like and unlike everything else, and there are no absolute judgments of
similarity, only qua-judgments. LLMs are more like humans qua language and reasoning,
chimpanzees more like humans qua perception and action.
</blockquote>
<p>
The contention is surely right in the abstract, but it invites an empirical question:
what does an LLM treat as similar when nobody tells it what to compare on?
</p>
<h2>What we did</h2>
<p>
A model writes sentences about each of five words — human, LLM, chimpanzee, frog,
virus. Each sentence is then rewritten with a different subject:
</p>
<blockquote className="pairq">
<span>“<i>Humans</i> seek meaning in their experiences”</span>
<span className="becomes">becomes</span>
<span>“<i>Chimpanzees</i> seek meaning in their experiences”</span>
</blockquote>
<p>
Each version is shown to the model on its own and rated 0 to 100 for how much sense it
makes.
</p>
<p>
It never sees the pair, so it is never choosing between them. And nothing tells it what
counts as similar. Sentences about language pull an LLM closer to a human; sentences
about bodies pull a chimpanzee closer.
</p>
<p>
A substitution only runs one way at a time, and the two ways do not agree. Put a human
into sentences written about LLMs and the result scores <b>85</b>; put an LLM into
sentences written about humans and it scores <b>43</b>.
</p>
{run && (
<p className="note dim">
Scored by <code>{run.judge}</code>{when ? ` on ${when}` : ""}.
</p>
)}
</div>
)
}
function Generate({ models, access, onRun }: {
models: TbModel[]; access: AiAccess
onRun: (m: TbModel, n: number, words: string[]) => void
}) {
const [pick, setPick] = useState(-1)
const [n, setN] = useState(15) // pooled both ways, so a row rests on 2n — see notes.md
const [wordText, setWordText] = useState(WORDS.join(", "))
/* Exactly five, so every run is the same shape and the same price — cost is quadratic
* (`words + words² × n`), and a grid built for five is what the whole page reads as.
*
* SPACES ARE ALLOWED ON PURPOSE. `language model` against `LLM` is one of this project's own
* results — 0.52 nats apart in the local arm, wider than the effect being argued about — so a
* letters-only rule would block a comparison worth running. It is 14 characters, which is why
* the limit is 15 and not 12.
*
* Nothing is silently dropped: a word that fails says so and the run stays disabled, because a
* quietly discarded word reads afterwards exactly like a word that was measured. */
const N_WORDS = 5, MAX_LEN = 15
const SHAPE = /^[A-Za-z][A-Za-z0-9'’ -]*$/
const parsed = useMemo(() => {
const seen = new Set<string>()
const words: string[] = []
const issues: string[] = []
let dupes = 0
for (const raw of wordText.split(",")) {
const w = raw.trim().replace(/\s+/g, " ")
if (!w) continue // trailing comma while typing is not an error
const k = w.toLowerCase()
if (seen.has(k)) { dupes++; continue }
seen.add(k)
if (w.length > MAX_LEN)
issues.push(`“${w}” is too long — ${MAX_LEN} characters at most.`)
else if (w.length < 2)
issues.push(`“${w}” is too short — two letters at least.`)
else if (!SHAPE.test(w))
issues.push(`“${w}” — letters, digits, spaces and hyphens only, starting with a letter.`)
else words.push(w)
}
if (dupes) issues.push(`${dupes} repeated word${dupes > 1 ? "s" : ""} ignored.`)
if (!issues.length && words.length !== N_WORDS)
issues.push(`Five words exactly — there ${words.length === 1 ? "is" : "are"} ${words.length}.`)
return { words, issues }
}, [wordText])
const words = parsed.words
const ready = parsed.issues.length === 0 // a wrong count is itself an issue
useEffect(() => {
if (pick >= 0 || !models.length) return
const luna = models.findIndex((m) => /luna/i.test(m.name))
setPick(luna >= 0 ? luna : 0)
}, [models, pick])
const calls = words.length + words.length * words.length * n
if (access === "none") return (
<p className="warn">
No AI access. This bulb needs <code>--trust</code> and an <code>OPENROUTER_API_KEY</code>
{" "}(or another provider key) in <code>.env</code>.
</p>
)
return (
<div className="setup">
<section className="fld">
<h2>Model</h2>
<select value={pick} aria-label="model"
onChange={(e) => setPick(+e.target.value)}>
{models.map((m, i) => (
<option key={i} value={i}>
{m.providerName ? `${m.providerName}: ${m.friendlyName}` : m.friendlyName}
</option>
))}
</select>
<p className="hint">
One model writes the sentences and scores them. It is shown one sentence at a time,
with nothing named, so it never knows a comparison is happening.
</p>
</section>
<section className="fld">
<h2>Corpus</h2>
{/* fixed choices rather than a number field: typing into a spinner means fighting the
OS over selection and increment, for a value that only ever takes a few sane sizes */}
<div className="choices" role="group" aria-label="sentences per word">
{[5, 10, 15, 20].map((v) => (
<button key={v} type="button" className={"choice" + (n === v ? " on" : "")}
aria-pressed={n === v} onClick={() => setN(v)}>
{v}
</button>
))}
<span className="choices-l">sentences per word</span>
</div>
<p className="hint">
A pair pools both directions, so each row rests on <b>{n * 2}</b> ratings, not {n}.
Sentences disagree with each other a great deal, so that doubling is what buys the
precision rather than a bigger corpus.
</p>
</section>
<section className="fld">
<h2>Words</h2>
<input className="wordbox" value={wordText} aria-label="words, comma separated"
spellCheck={false} onChange={(e) => setWordText(e.target.value)} />
{/* the chips are the parsed truth, shown as the SUBJECT each word will appear as — so a
plural that came out wrong is visible before a run is paid for, not after */}
<div className="chips">
{words.map((w) => <span className="chip" key={w}>{subject(w)}</span>)}
</div>
<p className="hint">
Every word gets its own set of sentences and is substituted into every other set, so
{" "}{words.length} words make <b>{(words.length * (words.length - 1)) / 2} pairs</b>.
The chips are what will actually be written, so check the plurals there, since
every rated sentence begins with one.
</p>
{parsed.issues.length > 0 && (
<div className="note warn">
{parsed.issues.map((m, i) => <div key={i}>{m}</div>)}
</div>
)}
</section>
<div className="action">
{/* courtesy access is gated, not just warned: even the smallest run (130 calls) is far
past the free quota, so it would stall partway and the partial run is discarded */}
<button className="go" disabled={pick < 0 || !ready || access !== "own"}
onClick={() => onRun(models[pick], n, words)}>
Run · {calls} calls
</button>
{access === "courtesy" && (
<span className="hint">
Courtesy model only: {calls} calls is far past its free quota, so the run is
disabled. To run it, use your own keys — locally with the <b>typebulb CLI</b>, or
sign in at <b>typebulb.com</b> and add your keys.
</span>
)}
</div>
</div>
)
}
// ── app ──────────────────────────────────────────────────────────────────────
function App() {
const first = useMemo(baked, []) // parsing the data block once, not on every render
const [run, setRun] = useState<Run | null>(first)
const [view, setView] = useState<"read" | "make" | "about">(first ? "read" : "make")
const [models, setModels] = useState<TbModel[]>([])
const [access, setAccess] = useState<AiAccess>("own")
const [busy, setBusy] = useState(false)
const [prog, setProg] = useState<Prog>({ done: 0, total: 0, label: "" })
const [err, setErr] = useState("")
const abort = useRef(false)
const busyRef = useRef(false)
useEffect(() => {
tb.aiAccess().then(setAccess)
tb.models().then(setModels)
}, [])
/* `data.txt` is for a published copy, where there is no disk to read. Locally the run that
* matters is the one on disk, so `latest.json` — written beside every run's timestamped
* archive — is consulted ALWAYS, and the NEWER of the two wins.
*
* It used to only be read when nothing was baked, which meant a stale bake silently beat a
* fresh 505-call run and the page looked like the run had been lost. Comparing timestamps
* also handles the other direction: bake a newer run, delete its batch, and the bake still
* wins. Whichever was measured last is what you see, and the log says which it took. */
useEffect(() => {
tb.fs.read("latest.json")
.then((t) => {
const d = normalize(JSON.parse(t))
if (!d) return
if (first && first.timestamp >= d.timestamp) {
tb.log(`[data] keeping the baked run (${first.timestamp}) — newer than latest.json`)
return
}
setRun(d); setView("read")
tb.log(`[data] loaded latest.json (${d.timestamp}, n=${d.n}) — newer than the bake`)
})
.catch(() => {})
}, [])
/* `words` comes from the Run view, not from the WORDS constant — that is now only the default
* the field is seeded with. A results file records the set it was measured on, so an old run
* still reads back in its own terms after the field is changed. */
async function go(m: TbModel, n: number, words: string[]) {
busyRef.current = true
abort.current = false
setBusy(true); setErr(""); setView("make")
try {
const corpus: Record<string, string[]> = {}
setProg({ done: 0, total: words.length, label: "writing the sentences…" })
// one writer per word — the passes inside a word are sequential, the words are not
await pool(words.map((a) => async () => { corpus[a] = await generate(m, a, n) }),
CONC, () => setProg((p) => ({ ...p, done: p.done + 1 })), () => abort.current)
if (abort.current) throw new Error("stopped")
// one job per unique (word, sentence) — every column of a row shares the diagonal's scores
const seen = new Map<string, { word: string; sentence: string }>()
for (const anchor of words)
for (const word of words)
for (const s of corpus[anchor] ?? []) seen.set(`${word} ${s}`, { word, sentence: s })
const scored = new Map<string, number>()
const jobs = [...seen.entries()].map(([k, j]) => async () => {
const v = await rate(m, say(j.word, j.sentence))
if (v != null) scored.set(k, v)
})
let done = 0
setProg({ done: 0, total: jobs.length, label: `rating ${jobs.length} sentences…` })
await pool(jobs, CONC, () => setProg((p) => ({ ...p, done: ++done })), () => abort.current)
if (abort.current) throw new Error("stopped")
const results: Cell[] = []
for (const anchor of words)
for (const subj of words)
results.push({
anchor, subject: subj,
scores: (corpus[anchor] ?? []).map((s) => scored.get(`${subj} ${s}`) ?? null),
})
const out: Run = {
timestamp: new Date().toISOString(), judge: m.name, provider: m.provider,
n, words, corpus, results,
}
setRun(out); setView("read"); setProg({ done: 0, total: 0, label: "" })
const stamp = out.timestamp.replace(/[:.]/g, "-")
try {
const body = toJson(out)
await tb.fs.write(`matrix-${stamp}.json`, body)
await tb.fs.write("latest.json", body) // what the page reloads from
tb.log(`[run] wrote matrix-${stamp}.json (n=${n}, ${results.length} cells, judge=${m.name})`)
} catch (e) {
tb.log(`[run] could not write results: ${String(e)}`)
}
} catch (e) {
setErr(String(e))
tb.log(`[run] error ${String(e)}`)
} finally {
busyRef.current = false
setBusy(false)
}
}
/* Terminal channel. `send … selftest --wait` prints what the read view is drawing, so a run
* can be checked without a screenshot; `send … make` / `read` switch views. Re-registered
* when the run changes so the handler reads the current one; `busy` arrives via a ref so a
* mid-run selftest answers truthfully without re-registering every progress tick. */
useEffect(() => tb.onMessage((msg: unknown) => {
if (msg === "read" || msg === "make" || msg === "about") { setView(msg); return `view: ${msg}` }
if (msg !== "selftest") return
if (!run) return { ok: false, reason: "no run loaded" }
const pairs = pairStats(run).map((s) => ({
pair: s.pair.key, score: +s.score.toFixed(1), n: s.ok.length,
xy: +s.xy.toFixed(1), yx: +s.yx.toFixed(1), asym: +s.asym.toFixed(1),
saturated: s.saturated, floored: s.floored,
}))
return { ok: true, busy: busyRef.current, judge: run.judge, n: run.n,
when: run.timestamp, words: run.words, pairs }
}), [run])
const download = () => {
if (!run) return
const blob = new Blob([toJson(run)], { type: "application/json" })
const a = document.createElement("a")
a.href = URL.createObjectURL(blob)
a.download = `luna-${run.timestamp.slice(0, 10)}.json`
a.click()
URL.revokeObjectURL(a.href)
}
const upload = () => {
const input = document.createElement("input")
input.type = "file"; input.accept = ".json"
input.onchange = async () => {
const f = input.files?.[0]
if (!f) return
try {
const d = normalize(JSON.parse(await f.text()))
if (!d) throw new Error("no results in that file")
setRun(d); setView("read"); setErr("")
} catch (e) { setErr(`could not read that file: ${String(e)}`) }
}
input.click()
}
return (
<div className="wrap viz-root">
<div className="bar">
<button className={"tab" + (view === "read" ? " on" : "")} disabled={!run}
onClick={() => setView("read")}>Results</button>
<button className={"tab" + (view === "make" ? " on" : "")}
onClick={() => setView("make")}>Run</button>
<button className={"tab" + (view === "about" ? " on" : "")}
onClick={() => setView("about")}>Method</button>
{busy
? <button className="tab act stop" onClick={() => { abort.current = true }}>Stop</button>
: <>
<button className="tab act" onClick={upload}>Upload</button>
{run && <button className="tab act" onClick={download}>Download</button>}
</>}
</div>
{busy && (
<div className="prog">
<div className="prog-track">
<div className="prog-fill"
style={{ width: `${prog.total ? (prog.done / prog.total) * 100 : 0}%` }} />
</div>
<p className="prog-lab">{prog.label} {prog.total ? `${prog.done}/${prog.total}` : ""}</p>
</div>
)}
{err && <p className="note warn">{err}</p>}
{view === "read" && run && <Read run={run} />}
{view === "read" && !run && (
<p className="note">No run baked in yet. Switch to <b>Run</b>, or upload a results file.</p>
)}
{view === "make" && (
models.length
? <Generate models={models} access={access} onRun={go} />
: <p className="note">Loading models…</p>
)}
{view === "about" && <Method run={run} />}
</div>
)
}
createRoot(document.getElementById("root")!).render(<App />)
```
**styles.css**
```css
.viz-root {
color-scheme: light;
--cell: 34px;
--ink: #0b0b0b; --ink-2: #45443f; --muted: #6f6e68;
--ink-rgb: 11,11,11;
--border: rgba(11,11,11,.12); --panel: rgba(11,11,11,.04);
--a: #2a78d6; --b: #eb6834; --surface: #fcfcfb;
}
@media (prefers-color-scheme: dark) {
:root:where(:not([data-theme="light"])) .viz-root {
color-scheme: dark;
--ink:#fff; --ink-2:#d8d7cd; --muted:#9d9c93;
--ink-rgb: 255,255,255;
--border:rgba(255,255,255,.14); --panel:rgba(255,255,255,.05);
--a:#3987e5; --b:#d95926; --surface:#1a1a19;
}
}
:root[data-theme="dark"] .viz-root {
color-scheme: dark;
--ink:#fff; --ink-2:#d8d7cd; --muted:#9d9c93;
--ink-rgb: 255,255,255;
--border:rgba(255,255,255,.14); --panel:rgba(255,255,255,.05);
--a:#3987e5; --b:#d95926; --surface:#1a1a19;
}
/* the charts earn the extra width; the prose does not, so each text block caps its own line
length rather than running to the full measure */
.wrap { max-width: 1080px; margin: 0 auto; padding: 20px 22px 60px;
font: 17px/1.55 system-ui, -apple-system, "Segoe UI", sans-serif; color: var(--ink); }
h1 { font-size: 31px; margin: 0; letter-spacing: -.025em; text-align: center; }
.sub { margin: 0; color: var(--muted); font-size: 16px; }
.t-a { color: var(--a); } .t-b { color: var(--b); }
.vs { font-weight: 400; color: var(--muted); }
.tagline { font-size: 25px; margin: 0 0 12px; letter-spacing: -.02em;
font-weight: 500; color: var(--ink-2); text-align: center; }
.tagline b { text-decoration: underline; text-underline-offset: 3px; }
/* the candidate glyph, sized to whatever text it sits in */
.glyph { display: inline-block; width: .78em; height: .78em; vertical-align: baseline; }
.glyph.a { fill: var(--a); } .glyph.b { fill: var(--b); }
/* Two equal columns, two rows: matrix and ranking above, their text beneath. Equal columns
rather than columns sized to the objects — a column only as wide as the matrix would squeeze
the paragraph under it to 250px, which is why the text used to have to span the whole page. */
.head { display: grid; grid-template-columns: minmax(0,1fr) minmax(0,1fr);
gap: 18px 34px; align-items: start; margin-top: 22px; }
@media (max-width: 760px) {
/* stacked, each picker still followed by its own text rather than by the other picker */
.head { grid-template-columns: minmax(0,1fr); }
.head > .mx-wrap { order: 1 }
.head > .pickhint { order: 2 }
/* on its own row it is a 290px block in a full-width column, so it centres like the matrix
above it rather than hugging the left edge */
.head > .ranks { order: 3; justify-self: center; }
.head > .sub { order: 4 }
}
/* the order the grid cannot show — ten shades of one colour do not sort by eye */
/* capped rather than filling the column: stretched to 400px the number drifts a quarter of a
page from the name it belongs to, and the two stop reading as one row */
.ranks { display: grid; gap: 1px; align-content: start; max-width: 290px; }
.ranks-h { font-size: 15.5px; font-weight: 640; letter-spacing: -.015em; color: var(--ink-2);
text-align: center; padding: 0 6px 8px; }
.rank { display: grid; grid-template-columns: minmax(0,1fr) auto; gap: 10px;
font: inherit; font-size: 13px; text-align: left; padding: 3px 6px; cursor: pointer;
border: 0; border-left: 3px solid transparent; border-radius: 3px;
background: transparent; color: var(--muted); white-space: nowrap; }
.rank:hover { background: var(--panel); color: var(--ink-2); }
.rank.a { color: var(--a); border-left-color: var(--a); font-weight: 640; }
.rank.b { color: var(--b); border-left-color: var(--b); font-weight: 640; }
.rank-v { font-variant-numeric: tabular-nums; }
/* chrome — one underlined row, thicker under the selected tab -------------- */
.bar { display: flex; align-items: flex-end; gap: 22px; margin-bottom: 20px;
border-bottom: 1px solid var(--border); }
.tab { font: inherit; font-size: 15px; padding: 6px 2px 9px; cursor: pointer;
background: transparent; color: var(--muted);
border: 0; border-bottom: 3px solid transparent; margin-bottom: -1px; }
.tab:hover:not(:disabled) { color: var(--ink); }
.tab.on { color: var(--ink); font-weight: 600; border-bottom-color: var(--ink); }
.tab:disabled { opacity: .4; cursor: default; }
/* the actions share the row but never take the selected-tab underline */
.tab.act { font-size: 14.5px; }
.tab.act:not(.stop):hover { color: var(--ink); }
.tab.stop { color: var(--b); }
.prog { margin: 0 0 16px; }
.prog-track { height: 6px; border-radius: 3px; background: var(--panel); overflow: hidden; }
.prog-fill { height: 100%; background: var(--b); border-radius: 3px; transition: width .2s; }
.prog-lab { margin: 6px 0 0; font-size: 14.5px; color: var(--muted);
font-variant-numeric: tabular-nums; }
/* matrix — the picker. Symmetric, so a ROW is every pair one word belongs to, and two cells in
it are the two candidates. The diagonal is blank: a word is not a pair with itself. ------- */
.mx { display: grid; gap: 2px; }
.mx-wrap { width: max-content; margin: 0 auto; }
.colLabel { writing-mode: vertical-rl; transform: rotate(180deg);
font-size: 13px; color: var(--muted); justify-self: center;
/* the rotation flips physical sides, so the gap under the label is padding-TOP */
max-height: 104px; white-space: nowrap; padding-top: 6px; }
.rowLabel { font: inherit; font-size: 13px; text-align: right; white-space: nowrap;
padding: 0 9px 0 0; border: 0; background: transparent; color: var(--muted);
cursor: pointer; }
.rowLabel:hover { color: var(--ink); }
.rowLabel.on { color: var(--ink); font-weight: 640; }
.cell { width: var(--cell); height: var(--cell); border-radius: 4px; border: 0; padding: 0;
cursor: pointer; }
/* the top of the scale by stipulation, not by measurement — so it is coloured but never picked */
.cell.diag { cursor: default; }
/* the selected pair is marked by its SHAPE, not an outline: an outline only says "selected",
while the shape also says which of the two candidates this square is */
.cmark { display: block; width: 100%; height: 100%;
fill: var(--surface); stroke-width: 2.2; stroke-linejoin: round; }
.cmark.a { stroke: var(--a); } .cmark.b { stroke: var(--b); }
.cell:focus-visible { outline: 2px solid var(--ink); outline-offset: 2px; }
.cell.none { background: transparent; cursor: default; border: 1px dashed var(--border); }
.legend { display: grid; gap: 5px; margin-top: 12px; }
/* width 0 + min-width 100%: the svg's intrinsic (viewBox) width must not size the max-content
matrix wrap — the matrix sizes the column, the bar stretches to it */
.lgBar { display: block; width: 0; min-width: 100%; height: 12px; }
.lgEnds { display: flex; justify-content: space-between; gap: 12px;
font-size: 13px; color: var(--muted); }
/* no max-width: the grid cell is the measure now. The 58ch cap was sized for the old
three-column row and, in a half-width quadrant, only stopped the text short of its own cell. */
.pickhint { margin: 0; font-size: 15px; color: var(--muted); line-height: 1.6; }
.chips { display: flex; flex-wrap: wrap; gap: 8px; }
.chip { font-size: 14.5px; padding: 3px 10px; border-radius: 999px;
border: 1px solid var(--border); color: var(--ink-2); }
/* strip ------------------------------------------------------------------- */
.strip-wrap { margin: 20px 0 24px; }
/* no min-width: below 640px the chart is re-laid-out rather than scrolled sideways */
.strip { width: 100%; height: auto; display: block; }
.ends { display: flex; justify-content: space-between; flex-wrap: wrap; gap: 4px 20px;
font-size: 16px; font-weight: 600; letter-spacing: -.01em; color: var(--muted);
margin-top: 0; }
/* the poles are values, the centre is the axis's name — weight makes the hierarchy */
.ends .mid, .lgEnds .mid { font-weight: 400; }
/* both gradient bars — the matrix legend and the strip's axis — are this one rect */
.axisbar { color: var(--ink); stroke: rgba(128,128,128,.25); }
.axis { stroke: var(--border); stroke-width: 1.5; }
.tick { fill: var(--muted); font-family: system-ui, sans-serif;
font-variant-numeric: tabular-nums; }
/* the summary of the whole page, so it is sized like one */
.meanlab { fill: var(--a); font-weight: 660; letter-spacing: -.02em;
font-family: system-ui, sans-serif; font-variant-numeric: tabular-nums; }
.meanlab.b { fill: var(--b); }
/* a position marker, not a mark of its own — it drops from the label to the axis and the dots
are drawn over it */
.meanline { stroke: var(--a); stroke-width: 2; opacity: .45; }
.meanline.b { stroke: var(--b); }
/* one hue: position carries the score, shape carries the direction, and a third channel here
would only repeat one of them */
/* hue AND shape both say which candidate — position already says the score, so colour is free
to do the job the geometry cannot */
.dot { cursor: pointer; stroke: var(--surface); stroke-width: 2.5; opacity: .55; }
.dot.a { fill: var(--a); } .dot.b { fill: var(--b); }
.dot:hover { opacity: .85; }
.dot.on { opacity: 1; stroke: var(--ink); stroke-width: 3; }
/* quote ------------------------------------------------------------------- */
.quote { padding: 20px 22px; border-radius: 12px;
border: 1px solid var(--border); background: var(--panel); }
.q-text { font-size: 16.5px; line-height: 1.5; }
/* the triad's two readings of one sentence, stacked so the swap is the only visible difference */
.q-line { display: grid; grid-template-columns: minmax(0,1fr) 46px; gap: 14px;
align-items: baseline; }
.q-line + .q-line { margin-top: 10px; }
/* the substitution, shown rather than explained: what was replaced, struck, then what replaced it */
.sw { font-weight: 640; color: var(--ink); }
.sw.a { color: var(--a); } .sw.b { color: var(--b); }
.sw-was { color: var(--muted); font-weight: 400; text-decoration-thickness: 2px; }
.qs-n { font-size: 17px; font-weight: 640; font-variant-numeric: tabular-nums;
text-align: right; }
.qs-n.a { color: var(--a); } .qs-n.b { color: var(--b); }
.q-cap { margin-top: 14px; padding-top: 12px; border-top: 1px solid var(--border);
font-size: 14.5px; color: var(--muted); line-height: 1.55; }
/* table view -------------------------------------------------------------- */
.tableview { margin-top: 22px; font-size: 15px; }
.tableview summary { cursor: pointer; color: var(--muted); }
.tableview table { display: block; overflow-x: auto; width: 100%; border-collapse: collapse;
margin-top: 12px; }
.tableview th, .tableview td { text-align: left; padding: 5px 8px; vertical-align: top;
border-bottom: 1px solid var(--border); }
.tableview th { font-weight: 600; font-size: 13px; letter-spacing: .03em; color: var(--muted); }
.tableview tbody tr { cursor: pointer; }
.tableview tbody tr:hover, .tableview tbody tr.on { background: var(--panel); }
.num { text-align: right; font-variant-numeric: tabular-nums; width: 70px; }
.num.dim { color: var(--muted); }
/* setup ------------------------------------------------------------------- */
.setup { display: grid; gap: 26px; }
.fld h2 { font-size: 17px; font-weight: 630; margin: 0 0 8px; letter-spacing: -.015em; }
input, select { font: inherit; font-size: 15px; padding: 5px 8px; border-radius: 6px;
border: 1px solid var(--border); background: Canvas; color: CanvasText; }
.wordbox { width: 100%; max-width: 46ch; margin-bottom: 10px; }
option { background: Canvas; color: CanvasText; }
/* a short row of sane sizes — the whole range this dial is ever set to */
.choices { display: flex; align-items: center; flex-wrap: wrap; gap: 8px; }
.choice { font: inherit; font-size: 15px; font-variant-numeric: tabular-nums;
min-width: 46px; padding: 6px 12px; border-radius: 7px; cursor: pointer;
border: 1px solid var(--border); background: transparent; color: var(--ink-2); }
.choice:hover { color: var(--ink); border-color: var(--muted); }
.choice.on { background: var(--ink); border-color: var(--ink); color: var(--surface);
font-weight: 640; }
.choices-l { margin-left: 6px; font-size: 15px; color: var(--ink-2); }
.hint { margin: 10px 0 0; font-size: 14.5px; color: var(--muted); line-height: 1.55; }
.action { display: flex; align-items: center; gap: 14px; flex-wrap: wrap; }
.go { font: inherit; font-size: 16px; padding: 9px 22px; border-radius: 7px; cursor: pointer;
border: 1px solid var(--ink-2); background: transparent; color: var(--ink); }
.go:disabled { opacity: .45; cursor: default; }
.note { font-size: 16px; color: var(--ink-2); line-height: 1.6; margin: 16px 0 0;
max-width: 88ch; }
/* Method — a page you arrive at on purpose, so it is prose measure rather than caption measure */
.method { max-width: 72ch; font-size: 16.5px; line-height: 1.65; color: var(--ink-2); }
.method h2 { font-size: 21px; font-weight: 640; letter-spacing: -.02em; color: var(--ink);
margin: 18px 0 6px; }
.method h2:first-child { margin-top: 4px; }
.method p { margin: 12px 0; }
.method b { color: var(--ink); }
.method blockquote { margin: 14px 0; padding: 2px 0 2px 16px;
border-left: 3px solid var(--border); color: var(--muted); font-style: italic; }
.method .pairq { display: grid; gap: 4px; font-style: normal; color: var(--ink-2); }
.method .pairq .becomes { color: var(--muted); font-size: 14.5px; }
.note.dim { color: var(--muted); font-size: 15px; }
.note.warn, .warn { color: var(--ink); border-left: 3px solid var(--b); padding-left: 12px; }
code { font-size: 14.5px; background: var(--panel); padding: 2px 6px; border-radius: 4px; }
```
**index.html**
```html
<div id="root"></div>
```
**data.txt**
```txt
{
"timestamp": "2026-08-03T00:48:03.097Z",
"judge": "openai/gpt-5.6-luna",
"provider": "openrouter",
"n": 20,
"words": [
"human",
"LLM",
"chimpanzee",
"frog",
"virus"
],
"corpus": {
"human": [
"communicate through language, gestures, expressions, and symbols.",
"seek food, water, shelter, and security.",
"create art to express ideas, emotions, and experiences.",
"form families, friendships, communities, and societies.",
"learn from observation, instruction, practice, and experience.",
"adapt their behavior to different environments and circumstances.",
"use tools to extend their physical and mental abilities.",
"remember some events while forgetting many others.",
"experience a wide range of emotions throughout their lives.",
"organize knowledge into stories, categories, theories, and traditions.",
"cooperate to accomplish goals that individuals cannot achieve alone.",
"compete for resources, recognition, influence, and opportunities.",
"change their surroundings through agriculture, construction, technology, and industry.",
"develop customs that vary across cultures and generations.",
"make decisions using both reasoning and intuition.",
"are influenced by their upbringing, surroundings, beliefs, and relationships.",
"investigate the natural world through observation and experimentation.",
"celebrate milestones, rituals, achievements, and shared identities.",
"face illness, aging, uncertainty, and mortality.",
"continue to invent new ways to solve problems and satisfy needs."
],
"LLM": [
"generate text by predicting likely sequences of tokens.",
"can summarize, translate, classify, and transform written content.",
"learn statistical patterns from large collections of training data.",
"may produce confident answers that contain factual errors.",
"can adapt their responses to different tones, formats, and audiences.",
"are used in customer support, education, software development, and research.",
"do not possess human experiences or personal beliefs.",
"can explain complex topics at different levels of difficulty.",
"may reflect biases present in their training data or evaluation processes.",
"can assist programmers by generating, reviewing, and debugging code.",
"often perform better when users provide clear context and specific instructions.",
"can process conversational context within the limits of their context windows.",
"require substantial computational resources during training and deployment.",
"can help users brainstorm ideas, outlines, names, and creative concepts.",
"may struggle with arithmetic, obscure facts, and multi-step reasoning.",
"can generate text in many languages, although quality varies across languages.",
"are evaluated using benchmarks, human judgments, and task-specific tests.",
"can be integrated into applications through programming interfaces and software tools.",
"do not automatically verify the accuracy or reliability of their outputs.",
"continue to evolve through advances in data, algorithms, hardware, and alignment methods."
],
"chimpanzee": [
"are great apes native to equatorial Africa.",
"share a large proportion of their DNA with humans.",
"live in communities that may include dozens of individuals.",
"communicate through vocalizations, gestures, facial expressions, and body language.",
"use tools such as sticks to extract termites from their nests.",
"sometimes use stones to crack open hard-shelled nuts.",
"eat fruits, leaves, seeds, insects, and occasionally small animals.",
"build sleeping nests in trees or on the ground.",
"have long arms that help them move through forest canopies.",
"can walk on two legs for short distances.",
"form strong social bonds with other members of their communities.",
"groom one another to strengthen relationships and reduce tension.",
"recognize themselves and other individuals over long periods.",
"learn many behaviors by observing older group members.",
"have complex systems of alliances and social status.",
"may show empathy toward injured or distressed companions.",
"inhabit forests, woodlands, and savanna mosaics.",
"face threats from habitat loss, hunting, and infectious diseases.",
"usually have a gestation period of about eight months.",
"are protected in many countries because their populations have declined."
],
"frog": [
"are amphibians that typically live part of their lives in water and part on land.",
"breathe through their lungs and, in many species, through their skin.",
"have long, muscular hind legs adapted for jumping.",
"use their sticky tongues to capture insects and other small prey.",
"begin life as aquatic tadpoles in most species.",
"undergo metamorphosis as they develop from tadpoles into adults.",
"can be found on every continent except Antarctica.",
"inhabit environments ranging from tropical rainforests to deserts.",
"absorb water through their skin rather than drinking it in the usual way.",
"produce a wide variety of calls for mating, territory, and warning.",
"are prey for animals such as snakes, birds, fish, and mammals.",
"help control insect populations in many ecosystems.",
"can serve as indicators of environmental health because their skin is sensitive to pollutants.",
"have eyes positioned to provide a broad field of vision.",
"vary greatly in size, color, behavior, and habitat.",
"may use camouflage to avoid predators.",
"in cold regions can survive winter by entering dormant states.",
"reproduce in different ways, including laying eggs in water or on land.",
"have permeable skin that makes them vulnerable to changes in moisture and water quality.",
"are represented in myths, folklore, literature, and popular culture around the world."
],
"virus": [
"are infectious agents that require host cells to reproduce.",
"contain genetic material made of DNA or RNA.",
"can infect animals, plants, fungi, bacteria, and archaea.",
"are generally much smaller than bacteria.",
"vary widely in shape, structure, and genetic complexity.",
"may spread through air, water, food, bodily fluids, or direct contact.",
"can cause diseases ranging from mild infections to severe illnesses.",
"sometimes produce no noticeable symptoms in infected hosts.",
"evolve through mutation, genetic recombination, and natural selection.",
"can evade immune defenses by changing their surface molecules.",
"may remain dormant in host cells for extended periods.",
"can be transmitted between different species under suitable conditions.",
"influence ecosystems by affecting the survival and abundance of organisms.",
"that infect bacteria are called bacteriophages.",
"are used in some medical treatments, including certain gene therapies.",
"can be detected through laboratory tests that identify their genetic material or proteins.",
"are studied to understand evolution, cellular biology, and disease transmission.",
"can be prevented by measures such as vaccination, hygiene, and isolation.",
"do not carry out independent metabolism outside host cells.",
"have contributed genetic material to the genomes of many organisms."
]
},
"results": [
{
"anchor": "human",
"subject": "human",
"scores": [100, 100, 100, 100, 99, 100, 99, 100, 100, 100, 100, 98, 100, 100, 95, 100, 100, 99, 100, 98]
},
{
"anchor": "human",
"subject": "LLM",
"scores": [25, 0, 35, 5, 65, 90, 85, 90, 0, 65, 75, 35, 35, 35, 55, 30, 20, 30, 5, 85]
},
{
"anchor": "human",
"subject": "chimpanzee",
"scores": [75, 99, 25, 95, 85, 95, 95, 95, 95, 35, 95, 95, 0, 95, 85, 95, 95, 45, 100, 90]
},
{
"anchor": "human",
"subject": "frog",
"scores": [35, 95, 2, 20, 45, 95, 10, 95, 35, 1, 40, 65, 0, 2, 35, 35, 40, 5, 98, 75]
},
{
"anchor": "human",
"subject": "virus",
"scores": [2, 5, 0, 2, 2, 85, 2, 45, 0, 0, 75, 55, 15, 5, 0, 5, 0, 0, 25, 35]
},
{
"anchor": "LLM",
"subject": "human",
"scores": [80, 100, 85, 100, 100, 100, 0, 100, 98, 100, 95, 95, 40, 100, 100, 100, 95, 90, 90, 40]
},
{
"anchor": "LLM",
"subject": "LLM",
"scores": [98, 98, 98, 100, 98, 100, 99, 95, 98, 100, 98, 98, 95, 100, 100, 98, 100, 100, 95, 98]
},
{
"anchor": "LLM",
"subject": "chimpanzee",
"scores": [0, 5, 70, 75, 75, 5, 75, 2, 20, 0, 75, 65, 2, 0, 95, 5, 90, 10, 85, 0]
},
{
"anchor": "LLM",
"subject": "frog",
"scores": [0, 0, 25, 1, 0, 35, 100, 0, 5, 1, 5, 0, 0, 5, 95, 1, 10, 20, 95, 0]
},
{
"anchor": "LLM",
"subject": "virus",
"scores": [0, 5, 0, 5, 0, 20, 100, 5, 5, 5, 2, 0, 0, 0, 80, 20, 35, 90, 88, 5]
},
{
"anchor": "chimpanzee",
"subject": "human",
"scores": [95, 100, 100, 100, 100, 100, 98, 65, 15, 100, 100, 92, 98, 98, 100, 100, 98, 95, 20, 2]
},
{
"anchor": "chimpanzee",
"subject": "LLM",
"scores": [0, 2, 0, 5, 0, 1, 0, 0, 0, 2, 5, 5, 25, 25, 5, 85, 0, 0, 0, 2]
},
{
"anchor": "chimpanzee",
"subject": "chimpanzee",
"scores": [98, 95, 100, 98, 98, 99, 98, 98, 98, 98, 98, 95, 95, 95, 98, 90, 100, 100, 80, 95]
},
{
"anchor": "chimpanzee",
"subject": "frog",
"scores": [0, 75, 65, 85, 5, 1, 25, 15, 2, 35, 15, 25, 15, 20, 25, 40, 95, 100, 2, 95]
},
{
"anchor": "chimpanzee",
"subject": "virus",
"scores": [0, 20, 60, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 2, 0, 95, 40, 1, 0]
},
{
"anchor": "frog",
"subject": "human",
"scores": [0, 90, 35, 0, 0, 0, 10, 100, 0, 85, 85, 95, 45, 95, 100, 96, 5, 0, 70, 100]
},
{
"anchor": "frog",
"subject": "LLM",
"scores": [0, 0, 0, 0, 0, 5, 78, 10, 0, 5, 0, 15, 0, 0, 20, 10, 10, 0, 0, 35]
},
{
"anchor": "frog",
"subject": "chimpanzee",
"scores": [0, 5, 20, 3, 0, 0, 98, 15, 0, 95, 40, 75, 30, 80, 95, 65, 5, 0, 25, 85]
},
{
"anchor": "frog",
"subject": "frog",
"scores": [100, 100, 100, 100, 98, 100, 100, 98, 95, 100, 99, 98, 95, 95, 100, 98, 98, 98, 99, 99]
},
{
"anchor": "frog",
"subject": "virus",
"scores": [0, 0, 0, 0, 0, 0, 2, 100, 0, 0, 20, 95, 5, 0, 85, 35, 95, 0, 25, 95]
},
{
"anchor": "virus",
"subject": "human",
"scores": [1, 100, 55, 0, 85, 95, 95, 95, 100, 65, 5, 10, 100, 2, 98, 100, 100, 5, 70, 85]
},
{
"anchor": "virus",
"subject": "LLM",
"scores": [0, 0, 0, 20, 20, 0, 5, 0, 25, 0, 0, 10, 75, 0, 30, 0, 15, 0, 100, 0]
},
{
"anchor": "virus",
"subject": "chimpanzee",
"scores": [0, 100, 10, 0, 55, 5, 95, 90, 100, 15, 2, 5, 95, 0, 65, 98, 95, 1, 5, 10]
},
{
"anchor": "virus",
"subject": "frog",
"scores": [0, 100, 5, 0, 90, 15, 80, 95, 100, 35, 2, 10, 98, 0, 65, 98, 98, 2, 0, 45]
},
{
"anchor": "virus",
"subject": "virus",
"scores": [100, 100, 100, 100, 100, 98, 100, 100, 98, 95, 100, 100, 100, 100, 100, 100, 100, 98, 98, 95]
}
]
}
```
**config.json**
```json
{
"description": "Is a chimpanzee or an LLM more like a human? Humans pass for LLMs; LLMs don't pass for humans. An experiment in subject-swapped, blind-rated sentences.",
"dependencies": {
"react": "^19.2.7",
"react-dom": "^19.2.7",
"pluralize": "^8.0.0"
}
}
```
**notes.md**
````md
One bulb, two views: **Run** generates a dataset, **Results** reads it.
## The question is a triad
> "there are no absolute judgments of similarity, only qua-judgments. LLMs are more like humans
> qua language and reasoning, chimpanzees more like humans qua perception and action."
Three words: an anchor and two candidates. **Given a human, chimpanzee or LLM?** That is the
question, and it is why the picker is a grid rather than a list — a list of pairs can only answer
it by making you compare two of its rows in your head.
Nothing here ever names a respect. The model is shown one sentence at a time, with no pair, no
candidate list and no topic, and asked only how much sense it makes as a claim about the world.
Whatever weighting it applies is its own; the design refuses to supply one.
## What each object carries
Three things are on screen and none of them repeats another:
| | what it is | symmetric? |
|---|---|---|
| **the grid** | the column word read in the row word's sentences | **no** — a square against its mirror is the direction gap |
| **the ranking** | both directions pooled, one number per pair | **yes** — the only place "likeness" is honest |
| **the chart** | both candidates read into the anchor's own sentences | paired; its two averages *are* the two picked squares |
**Selection is positional, not tied to what the colour encodes.** Row `w` is `w`'s own sentences,
so two cells in that row are two candidates for `w`'s place. Click a square to move; click a row's
name to ask the same of a different word. A candidate can never be its own row, so a move that
would leave fewer than two tops up with the word that row is most similar to. The selected
sentence is remembered by its text, so sliding along a row keeps the same sentence on screen with
different words substituted into it.
**Two earlier versions of this grid failed, for different reasons.** The first forced one side of
every comparison onto the **diagonal** — the corpus's own word — so it could not express the
headline question at all and pinned a 96–99 word into every reading. The second pooled the cells
to fix that, which fixed the wrong thing: it was never the directedness that hurt, and pooling
cost the asymmetry *and* made the squares disagree with the chart below them.
## The diagonal
A word read in its own sentences: **96–99 across all five**. It is drawn, at the top of the scale
where it belongs, but not selectable — a word is not a candidate to replace itself. It is a
control that could have failed and did not, and it is what the quote panel reports as "as
written".
## Why the chart is the anchor's sentences only
Both candidates are read into the *same* sentences, so the comparison is paired and the two
averages are directly subtractable. That costs precision — the chart rests on `n` sentences where
a ranking row rests on `2n` — and it is the most important number on the page, so it is also the
least precise one. Size the run accordingly.
## Words, and the plural trap
Five words exactly, 2–15 characters, letters/digits/spaces/hyphens, starting with a letter.
Duplicates are ignored case-insensitively and nothing is dropped silently — a failing word says so
and the run stays disabled.
**Spaces are allowed on purpose.** `language model` against `LLM` is one of this project's own
results — 0.52 nats apart in the local arm, wider than the effect being argued about — so a
letters-only rule would block a comparison worth running. It is 14 characters, which is why the
limit is 15.
**Plurals are not cosmetic**: every rated sentence begins with one, so a wrong plural means a whole
word's worth of ungrammatical sentences. That has already cost this project a row — a `/s$/` guard
turned `virus` into `Virus` and invalidated an entire n=25 column. Hand-rolled rules cannot survive
typed input (`person`→`people`, `sheep`→`sheep`, an already-plural `humans` must stay put), so
`pluralize` does it, with two overrides:
- an all-caps initialism takes a bare `s`, because the library restores case and returns `LLMS`
- `virus`, because the library returns **`viri`** — Latin *virus* had no plural and *viri* is the
plural of *vir*, "man". Its `-us → -i` family (`cactus`, `fungus`, `radius`) is where to look
first if another word comes out strange.
The chips under the field show the **subject form each word will actually appear as**, so a bad
plural is visible before a run is paid for rather than discovered in the data afterwards.
## How many sentences
5 · 10 · 15 · 20, defaulting to 15. A ranking row pools both directions and so rests on `2n`
ratings; the chart rests on `n`. Cost is `words + words² × n` — **380 calls at 15**, 505 at 20.
There is deliberately **no repeat-rating control**: rating one sentence three times attacks noise
in the model, while the scatter here is between sentences.
## Results so far
Two runs, `openai/gpt-5.6-luna`, 2026-08-03. Each run writes its own corpus, so these share no
sentences at all — which makes the comparison a real replication rather than a re-read.
| pair, pooled | n=10 | n=20 |
|---|---:|---:|
| human · chimpanzee | 83.2 | **81.6** |
| human · LLM | 63.6 | **64.3** |
| human · frog | 36.4 | 46.0 |
| human · virus | 37.6 | 40.6 |
| frog · virus | 30.4 | 37.4 |
| chimpanzee · frog | 40.6 | 36.9 |
| chimpanzee · virus | 30.1 | 26.6 |
| LLM · chimpanzee | 15.0 | 22.9 |
| LLM · virus | 15.7 | 19.1 |
| LLM · frog | 14.7 | 14.7 |
**The top two held and the middle did not.** `human · chimpanzee` and `human · LLM` moved 1.6 and
0.7 points across different corpora, and the gap between them went 19.6 → 17.3. Meanwhile
`human · frog` jumped 9.6 and overtook `chimpanzee · frog`, so that middle ordering was never real.
The three LLM pairs stayed the bottom cluster but reordered within it.
**Quote the top two and the floor. Do not quote the middle five.**
The direction gaps, which the grid shows and the ranking pools away:
| | n=10 | n=20 |
|---|---:|---:|
| `human ↔ LLM` | 27.3 | **42.2** |
| `human ↔ chimpanzee` | 10.2 | **4.3** |
A human reads at 85 in LLM sentences; an LLM reads at 43 in human ones. A human and a chimpanzee
stand in for each other about equally well in both directions. That pattern reproduced across both
runs and agrees with the local logprob arm — but it is a finding about the *measure*, not an answer
to the question, and the page treats it accordingly.
## Running it
```
npx typebulb --batch <name> typebulbs/u/lab/chimp-vs-chatbot.bulb.md
```
**Use `--batch` every time.** It re-points `tb.dir`, and the results files with it, at
`typebulbs/u/lab/chimp-vs-chatbot/batches/<name>/`. Runs are written as `matrix-<ISO timestamp>.json`
plus a `latest.json`.
**The page reads `latest.json` always and takes whichever is newer** — it or the baked `data.txt`
— and logs which it took. It used to consult disk only when nothing was baked, which let a stale
bake silently beat a fresh 505-call run and made the run look lost.
Check a run without a screenshot:
```
npx typebulb send typebulbs/u/lab/chimp-vs-chatbot.bulb.md selftest --wait
```
returns every pair's `score`, `n`, `xy`, `yx`, `asym` and the two saturation flags as JSON.
`send … read` / `send … make` switch views.
## Baking a run into the bulb
A published copy has no disk to read, so its Results view shows whatever `data.txt` holds.
Transfer the newest results file into that block with:
```
node typebulbs/u/lab/refresh-data.mjs typebulbs/u/lab/chimp-vs-chatbot.bulb.md
```
It picks the newest `.json` under the bulb's folder by mtime (searching batches recursively),
sanity-checks it, and `typebulb put`s it into `data.txt`. `--dry-run` reports what it would do.
Locally a stale bake is caught — the page takes the newer of the bake and `latest.json` and
logs which — but that guard cannot run on a published copy, so bake before publishing and
check the judge and date on the Method tab.
## Limits, stated rather than glossed
- **The zero is not meaningful** the way the local arm's was. Lifts are in nats and additive; a
0–100 rating is not. The ordering survives, the arithmetic does not.
- **The model is a black box** that can change without notice. Its id and the date are the only
provenance this arm has.
- **The corpus is model-written**, and a model grading text it wrote is the largest outstanding
threat to the design. The fix is sourcing sentences from real text.
- **Ceiling and floor**: a pair whose candidates both read as plainly sensible piles up near 100
and reads as a tie that was never measured. The page flags a comparison averaging above 90, and
below 10 for the same failure upside down. Resolving a saturated pair needs a forced choice
between the two sentences rather than a rating of each.
````