Matchline

The problem

I started Matchline with two premises about the 2026 job market. If anyone can generate a tailored résumé and cover letter in seconds, polish stops telling a hiring manager much. Specificity that a candidate can defend might still tell them something. A generative writer doesn’t know which of my projects mattered, which numbers are honest, or which claims I can defend in an interview.

An application doesn’t need a lie to be empty. Matchline’s validator refuses twelve phrases, including collaborated cross-functionally, drove results, and moved the needle. I can trace those back to something I did somewhere. They can be true and still tell a reader nothing.

A typical AI career tool takes a résumé and a job description and generates prose. I wanted to build the evidence first: a structured graph of capabilities, outcomes, and evidence that I could keep refining. AI would select from it for a particular opportunity. Its job was selection, sequencing, and framing. A claim that didn’t trace to something I’d documented and approved was a defect to catch.

This was a hypothesis about my own job search. I did no external user research or market validation. I kept V1 to finding out whether the workflow worked for me before asking whether it worked for anyone else.

• 1 userme; I deliberately kept V1 to my own job search
• $25/momy Anthropic cap, enough for about twelve tuning runs
• 10 × 12résumés by job descriptions in the evaluation corpus on 2026-10-06

What got built

Capability	State
Résumé into Experience Units	Built
Approved evidence only, into matching	Built
Job description into requirements, with gaps surfaced	Built
Fabrication gate blocking export	Built, never user-validated; the export it gates is a Phase 2 stub
Latency and cost targets	Measured once (2026-04-27): cost met, latency 11.8× over

LinkedIn and long-form ingestion were deferred. Uploaded PRDs, decks, and retrospectives never existed—an earlier version of this page listed them as inputs, incorrectly. They were specified, but never built. I’m keeping the correction here because the earlier page claimed more than the product did.

The evidence model

An Experience Unit is one checkable claim about work. I gave it a strict schema, so the model can’t return a Unit without the evidence the contract requires:

export const ExtractedUnitV1Schema = z
  .object({
    raw_text: z.string().trim().min(1),
    normalized_summary: z.string().trim().min(1),
    unit_type: UnitTypeSchema,

    skills: z.array(z.string().trim().min(1)),
    tools: z.array(z.string().trim().min(1)),
    domains: z.array(z.string().trim().min(1)),
    seniority_signals: z.array(z.string().trim().min(1)),
    scope_signals: z.array(z.string().trim().min(1)),
    business_outcomes: z.array(z.string().trim().min(1)),
    metrics: z.array(MetricSchema),

    evidence_type: EvidenceTypeSchema,
    // Prompt says "below 0.50 should not be emitted" — enforce at
    // schema level too so a hallucination flood is rejected.
    confidence_score: z.number().min(0.5).max(1),

    date_range: DateRangeSchema.optional(),
  })
  .strict();

This one is mine. I labeled it from my own résumé on April 26, 2026, and the exhibit quotes the fixture. The same claim is on my published résumé:

Claim

Introduced AI-powered PR review reducing median PR cycle time from 7.4 to 4 days, beating the 30% Q2 OKR target

Skills

engineering productivity · AI tooling adoption · OKR ownership · process change

Domains

developer tools · AI-augmented engineering

Tools

GitHub · AI code review

“Improved developer productivity” wouldn’t give the matcher much to work with. This Unit names the change, the before and after, and the target it beat. It separates those details so the matcher can ask whether this claim answers a particular requirement.

The fixture carries four fields. The scorer compares two: the summary and the skills. A production Unit also carries metrics, seniority and scope signals, an evidence type, and a confidence score. The model produces them; the server validates them against the contract above.

I kept approval out of the extractor’s output. It can’t return that field, and the server stamps every extracted Unit unapproved. Anything the model proposes has to pass review before matching can use it. The enforcing filter runs server-side wherever evidence is consumed, four places as of 2026-10-06.

I can also type a Unit myself. Those arrive pre-approved and marked user_confirmed, because I’ve already made the assertion. Another review would add nothing.

A pasted resume is extracted into Experience Units, each of which is written to the capability graph already, but unapproved. Human review flips that approval flag in place. The gate is not on the way into the graph but on the way out of it: a server-side filter passes only approved Units into matching, so an unapproved Unit sits in the store and is never read. Units the user types by hand arrive pre-approved instead. Separately, a job description is parsed into requirements. The matching engine scores approved Units against those requirements and surfaces requirements with no qualifying evidence as gaps rather than hiding them. Generation draws only on approved matches, and a traceability check sits between the generated draft and export: a claim that cannot be traced back to an approved Unit is stopped at the gate until the user resolves it.

The two orange nodes are the whole product. The first sits after the graph: unapproved Units are stored immediately, then filtered out when something reads the evidence. The audit calls it a read gate because it doesn’t stop a Unit from being written. Those two checks carry the thesis. The rest is plumbing a competent team would build the same way.

I put requirements with no qualifying evidence in a gaps panel above the match list, inside the same tab. I wanted to see the gaps when I looked at the matches. They were the point, not an embarrassment to bury.

The decisions

1. TrustPending

   Make zero fabrication a gate, not a warning

   What I encountered

   An application I send is something I may have to defend out loud, in a room, weeks later. A good sentence about something that never happened can sound confident right up to the follow-up question.

   Over

   A confidence score and a warning banner for risky claims. That's the standard pattern, and it still lets someone ship when a deadline matters more than the warning.

   What I decided

   I required every claim in a draft to trace back to its Experience Unit and disabled export while any claim failed. To clear it, I have to edit the claim, remove it, or supply a Unit that supports it.

   Why

   I didn't want to leave that judgment to someone in a hurry, reading prose that sounds like their own. A block needs a disqualifying reason. An unprovable claim in a job application qualifies.

   Cost

   The tool gets in my way most when I'm in a hurry. It also blocks a true accomplishment if I haven't documented it. I have to write the evidence down before I can use it, so an honest missing record and a careless invented claim pay the same tax.

   Validation boundary

   The gate exists in code, with a fixture designed to break it. That test proves the orchestration blocks a known fabricated claim. The language-model checks are mocked, so it doesn't measure whether the detector reliably catches a new one. The product records no use of a generated output by a user. The export behind the gate remains a stub.

2. ScopePending

   Build for one user and refuse to generalize

   What I encountered

   I could specify outcome-driven tuning, a referral graph, and a should-I-apply score on day one. All those V2 layers were more interesting to build than the four steps that had to work first.

   Over

   Building the multi-user foundation first, on the usual argument that tenancy is cheap to add up front and expensive to retrofit.

   What I decided

   I kept V1 to one user and one job search. The spec explicitly put multi-user, sharing, teams, job-board integrations, and every learning layer out of scope. I didn't leave them waiting in a backlog.

   Why

   I needed to find out whether evidence-grounded generation could produce an application I'd send. One user could answer that question. A thousand users wouldn't answer it any better. Whether the architecture could hold them was a different question.

   Cost

   I'm the only customer and the author. The evaluation corpus comes from my prospect list. Every signal is one I generated myself, and I can't separate a product that works from one its builder is willing to tolerate.

3. MethodMixed

   Fix the ruler before trusting the measurement

   What I encountered

   I was tuning the matching layer against match accuracy. One run reported 18.1%, with three samples of a single fixture ranging from 12.5% to 25.0%. That spread was wider than the improvement I was trying to measure.

   Over

   Continuing to tune against the noisy metric and averaging more samples. It would cost less engineering time and was affordable within the cap.

   What I decided

   I stopped tuning the matcher and repaired the measurement. I replaced the absolute 0.30 similarity threshold with relative best-match mapping and a 0.10 sanity floor. Later, I built a content-addressed stage cache so unchanged stages could replay for free.

   Why

   When a metric's run-to-run noise is larger than the effect I'm measuring, I can't tell whether a change helped. Averaging costs another run. At three samples, a run cost $2.06 against my $25 monthly cap, enough for about twelve tuning runs a month.

   Cost

   My final session changed no user-facing feature. I paused with better instruments and the four-pair baseline unchanged. The one run after the mapping repair showed less noise and a mean near 19%. The work someone would have seen in a demo was the work I didn't do.

   What it changed

   Both repairs landed, but I measured only the first. In its own pull request, the mapping fix ran once on one fixture at three samples. The range fell from 12.5 points to 4.1, and the mean was 19.4%. Eight weeks of product work separated that repair from the cache, which landed in the final session at the end of July. No run under the repaired mapping touched the four-pair corpus, and none followed the cache. I stopped with a better-designed instrument and one fixture's evidence that it was better.

4. PrioritizationMixed

   A deadline someone else depends on wins

   What I encountered

   In early July, Matchline and Five Across wanted the same summer. Matchline had one user, no launch date, and nobody waiting. Five Across was a live multiplayer bingo game for a nine-night cruise. The sailing date wasn't going to move, and the players would either have the game or not.

   Over

   Pushing Matchline to a usable V1 first, on the argument that a tool this close to working shouldn't stop mid-flight.

   What I decided

   I stopped Matchline to ship the game. I spent the one remaining Matchline session on measurement infrastructure instead of features.

   Why

   A date someone else set is not negotiable and a date I set is. That's my reasoning. The commits corroborate the sequence, not its cause: one project stopped and another started. Their timestamps can't tell anyone why I chose it.

   Cost

   The instruments were finally cheap, but accuracy wasn't there. I never made the four-pair run under the repaired mapping that the cache was built to make cheap. That was the next step, though it wouldn't alone establish that matching works. I still needed a finished corpus and a real application end to end.

   What it changed

   Product work stopped the day before the Five Across repository was created, apart from one final session on the evaluation cache. The cache closed that phase. The next product commit landed a month later.

The bar before the ruler

I set the bar before there was anything to measure. The repository’s first commit carried it: 80% extraction accuracy and 80% match accuracy. I hadn’t yet built the ruler I would read it with. Matchline reached neither bar. How the numbers moved matters more than where they stopped.

The first scorer compared generated summaries with hand-labeled ones using string equality. The model paraphrased, and the score was 3.3%. Token overlap brought it to 34%. A coverage coefficient gave full credit when a longer generated set covered the labeled set, bringing it to 50.3%. I had changed the ruler twice.

The next change was to the product: expanding the matching vocabulary from roughly 220 entries to 415. That added a single point, reaching 51.3%.

Extraction accuracy rose from 3.3 percent to 51.3 percent across four measurements, against an 80 percent target that was never reached. Three of those four measurements came from different definitions of the scoring metric rather than from any change to the product: string equality scored 3.3 percent, token overlap 34 percent, and a coverage coefficient 50.3 percent. Only the last step was a change to the system itself, an expansion of the matching vocabulary, and it added a single point to reach 51.3 percent.

All four measurements came from one résumé-and-job pair. Later the same day, a run across all four labeled pairs, three samples each, scored 48.4% extraction and 19.1% match. That aggregate remains the baseline of record.

The vocabulary work also moved match accuracy from 4.2% to 16.7%. After that, I couldn’t read the signal. One run reported 18.1%, with three samples of that one fixture ranging from 12.5% to 25.0%. Eight to thirteen points of noise was more than the improvement I was trying to measure.

I could average more runs, but a single run cost $2.06 against my $25 monthly cap. That bought about twelve runs a month.

I spent the last working session on the instrument rather than the product.

The artifact

A validator that says “looks fine” can score well if every input is honest. I needed a case that should fail. The repository has a hand-labeled fabricated paragraph attached to my own résumé, the one the Unit above comes from. It claims a multi-year x86 and Arm CPU roadmap, a hundred-million-VM fleet, a 40% throughput gain on SAP HANA, and $400M in contracts displacing a competitor.

Both claims carry a number. The 7.4-to-4-day cycle time traces to the Unit above. I’ve never worked on SAP HANA, so a 40% throughput gain on it is a claim I couldn’t defend, however plausible the sentence sounds. None of those four claims traces to an Experience Unit. The fixture lists all four, explains why each fails, and expects validation to fail.

That test runs in continuous integration against the Firestore emulator. Passing means the gate blocked the Export control, whose handler is still a stub. It proves the orchestration refuses a known untraceable claim. The model checks are mocked, so it doesn’t establish that the model catches a new one. The integration reaches the database, not the model API.

How it was built

Agents wrote the implementation, and independent agent identities reviewed it under the enforcement system documented in Mergepath. They caught three defects that materially changed the product:

• A seniority scorer zeroed every Experience Unit because extraction emitted verbs and the ladder matched nouns.
• A schema converter silently returned an empty object, disabling model-input validation across four subsystems.
• A matching rerun discarded the user’s rejections.

On that last one, my first reply to the reviewer was wrong. I cited a test that didn’t cover the case. The reviewer pushed back a second time before it was fixed.

The eval stage cache landed on the evening of 2026-07-31, Pacific. That dates the pause. For the month after it, maintenance continued: dependency bumps, repository-template syncs, and identity and CI tooling. None touched the product.

Product commits resumed on 2026-08-31 and, as of 2026-09-01, were still landing. The fixes below belong to that later work. Paused means I hadn’t launched or decided to restart; it doesn’t mean the repository stopped receiving product fixes.

A development build is deployed behind a sign-in wall, and the “View Live Product” button above goes to it. It was redeployed on the evening of 2026-08-31, replacing the 2026-05-02 build it had served until then, and again on 2026-10-02. The served build checked for this page carries the June and July work and the fixes that followed.

Since 2026-09-30, the source rules and callable gates restrict reads and calls to my owner allowlist. The audit read the anonymous sign-in shell and its bundle; it didn’t verify the deployed rules or try another account. The link shows that shell, not the corpus.

What would restart it

I hadn’t written restart or kill criteria while I was building. The repository had exit criteria for finishing V1, but none for deciding whether to continue. I had criteria for building, not for deciding.

Resume it when:

• I’ve finished and labeled the evaluation corpus.
• I can trust the measurement enough to distinguish improvement from noise.
• I’ve taken one real application through the product end to end. The second build phase required it, and I never met that condition.

Kill it when:

• Matching is still fundamentally poor after a full tuning cycle on the cheap evaluation infrastructure. Then I’d be treating the wrong approach as something that just needs more tuning.
• I finish a job search without reaching for it. The success metric was making Matchline my primary tool, instead of another system I abandon for a spreadsheet. If I don’t use it under real pressure, that answers the question.

Where it stands

As of September 2026, my job search had resumed. Matchline finally had the situation it was built to test.

Writing this page also exposed three stacked production failures:

• missing Cloud Run invoker bindings blocked CORS preflights;
• trailing newlines in both provider secrets broke authentication;
• extraction had a sixty-second timeout for three long model calls.

Once those were fixed, résumé extraction and JD parsing ran end to end. That exposed the next problem: the matching ontology under-recognizes out-of-domain requirements. Until a fix on 2026-09-01, jaccard() zeroed 45% of the score before fit was even considered.

The eval harness can now route tuning runs through a subscription CLI. Those runs don’t draw on the metered Anthropic cap that had limited me to roughly twelve runs a month.

The project remains paused. The infrastructure works, experiments are cheap, and my job search supplies the test. I still need to decide to restart against the criteria above. Continuing to make commits isn’t that decision.

What I learned

Getting the model to produce something plausible wasn’t the hardest part. I had to decide what evidence would make its output trustworthy, then build a measurement system I could trust to tell me whether the product was improving.

I got further with the evidence than with the measurement. Matchline can structure evidence, ground claims, and stop an untraceable claim at the gate. Matching stayed well below the quality bar.

Setting the bar first was right: 19% against 80% can’t be argued into a pass. Reading results before I’d tested the ruler was the mistake. On extraction, two changes to the scorer moved the number 47 points; the one change to the product moved it one. Next time the ruler comes first: a labeled corpus, a scorer checked against my own judgment, and run-to-run noise smaller than the change I’m looking for. This time I never reran the four-pair baseline under the repaired mapping, so I still can’t say how much of the gap to 80% was the product and how much was the instrument.

That’s why I paused before shipping. A product that exists is not necessarily a product that works, and a metric that moves is not necessarily evidence that anything got better.