ProblemsConfidence scores that dont move with being right, so the threshold is decoration

Confidence scores that dont move with being right, so the threshold is decoration

Asked by kbriggs81·Jul 7, 2026

dataconfidencecalibrationintermediate

fieldfare is 240 extracted fields off 60 documents, four fields each, every one checked against what is actually printed on the page. The prompt asks for a value and a confidence 0 to 100 next to it, which is what everybody does and what i have been doing since March. Banded. The bands are mine, not the model's. >= 80: n=118, right on 84. 71.2% 60 to 79: n=58, right on 42. 72.4% 40 to 59: n=45, right on 31. 68.9% under 40: n=19, right on 9. 47.4% So the top band comes in below the band underneath it, and the two hundred odd items sat between 40 and 100 fit inside three and a half points of each other. The only band doing any work is the bottom one, and it is 19 items, and i am not putting a process on 19 items. We route on >= 60 at the moment. That gate passes 176 of 240 and the things it drops are no worse than the things it keeps, so it is a gate in name and in nothing else. I am not asking how to calibrate a number, i have read enough to know that is not happening inside a prompt. What i want is a field the caller can threshold on that is tied to something checkable, whatever that turns out to be. Rerunning fieldfare is about 4 pounds and forty minutes, so i can test things, within reason.

2 Prompt Submissions

New Submission

Executed Calls

—

Not run against a model yet

Works Rate

0 works · 1 fails

Total Copies

Times copied by users

Problem Instructions

A field next to each extracted value that a caller can threshold on and that moves with being right. The 0 to 100 confidence does not, so the >= 60 gate passes 176 of 240 and drops nothing worse than it keeps.

•On fieldfare the top band comes in above the band underneath it, which >= 80 currently does not.
•The spread between the best and worst band is wider than the 3.5 points the 221 items between 40 and 100 sit inside now.
•The field is derived off something on the page, so I can check it against the document rather than take the number on trust.

🏆 Best Current Solution

pat has the most upvoted solution, at 3.

0% Worked·0 Forks

Fork & Improve Compare Solutions

Prompt Submissions(2)

Sort by:Best Top Rated Most Improved Newest

delete the number and emit something the caller can check itself

PromptJul 8, 2026by pat

0% Works

You extract fields from a document. You do not report confidence. No number expressing certainty, likelihood, probability or reliability may appear anywhere in your output, on any scale, under any key name. For each requested field emit one record with these keys and no others. value The extracted value, exactly as printed. Null where there is none. quote A contiguous run of characters copied verbatim from the document, containing the value. Copied, not retyped, not tidied, not shortened with an ellipsis. Null where value is null. quote_contains_value Boolean. Your assertion that the string in value occurs inside the string in quote. The caller tests this independently. Do not assert true unless it is true as a plain substring test, with no allowance for formatting differences. support Exactly one token from this closed list: PRINTED_LABELLED the value is printed on the page next to a label naming this field PRINTED_UNLABELLED the value is printed on the page and you identified the field from position or context rather than a label HANDWRITTEN the value is written by hand DERIVED you calculated the value from other values on the page rather than reading it INFERRED you concluded the value from context without a printed value to read NOT_FOUND there is no value occurrence Exactly one token: SINGLE the value appears once on the page REPEATED it appears more than once with the same value CONFLICTING it appears more than once with different values NONE it does not appear conflicting_values Array of the differing values where occurrence is CONFLICTING. Empty otherwise. RULES DERIVED requires an operands array naming the values you calculated from, each with its own quote. INFERRED requires value non-null and quote null. It is the only support class permitted to carry a value with no quote, and it exists so the caller can drop those rows without guessing which ones they were. NOT_FOUND requires value null, quote null, occurrence NONE. A value you cannot quote and cannot justify as DERIVED is INFERRED. It is not PRINTED_UNLABELLED. Do not choose a support class by how sure you feel. The classes describe where the characters came from, which is a fact about the page rather than a fact about you. Do not soften CONFLICTING into SINGLE by picking the value that looks more correct. OUTPUT One JSON array, one object per requested field, in the order requested. No fence, no prose. [{"name":"...","value":"...","quote":"...","quote_contains_value":true,"support":"PRINTED_LABELLED","occurrence":"SINGLE","conflicting_values":[],"operands":[]}] The caller applies its own gate using: whether quote occurs in the source document as a substring, whether value occurs in quote as a substring, whether support is in the caller's permitted set, and whether occurrence is CONFLICTING. All four are decidable outside this prompt without a model.

Tested

–

Works

Fails

Model

Unspecified

View Details

Support token first in the object, fixed cut, no negotiation

PromptJul 20, 2026by rilaga

No reports yet

You extract fields from a document. Output is consumed by a client that may terminate the stream before the response is complete. Key order is therefore part of the contract and is not negotiable. For each requested field, the object is emitted with the keys in exactly this order: 1. n the field name 2. t the support token 3. v the value The support token is emitted before the value it applies to. This is required. A client that cuts mid-object must be able to act on every field that arrived complete, and a token arriving after its value is a token that does not arrive. SUPPORT TOKEN. Three levels. Closed set. DIRECT The value is printed on the page and you read it. A label names the field or the position makes the field unambiguous. No calculation, no inference, no reconstruction of missing characters. DERIVED You produced the value by calculation from other values on the page, by expanding an abbreviation, by normalising a format, or by reconstructing characters that were partly illegible. GUESS The value is not printed and you concluded it from context, or the printed value is illegible and you supplied the most likely reading, or the field is absent and you supplied what a document of this kind usually carries. Three levels only. Do not introduce a fourth. Do not qualify a token with a modifier. Do not emit a percentage, a score, a decimal, or a word such as high, medium or low anywhere in the output. A field with no value emits t as GUESS and v as null. Absence is not a fourth level. TOKEN DISCIPLINE The token is decided before the value is written and is not revised after writing it. If deciding the value changes which token applies, the higher uncertainty token wins: DIRECT becomes DERIVED, DERIVED becomes GUESS. The token never moves the other way inside a single response. Tokens are emitted in upper case exactly as written above. OUTPUT One JSON array. Fields in the order requested. No fence, no preamble, no closing text. [{"n":"policy_number","t":"DIRECT","v":"..."},{"n":"sum_insured","t":"DERIVED","v":"..."}] Emit fields in the order requested so that a truncated array is a prefix of the full one and the client knows which fields it did not receive.