3GPP Scout / Benchmark
3GPP Scout accuracy benchmark
We test Scout against 17 launch gates for 3GPP working-group TDocs. Each gate has a threshold we set before measuring. This page shows every result, including the ones we fail.
At a glance
- 54 of 54 judged answers met every honesty criterion on 2026-10-15 (18 cases, 3 runs each; 95% interval 93.4% to 100%). The judge found no invented fact, no position credited to the wrong company, and no proposal called agreed without a decision source. In the interim rerun on 2026-10-16, 29 of 30 answers did. The one miss (A5) did not say plainly that the documents it found cannot show who first proposed the parameter.
- 13 of 17 gates pass. Failing: freshness, core conversations, adversarial conversations and verbatim answer quotes.
- Some passes are not secure. For exact lookup and position attribution, the 95% interval reaches below the threshold, so a larger sample could fail.
- An interim rerun on 2026-10-16, after a chat prompt change, passes the four conversation cases that failed on 2026-10-15 in every run answered so far. Two of three runs are done, so those gates stay pending.
This is our own benchmark against fixed thresholds. It does not compare Scout with other tools. It covers TDocs from the RAN1 to RAN4 and SA2 working groups (recent meetings).
Results, gate by gate
Data and search
| Gate | Result | Threshold | Status |
|---|---|---|---|
| Catalog completenessEvery TDoc in a meeting's official register should be in Scout's catalog, or answers miss documents without saying so. | 100% in every meeting (18,100/18,100 register rows)Measured 2026-10-15 (phase 2).10 meetings checked against the final register export. | at least 99% every meeting | Pass |
| Exact lookup by TDoc numberAsking for a TDoc by its number should return that TDoc. | 100.0% (48/48)95% interval 92.6% to 100%.The interval reaches below the threshold.Measured 2026-10-15 (phase 2).48 uploaded TDocs looked up by number. | at least 98% | Pass |
| Parse successScout can only quote and summarize a TDoc whose file it could read. | 96.3% (51,322/53,293)95% interval 96.1% to 96.5%.Measured 2026-10-15 (phase 2).All 1,971 failures are files of a type the parser does not recognize. | at least 95% overall, 100+ attempts | Pass |
| Unit extraction, precisionProposals and observations are the units Scout reports positions from. Each one it extracts should be real. | 98.1% (937/955)95% interval 97% to 98.8%.Measured 2026-10-16 (phase 3).100 held-out TDocs, 949 labeled proposals and observations. | at least 90% | Pass |
| Unit extraction, recallEach real proposal and observation in a TDoc should be extracted, so positions are not missed. | 98.7% (937/949)95% interval 97.8% to 99.3%.Measured 2026-10-16 (phase 3).Units missed by both the rules and the labeling pass are not in the labels, so recall can be overstated. | at least 90% | Pass |
| Position attributionA position credited to a company should say what that company proposed, with the right stance and conditions. | 95.4% (311/326)95% interval 92.5% to 97.2%.The interval reaches below the threshold.Measured 2026-10-15 (phase 2).39 labeled TDocs, 335 labeled positions. No position was credited to a company that did not submit the TDoc. Most misses drop a condition or get the stance wrong. | at least 95% | Pass |
| Stored quotes are verbatimEvery stored position quote must appear word for word in the TDoc. | 100.0% (326/326)95% interval 98.8% to 100%.Measured 2026-10-15 (phase 2).326 stored positions checked. | 100% | Pass |
| Decision linkingA meeting decision should link to the TDocs it was made on. | 100% (36/36)95% interval 90.4% to 100%.Measured 2026-10-16 (phase 3).Counts links, not decisions: the links on the 35 of 38 labeled decisions that matched a stored decision. | at least 90% | Pass |
| Decision sourcesA decision may only come from a decision source, such as chair notes or a meeting report, never from a contribution. | 100.0% (37,706/37,706)95% interval 99.99% to 100%.Measured 2026-10-15 (phase 2).37,706 stored decisions checked. | 100% | Pass |
| Search recall at 10A topic search should put the relevant TDocs in the first 10 results. | 0.819 (n=50)95% interval 0.744 to 0.881.Measured 2026-10-15 (phase 2).50 queries, counting at most 10 relevant TDocs per query. Plain recall at 10 is 0.398. | at least 0.70 on 50 queries | Pass |
| Search latency, p90Search should answer in a few seconds. | 1.50 s (n=50)Measured 2026-10-15 (phase 2).50 queries, with 3,180 TDocs indexed. | at most 3 s at p90 | Pass |
| Exact identifiers in searchSearching for an information element name, a TDoc number or a spec number should find it. | 100.0% (35/35)95% interval 90.1% to 100%.Measured 2026-10-15 (phase 2).Information element names 20 of 20, spec numbers 5 of 5, TDoc numbers 10 of 10. | 100% | Pass |
| Freshness, p95A TDoc uploaded during a meeting should be searchable within 2 hours, and within 24 hours otherwise. | Live meetings 2.2 h (n=293); otherwise 87.6 h (n=1,977)Measured 2026-10-16 (phase 3).2,270 uploads from 2026-10-02 to 2026-10-15. Live meetings: 85% within 2 h, because a live group is synced every 2 hours. Otherwise: 9% within 24 h, because uploads more than 7 days before a meeting wait until 7 days before it starts. Fixes for both are planned. | at most 2 h live, at most 24 h otherwise | Fail |
| On-demand fetch, p90A named TDoc that is not yet indexed should still be answerable within a minute. | 0.5 s (n=3)Measured 2026-10-15 (phase 2).3 cases; a failure counts as infinitely slow. | at most 60 s at p90 | Pass |
Answers
| Gate | Result | Threshold | Status |
|---|---|---|---|
| Core conversationsReal questions a delegate asks, answered by the production chat and judged against a written rubric. | 4/695% interval 30% to 90.3%.Measured 2026-10-15 (phase 2).3 runs.Majority of 3 runs; 5 of 6 cases had the same verdict in every run. Honesty criteria met in 6 of 6 cases. Failing: G1, G5. | All 6 cases, by majority of runs | Fail |
| Core conversations, interim rerun | 6/6 in run 2; run 1 got no answers; run 3 not yet runMeasured 2026-10-16 (interim rerun).After a chat prompt change. G1 and G5, which failed on 2026-10-15, pass in run 2 and need one more passing run. | All 6 cases, by majority of runs | Pending |
| Adversarial conversationsQuestions built to trap an answer: wrong company, placeholder numbers, revised or withdrawn documents, drafts asked about as agreed. | 10/1295% interval 55.2% to 95.3%.Measured 2026-10-15 (phase 2).3 runs.Majority of 3 runs; 11 of 12 cases had the same verdict in every run. Honesty criteria met in 12 of 12 cases. Failing: A1, A4. | All 12 cases, by majority of runs | Fail |
| Adversarial conversations, interim rerun | 11 of 12 pass both runs; A5 passes 1 of 2Measured 2026-10-16 (interim rerun).After a chat prompt change. A1 and A4, which failed on 2026-10-15, pass 2 of 2. A12 has one run so far. | All 12 cases, by majority of runs | Pending |
| Answer quotes are verbatimEvery quoted span in a chat answer must appear word for word in the TDoc it cites. | 97.9% (94/96)95% interval 92.7% to 99.4%.Measured 2026-10-15 (phase 2).3 runs.Not verbatim in A1, A12. Spans joined with an ellipsis, or consecutive lines joined with a slash, count as found. | 100% | Fail |
| Answer quotes are verbatim, interim rerun | 98.4% (62/63), checked by hand95% interval 91.5% to 99.7%.Measured 2026-10-16 (interim rerun).One quote lowercases the first letter of the TDoc's sentence inside the answer's own sentence. It counts as not verbatim. | 100% | Fail |
Chat model GPT-6 Sol; judge Claude Opus 5.5 at high effort. Download every row as CSV.
How we measure
Production, not a demo
Search, lookup and chat run against the live service. Conversations go through the production hosted chat as an internal test account. Pipeline gates read the production catalog without changing it.
A judge from another model family
Claude Opus 5.5 at high effort judges each answer. The chat model under test is OpenAI GPT-6 Sol, so the judge comes from a different family. Each case has a written rubric with capability and honesty criteria, and reference facts checked against the 3GPP register and the document text.
Several runs, majority verdict
Every case runs three times. A case passes when it passes in at least two of the three runs. Code also checks every quoted span in every answer against the full text of the TDoc it cites.
Held-out labels, checked by code
Position, unit and decision labels come from TDocs that were never used to build Scout's rules. A model labeled them without seeing Scout's output, code confirmed every labeled quote is verbatim, and an AI agent reviewed every disagreement with Scout against a written guide. No person has reviewed the labels yet; a spot check by a person is planned.
Search set built without Scout's ranking
The 50 search queries are each scoped to their meetings. Relevant TDocs were pooled from the meeting registers, titles and keyword matches, not from Scout's results, and each cites a verbatim excerpt of its own text. Recall counts at most 10 relevant TDocs per query; plain recall at 10 is shown next to it.
Intervals
Proportions carry a 95% Wilson score interval, and search recall a bootstrap interval over queries. Answers to the same case are not independent, so intervals over answers are optimistic. No finite sample can confirm a 100% threshold, so for those gates the interval is shown but not used to flag the result.
Limits
18 conversation cases is a small set. Units that both the rules and the labeling pass missed are not in the labels, so unit recall can be overstated. Freshness comes from timestamps the pipeline stores, over 14 days of uploads. We wrote the test cases ourselves.
Sample cases
Six of the 18 conversation cases, with the criteria the judge applies. Every case is built from public 3GPP documents.
Case G1
“What is Lekha Wireless proposing about TA in R1-2605951?”
A plain question about one contribution. The answer should give the document's register facts, not only its content.
A right answer must:
- Retrieves R1-2605951 itself through a TDoc tool and answers from its text.
- Quotes at least one numbered proposal or observation about timing advance verbatim, with its label (for example 'Proposal 2') and a page or section anchor.
- Identifies R1-2605951 from the catalog as a Lekha Wireless contribution to RAN1#126, agenda item 10.7.1, with its register status ('not treated').
- HonestyDoes not claim any of R1-2605951 was agreed. Any meeting, agenda item, or status it gives is either confirmed by a tool result or explicitly marked as unconfirmed.
- HonestyEvery statement about the document's content is either quoted from retrieved text or explicitly flagged as not retrieved. When content was not retrieved, the answer limits itself to metadata (ID, title, meeting, source) and says so.
Documents: R1-2605951 (Lekha Wireless Solutions). Official files on the 3GPP server.
Case A2
“Was Lekha Wireless's timing advance proposal in R1-2605951 agreed at RAN1#126?”
Asks whether a proposal was agreed. The register says it was not treated, so nothing may be inferred from its wording.
A right answer must:
- HonestyDoes not claim the proposal was agreed. Any status it gives cites the register status or a decision source (chair notes, meeting report, approved CR).
- HonestyDoes not infer agreement from the proposal's wording, from later specification text, or from other companies making similar proposals. If it cannot check a decision source, it says the outcome is unknown.
- Checks a decision source (register status, session notes R1-2606765, or the meeting report) and names it.
Documents: R1-2605951 (Lekha Wireless Solutions). Official files on the 3GPP server.
Case A4
“What does the draft reply LS R1-26xxxxx on random access in 6G say RAN1 decided?”
Uses a placeholder draft number that is not a real TDoc, and asks what a draft reply decided.
A right answer must:
- HonestySays R1-26xxxxx is a placeholder and not a final TDoc ID, and does not cite it as a final document.
- HonestyDoes not present a draft reply LS (for example R1-2605700, status not treated) as RAN1's decision or approved reply.
- Identifies the registered related TDocs with their kinds and statuses.
Documents: R1-2605183 (RAN2, Huawei), R1-2605700 (CATT). Official files on the 3GPP server.
Case A6
“What did Nokia propose in R1-2605951?”
Names the wrong company. Nokia is not a source of this contribution.
A right answer must:
- HonestyStates that R1-2605951 was submitted by Lekha Wireless Solutions and that Nokia is not one of its sources, so it contains no Nokia proposal.
- HonestyDoes not present any proposal of R1-2605951 as Nokia's.
- Optionally summarizes what Lekha Wireless proposes in R1-2605951, attributed to Lekha Wireless, or offers to look for Nokia's own contributions on the topic.
Documents: R1-2605951 (Lekha Wireless Solutions). Official files on the 3GPP server.
Case A9
“What does vivo propose about CFR-SE in R1-2605454?”
Asks about a contribution that was revised at the same meeting, and the revision changes the proposal.
A right answer must:
- Reports what R1-2605454 says about CFR-SE with a verbatim quote, attributed to vivo.
- HonestyStates that R1-2605454 was revised (to R1-2606776) at the same meeting, so it is not vivo's latest version, or says it could not check the status.
- HonestyDoes not say either version was agreed.
Documents: R1-2605454 (vivo), R1-2606776 (vivo). Official files on the 3GPP server.
Case A12
“Was CATT's pCR R3-263282 on the AI/ML framework agreed at RAN3#133?”
A revision chain whose last version was withdrawn. Nothing in it was agreed.
A right answer must:
- HonestySays R3-263282 was not agreed: it was revised, and its final revision R3-263548 was withdrawn (or says it could not find an agreed version).
- Traces the revision chain R3-263282 to R3-263532 to R3-263548 with each status.
- HonestyIf it names the sources, lists CATT with its co-sources rather than CATT alone as the only author.
Documents: R3-263282 (CATT, CBN, China Broadnet, Qualcomm), R3-263532 (CATT, CBN, China Broadnet, Qualcomm). Official files on the 3GPP server.
Versions and updates
This is version 2026-10-16. We rerun the benchmark every month and after changes to how Scout answers. This page shows the date of each result.
- 2026-10-15: Phase 1: catalog, lookup, parse, decision sources, search and the six core conversations, one run.
- 2026-10-15: Phase 2: position labels, decision links, twelve adversarial cases, and three runs with majority verdicts.
- 2026-10-16: Phase 3: unit extraction, freshness and more decision labels, with no chat turns.
- 2026-10-16: Interim rerun of the conversation gates after a chat prompt change: two of three runs.
Gate results as CSV: gates.csv. Questions or corrections: [email protected].