Pebble 1 30B, a model that knows what it knows

September 13, 2026 | Asa Shepard

Socrates is supposed to have said he was the wisest man in Athens only because he alone knew that he knew nothing. Pebble knows rather more than nothing, and the part we trained is the second half. Pebble 1 30B answers factual questions when it is confident and says it does not know when it is not. It is a fine-tune of Meta’s open Muse-Glimmer-30B, a 30-billion-parameter reasoning model, and the weights are open under the same Apache 2.0 license.

We measure it on AA-Omniscience, Artificial Analysis’s benchmark whose scoring rule is: a wrong answer costs (-1) as much as a right one earns (+1), and declining to answer costs nothing (0). The base model, Glimmer, scores -33. Pebble 1 30B scores +10 on the open subset of AA-Omniscience, and its hallucination rate falls from the base model’s 82% to just 6%.

On AbstentionBench, Meta's benchmark for knowing when not to answer, Pebble 1 abstains with 91% precision and still answers 96% of answerable questions, scoring above every other reasoning model in the published table.

We used supervised fine-tuning and preference optimization to teach Pebble 1 30B to answer when it knows and to decline when it does not, without shortening its reasoning. The benchmark’s public questions were held out for evaluation only.

Results

(AA-Omniscience , 600 public questions, graded with a replica of the official rubric. The official leaderboard uses a private set as well; the base model’s official score is shown for comparison.)

What happens to each question

Correct
Declined
Wrong

Every question ends one of three ways: a correct answer, an incorrect answer, or a decline to answer. The base model attempts almost everything and is wrong most of the time (hence its 82% hallucination rate). Pebble 1 30B declines most of the time and is wrong on approximately one question in sixteen.

Omniscience index

The index is scored as # correct minus # wrong, out of a hundred. A second grader, Gemini 3.6 Flash on the same rubric, scores Pebble 1 30B at plus 11.5. Official numbers will differ slightly, since the official grader and question set are not the ones used here.

Against the field

Pebble 1 30B, public set, replica grader
Muse-Glimmer-30B, official
Top 30 under 150B parameters on each metric, official leaderboard, 2026-09-12
Frontier reference, official

Index, higher is better

Hallucination rate, lower is better

The grey rows are the thirty best models under 150 billion parameters on each metric, from the official leaderboard: the official grader and the full question set of 6000. The hollow rows are today’s best frontier systems, for scale. Our two coloured rows are not official (pending measurement by Artificial Analysis). For those measurements, we used the 600 public questions under a replica grader, which lands within 1.3 points of the official score on the base model. Pebble 1 30B has not been scored officially yet, so read its position as an estimate with the interval drawn. On that estimate, it has the highest index of any model under 150 billion parameters while itself having only 30 billion parameters, and our 95% confidence interval clears the field. On hallucination, only one model under 150 billion is lower, a 1-billion-parameter non-reasoning model whose index is -1; the next lowest is double Pebble’s rate. Not every model and fine-tune is included in the leaderboard we draw from.

Pebble 1 30B is tuned and evaluated on English short-answer questions. Long-form, multi-step, and agentic behaviour inherit from the base model and were not specifically tuned.

Ask it something

Free, no account. Conversations stay in your browser.

Afterword

Pebble 1 30B is a proof that one property we care about can be trained into weights rather than enforced by a harness. We think models that go deep on a bounded domain beat models trained to be broad, and at the same time, going deep is reliant on reliability. On this benchmark, a frontier model answers incorrectly on 68.8% of the questions it does not get right, and that does not sit well with depth. For example, a professor who possesses great depth in one subfield could be expected to accurately answer about that subfield at least nine times in ten, and be wrong certainly far less often than 68.8% of the time.

Training abstention into the weights turns out to be a useful exercise, and it is not one we have seen done to this degree. Anecdotally, a hallucination rate near six percent changes how it feels to rely on the model, which still answers a meaningful share of its requests.

We’re exploring two primary options for what we’ll try next. First, the same calibration under a narrower target: a model taught to answer in one area, such as a particular unsolved cipher or genre thereof, an open problem in mathematics, or a corner of biology, with abstention trained alongside the area. Second, the same techniques we used to create Pebble 1 30B on a larger base. Muse-Glimmer-30B is weaker than other models, and the gain on it was large. On a base like GPT-6 Astra, Claude Fable 5.1, GLM-5.3, Kimi K3, or DeepSeek V4.1 Flash, the ceiling is much higher, since a model that never hallucinates can score no higher on the index than its base accuracy, without training in more knowledge. One relevant open question is what trained abstention does to other model capabilities, such as long-running tasks.

Acknowledgements

The idea that models carry usable knowledge of what they know goes back to Kadavath and colleagues at Anthropic in 2022, and to Lin, Hilton and Evans teaching models to say their uncertainty in words the same year. Refusal-aware tuning, teaching a model to say it does not know on the questions it would get wrong, was set out by Zhang and colleagues in R-Tuning and by Yang and colleagues in Alignment for Honesty in 2023. Kalai and colleagues argued in 2025 that hallucination persists because evaluation rewards guessing, which is the same argument AA-Omniscience makes with its scoring rule.