Scaling the test-time compute of large language models has demonstrated impressive performance on reasoning benchmarks. However, existing evaluations of test-time scaling make the strong assumption that a reasoning system should always give an answer to any question provided. This overlooks concerns about whether a model is confident in its answer, and whether it is appropriate to always provide a response. To address these concerns, we extract confidence scores during reasoning for thresholding model responses. We find that increasing compute budget at inference time not only helps models answer more questions correctly, but also increases confidence in correct responses. We then extend the current paradigm of zero-risk responses during evaluation by considering settings with non-zero levels of response risk, and suggest a recipe for reporting evaluations under these settings.
\n","updatedAt":"2025-02-20T19:01:29.317Z","author":{"_id":"6372bc95c4267fd7cd77f4d0","avatarUrl":"/avatars/17a24af68f45487e601687d777b352b6.svg","fullname":"William Jurayj","name":"wjurayj","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":6,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9453322887420654},"editors":["wjurayj"],"editorAvatarUrls":["/avatars/17a24af68f45487e601687d777b352b6.svg"],"reactions":[],"isReport":false}},{"id":"67b7d83a6c3182e96bbda789","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2025-02-21T01:34:50.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning](https://huggingface.co/papers/2502.07154) (2025)\n* [Multiple Choice Questions: Reasoning Makes Large Language Models (LLMs) More Self-Confident Even When They Are Wrong](https://huggingface.co/papers/2501.09775) (2025)\n* [Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach](https://huggingface.co/papers/2502.05171) (2025)\n* [Commonsense Video Question Answering through Video-Grounded Entailment Tree Reasoning](https://huggingface.co/papers/2501.05069) (2025)\n* [ARR: Question Answering with Large Language Models via Analyzing, Retrieving, and Reasoning](https://huggingface.co/papers/2502.04689) (2025)\n* [Benchmarking Large Language Models via Random Variables](https://huggingface.co/papers/2501.11790) (2025)\n* [Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling](https://huggingface.co/papers/2502.06703) (2025)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\nThe following papers were recommended by the Semantic Scholar API
\n- \n
- Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning (2025) \n
- Multiple Choice Questions: Reasoning Makes Large Language Models (LLMs) More Self-Confident Even When They Are Wrong (2025) \n
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach (2025) \n
- Commonsense Video Question Answering through Video-Grounded Entailment Tree Reasoning (2025) \n
- ARR: Question Answering with Large Language Models via Analyzing, Retrieving, and Reasoning (2025) \n
- Benchmarking Large Language Models via Random Variables (2025) \n
- Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling (2025) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
Is That Your Final Answer? Test-Time Scaling Improves Selective Question Answering
Abstract
Evaluating large language models includes confidence thresholding to manage response risk at test time, showing that increased compute improves both accuracy and confidence.
Scaling the test-time compute of large language models has demonstrated impressive performance on reasoning benchmarks. However, existing evaluations of test-time scaling make the strong assumption that a reasoning system should always give an answer to any question provided. This overlooks concerns about whether a model is confident in its answer, and whether it is appropriate to always provide a response. To address these concerns, we extract confidence scores during reasoning for thresholding model responses. We find that increasing compute budget at inference time not only helps models answer more questions correctly, but also increases confidence in correct responses. We then extend the current paradigm of zero-risk responses during evaluation by considering settings with non-zero levels of response risk, and suggest a recipe for reporting evaluations under these settings.
Community
Scaling the test-time compute of large language models has demonstrated impressive performance on reasoning benchmarks. However, existing evaluations of test-time scaling make the strong assumption that a reasoning system should always give an answer to any question provided. This overlooks concerns about whether a model is confident in its answer, and whether it is appropriate to always provide a response. To address these concerns, we extract confidence scores during reasoning for thresholding model responses. We find that increasing compute budget at inference time not only helps models answer more questions correctly, but also increases confidence in correct responses. We then extend the current paradigm of zero-risk responses during evaluation by considering settings with non-zero levels of response risk, and suggest a recipe for reporting evaluations under these settings.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning (2025)
- Multiple Choice Questions: Reasoning Makes Large Language Models (LLMs) More Self-Confident Even When They Are Wrong (2025)
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach (2025)
- Commonsense Video Question Answering through Video-Grounded Entailment Tree Reasoning (2025)
- ARR: Question Answering with Large Language Models via Analyzing, Retrieving, and Reasoning (2025)
- Benchmarking Large Language Models via Random Variables (2025)
- Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper