Congrats on the feature release \n\n@sumuks\n\t
\nBut data generation and judges for task-specific skills assessment have been around for at least a year, people are struggling to choose the best foundation model for their application.
\nThat's because offline evaluation techniques like this are unreliable predictors for how users will respond to changes in the app. It's a nuanced engineering decision considering factors such as latency or other real-world constraints to deploying the model.
\nIn fact, most \"great ideas\" for software changes across industry fail to meaningfully impact business metrics and user engagement when measured with online evaluation methods like A/B testing.
\nTools like this may help in the ramp up to the real evaluation, but only if you're applying the scientific method to identify these predictors, closing the loop in your AI development and evaluation.
\nWrote more on trustworthy AI experiments here: https://www.remyx.ai/blog/trustworthy-ai-experiments
\n","updatedAt":"2025-04-03T14:59:38.254Z","author":{"_id":"647777304ae93470ffc28913","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/BywZYAPzsyBSCf8yJFiUf.jpeg","fullname":"Salma Mayorquin","name":"salma-remyx","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":102,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9215332269668579},"editors":["salma-remyx"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/BywZYAPzsyBSCf8yJFiUf.jpeg"],"reactions":[],"isReport":false,"parentCommentId":"67ede02afd915b0739de843c"}}]},{"id":"67ef38151de277aadd52817b","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2025-04-04T01:38:29.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [WritingBench: A Comprehensive Benchmark for Generative Writing](https://huggingface.co/papers/2503.05244) (2025)\n* [Dynamic-KGQA: A Scalable Framework for Generating Adaptive Question Answering Datasets](https://huggingface.co/papers/2503.05049) (2025)\n* [Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Models](https://huggingface.co/papers/2504.01001) (2025)\n* [Investigating the Impact of Quantization Methods on the Safety and Reliability of Large Language Models](https://huggingface.co/papers/2502.15799) (2025)\n* [LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs - No Silver Bullet for LC or RAG Routing](https://huggingface.co/papers/2502.09977) (2025)\n* [Adaptively evaluating models with task elicitation](https://huggingface.co/papers/2503.01986) (2025)\n* [TLUE: A Tibetan Language Understanding Evaluation Benchmark](https://huggingface.co/papers/2503.12051) (2025)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\nThe following papers were recommended by the Semantic Scholar API
\n- \n
- WritingBench: A Comprehensive Benchmark for Generative Writing (2025) \n
- Dynamic-KGQA: A Scalable Framework for Generating Adaptive Question Answering Datasets (2025) \n
- Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Models (2025) \n
- Investigating the Impact of Quantization Methods on the Safety and Reliability of Large Language Models (2025) \n
- LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs - No Silver Bullet for LC or RAG Routing (2025) \n
- Adaptively evaluating models with task elicitation (2025) \n
- TLUE: A Tibetan Language Understanding Evaluation Benchmark (2025) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
YourBench: Easy Custom Evaluation Sets for Everyone
Abstract
YourBench introduces an automated framework for generating domain-specific LLM benchmarks dynamically and cost-effectively from user documents, ensuring reliable and up-to-date evaluations.
Evaluating large language models (LLMs) effectively remains a critical bottleneck, as traditional static benchmarks suffer from saturation and contamination, while human evaluations are costly and slow. This hinders timely or domain-specific assessment, crucial for real-world applications. We introduce YourBench, a novel, open-source framework that addresses these limitations by enabling dynamic, automated generation of reliable, up-to-date, and domain-tailored benchmarks cheaply and without manual annotation, directly from user-provided documents. We demonstrate its efficacy by replicating 7 diverse MMLU subsets using minimal source text, achieving this for under 15 USD in total inference costs while perfectly preserving the relative model performance rankings (Spearman Rho = 1) observed on the original benchmark. To ensure that YourBench generates data grounded in provided input instead of relying on posterior parametric knowledge in models, we also introduce Tempora-0325, a novel dataset of over 7K diverse documents, published exclusively after March 2025. Our comprehensive analysis spans 26 SoTA models from 7 major families across varying scales (3-671B parameters) to validate the quality of generated evaluations through rigorous algorithmic checks (e.g., citation grounding) and human assessments. We release the YourBench library, the Tempora-0325 dataset, 150k+ question answer pairs based on Tempora and all evaluation and inference traces to facilitate reproducible research and empower the community to generate bespoke benchmarks on demand, fostering more relevant and trustworthy LLM evaluation.
Community
we're launching 🤗 yourbench today, an open source tool for custom benchmarking and synthetic data generation from ANY of your documents. it's a big step towards improving how model evaluations work
most benchmarks test general capabilities, but we know that for many use cases what really matters is how well a model performs your specific task. yourbench lets you evaluate models on what matters to you.
you can try it with your own docs today!: https://huggingface.co/spaces/yourbench/demo
Congrats on the feature release @sumuks
But data generation and judges for task-specific skills assessment have been around for at least a year, people are struggling to choose the best foundation model for their application.
That's because offline evaluation techniques like this are unreliable predictors for how users will respond to changes in the app. It's a nuanced engineering decision considering factors such as latency or other real-world constraints to deploying the model.
In fact, most "great ideas" for software changes across industry fail to meaningfully impact business metrics and user engagement when measured with online evaluation methods like A/B testing.
Tools like this may help in the ramp up to the real evaluation, but only if you're applying the scientific method to identify these predictors, closing the loop in your AI development and evaluation.
Wrote more on trustworthy AI experiments here: https://www.remyx.ai/blog/trustworthy-ai-experiments
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- WritingBench: A Comprehensive Benchmark for Generative Writing (2025)
- Dynamic-KGQA: A Scalable Framework for Generating Adaptive Question Answering Datasets (2025)
- Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Models (2025)
- Investigating the Impact of Quantization Methods on the Safety and Reliability of Large Language Models (2025)
- LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs - No Silver Bullet for LC or RAG Routing (2025)
- Adaptively evaluating models with task elicitation (2025)
- TLUE: A Tibetan Language Understanding Evaluation Benchmark (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 0
No model linking this paper
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper