Librarian Bot. I found the following papers similar to this paper. \n
The following papers were recommended by the Semantic Scholar API
\n- \n
- LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs (2024) \n
- LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs (2024) \n
- Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form Text (2024) \n
- See What LLMs Cannot Answer: A Self-Challenge Framework for Uncovering LLM Weaknesses (2024) \n
- DHP Benchmark: Are LLMs Good NLG Evaluators? (2024) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
Papers
arxiv:2409.16191
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
Abstract
A comprehensive benchmark, HelloBench, and evaluation method, HelloEval, are introduced to assess the long text generation capabilities of Large Language Models, revealing limitations and providing human-aligned evaluations.
AI-generated summary
In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities in various tasks (e.g., long-context understanding), and many benchmarks have been proposed. However, we observe that long text generation capabilities are not well investigated. Therefore, we introduce the Hierarchical Long Text Generation Benchmark (HelloBench), a comprehensive, in-the-wild, and open-ended benchmark to evaluate LLMs' performance in generating long text. Based on Bloom's Taxonomy, HelloBench categorizes long text generation tasks into five subtasks: open-ended QA, summarization, chat, text completion, and heuristic text generation. Besides, we propose Hierarchical Long Text Evaluation (HelloEval), a human-aligned evaluation method that significantly reduces the time and effort required for human evaluation while maintaining a high correlation with human evaluation. We have conducted extensive experiments across around 30 mainstream LLMs and observed that the current LLMs lack long text generation capabilities. Specifically, first, regardless of whether the instructions include explicit or implicit length constraints, we observe that most LLMs cannot generate text that is longer than 4000 words. Second, we observe that while some LLMs can generate longer text, many issues exist (e.g., severe repetition and quality degradation). Third, to demonstrate the effectiveness of HelloEval, we compare HelloEval with traditional metrics (e.g., ROUGE, BLEU, etc.) and LLM-as-a-Judge methods, which show that HelloEval has the highest correlation with human evaluation. We release our code in https://github.com/Quehry/HelloBench.
Community
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs (2024)
- LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs (2024)
- Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form Text (2024)
- See What LLMs Cannot Answer: A Self-Challenge Framework for Uncovering LLM Weaknesses (2024)
- DHP Benchmark: Are LLMs Good NLG Evaluators? (2024)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
This comment has been hidden
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2409.16191 in a model README.md to link it from this page.
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2409.16191 in a Space README.md to link it from this page.