Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models (2024)
EvalGIM: A Library for Evaluating Generative Image Models (2024)
Accelerating Video Diffusion Models via Distribution Matching (2024)
FastRM: An efficient and automatic explainability framework for multimodal generative models (2024)
An Efficient Framework for Enhancing Discriminative Models via Diffusion Techniques (2024)
SPAgent: Adaptive Task Decomposition and Model Selection for General Video Generation and Editing (2024)
Efficient Generative Modeling with Residual Vector Quantization-Based Tokens (2024)

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2024-12-18T01:36:24.496Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.6931726336479187},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2412.09645","authors":[{"_id":"67610b5eb04baaf63514fcc9","user":{"_id":"61f24cbb88b9b5abbe184a85","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/61f24cbb88b9b5abbe184a85/OvcJRU51yI8pdO77NBHLb.jpeg","isPro":false,"fullname":"zhangfan","user":"Fan-s","type":"user"},"name":"Fan Zhang","status":"claimed_verified","statusLastChangedAt":"2024-12-17T08:03:36.826Z","hidden":false},{"_id":"67610b5eb04baaf63514fcca","user":{"_id":"6658d01c6f1a71ba56d6c273","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tc4nZrMuZQLfgt5aVxtH4.jpeg","isPro":false,"fullname":"Tian Shulin","user":"shulin16","type":"user"},"name":"Shulin Tian","status":"admin_assigned","statusLastChangedAt":"2024-12-17T09:23:16.414Z","hidden":false},{"_id":"67610b5eb04baaf63514fccb","user":{"_id":"60efe7fa0d920bc7805cada5","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/60efe7fa0d920bc7805cada5/2LBrJBjSCOP5ilZIpWLHl.png","isPro":false,"fullname":"Ziqi Huang","user":"Ziqi","type":"user"},"name":"Ziqi Huang","status":"admin_assigned","statusLastChangedAt":"2024-12-17T09:23:04.978Z","hidden":false},{"_id":"67610b5eb04baaf63514fccc","name":"Yu Qiao","hidden":false},{"_id":"67610b5eb04baaf63514fccd","user":{"_id":"62ab1ac1d48b4d8b048a3473","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png","isPro":false,"fullname":"Ziwei Liu","user":"liuziwei7","type":"user"},"name":"Ziwei Liu","status":"admin_assigned","statusLastChangedAt":"2024-12-17T09:22:56.546Z","hidden":false}],"publishedAt":"2024-12-10T18:52:39.000Z","submittedOnDailyAt":"2024-12-17T03:04:04.556Z","title":"Evaluation Agent: Efficient and Promptable Evaluation Framework for\n Visual Generative Models","submittedOnDailyBy":{"_id":"60efe7fa0d920bc7805cada5","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/60efe7fa0d920bc7805cada5/2LBrJBjSCOP5ilZIpWLHl.png","isPro":false,"fullname":"Ziqi Huang","user":"Ziqi","type":"user"},"summary":"Recent advancements in visual generative models have enabled high-quality\nimage and video generation, opening diverse applications. However, evaluating\nthese models often demands sampling hundreds or thousands of images or videos,\nmaking the process computationally expensive, especially for diffusion-based\nmodels with inherently slow sampling. Moreover, existing evaluation methods\nrely on rigid pipelines that overlook specific user needs and provide numerical\nresults without clear explanations. In contrast, humans can quickly form\nimpressions of a model's capabilities by observing only a few samples. To mimic\nthis, we propose the Evaluation Agent framework, which employs human-like\nstrategies for efficient, dynamic, multi-round evaluations using only a few\nsamples per round, while offering detailed, user-tailored analyses. It offers\nfour key advantages: 1) efficiency, 2) promptable evaluation tailored to\ndiverse user needs, 3) explainability beyond single numerical scores, and 4)\nscalability across various models and tools. Experiments show that Evaluation\nAgent reduces evaluation time to 10% of traditional methods while delivering\ncomparable results. The Evaluation Agent framework is fully open-sourced to\nadvance research in visual generative models and their efficient evaluation.","upvotes":36,"discussionId":"67610b60b04baaf63514fd5d","projectPage":"https://vchitect.github.io/Evaluation-Agent-project","githubRepo":"https://github.com/Vchitect/Evaluation-Agent","githubRepoAddedBy":"auto","ai_summary":"The Evaluation Agent framework efficiently evaluates visual generative models using human-like strategies, reducing time to 10% compared to traditional methods while maintaining quality.","ai_keywords":["diffusion-based models"],"githubStars":121},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"61f24cbb88b9b5abbe184a85","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/61f24cbb88b9b5abbe184a85/OvcJRU51yI8pdO77NBHLb.jpeg","isPro":false,"fullname":"zhangfan","user":"Fan-s","type":"user"},{"_id":"6658d01c6f1a71ba56d6c273","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tc4nZrMuZQLfgt5aVxtH4.jpeg","isPro":false,"fullname":"Tian Shulin","user":"shulin16","type":"user"},{"_id":"62ab1ac1d48b4d8b048a3473","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png","isPro":false,"fullname":"Ziwei Liu","user":"liuziwei7","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"638d7d33c8912be69c1a5849","avatarUrl":"/avatars/475efc364e10f3594467bd45260d8999.svg","isPro":false,"fullname":"Xin Huang","user":"xanderhuang","type":"user"},{"_id":"6448b2f53e7b3c11be684348","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6448b2f53e7b3c11be684348/QvlUQG3pWf8ZyEVBV6F7w.jpeg","isPro":true,"fullname":"Qianli Ma","user":"Mqleet","type":"user"},{"_id":"6415d088107962562e99517c","avatarUrl":"/avatars/c2fa60334080fc238016b49b1a436c00.svg","isPro":false,"fullname":"Qi Chen-SII","user":"qc316","type":"user"},{"_id":"643815c4961bb61e463c5896","avatarUrl":"/avatars/3b44592472f16c56105bff8c314d9939.svg","isPro":false,"fullname":"Jianxiong Gao","user":"Jianxiong","type":"user"},{"_id":"635a46ae6365824bdf0aadb3","avatarUrl":"/avatars/4b07c9dcdfbe726581a99f0b0e0f9007.svg","isPro":false,"fullname":"dzy","user":"zy1111","type":"user"},{"_id":"6351463b8445bbe32e944f6c","avatarUrl":"/avatars/ec0e8f378d5314d4af97d6c488771b3d.svg","isPro":false,"fullname":"Yuhao Liu","user":"LeoLau","type":"user"},{"_id":"60efe7fa0d920bc7805cada5","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/60efe7fa0d920bc7805cada5/2LBrJBjSCOP5ilZIpWLHl.png","isPro":false,"fullname":"Ziqi Huang","user":"Ziqi","type":"user"},{"_id":"6683a05e74fb1736a4b7c934","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6683a05e74fb1736a4b7c934/eiz6qlqIUjAWGy5zfg8Cs.jpeg","isPro":false,"fullname":"QRQ","user":"RichardQRQ","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":3}">

Papers

arxiv:2412.09645

Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models

Published on Dec 10, 2024

· Submitted by

Ziqi Huang on Dec 17, 2024

#3 Paper of the day

Upvote

Authors:

Fan Zhang ,

Shulin Tian ,

Ziqi Huang ,

Ziwei Liu

Abstract

The Evaluation Agent framework efficiently evaluates visual generative models using human-like strategies, reducing time to 10% compared to traditional methods while maintaining quality.

AI-generated summary

Recent advancements in visual generative models have enabled high-quality image and video generation, opening diverse applications. However, evaluating these models often demands sampling hundreds or thousands of images or videos, making the process computationally expensive, especially for diffusion-based models with inherently slow sampling. Moreover, existing evaluation methods rely on rigid pipelines that overlook specific user needs and provide numerical results without clear explanations. In contrast, humans can quickly form impressions of a model's capabilities by observing only a few samples. To mimic this, we propose the Evaluation Agent framework, which employs human-like strategies for efficient, dynamic, multi-round evaluations using only a few samples per round, while offering detailed, user-tailored analyses. It offers four key advantages: 1) efficiency, 2) promptable evaluation tailored to diverse user needs, 3) explainability beyond single numerical scores, and 4) scalability across various models and tools. Experiments show that Evaluation Agent reduces evaluation time to 10% of traditional methods while delivering comparable results. The Evaluation Agent framework is fully open-sourced to advance research in visual generative models and their efficient evaluation.

View arXiv page View PDF Project page GitHub 121 auto Add to collection

Community

Ziqi

Paper author Paper submitter Dec 17, 2024

Evaluation Agent - It provides an efficient and promptable evaluation framework for visual generative models, mimicking human-like strategies with only a few samples.

Key advantages: