Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (2024)
Distilling Vision-Language Models on Millions of Videos (2024)
PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong Vision-language Adapter (2024)
Memory Consolidation Enables Long-Context Video Understanding (2024)
Question-Instructed Visual Descriptions for Zero-Shot Video Question Answering (2024)

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2024-02-22T01:22:14.988Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.721083402633667},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}},{"id":"6639d51163e71558f70f1273","author":{"_id":"65a12cdb0c8d993b174ffe90","avatarUrl":"/avatars/1c9b615a07f1d51bf3de8d994990ff76.svg","fullname":"rasem","name":"valentinbvro83","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false},"createdAt":"2024-05-07T07:15:29.000Z","type":"comment","data":{"edited":true,"hidden":false,"latest":{"raw":"great work\n\n","html":"

great work

\n","updatedAt":"2024-05-07T07:15:49.018Z","author":{"_id":"65a12cdb0c8d993b174ffe90","avatarUrl":"/avatars/1c9b615a07f1d51bf3de8d994990ff76.svg","fullname":"rasem","name":"valentinbvro83","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.35448166728019714},"editors":["valentinbvro83"],"editorAvatarUrls":["/avatars/1c9b615a07f1d51bf3de8d994990ff76.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2402.13217","authors":[{"_id":"65d573c188f4a905891d40cc","user":{"_id":"650c249887dcda6616baa040","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tIVfxoAHAJ0sWyDMkaarA.jpeg","isPro":false,"fullname":"Long Zhao","user":"garyzhao9012","type":"user"},"name":"Long Zhao","status":"extracted_confirmed","statusLastChangedAt":"2024-02-21T04:41:13.955Z","hidden":false},{"_id":"65d573c188f4a905891d40cd","name":"Nitesh B. Gundavarapu","hidden":false},{"_id":"65d573c188f4a905891d40ce","name":"Liangzhe Yuan","hidden":false},{"_id":"65d573c188f4a905891d40cf","name":"Hao Zhou","hidden":false},{"_id":"65d573c188f4a905891d40d0","name":"Shen Yan","hidden":false},{"_id":"65d573c188f4a905891d40d1","user":{"_id":"64cec85201931c6016f6f1b3","avatarUrl":"/avatars/77a4ebbb3995600cd01d385b2a6084c4.svg","isPro":false,"fullname":"Jennifer J. Sun","user":"JenJSun","type":"user"},"name":"Jennifer J. Sun","status":"admin_assigned","statusLastChangedAt":"2024-02-21T10:15:45.333Z","hidden":false},{"_id":"65d573c188f4a905891d40d2","name":"Luke Friedman","hidden":false},{"_id":"65d573c188f4a905891d40d3","name":"Rui Qian","hidden":false},{"_id":"65d573c188f4a905891d40d4","name":"Tobias Weyand","hidden":false},{"_id":"65d573c188f4a905891d40d5","user":{"_id":"638fe91639f7e2a7f9d2a8c6","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/638fe91639f7e2a7f9d2a8c6/hB7DMVODcdAEUdQnXxWA8.jpeg","isPro":false,"fullname":"Yue Zhao","user":"zhaoyue-zephyrus","type":"user"},"name":"Yue Zhao","status":"claimed_verified","statusLastChangedAt":"2024-06-15T07:46:18.796Z","hidden":false},{"_id":"65d573c188f4a905891d40d6","name":"Rachel Hornung","hidden":false},{"_id":"65d573c188f4a905891d40d7","name":"Florian Schroff","hidden":false},{"_id":"65d573c188f4a905891d40d8","name":"Ming-Hsuan Yang","hidden":false},{"_id":"65d573c188f4a905891d40d9","name":"David A. Ross","hidden":false},{"_id":"65d573c188f4a905891d40da","name":"Huisheng Wang","hidden":false},{"_id":"65d573c188f4a905891d40db","name":"Hartwig Adam","hidden":false},{"_id":"65d573c188f4a905891d40dc","user":{"_id":"65b13272dfa3789ef39937ce","avatarUrl":"/avatars/c7803adb1b82b881946b3b0aeaee5f4c.svg","isPro":false,"fullname":"Mikhail Sirotenko","user":"msirotenko-g","type":"user"},"name":"Mikhail Sirotenko","status":"extracted_confirmed","statusLastChangedAt":"2024-02-21T14:33:00.920Z","hidden":false},{"_id":"65d573c188f4a905891d40dd","user":{"_id":"64a78e9bacc69c3bbaf12b3d","avatarUrl":"/avatars/09cd26234e0e0c82f5cc2c02e710163c.svg","isPro":false,"fullname":"Ting Liu","user":"tingianliu","type":"user"},"name":"Ting Liu","status":"claimed_verified","statusLastChangedAt":"2024-02-22T22:27:59.395Z","hidden":false},{"_id":"65d573c188f4a905891d40de","name":"Boqing Gong","hidden":false}],"publishedAt":"2024-02-20T18:29:49.000Z","submittedOnDailyAt":"2024-02-21T01:23:38.694Z","title":"VideoPrism: A Foundational Visual Encoder for Video Understanding","submittedOnDailyBy":{"_id":"60f1abe7544c2adfd699860c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674929746905-60f1abe7544c2adfd699860c.jpeg","isPro":false,"fullname":"AK","user":"akhaliq","type":"user"},"summary":"We introduce VideoPrism, a general-purpose video encoder that tackles diverse\nvideo understanding tasks with a single frozen model. We pretrain VideoPrism on\na heterogeneous corpus containing 36M high-quality video-caption pairs and 582M\nvideo clips with noisy parallel text (e.g., ASR transcripts). The pretraining\napproach improves upon masked autoencoding by global-local distillation of\nsemantic video embeddings and a token shuffling scheme, enabling VideoPrism to\nfocus primarily on the video modality while leveraging the invaluable text\nassociated with videos. We extensively test VideoPrism on four broad groups of\nvideo understanding tasks, from web video question answering to CV for science,\nachieving state-of-the-art performance on 30 out of 33 video understanding\nbenchmarks.","upvotes":38,"discussionId":"65d573c288f4a905891d4113","githubRepo":"https://github.com/google-deepmind/videoprism","githubRepoAddedBy":"auto","ai_summary":"VideoPrism, a pretrained video encoder, achieves top performance across various video understanding tasks by utilizing global-local distillation and token shuffling of semantic video embeddings enhanced with associated text.","ai_keywords":["VideoPrism","masked autoencoding","global-local distillation","semantic video embeddings","token shuffling scheme","video understanding tasks","web video question answering","CV for science","video understanding benchmarks"],"githubStars":351},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"635cada2c017767a629db012","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1667018139063-noauth.jpeg","isPro":false,"fullname":"Ojasvi Singh Yadav","user":"ojasvisingh786","type":"user"},{"_id":"6039478ab3ecf716b1a5fd4d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg","isPro":true,"fullname":"taesiri","user":"taesiri","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"63d80e9807cd1aa3c49ecf9e","avatarUrl":"/avatars/b4cf2ac91ae2998358fb4b9269e68dd5.svg","isPro":false,"fullname":"Ma Ia","user":"magoli","type":"user"},{"_id":"630412d57373aacccd88af95","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1670594087059-630412d57373aacccd88af95.jpeg","isPro":true,"fullname":"Yasunori Ozaki","user":"alfredplpl","type":"user"},{"_id":"6538119803519fddb4a17e10","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6538119803519fddb4a17e10/ffJMkdx-rM7VvLTCM6ri_.jpeg","isPro":false,"fullname":"samusenps","user":"samusenps","type":"user"},{"_id":"655ac762cb17ec19ef82719b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/655ac762cb17ec19ef82719b/1kDncYrGLYS_2SR8cNdAL.png","isPro":false,"fullname":"Welcome to matlok","user":"matlok","type":"user"},{"_id":"635964636a61954080850e1d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/635964636a61954080850e1d/0bfExuDTrHTtm8c-40cDM.png","isPro":false,"fullname":"William Lamkin","user":"phanes","type":"user"},{"_id":"61868ce808aae0b5499a2a95","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/61868ce808aae0b5499a2a95/F6BA0anbsoY_Z7M1JrwOe.jpeg","isPro":true,"fullname":"Sylvain Filoni","user":"fffiloni","type":"user"},{"_id":"6344397d73b4f9cedab5ece7","avatarUrl":"/avatars/9905252b2cbd6f9fee64f3d607e85931.svg","isPro":false,"fullname":"Sidharth","user":"sidhusmart","type":"user"},{"_id":"6530274264599cf1271aa94f","avatarUrl":"/avatars/c918abcb3e54d65bb9c6f171153e7196.svg","isPro":false,"fullname":"Qifei Wang","user":"raccoon1119","type":"user"},{"_id":"649b184ab9368db7f5bcab69","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/649b184ab9368db7f5bcab69/UOSWgMd1bgM0ksuVRCNjx.jpeg","isPro":false,"fullname":"Bharat","user":"bsbarkur","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":3}">

Papers

arxiv:2402.13217

VideoPrism: A Foundational Visual Encoder for Video Understanding

Published on Feb 20, 2024

· Submitted by

AK on Feb 21, 2024

#3 Paper of the day

Upvote

Authors:

Long Zhao ,

Jennifer J. Sun ,

Yue Zhao ,

Mikhail Sirotenko ,

Ting Liu ,

Abstract

VideoPrism, a pretrained video encoder, achieves top performance across various video understanding tasks by utilizing global-local distillation and token shuffling of semantic video embeddings enhanced with associated text.

AI-generated summary

We introduce VideoPrism, a general-purpose video encoder that tackles diverse video understanding tasks with a single frozen model. We pretrain VideoPrism on a heterogeneous corpus containing 36M high-quality video-caption pairs and 582M video clips with noisy parallel text (e.g., ASR transcripts). The pretraining approach improves upon masked autoencoding by global-local distillation of semantic video embeddings and a token shuffling scheme, enabling VideoPrism to focus primarily on the video modality while leveraging the invaluable text associated with videos. We extensively test VideoPrism on four broad groups of video understanding tasks, from web video question answering to CV for science, achieving state-of-the-art performance on 30 out of 33 video understanding benchmarks.