Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Supervised Knowledge Makes Large Language Models Better In-context Learners (2023)
Evaluating Large Language Models in Semantic Parsing for Conversational Question Answering over Knowledge Graphs (2024)
INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning (2024)
PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities (2024)
kNN-ICL: Compositional Task-Oriented Parsing Generalization with Nearest Neighbor In-Context Learning (2023)

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2024-02-03T01:21:11.489Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7506274580955505},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2402.00858","authors":[{"_id":"65bc6cec358734fd096bc182","name":"Yilun Zhu","hidden":false},{"_id":"65bc6cec358734fd096bc183","user":{"_id":"6650e6b9e29125c00a919a4d","avatarUrl":"/avatars/b0093fa9da51e42168545e83750c3e81.svg","isPro":false,"fullname":"Joel Moniz","user":"joelmoniz","type":"user"},"name":"Joel Ruben Antony Moniz","status":"claimed_verified","statusLastChangedAt":"2025-03-13T08:25:22.272Z","hidden":false},{"_id":"65bc6cec358734fd096bc184","name":"Shruti Bhargava","hidden":false},{"_id":"65bc6cec358734fd096bc185","user":{"_id":"64bad07938953777fe3ed5e9","avatarUrl":"/avatars/91c07a176d8d51111873751e82a4e0ea.svg","isPro":false,"fullname":"Jiarui Lu","user":"jiarui-lu","type":"user"},"name":"Jiarui Lu","status":"admin_assigned","statusLastChangedAt":"2024-02-02T11:36:16.850Z","hidden":false},{"_id":"65bc6cec358734fd096bc186","name":"Dhivya Piraviperumal","hidden":false},{"_id":"65bc6cec358734fd096bc187","name":"Site Li","hidden":false},{"_id":"65bc6cec358734fd096bc188","name":"Yuan Zhang","hidden":false},{"_id":"65bc6cec358734fd096bc189","name":"Hong Yu","hidden":false},{"_id":"65bc6cec358734fd096bc18a","name":"Bo-Hsiang Tseng","hidden":false}],"publishedAt":"2024-02-01T18:55:29.000Z","submittedOnDailyAt":"2024-02-02T01:47:50.177Z","title":"Can Large Language Models Understand Context?","submittedOnDailyBy":{"_id":"60f1abe7544c2adfd699860c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674929746905-60f1abe7544c2adfd699860c.jpeg","isPro":false,"fullname":"AK","user":"akhaliq","type":"user"},"summary":"Understanding context is key to understanding human language, an ability\nwhich Large Language Models (LLMs) have been increasingly seen to demonstrate\nto an impressive extent. However, though the evaluation of LLMs encompasses\nvarious domains within the realm of Natural Language Processing, limited\nattention has been paid to probing their linguistic capability of understanding\ncontextual features. This paper introduces a context understanding benchmark by\nadapting existing datasets to suit the evaluation of generative models. This\nbenchmark comprises of four distinct tasks and nine datasets, all featuring\nprompts designed to assess the models' ability to understand context. First, we\nevaluate the performance of LLMs under the in-context learning pretraining\nscenario. Experimental results indicate that pre-trained dense models struggle\nwith understanding more nuanced contextual features when compared to\nstate-of-the-art fine-tuned models. Second, as LLM compression holds growing\nsignificance in both research and real-world applications, we assess the\ncontext understanding of quantized models under in-context-learning settings.\nWe find that 3-bit post-training quantization leads to varying degrees of\nperformance reduction on our benchmark. We conduct an extensive analysis of\nthese scenarios to substantiate our experimental results.","upvotes":24,"discussionId":"65bc6cee358734fd096bc1c5","ai_summary":"The benchmark evaluates LLMs' context understanding by assessing pre-trained and quantized models across four tasks and nine datasets, highlighting the performance differences between dense and fine-tuned models.","ai_keywords":["Large Language Models (LLMs)","in-context learning","pretraining","fine-tuning","contextual features","prompt design","benchmark","3-bit post-training quantization"]},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"656ec0b0c657c4341dd34ff6","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/jQ0au6uklu500MOUSXWuw.jpeg","isPro":false,"fullname":"Everett Kleven","user":"everettVT","type":"user"},{"_id":"6578f357e390cfd409bda675","avatarUrl":"/avatars/c513b5f953dd787ed8bdb9e7cf31ed9e.svg","isPro":false,"fullname":"Chris Concannon","user":"choncan","type":"user"},{"_id":"657217faabb25ed8aedd5e48","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/657217faabb25ed8aedd5e48/UUHAXeGtOnQBXFD3nYtf2.jpeg","isPro":false,"fullname":"Vlad Bogolin","user":"vladbogo","type":"user"},{"_id":"6538119803519fddb4a17e10","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6538119803519fddb4a17e10/ffJMkdx-rM7VvLTCM6ri_.jpeg","isPro":false,"fullname":"samusenps","user":"samusenps","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"64747f7e33192631bacd8831","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64747f7e33192631bacd8831/dstkZJ4sHJSeqLesV5cOC.jpeg","isPro":false,"fullname":"Taufiq Dwi Purnomo","user":"taufiqdp","type":"user"},{"_id":"653a18000dfecb4b26dd2876","avatarUrl":"/avatars/fcf8a2ea58f6eca0a6196299c68fc8ad.svg","isPro":false,"fullname":"James Chang","user":"strategist922","type":"user"},{"_id":"60d344d013f774189902f542","avatarUrl":"/avatars/a2462d50d7a85dc3896d8879a288e4a0.svg","isPro":false,"fullname":"Yener Karaca","user":"Yener","type":"user"},{"_id":"65a3a1763581a68c4198b6f2","avatarUrl":"/avatars/2b238bd9605b53c597eade13e29e3778.svg","isPro":false,"fullname":"dude","user":"asawq2006","type":"user"},{"_id":"64fb20b6010f41e43528e26f","avatarUrl":"/avatars/9a8075da5cdaf832e011a7fc9ad4c690.svg","isPro":false,"fullname":"Nobody","user":"n3rdium","type":"user"},{"_id":"648eb1eb59c4e5c87dc116e0","avatarUrl":"/avatars/c636cea39c2c0937f01398c94ead5dad.svg","isPro":false,"fullname":"fdsqefsgergd","user":"T-representer","type":"user"},{"_id":"6350c89759bfa9a85d434138","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1666238674117-6350c89759bfa9a85d434138.jpeg","isPro":false,"fullname":"Yang Lee","user":"innovation64","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0}">

Papers

arxiv:2402.00858

Can Large Language Models Understand Context?

Published on Feb 1, 2024

· Submitted by

AK on Feb 2, 2024

Upvote

Authors:

Joel Ruben Antony Moniz ,

Jiarui Lu ,

Abstract

The benchmark evaluates LLMs' context understanding by assessing pre-trained and quantized models across four tasks and nine datasets, highlighting the performance differences between dense and fine-tuned models.

AI-generated summary

Understanding context is key to understanding human language, an ability which Large Language Models (LLMs) have been increasingly seen to demonstrate to an impressive extent. However, though the evaluation of LLMs encompasses various domains within the realm of Natural Language Processing, limited attention has been paid to probing their linguistic capability of understanding contextual features. This paper introduces a context understanding benchmark by adapting existing datasets to suit the evaluation of generative models. This benchmark comprises of four distinct tasks and nine datasets, all featuring prompts designed to assess the models' ability to understand context. First, we evaluate the performance of LLMs under the in-context learning pretraining scenario. Experimental results indicate that pre-trained dense models struggle with understanding more nuanced contextual features when compared to state-of-the-art fine-tuned models. Second, as LLM compression holds growing significance in both research and real-world applications, we assess the context understanding of quantized models under in-context-learning settings. We find that 3-bit post-training quantization leads to varying degrees of performance reduction on our benchmark. We conduct an extensive analysis of these scenarios to substantiate our experimental results.