https://huggingface.co/spaces/m42-health/clinical_ner_leaderboard

\n","updatedAt":"2024-10-08T10:50:30.059Z","author":{"_id":"628e39f4b1596566033b8d7b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/628e39f4b1596566033b8d7b/-Y807up1cgMmAQsczdOPn.jpeg","fullname":"Clément Christophe","name":"cchristophe","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":9,"isUserFollowing":false}},"numEdits":0,"editors":["cchristophe"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/628e39f4b1596566033b8d7b/-Y807up1cgMmAQsczdOPn.jpeg"],"reactions":[{"reaction":"🔥","users":["AdinaY"],"count":1},{"reaction":"👍","users":["AdinaY"],"count":1}],"isReport":false},"replies":[{"id":"670682dff95c7ec727d8dabd","author":{"_id":"63a369d98c0c89dcae3b8329","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63a369d98c0c89dcae3b8329/AiH2zjy1cnt9OADAAZMLD.jpeg","fullname":"Adina Yakefu","name":"AdinaY","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":1145,"isUserFollowing":false},"createdAt":"2024-10-09T13:19:27.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Super cool work🔥 Congrats! ","html":"

Super cool work🔥 Congrats!

\n","updatedAt":"2024-10-09T13:19:27.765Z","author":{"_id":"63a369d98c0c89dcae3b8329","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63a369d98c0c89dcae3b8329/AiH2zjy1cnt9OADAAZMLD.jpeg","fullname":"Adina Yakefu","name":"AdinaY","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":1145,"isUserFollowing":false}},"numEdits":0,"editors":["AdinaY"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/63a369d98c0c89dcae3b8329/AiH2zjy1cnt9OADAAZMLD.jpeg"],"reactions":[{"reaction":"👍","users":["mpimentel","cchristophe"],"count":2}],"isReport":false,"parentCommentId":"67050e76815aea7b41426fe0"}}]},{"id":"6705dd648e2c685057a454a0","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2024-10-09T01:33:24.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Evaluating Named Entity Recognition Using Few-Shot Prompting with Large Language Models](https://huggingface.co/papers/2408.15796) (2024)\n* [MEDIC: Towards a Comprehensive Framework for Evaluating LLMs in Clinical Applications](https://huggingface.co/papers/2409.07314) (2024)\n* [A Survey for Large Language Models in Biomedicine](https://huggingface.co/papers/2409.00133) (2024)\n* [LLMs are not Zero-Shot Reasoners for Biomedical Information Extraction](https://huggingface.co/papers/2408.12249) (2024)\n* [Clinical Insights: A Comprehensive Review of Language Models in Medicine](https://huggingface.co/papers/2408.11735) (2024)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2024-10-09T01:33:24.441Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false}},"numEdits":0,"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2410.05046","authors":[{"_id":"67050c6448db87e174db722b","user":{"_id":"5f27a163e923d665e616271c","avatarUrl":"/avatars/2b10d5846cef8a01b5db4c435b6f2628.svg","isPro":false,"fullname":"Wadood Abdul","user":"wadood","type":"user"},"name":"Wadood M Abdul","status":"claimed_verified","statusLastChangedAt":"2024-10-09T07:37:56.785Z","hidden":false},{"_id":"67050c6448db87e174db722c","user":{"_id":"6454faafa13edf669cd74f36","avatarUrl":"/avatars/355d16e28ca9cf5891368e43bcda6de5.svg","isPro":false,"fullname":"Marco Pimentel","user":"mpimentel","type":"user"},"name":"Marco AF Pimentel","status":"claimed_verified","statusLastChangedAt":"2024-10-08T12:45:56.120Z","hidden":false},{"_id":"67050c6448db87e174db722d","user":{"_id":"65f012b513700cbfce47c0f9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65f012b513700cbfce47c0f9/8EW1TkevArZOaWHYOZ859.jpeg","isPro":false,"fullname":"Muhammad Umar Salman","user":"umarsalman","type":"user"},"name":"Muhammad Umar Salman","status":"admin_assigned","statusLastChangedAt":"2024-10-09T13:16:59.116Z","hidden":false},{"_id":"67050c6448db87e174db722e","user":{"_id":"5f5f6c113c67af20d9945afb","avatarUrl":"/avatars/06b2eb3a5d27864280d4d02e6d00d782.svg","isPro":false,"fullname":"Tathagata Raha","user":"tathagataraha","type":"user"},"name":"Tathagata Raha","status":"admin_assigned","statusLastChangedAt":"2024-10-09T13:17:05.096Z","hidden":false},{"_id":"67050c6448db87e174db722f","user":{"_id":"628e39f4b1596566033b8d7b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/628e39f4b1596566033b8d7b/-Y807up1cgMmAQsczdOPn.jpeg","isPro":false,"fullname":"Clément Christophe","user":"cchristophe","type":"user"},"name":"Clément Christophe","status":"admin_assigned","statusLastChangedAt":"2024-10-09T13:17:11.030Z","hidden":false},{"_id":"67050c6448db87e174db7230","user":{"_id":"65280984b794fe3d06544d77","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65280984b794fe3d06544d77/tyrxbxtDG02On1uiRaVbL.jpeg","isPro":false,"fullname":"Praveenkumar","user":"pkanithi","type":"user"},"name":"Praveen K Kanithi","status":"claimed_verified","statusLastChangedAt":"2024-10-08T12:46:16.946Z","hidden":false},{"_id":"67050c6448db87e174db7231","name":"Nasir Hayat","hidden":false},{"_id":"67050c6448db87e174db7232","user":{"_id":"65281d6ef61ca80b9c2ee707","avatarUrl":"/avatars/090ea7210a4bb6549b0f7fee71525625.svg","isPro":false,"fullname":"Ronnie Rajan","user":"ronnierajan","type":"user"},"name":"Ronnie Rajan","status":"admin_assigned","statusLastChangedAt":"2024-10-09T13:17:31.565Z","hidden":false},{"_id":"67050c6448db87e174db7233","name":"Shadab Khan","hidden":false}],"publishedAt":"2024-10-07T14:00:18.000Z","submittedOnDailyAt":"2024-10-08T09:20:30.054Z","title":"Named Clinical Entity Recognition Benchmark","submittedOnDailyBy":{"_id":"628e39f4b1596566033b8d7b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/628e39f4b1596566033b8d7b/-Y807up1cgMmAQsczdOPn.jpeg","isPro":false,"fullname":"Clément Christophe","user":"cchristophe","type":"user"},"summary":"This technical report introduces a Named Clinical Entity Recognition\nBenchmark for evaluating language models in healthcare, addressing the crucial\nnatural language processing (NLP) task of extracting structured information\nfrom clinical narratives to support applications like automated coding,\nclinical trial cohort identification, and clinical decision support.\n The leaderboard provides a standardized platform for assessing diverse\nlanguage models, including encoder and decoder architectures, on their ability\nto identify and classify clinical entities across multiple medical domains. A\ncurated collection of openly available clinical datasets is utilized,\nencompassing entities such as diseases, symptoms, medications, procedures, and\nlaboratory measurements. Importantly, these entities are standardized according\nto the Observational Medical Outcomes Partnership (OMOP) Common Data Model,\nensuring consistency and interoperability across different healthcare systems\nand datasets, and a comprehensive evaluation of model performance. Performance\nof models is primarily assessed using the F1-score, and it is complemented by\nvarious assessment modes to provide comprehensive insights into model\nperformance. The report also includes a brief analysis of models evaluated to\ndate, highlighting observed trends and limitations.\n By establishing this benchmarking framework, the leaderboard aims to promote\ntransparency, facilitate comparative analyses, and drive innovation in clinical\nentity recognition tasks, addressing the need for robust evaluation methods in\nhealthcare NLP.","upvotes":16,"discussionId":"67050c6448db87e174db726e","ai_summary":"A benchmark for evaluating language models in clinical entity recognition uses the OMOP Common Data Model for standardized assessment of diverse architectures across multiple medical domains.","ai_keywords":["Named Clinical Entity Recognition","encoder architectures","decoder architectures","clinical narratives","automated coding","clinical trial cohort identification","clinical decision support","F1-score"]},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"5f27a163e923d665e616271c","avatarUrl":"/avatars/2b10d5846cef8a01b5db4c435b6f2628.svg","isPro":false,"fullname":"Wadood Abdul","user":"wadood","type":"user"},{"_id":"65281d6ef61ca80b9c2ee707","avatarUrl":"/avatars/090ea7210a4bb6549b0f7fee71525625.svg","isPro":false,"fullname":"Ronnie Rajan","user":"ronnierajan","type":"user"},{"_id":"628e39f4b1596566033b8d7b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/628e39f4b1596566033b8d7b/-Y807up1cgMmAQsczdOPn.jpeg","isPro":false,"fullname":"Clément Christophe","user":"cchristophe","type":"user"},{"_id":"65280984b794fe3d06544d77","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65280984b794fe3d06544d77/tyrxbxtDG02On1uiRaVbL.jpeg","isPro":false,"fullname":"Praveenkumar","user":"pkanithi","type":"user"},{"_id":"5f5f6c113c67af20d9945afb","avatarUrl":"/avatars/06b2eb3a5d27864280d4d02e6d00d782.svg","isPro":false,"fullname":"Tathagata Raha","user":"tathagataraha","type":"user"},{"_id":"6454faafa13edf669cd74f36","avatarUrl":"/avatars/355d16e28ca9cf5891368e43bcda6de5.svg","isPro":false,"fullname":"Marco Pimentel","user":"mpimentel","type":"user"},{"_id":"65f012b513700cbfce47c0f9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65f012b513700cbfce47c0f9/8EW1TkevArZOaWHYOZ859.jpeg","isPro":false,"fullname":"Muhammad Umar Salman","user":"umarsalman","type":"user"},{"_id":"6405f62ba577649430be5124","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6405f62ba577649430be5124/WMSqNvDqaeydFkHhNCU_K.png","isPro":false,"fullname":"Mykhailo Shtopko","user":"BioMike","type":"user"},{"_id":"6512cdbd332b85e7cf86acfe","avatarUrl":"/avatars/553dd4bf1ca28717a29f4cd325861b4d.svg","isPro":false,"fullname":"Prateek Munjal","user":"PrateekMunjal","type":"user"},{"_id":"6506cfafd55dd4e15caeea09","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/uTkC6G8Cj51av4i7kAaI8.png","isPro":false,"fullname":"Svetlana Maslenkova","user":"maslenkovas","type":"user"},{"_id":"64d324af4eb2ea6d5d910810","avatarUrl":"/avatars/df775a4a594fad72980cc2744c7c56dc.svg","isPro":false,"fullname":"Ahmed Al Mahrooqi","user":"ahmed1996said","type":"user"},{"_id":"638da3b5bb255c6a0fee4c5e","avatarUrl":"/avatars/d1de9430b78a6a32f1be53780bec1941.svg","isPro":false,"fullname":"Shirin P","user":"ShirinP","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0}">

Papers

arxiv:2410.05046

Named Clinical Entity Recognition Benchmark

Published on Oct 7, 2024

· Submitted by

Clément Christophe on Oct 8, 2024

Upvote

Authors:

Wadood M Abdul ,

Marco AF Pimentel ,

Muhammad Umar Salman ,

Tathagata Raha ,

Clément Christophe ,

Praveen K Kanithi ,

Ronnie Rajan ,

Abstract

A benchmark for evaluating language models in clinical entity recognition uses the OMOP Common Data Model for standardized assessment of diverse architectures across multiple medical domains.

AI-generated summary

This technical report introduces a Named Clinical Entity Recognition Benchmark for evaluating language models in healthcare, addressing the crucial natural language processing (NLP) task of extracting structured information from clinical narratives to support applications like automated coding, clinical trial cohort identification, and clinical decision support. The leaderboard provides a standardized platform for assessing diverse language models, including encoder and decoder architectures, on their ability to identify and classify clinical entities across multiple medical domains. A curated collection of openly available clinical datasets is utilized, encompassing entities such as diseases, symptoms, medications, procedures, and laboratory measurements. Importantly, these entities are standardized according to the Observational Medical Outcomes Partnership (OMOP) Common Data Model, ensuring consistency and interoperability across different healthcare systems and datasets, and a comprehensive evaluation of model performance. Performance of models is primarily assessed using the F1-score, and it is complemented by various assessment modes to provide comprehensive insights into model performance. The report also includes a brief analysis of models evaluated to date, highlighting observed trends and limitations. By establishing this benchmarking framework, the leaderboard aims to promote transparency, facilitate comparative analyses, and drive innovation in clinical entity recognition tasks, addressing the need for robust evaluation methods in healthcare NLP.