https://medarc-ai.github.io/mindeye/

\n","updatedAt":"2023-05-30T09:11:00.067Z","author":{"_id":"6057b823861b9d53d9c4b8df","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1625184855691-6057b823861b9d53d9c4b8df.jpeg","fullname":"Tanishq Abraham","name":"tmabraham","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":77,"isUserFollowing":false}},"numEdits":0,"editors":["tmabraham"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1625184855691-6057b823861b9d53d9c4b8df.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2305.18274","authors":[{"_id":"64756bd7b68461d5cf7ef512","user":{"_id":"62b707fbdd998a8b1e424fc3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62b707fbdd998a8b1e424fc3/DW_YvHw06y69jOUKz9gel.png","isPro":false,"fullname":"Paul Scotti","user":"pscotti","type":"user"},"name":"Paul S. Scotti","status":"claimed_verified","statusLastChangedAt":"2023-05-30T11:11:33.284Z","hidden":false},{"_id":"64756bd7b68461d5cf7ef513","name":"Atmadeep Banerjee","hidden":false},{"_id":"64756bd7b68461d5cf7ef514","user":{"_id":"613e5a19fdb9ea2339978745","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/613e5a19fdb9ea2339978745/Iv0UjOrXoxom8EU-EEdk3.png","isPro":false,"fullname":"Jimmie Goode","user":"jimgoo","type":"user"},"name":"Jimmie Goode","status":"claimed_verified","statusLastChangedAt":"2023-05-30T14:15:17.526Z","hidden":false},{"_id":"64756bd7b68461d5cf7ef515","user":{"_id":"613dbca582b4af22cbd7fdb9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1631435931286-noauth.jpeg","isPro":false,"fullname":"nev","user":"nev","type":"user"},"name":"Stepan Shabalin","status":"admin_assigned","statusLastChangedAt":"2023-05-30T08:58:52.757Z","hidden":false},{"_id":"64756bd7b68461d5cf7ef516","name":"Alex Nguyen","hidden":false},{"_id":"64756bd7b68461d5cf7ef517","name":"Ethan Cohen","hidden":false},{"_id":"64756bd7b68461d5cf7ef518","user":{"_id":"62b0d296476825d4e772e762","avatarUrl":"/avatars/9df351257b0d5f2ffcb3e1a48aff6aa7.svg","isPro":false,"fullname":"Aidan Dempster","user":"Veldrovive","type":"user"},"name":"Aidan J. Dempster","status":"claimed_verified","statusLastChangedAt":"2023-05-30T09:10:26.072Z","hidden":false},{"_id":"64756bd7b68461d5cf7ef519","name":"Nathalie Verlinde","hidden":false},{"_id":"64756bd7b68461d5cf7ef51a","user":{"_id":"6303d5350907b9a115c42a5f","avatarUrl":"/avatars/eba35bf756ae1703e10cace3aa6674a5.svg","isPro":false,"fullname":"Elad Yundler","user":"elad619","type":"user"},"name":"Elad Yundler","status":"claimed_verified","statusLastChangedAt":"2023-05-30T18:00:18.278Z","hidden":false},{"_id":"64756bd7b68461d5cf7ef51b","name":"David Weisberg","hidden":false},{"_id":"64756bd7b68461d5cf7ef51c","name":"Kenneth A. Norman","hidden":false},{"_id":"64756bd7b68461d5cf7ef51d","user":{"_id":"6057b823861b9d53d9c4b8df","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1625184855691-6057b823861b9d53d9c4b8df.jpeg","isPro":false,"fullname":"Tanishq Abraham","user":"tmabraham","type":"user"},"name":"Tanishq Mathew Abraham","status":"claimed_verified","statusLastChangedAt":"2023-05-30T09:10:55.780Z","hidden":false}],"publishedAt":"2023-05-29T17:49:00.000Z","submittedOnDailyAt":"2023-05-30T01:52:00.399Z","title":"Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning\n and Diffusion Priors","submittedOnDailyBy":{"_id":"60f1abe7544c2adfd699860c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674929746905-60f1abe7544c2adfd699860c.jpeg","isPro":false,"fullname":"AK","user":"akhaliq","type":"user"},"summary":"We present MindEye, a novel fMRI-to-image approach to retrieve and\nreconstruct viewed images from brain activity. Our model comprises two parallel\nsubmodules that are specialized for retrieval (using contrastive learning) and\nreconstruction (using a diffusion prior). MindEye can map fMRI brain activity\nto any high dimensional multimodal latent space, like CLIP image space,\nenabling image reconstruction using generative models that accept embeddings\nfrom this latent space. We comprehensively compare our approach with other\nexisting methods, using both qualitative side-by-side comparisons and\nquantitative evaluations, and show that MindEye achieves state-of-the-art\nperformance in both reconstruction and retrieval tasks. In particular, MindEye\ncan retrieve the exact original image even among highly similar candidates\nindicating that its brain embeddings retain fine-grained image-specific\ninformation. This allows us to accurately retrieve images even from large-scale\ndatabases like LAION-5B. We demonstrate through ablations that MindEye's\nperformance improvements over previous methods result from specialized\nsubmodules for retrieval and reconstruction, improved training techniques, and\ntraining models with orders of magnitude more parameters. Furthermore, we show\nthat MindEye can better preserve low-level image features in the\nreconstructions by using img2img, with outputs from a separate autoencoder. All\ncode is available on GitHub.","upvotes":5,"discussionId":"64756bd8b68461d5cf7ef523","githubRepo":"https://github.com/medarc-ai/fmri-reconstruction-nsd","githubRepoAddedBy":"auto","ai_summary":"MindEye, a novel fMRI-to-image approach, retrieves and reconstructs viewed images with high accuracy using specialized submodules and diffusion prior in latent space.","ai_keywords":["fMRI","contrastive learning","diffusion prior","latent space","CLIP image space","generative models","retrieval","reconstruction","LAION-5B","img2img","autoencoder"],"githubStars":366},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"60a551a34ecc5d054c8ad93e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/60a551a34ecc5d054c8ad93e/dhcBFtwNLcKqqASxniyVw.jpeg","isPro":false,"fullname":"Mishig Davaadorj","user":"mishig","type":"user"},{"_id":"6057b823861b9d53d9c4b8df","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1625184855691-6057b823861b9d53d9c4b8df.jpeg","isPro":false,"fullname":"Tanishq Abraham","user":"tmabraham","type":"user"},{"_id":"6538119803519fddb4a17e10","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6538119803519fddb4a17e10/ffJMkdx-rM7VvLTCM6ri_.jpeg","isPro":false,"fullname":"samusenps","user":"samusenps","type":"user"},{"_id":"62b707fbdd998a8b1e424fc3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62b707fbdd998a8b1e424fc3/DW_YvHw06y69jOUKz9gel.png","isPro":false,"fullname":"Paul Scotti","user":"pscotti","type":"user"},{"_id":"68dff9449ab7f4d313d5d74e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/68dff9449ab7f4d313d5d74e/_TS11DlSrSH29w1c4pkFh.jpeg","isPro":false,"fullname":"Ujjwal Tyagi","user":"Ujjwal-Tyagi","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0}">

arxiv:2305.18274

Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors

Published on May 29, 2023

· Submitted by

AK on May 30, 2023

Upvote

Authors:

Paul S. Scotti ,

Jimmie Goode ,

Stepan Shabalin ,

Aidan J. Dempster ,

Elad Yundler ,

Tanishq Mathew Abraham

Abstract

MindEye, a novel fMRI-to-image approach, retrieves and reconstructs viewed images with high accuracy using specialized submodules and diffusion prior in latent space.

AI-generated summary

We present MindEye, a novel fMRI-to-image approach to retrieve and reconstruct viewed images from brain activity. Our model comprises two parallel submodules that are specialized for retrieval (using contrastive learning) and reconstruction (using a diffusion prior). MindEye can map fMRI brain activity to any high dimensional multimodal latent space, like CLIP image space, enabling image reconstruction using generative models that accept embeddings from this latent space. We comprehensively compare our approach with other existing methods, using both qualitative side-by-side comparisons and quantitative evaluations, and show that MindEye achieves state-of-the-art performance in both reconstruction and retrieval tasks. In particular, MindEye can retrieve the exact original image even among highly similar candidates indicating that its brain embeddings retain fine-grained image-specific information. This allows us to accurately retrieve images even from large-scale databases like LAION-5B. We demonstrate through ablations that MindEye's performance improvements over previous methods result from specialized submodules for retrieval and reconstruction, improved training techniques, and training models with orders of magnitude more parameters. Furthermore, we show that MindEye can better preserve low-level image features in the reconstructions by using img2img, with outputs from a separate autoencoder. All code is available on GitHub.