← Main README · Our Survey
III. Human Interaction Resources
Resources that capture human activity in relation to procedures, objects, scenes, and other people.
## Browse Categories
---
## III.1 Egocentric Procedural Activities
*First-person task recordings for step understanding, anticipation, assistance, and long-horizon reasoning.*
| Resource | Type | Venue | Paper | Paper Page | Website |
|---|:---:|:---:|---|:---:|:---:|
| Coherent4D | Dataset + Benchmark | arXiv 2026 | From Where to How: Continuous 4D Interaction Forecasting from Egocentric Video | [:page_facing_up:](https://arxiv.org/abs/2609.08636 "Paper page") | [:house:](https://corrineqiu.github.io/from-where-to-how/ "Homepage") [:octocat:](https://github.com/CorrineQiu/from-where-to-how "GitHub") |
| EgoSAT | Benchmark | ECCV 2026 | EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understanding | [:page_facing_up:](https://arxiv.org/abs/2606.24422 "Paper page") | [:house:](https://leiyj23.github.io/EgoSAT/ "Homepage") |
| EgoProactive/Pro2Bench | Dataset + Benchmark | arXiv 2026 | Plan, Watch, Recover: A Benchmark and Architectures for Proactive Procedural Assistance | [:page_facing_up:](https://arxiv.org/abs/2606.04970 "Paper page") | [🤗](https://huggingface.co/datasets/facebook/wearable-ai "Hugging Face") |
| EgoPro-Bench | Benchmark | arXiv 2026 | EgoPro-Bench: Benchmarking Personalized Proactive Interaction in Egocentric Video Streams | [:page_facing_up:](https://arxiv.org/abs/2605.07299 "Paper page") | - |
| Minerva-Ego | Benchmark | CVPR 2026 | Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding | [:page_facing_up:](https://openaccess.thecvf.com/content/CVPR2026/html/Nagrani_Minerva-Ego_Spatiotemporal_Hints_for_Egocentric_Video_Understanding_CVPR_2026_paper.html "Paper page") | [:octocat:](https://github.com/google-deepmind/neptune "GitHub") |
| ExAct | Benchmark | NeurIPS 2025 | ExAct: A Video-Language Benchmark for Expert Action Analysis | [:page_facing_up:](https://arxiv.org/abs/2506.06277 "Paper page") | [:octocat:](https://github.com/Texaser/Exact "GitHub") |
| EgoExoBench | Benchmark | NeurIPS 2025 | EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs | [:page_facing_up:](https://proceedings.neurips.cc/paper_files/paper/2025/hash/a974bceb4d1af7d46673d1876482bd4f-Abstract-Datasets_and_Benchmarks_Track.html "Paper page") | [:octocat:](https://github.com/ayiyayi/EgoExoBench "GitHub") |
| EgoLife | Dataset + Benchmark | CVPR 2025 | EgoLife: Towards Egocentric Life Assistant | [:page_facing_up:](https://openaccess.thecvf.com/content/CVPR2025/html/Lin_EgoLife_Towards_Egocentric_Life_Assistant_CVPR_2025_paper.html "Paper page") | [:house:](https://egolife-ai.github.io/ "Homepage") |
| EgoVid-5M | Dataset + Benchmark | NeurIPS 2025 | EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Videos Generation | [:page_facing_up:](https://proceedings.neurips.cc/paper_files/paper/2025/file/3ef2b740cb22dcce67c20989cb3d3fce-Paper-Datasets_and_Benchmarks_Track.pdf "Paper page") | [:house:](https://egovid.github.io "Homepage") |
| HD-EPIC | Dataset + Benchmark | CVPR 2025 | HD-EPIC: A Highly-Detailed Egocentric Video Dataset | [:page_facing_up:](https://openaccess.thecvf.com/content/CVPR2025/html/Damen_HD-EPIC_A_Highly-Detailed_Egocentric_Video_Dataset_CVPR_2025_paper.html "Paper page") | [:house:](http://hd-epic.github.io "Homepage") |
| InterVLA | Dataset + Benchmark | ICCV 2025 | Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions | [:page_facing_up:](https://openaccess.thecvf.com/content/ICCV2025/html/Xu_Perceiving_and_Acting_in_First-Person_A_Dataset_and_Benchmark_for_Egocentric_ICCV_2025_paper.html "Paper page") | [:house:](https://liangxuy.github.io/InterVLA/ "Homepage") |
| Nymeria | Dataset + Benchmark | ECCV 2024 | Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wild | [:page_facing_up:](https://arxiv.org/abs/2406.09905 "Paper page") | [:house:](https://www.projectaria.com/datasets/nymeria/ "Homepage") |
| CaptainCook4D | Dataset | NeurIPS 2024 | CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities | [:page_facing_up:](https://proceedings.neurips.cc/paper_files/paper/2024/hash/7f55e6b053cc5f7d024b111c0a13f604-Abstract-Datasets_and_Benchmarks_Track.html "Paper page") | [:octocat:](https://github.com/CaptainCook4D "GitHub") [:house:](https://captaincook4d.github.io/captain-cook/ "Homepage") |
| Ego-Exo4D | Dataset | CVPR 2024 | Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives | [:page_facing_up:](https://openaccess.thecvf.com/content/CVPR2024/html/Grauman_Ego-Exo4D_Understanding_Skilled_Human_Activity_from_First-_and_Third-Person_Perspectives_CVPR_2024_paper.html "Paper page") | [:house:](https://ego-exo4d-data.org/ "Homepage") |
| EgoExoLearn | Dataset | CVPR 2024 | EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World | [:page_facing_up:](https://openaccess.thecvf.com/content/CVPR2024/html/Li_EgoExoLearn_A_Diverse_Large-Scale_Dataset_for_Egocentric_and_Exocentric_Video_CVPR_2024_paper.html "Paper page") | [:octocat:](https://github.com/OpenGVLab/EgoExoLearn "GitHub") |
| Aria Digital Twin | Dataset + Benchmark | ICCV 2023 | Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perception | [:page_facing_up:](https://openaccess.thecvf.com/content/ICCV2023/html/Pan_Aria_Digital_Twin_A_New_Benchmark_Dataset_for_Egocentric_3D_Machine_ICCV_2023_paper.html "Paper page") | [:house:](https://www.projectaria.com/datasets/adt/ "Homepage") |
| AssemblyHands | Dataset + Benchmark | CVPR 2023 | AssemblyHands: Towards Egocentric Activity Understanding via 3D Hand Pose Estimation | [:page_facing_up:](https://openaccess.thecvf.com/content/CVPR2023/html/Ohkawa_AssemblyHands_Towards_Egocentric_Activity_Understanding_via_3D_Hand_Pose_Estimation_CVPR_2023_paper.html "Paper page") | [:house:](https://assemblyhands.github.io/ "Homepage") |
| HoloAssist | Dataset | ICCV 2023 | HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World | [:page_facing_up:](https://openaccess.thecvf.com/content/ICCV2023/html/Wang_HoloAssist_an_Egocentric_Human_Interaction_Dataset_for_Interactive_AI_Assistants_ICCV_2023_paper.html "Paper page") | [:house:](https://holoassist.github.io/ "Homepage") |
| Assembly101 | Dataset + Benchmark | CVPR 2022 | Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities | [:page_facing_up:](https://openaccess.thecvf.com/content/CVPR2022/html/Sener_Assembly101_A_Large-Scale_Multi-View_Video_Dataset_for_Understanding_Procedural_Activities_CVPR_2022_paper.html "Paper page") | [:house:](https://assembly-101.github.io/ "Homepage") |
| Ego4D | Dataset | CVPR 2022 | Ego4D: Around the World in 3,000 Hours of Egocentric Video | [:page_facing_up:](https://openaccess.thecvf.com/content/CVPR2022/html/Grauman_Ego4D_Around_the_World_in_3_000_Hours_of_Egocentric_Video_CVPR_2022_paper.html "Paper page") | [:house:](https://ego4d-data.org/ "Homepage") |
| EgoTaskQA | Benchmark | NeurIPS 2022 | EgoTaskQA: Understanding Human Tasks in Egocentric Videos | [:page_facing_up:](https://openreview.net/forum?id=ttxAvIQA4i_ "Paper page") | [:house:](https://sites.google.com/view/egotaskqa "Homepage") |
| EPIC-KITCHENS-100 | Benchmark | IJCV 2022 | Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 | [:page_facing_up:](https://link.springer.com/article/10.1007/s11263-021-01531-2 "Paper page") | [:house:](https://epic-kitchens.github.io/ "Homepage") |
Back to top ↑
---
## III.2 Human-Object Interaction Data
*Hands, bodies, objects, contact, and affordance during manipulation and physical interaction.*
| Resource | Type | Venue | Paper | Paper Page | Website |
|---|:---:|:---:|---|:---:|:---:|
| CrossHOI-Bench | Benchmark | CVPR 2026 | CrossHOI-Bench: A Unified Benchmark for HOI Evaluation across Vision-Language Models and HOI-Specific Methods | [:page_facing_up:](https://doi.org/10.48550/arxiv.2508.18753 "Paper page") | [:octocat:](https://github.com/ChelsieLei/CrossHOI-Bench "GitHub") |
| HanDyVQA | Benchmark | CVPR 2026 | HanDyVQA: A Video QA Benchmark for Fine-Grained Hand-Object Interaction Dynamics | [:page_facing_up:](https://doi.org/10.48550/arxiv.2512.00885 "Paper page") | [:house:](https://masatate.github.io/HanDyVQA-project-page/ "Homepage") |
| CORE4D | Dataset | CVPR 2025 | CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object Rearrangement | [:page_facing_up:](https://doi.org/10.1109/cvpr52734.2025.00172 "Paper page") | [:house:](https://core4d.github.io/ "Homepage") |
| GigaHands | Dataset | CVPR 2025 | GigaHands: A Massive Annotated Dataset of Bimanual Hand Activities | [:page_facing_up:](https://doi.org/10.1109/cvpr52734.2025.01627 "Paper page") | [:octocat:](https://github.com/brown-ivl/GigaHands "GitHub") |
| HOIGen-1M | Dataset | CVPR 2025 | HOIGen-1M: A Large-scale Dataset for Human-Object Interaction Video Generation | [:page_facing_up:](https://doi.org/10.1109/cvpr52734.2025.02235 "Paper page") | [:house:](https://liuqi-creat.github.io/HOIGen.github.io/ "Homepage") [🤗](https://huggingface.co/datasets/HOIGen/HOIGen-1M "Hugging Face") |
| HOT3D | Dataset | CVPR 2025 | HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos | [:page_facing_up:](https://doi.org/10.1109/cvpr52734.2025.00662 "Paper page") | [:house:](https://facebookresearch.github.io/hot3d/ "Homepage") |
| HUMOTO | Dataset | ICCV 2025 | HUMOTO: A 4D Dataset of Mocap Human Object Interactions | [:page_facing_up:](https://doi.org/10.1109/iccv51701.2025.01013 "Paper page") | [:octocat:](https://github.com/adobe-research/humoto "GitHub") [:house:](https://jiaxin-lu.github.io/humoto/ "Homepage") |
| ParaHome | Dataset | CVPR 2025 | ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactions | [:page_facing_up:](https://doi.org/10.1109/cvpr52734.2025.00176 "Paper page") | [:house:](https://jlogkim.github.io/parahome/ "Homepage") |
| InterCap | Dataset | IJCV 2024 | InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interaction from Multi-view RGB-D Images | [:page_facing_up:](https://doi.org/10.1007/s11263-024-01984-1 "Paper page") | [:house:](https://intercap.is.tue.mpg.de/ "Homepage") |
| OAKINK2 | Dataset | CVPR 2024 | OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion | [:page_facing_up:](https://doi.org/10.1109/cvpr52733.2024.00050 "Paper page") | [:octocat:](https://github.com/oakink/OakInk2 "GitHub") [:house:](https://oakink.net/ "Homepage") |
| TACO | Benchmark | CVPR 2024 | TACO: Benchmarking Generalizable Bimanual Tool-Action-Object Understanding | [:page_facing_up:](https://doi.org/10.1109/cvpr52733.2024.02054 "Paper page") | [:house:](https://taco2024.github.io/ "Homepage") |
| ARCTIC | Dataset | CVPR 2023 | ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation | [:page_facing_up:](https://doi.org/10.1109/cvpr52729.2023.01244 "Paper page") | [:house:](https://arctic.is.tue.mpg.de/ "Homepage") |
| CHAIRS | Dataset | ICCV 2023 | Full-Body Articulated Human-Object Interaction | [:page_facing_up:](https://doi.org/10.1109/iccv51070.2023.00859 "Paper page") | [:house:](https://eng.bigai.ai/paper/full-body-articulated-human-object-interaction/ "Homepage") |
| BEHAVE | Dataset | CVPR 2022 | BEHAVE: Dataset and Method for Tracking Human Object Interactions | [:page_facing_up:](https://doi.org/10.1109/cvpr52688.2022.01547 "Paper page") | [:house:](https://virtualhumans.mpi-inf.mpg.de/behave/ "Homepage") |
| HOI4D | Dataset | CVPR 2022 | HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction | [:page_facing_up:](https://doi.org/10.1109/cvpr52688.2022.02034 "Paper page") | [:house:](https://hoi4d.github.io/ "Homepage") |
| DexYCB | Benchmark | CVPR 2021 | DexYCB: A Benchmark for Capturing Hand Grasping of Objects | [:page_facing_up:](https://openaccess.thecvf.com/content/CVPR2021/html/Chao_DexYCB_A_Benchmark_for_Capturing_Hand_Grasping_of_Objects_CVPR_2021_paper.html "Paper page") | [:house:](https://dex-ycb.github.io/ "Homepage") |
| H2O | Dataset | ICCV 2021 | H2O: Two Hands Manipulating Objects for First Person Interaction Recognition | [:page_facing_up:](https://doi.org/10.1109/iccv48922.2021.00998 "Paper page") | [:octocat:](https://github.com/taeinkwon/h2odataset "GitHub") [:house:](https://taeinkwon.com/projects/h2o/ "Homepage") |
Back to top ↑
---
## III.3 Human-Scene Interaction Data
*Human occupancy, contact, motion, and functional behavior grounded in three-dimensional environments.*
| Resource | Type | Venue | Paper | Paper Page | Website |
|---|:---:|:---:|---|:---:|:---:|
| EgoHTR | Dataset + Benchmark | CoRL 2026 | EgoHTR: Egocentric 4D Demonstrations of Human Terrain Traversal | [:page_facing_up:](https://arxiv.org/abs/2607.13472 "Paper page") | [:house:](https://egohtr.github.io/ "Homepage") [🤗](https://huggingface.co/datasets/leggedrobotics/egohtr "Hugging Face") |
| TRUMANS | Dataset | CVPR 2024 | Scaling Up Dynamic Human-Scene Interaction Modeling | [:page_facing_up:](https://openaccess.thecvf.com/content/CVPR2024/html/Jiang_Scaling_Up_Dynamic_Human-Scene_Interaction_Modeling_CVPR_2024_paper.html "Paper page") | [:octocat:](https://github.com/jnnan/trumans_utils "GitHub") |
| HUMANISE | Dataset | NeurIPS 2022 | HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes | [:page_facing_up:](https://doi.org/10.48550/arxiv.2210.09729 "Paper page") | [:octocat:](https://github.com/Silverster98/HUMANISE "GitHub") [:house:](https://silverster98.github.io/HUMANISE/ "Homepage") |
| RICH | Dataset | CVPR 2022 | Capturing and Inferring Dense Full-Body Human-Scene Contact | [:page_facing_up:](https://doi.org/10.1109/cvpr52688.2022.01292 "Paper page") | [:octocat:](https://github.com/paulchhuang/rich_toolkit "GitHub") [:house:](https://rich.is.tue.mpg.de/ "Homepage") |
| EgoBody | Dataset | ECCV 2022 | EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices | [:page_facing_up:](https://doi.org/10.1007/978-3-031-20068-7_11 "Paper page") | [:octocat:](https://github.com/sanweiliti/EgoBody "GitHub") [:house:](https://sanweiliti.github.io/egobody/egobody.html "Homepage") |
Back to top ↑
---
## III.4 Social Interaction Data
*Conversational, gestural, physical, and multimodal relations among multiple people.*
| Resource | Type | Venue | Paper | Paper Page | Website |
|---|:---:|:---:|---|:---:|:---:|
| EmotionDialogCN | Dataset | arXiv 2026 | EmotionDialogCN: A Spontaneous Multimodal Dataset for Mandarin Emotional Dialogue | [:page_facing_up:](https://arxiv.org/abs/2608.20905 "Paper page") | - |
| Inter-X++ | Dataset + Benchmark | arXiv 2026 | Inter-X++: A Comprehensive Benchmark for Multimodal Human-Human Interaction Analysis | [:page_facing_up:](https://arxiv.org/abs/2608.20312 "Paper page") | - |
| MPIE-Bench | Benchmark | arXiv 2026 | MPIE-Bench: Benchmarking Anatomically Plausible Multi-Person Interaction Editing | [:page_facing_up:](https://arxiv.org/abs/2607.27616 "Paper page") | [:octocat:](https://github.com/AnnLin0628/mpie-bench "GitHub") |
| SentiAvatar | Dataset | arXiv 2026 | SentiAvatar: Towards Expressive and Interactive Digital Humans | [:page_facing_up:](https://arxiv.org/pdf/2604.02908 "Paper page") | [:house:](https://sentiavatar.github.io/ "Homepage") |
| SpeakerVid-5M | Dataset | ICLR 2026 | SpeakerVid-5M: A Large-Scale High-Quality Dataset for Audio-Visual Dyadic Interactive Human Generation | [:page_facing_up:](https://arxiv.org/abs/2507.09862 "Paper page") | [:octocat:](https://github.com/Dorniwang/SpeakerVid-5M-Code "GitHub") [:house:](https://dorniwang.github.io/SpeakerVid-5M/ "Homepage") |
| Embody 3D | Dataset | arXiv 2025 | Embody 3D: A Large-scale Multimodal Motion and Behavior Dataset | [:page_facing_up:](https://arxiv.org/abs/2510.16258 "Paper page") | [:house:](https://www.meta.com/emerging-tech/codec-avatars/embody-3d/ "Homepage") |
| Seamless Interaction | Dataset | arXiv 2025 | Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset | [:page_facing_up:](https://arxiv.org/abs/2506.22554 "Paper page") | [:octocat:](https://github.com/facebookresearch/seamless_interaction "GitHub") [🤗](https://huggingface.co/datasets/facebook/seamless-interaction "Hugging Face") |
| INFP | Dataset | CVPR 2025 | INFP: Audio-Driven Interactive Head Generation in Dyadic Conversations | [:page_facing_up:](https://doi.org/10.1109/cvpr52734.2025.00997 "Paper page") | [:house:](https://grisoon.github.io/INFP/ "Homepage") |
| InterAct | Dataset | SIGGRAPH 2025 | InterAct: A Large-Scale Dataset of Dynamic, Expressive and Interactive Activities between Two People in Daily Scenarios | [:page_facing_up:](https://doi.org/10.1145/3747871 "Paper page") | [:house:](https://hku-cg.github.io/interact/ "Homepage") |
| SocialGesture | Dataset | CVPR 2025 | SocialGesture: Delving into Multi-person Gesture Understanding | [:page_facing_up:](https://doi.org/10.1109/cvpr52734.2025.01817 "Paper page") | [:house:](https://www.irohxucao.com/SocialGesture/ "Homepage") [🤗](https://huggingface.co/datasets/IrohXu/SocialGesture "Hugging Face") |
| From audio to photoreal embodiment | Dataset | CVPR 2024 | From audio to photoreal embodiment: Synthesizing humans in conversations | [:page_facing_up:](https://doi.org/10.1109/cvpr52733.2024.00101 "Paper page") | [:octocat:](https://github.com/facebookresearch/audio2photoreal "GitHub") [:house:](https://evonneng.github.io/projects/audio2photoreal/index.html "Homepage") |
| Harmony4D | Dataset | NeurIPS 2024 | Harmony4D: A Video Dataset for In-The-Wild Close Human Interactions | [:page_facing_up:](https://doi.org/10.52202/079017-3407 "Paper page") | [:house:](https://jyuntins.github.io/harmony4d/ "Homepage") |
| Inter-X | Dataset | CVPR 2024 | Inter-X: Towards Versatile Human-Human Interaction Analysis | [:page_facing_up:](https://doi.org/10.1109/cvpr52733.2024.02101 "Paper page") | [:house:](https://liangxuy.github.io/inter-x/ "Homepage") |
| EgoHumans | Benchmark | ICCV 2023 | EgoHumans: An Egocentric 3D Multi-Human Benchmark | [:page_facing_up:](https://doi.org/10.1109/iccv51070.2023.01814 "Paper page") | [:octocat:](https://github.com/rawalkhirodkar/egohumans "GitHub") [:house:](https://rawalkhirodkar.github.io/egohumans/ "Homepage") |
| Hi4D | Dataset | CVPR 2023 | Hi4D: 4D Instance Segmentation of Close Human Interaction | [:page_facing_up:](https://doi.org/10.1109/cvpr52729.2023.01632 "Paper page") | [:octocat:](https://github.com/yifeiyin04/Hi4D "GitHub") [:house:](https://yifeiyin04.github.io/Hi4D/ "Homepage") |
| The CANDOR Corpus | Dataset | Science Advances 2023 | The CANDOR Corpus: Insights from a Large Multimodal Dataset of Naturalistic Conversation | [:page_facing_up:](https://doi.org/10.1126/sciadv.adf3197 "Paper page") | [:house:](https://guscooney.com/candor-dataset/ "Homepage") |
| BEAT | Dataset | ECCV 2022 | BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis | [:page_facing_up:](https://doi.org/10.1007/978-3-031-20071-7_36 "Paper page") | [:house:](https://pantomatrix.github.io/BEAT/ "Homepage") |
Back to top ↑
---
← Human Dynamics Resources · All Survey Resources · Human Embodiment Resources →