Egocentric vs Exocentric Datasets

A Complete Guide to First- and Third-Person AI Training Data

Published on Jul 15, 2026
Back to Blogs
Egocentric and exocentric views of the same moment, side by side

Picture someone repairing a bicycle. Now picture two cameras recording it. One is strapped to the mechanic's head and sees exactly what they see: their hands, the chain, the tool, the grease. The other sits on a tripod across the workshop and sees the whole person, the bike, the bench and the room.

Same repair. Same instant. Two completely different recordings, each capturing something the other misses. That difference is the whole subject of this article.

The first camera produces egocentric data. The second produces exocentric data. As AI moves from labelling photographs to acting in the physical world, the demand for both has grown sharply, and so has the difficulty of collecting them well.

The Definition, Precisely

The distinction rests on one thing: where the camera is.

  • Egocentric (first-person) data is captured from a sensor attached to, worn by, or embedded within the agent. The camera moves as the actor moves. Think a GoPro on a helmet, smart glasses, an AR headset, or a camera mounted on a robot.
  • Exocentric (third-person) data is captured from a sensor positioned outside the agent, observing them. Think a CCTV camera, a tripod in a studio, or a broadcast camera at a match.

The terms come from cognitive science, where they describe two frames of reference the human brain uses: one anchored to your body, one anchored to the world. Both matter, and you switch between them constantly without noticing.

What Egocentric Captures Well

  • Hands, tools and the object being manipulated, in close detail
  • Attention and intent, especially when eye gaze is recorded alongside
  • The scene changes that follow directly from an action
  • Head motion and body dynamics, via IMU
And what makes it hard

Constant motion blur. Objects leaving frame the moment the head turns. Severe self-occlusion, because the hand doing the work hides the thing being worked on. And no stable global frame of reference, since the camera never stops moving.

What Exocentric Captures Well

  • Full-body pose and whole-body motion
  • Workspace geometry and the actor's position within it
  • Interactions between multiple people
  • Objects that the actor's own body would occlude
And what makes it hard

The fine detail of a grasp may be a handful of pixels. Intent is invisible, because you cannot see where the person is looking. And a fixed camera has fixed blind spots that the actor will inevitably turn their back on.

Why You Want Both, Captured Together

Read those two lists again and the pattern is hard to miss: each viewpoint's weakness is the other's strength. Ego sees the grasp but not the body. Exo sees the body but not the grasp. Ego knows intent but not position. Exo knows position but not intent.

This is why the most valuable ego/exo datasets are not two separate collections stapled together. They are simultaneous, time-synced captures of the same event. That synchronisation is the entire point, and it is what makes the data expensive.

When every stream shares one clock, any frame in the ego view maps to the exact same instant in every exo view. A model can then learn that this blurry close-up of a hand and that distant figure across the room are the same action, seen twice. Learn that correspondence across enough examples and the model starts to build a representation of the action that is independent of where the camera was standing.

A time-synced capture rig with one ego camera and four exo cameras sharing a single clock

That viewpoint-invariance is not an academic nicety. It is the thing that lets a robot watch a human do something and then do it itself, despite the fact that its cameras are nowhere near where the human's eyes were.

Why This Data Is Collected, and How It Gets Used

For most of computer vision's history, training data was exocentric almost by default, because that is how cameras were pointed: at things, from outside. The shift toward egocentric collection tracks a shift in what we are asking AI to do. A model that captions photographs can sit outside the scene. A model that is meant to act in the world needs to know what acting in the world looks like from the inside.

Four things this data is actually used for:

  • Teaching robots from human demonstration. Rather than hand-programming a manipulation task, you record humans doing it and let the policy learn. Ego data is the closest available proxy for what a robot's own sensors will see.
  • Building assistants that understand context. An AR assistant that can say "you have skipped a step" must first understand what the wearer is looking at and what they just did. That capability is trained on first-person footage of the same task done correctly and incorrectly.
  • Assessing skill. Comparing an expert and a novice performing the same task is a natural fit for paired viewpoints. The ego view captures technique; the exo view captures posture and timing. Ego-Exo4D leans into this deliberately, pairing its video with expert commentary from coaches and teachers.
  • Cross-view transfer. Exocentric video is abundant, since decades of it already exist. Egocentric video is scarce and costly. A live research thread is learning to transfer knowledge from plentiful exo data to ego tasks, which makes paired data valuable well beyond its own hours.

The Landmark Public Datasets

A handful of research datasets define the field, and they are worth knowing even if you end up collecting your own.

Dataset Scale What makes it notable
Ego4D 3,025 hours, 855 camera wearers, 74 locations, 9 countries The breadth benchmark for first-person daily-life activity. Egocentric only.
Ego-Exo4D 1,286 hours, 740 participants, 13 cities, 123 scenes Simultaneous, time-synced ego and exo video of skilled activity. Adds multichannel audio, eye gaze, 3D point clouds, camera poses, IMU and expert commentary. Released Dec 2023.
EPIC-KITCHENS Hundreds of hours of unscripted kitchen activity The long-running fine-grained benchmark for hand-object interaction, with a large verb and noun taxonomy.
Project Aria Research glasses platform Not a dataset but the capture hardware behind several of them, recording gaze, IMU and spatial data alongside video.
One caveat worth taking seriously

These datasets exist because participants consented to a specific, reviewed research protocol. A public dataset is not commercial training supply unless the source terms, the consent basis and the downstream licence all permit that use. Check the licence before the data reaches a production model, not after. When the terms do not cover your case, or when your domain simply is not in public data, commissioned collection is the route.

Where Ego and Exo Datasets Get Used

Robotics and Embodied AI

Imitation learning from human demonstration, manipulation policies, and navigation. The fastest-growing consumer of ego data by some distance.

AR / VR Assistants

Context-aware guidance on headsets that already capture first-person data as a by-product of being worn. Step tracking, object recall, hands-free help.

Skill Assessment and Training

Proficiency scoring, expert-versus-novice comparison, and automated coaching. Paired viewpoints are close to ideal here.

Industrial Safety and Ergonomics

Procedure compliance, near-miss analysis and posture assessment on the line. Ego shows the task; exo shows the body doing it.

Healthcare Procedures

Surgical and clinical technique capture for training and review, where the operator's own viewpoint carries most of the information.

Sports Analytics

Technique breakdown and performance measurement, combining the athlete's own view with the broadcast or sideline view.

Six use cases for ego and exo centric datasets

Why Collecting This Data Is Genuinely Hard

Everything above describes data that is valuable. What it does not convey is how much of an ego/exo project is spent not on the interesting parts, but on logistics. The failure modes are consistent enough to list.

  • Configuration drift. Fifty collectors, fifty phones, fifty chances for someone to record at the wrong resolution or sampling rate. You usually find out weeks later, when the data lands and half of it does not match the spec.
  • Consent at scale. Egocentric capture records the wearer's home, workplace and the people around them. Consent cannot be a folder of scanned PDFs that someone has to reconcile against filenames later. It has to be captured at the moment of recording and bound to the file.
  • Metadata inconsistency. If collectors type metadata by hand, you get typos, blanks and creative interpretations of what a field means. Metadata is what makes a dataset queryable; inconsistent metadata quietly reduces its value to near zero.
  • Late quality control. QC that runs after collection ends is QC that finds problems when the participants have gone home and the location is no longer available. Re-collection is the single most expensive line item on a data project.
  • Data residency. Footage of people's homes and faces, collected across borders, lands squarely inside local data protection law. Where the bytes physically sit is not a detail you can settle afterwards.
  • Connectivity. The most interesting collection environments (factories, remote locations, moving vehicles) tend to be the ones with the worst network coverage.

None of these are research problems. They are tooling problems, and they are the reason a collection project that looks straightforward on paper takes three times as long as planned.

AIDAC: The AI Data Collection Platform

AIDAC is Haidata's AI data collection platform: a mobile app for Android and iOS paired with a web dashboard. It handles collection of Audio, Video, Image, Text, PDF and Email files at scale.

AIDAC is not an ego/exo-specific product, and it is worth being straight about that. What it is, is a platform built around exactly the six problems listed above, because those problems are not unique to ego/exo work. They show up on every data collection project that involves more than a handful of people. An ego/exo project just hits them harder.

The AIDAC collection workflow from dashboard configuration through to delivery in your own cloud storage

Mapping each problem to what AIDAC actually does:

  • Configuration drift becomes impossible. Image resolution, audio sampling rate, bits per sample and channel count are all set from the dashboard, not on the device. Collectors never touch them. They open the app and record; the spec is already enforced.
  • Consent is generated and bound in-app. Participant name and signature are captured on the device at the time of recording, and the consent form is generated automatically against that session.
  • Metadata is generated, not typed. File name and size, phone model, OS version and the media properties are captured automatically. Collectors only fill the custom fields you defined in the dashboard, which removes most of the surface area for human error.
  • QC runs live, while collection is still happening. You configure how many QC levels you want, who owns each one (your team, the vendor, or your own org), and what percentage gets reviewed at each level. Problems surface while the participant is still in the room.
  • The data goes to your storage, in your geography. Files upload directly from the app to your own cloud storage, wherever local regulation requires it to live, and are permanently deleted from the phone afterwards. There is also a live audit report so you can see upload and QC status as it happens.
  • Offline mode handles bad connectivity. The app pre-fetches project details, works with no internet at all, and uploads when the connection returns.

For the audio side of a multimodal capture, AIDAC also records multi-party conference calls in WAV with up to four participants, and can save each participant's voice to a separate file, which is a way of solving speaker diarization at source rather than trying to unpick a mixed recording later.

The underlying idea is simple: put the configuration in the dashboard and the enforcement in the tooling, so that scaling from five collectors to five hundred does not scale your quality problems with it.

Annotating Ego and Exo Datasets

Collected footage is raw material. What makes it a training set is the annotation, and ego/exo video is among the more demanding things to annotate well.

Annotation layers on an egocentric frame including hand boxes, object tracking, gaze and temporal segmentation

Haidata's annotation teams work across the layers this data needs:

  • Temporal action segmentation. Marking the start and end of each action, including the atomic steps and the point-of-no-return moments that fine-grained benchmarks depend on.
  • Hand-object interaction. Contact state, grasp type and manipulation phase. This is the layer that carries most of the signal in first-person video, and the layer where annotator judgement matters most.
  • Object detection and tracking. Bounding boxes on manipulated objects, held through the occlusion and motion blur that egocentric footage produces constantly.
  • Natural language narration. Describing what the person is doing, and where useful, why. Narration quality is what separates a dataset that supports language-conditioned models from one that does not.
  • Ego-exo correspondence. Linking the same object and the same action across the two views. This layer exists only in paired data and it is the one that makes paired data worth collecting.
  • Gaze and pose. Where recorded, aligning eye gaze and body pose with the visual annotation so the layers agree with each other.

Two things make this work harder than standard video annotation. First, taxonomies get large fast, running to hundreds of verbs and thousands of nouns in datasets like Ego4D, which puts real weight on annotator training and guideline discipline. Second, annotations that are too coarse are close to useless: a label like "moves both hands" is technically correct and tells a model almost nothing. Fine-grained hand-object dynamics are where the value is, and getting there is a function of how well the annotation team understands the domain.

Haidata has been doing annotation work since 2020 from the Nilgiris Hills in India, and the same teams that handle human-in-the-loop and medical annotation work take on ego/exo projects. If you are collecting with AIDAC, collection and annotation run as one pipeline rather than two vendors and a handoff.

Frequently Asked Questions

Egocentric data is captured from a camera attached to or worn by the actor, so it moves with them and shows what they see, including their own hands and tools. Exocentric data is captured from a camera positioned outside the actor, such as a tripod or ceiling mount, and shows the whole body, the workspace and the surrounding scene. The simplest test is where the camera is: on the actor is ego, off the actor is exo.
Each view answers a question the other cannot. The egocentric view shows fine hand and object detail, intent and gaze, but has no global frame of reference and suffers heavy motion and occlusion. The exocentric view gives full-body pose and spatial context, but the fine detail of a grasp may be tiny or hidden. Captured together and time-synced, they let a model learn a viewpoint-invariant representation of the same action, which is what makes it possible to transfer a human demonstration to a robot that sees the world differently.
Ego4D is a large egocentric-only dataset of daily-life activity, with 3,025 hours of video from 855 camera wearers across 74 locations in 9 countries. Ego-Exo4D, released in December 2023, is a multiview dataset built around skilled activities such as sports, music, dance and bike repair. Its defining feature is that egocentric and exocentric video are captured simultaneously and time-synced, with 1,286 hours from 740 participants in 13 cities across 123 natural scenes, plus multichannel audio, eye gaze, 3D point clouds, camera poses, IMU and expert commentary from coaches and teachers. In short, Ego4D is first-person breadth and Ego-Exo4D is paired-viewpoint depth.
Not automatically. Public research datasets such as Ego4D, Ego-Exo4D and EPIC-KITCHENS are released under licences and consent bases that are usually framed for research use. A public dataset should not be treated as commercial training supply unless the source terms, the participant consent basis and the downstream licence all permit that use. Check each dataset's licence before it reaches a production model, and commission your own collection when the terms do not cover your case or when your domain is not represented in public data.
It depends on what your model needs. Research rigs such as Project Aria capture gaze, IMU and spatial data alongside video, and AR headsets like Quest or HoloLens capture first-person data as a by-product of normal use. But a large share of useful egocentric data can be collected with a head or chest mounted smartphone, which is far cheaper and easier to scale across many participants and locations. Start from the labels your model needs and work backwards to the minimum sensor set, rather than starting from the most capable rig.
Egocentric capture is uniquely sensitive because the camera records the wearer's home, workplace and the people around them. Established datasets handle this with formal independent review of the collection protocol, explicit informed consent from participants, de-identification where bystanders may appear, and a licence prescribing proper use. Ego-Exo4D reduced the problem partly by recording in closed environments with no passers-by, which meant nearly all of its video needed no de-identification. In practice you need consent captured and bound to each recording, control over where the data is stored, and a protocol reviewed before collection starts, not after.
The common layers are temporal action segmentation marking the start and end of each action, object detection and tracking on manipulated objects, hand-object interaction labels covering contact state, grasp type and manipulation phase, natural language narrations describing what the person is doing, and spatial annotations such as gaze, depth and body pose. Paired ego-exo data adds a layer of its own: ego-exo correspondence, which links the same object or action across the two views. Taxonomies get large, with hundreds of verbs and thousands of nouns in datasets like Ego4D.
It varies with scenario count, participant recruitment and how many viewpoints you capture, so timelines are best set against a specific brief rather than a rule of thumb. The pattern worth knowing is where the time actually goes: pilots are quick, but scaling to many collectors and locations is where projects slow down, because configuration drift, inconsistent metadata and quality problems found late force re-collection. Fixing those at the tooling level compresses the schedule far more than adding collectors does. Write to info@haidata.ai with your scenario list and target hours for an estimate.

Closing Thought

Ego and exo centric data is not a new idea, but the reason to collect it has changed. We are no longer building models that describe the world from a distance. We are building models that have to operate in it, and that means teaching them what a task looks like from the inside of it, and from the outside, at the same instant.

The research is moving quickly. The hard part, as usual, is not the research. It is getting a thousand hours of consistent, consented, well-labelled footage out of the real world and into a form a model can learn from.

For more information on how AIDAC and Haidata's annotation teams could help with your next ego/exo data collection project, please write to info@haidata.ai
Back to Blogs