Technical annotation layers and QA validation.
We deliver structured metadata maps alongside raw video streams. We partition our labeling work into seven explicit abstraction levels.
Metadata Ontology
Seven levels of abstraction.
Annotyx AI structures task demonstrations beyond simple video formats to enable structured learning.
Level 1: Episode Metadata
- Episode ID reference
- Task family definition
- Natural language instructions
- Anonymized contributor alias
- Environment mockup ID
- Camera specifications & sync offsets
- Duration & frame counts
- Clearance status
Level 2: Temporal Actions
- Start and end timestamps (ms)
- Action verb classification
- Active object identifiers
- Active hand labels (left/right/bimanual)
- Subtask outcome (success/failure)
- Annotation confidence score
{
"episode_id": "AAI-SORT-000042",
"start_time_s": 4.20,
"end_time_s": 6.85,
"action": "pick_up",
"active_object": "plastic_bottle",
"active_hand": "right",
"outcome": "success"
}Level 3: Hand-Object Interaction
- Approach phases
- Contact start & release frames
- Grasp type tags (pinch, power, wrap)
- Manipulation force direction tags
- Object state transitions (e.g. open/closed)
Level 4: Computer Vision Labels (Optional)
- 2D/3D hand bounding boxes
- Object bounding box tracks
- Segmentation mask sequences
- 2D hand keypoint coordinates
- Occlusion status checks
Level 5: Spatial Labels (Optional)
- 3D hand pose sequences
- 3D object trajectories
- Static camera pose matrices
- Depth coordinate arrays
- SLAM coordinate alignments
Level 6: Language & Reasoning (Optional)
- Natural language step guides
- Procedural plan layouts
- Subtask logic decompositions
- Failure visual explanation Q&As
- Occlusion reasoning descriptions
Level 7: Outcome & Recovery Logs
- Task success markers
- Partial success indicators
- Failure mode classifications
- Object drops or slip logs
- Recovery trajectory segments
Core Schema
Our standard delivery schema fields.
A project-specific delivery schema can include the following core dictionary fields.
| Field Name | Data Type | Description |
|---|---|---|
| episode_id | string | Unique episode identifier within the project collection. |
| task_instruction | string | The natural-language action instruction given to the contributor. |
| action_label | string | Verbal classification of active segment (e.g. wipe, grasp, sort). |
| start_timestamp_ms | int | Millisecond offset representing start of action segment. |
| end_timestamp_ms | int | Millisecond offset representing end of action segment. |
| active_object | string | Classification name of physical object being manipulated. |
| active_hand | enum | The hand used: left, right, or bimanual. |
| contact_state | boolean | True if physical contact with active object is detected. |
| outcome | enum | Success, failure, or incomplete task status. |
| qa_grade | string | Assigned quality grade: A, B, or C. |
Quality Control
The nine-point human QA checklist.
A delivery workflow can use this manual checklist alongside project-specific consistency and schema checks.
Dataset release grades.
Meets the agreed technical, visibility, task-completion, privacy-review, and temporal-labeling criteria for the intended delivery scope.
Minor lighting fluctuations or brief minor occlusions. Highly usable for broad training datasets, benchmarking, and representation learning.
Noticeable quality fluctuations, partial camera views, or significant hand occlusion. Usable only for target procedural understanding, task planning, or subtask decomposition pretraining.
Fails basic protocol guidelines, features unredacted privacy fields, lacks visible start/end states, or has damaged frames. Permanently excluded from delivery packages.
Interested in custom label mapping?
We work with perception teams to integrate specialized bounding boxes, keypoints, or custom segment parameters into the delivery card.