From Handheld Demos to Searchable Training Data: Annotating TRumi Episodes with DecisionFacts and Foxglove
Collecting handheld demonstrations is only half the job. The other half is robotics data annotation: turning hours of raw episodes into multimodal training data a team can actually search, review, and train on.
Our partners at DecisionFacts just showed what that second half can look like. They built a pipeline that takes episodes captured on TRumi, Trossen Robotics' handheld data collection device, and turns them into annotated, synchronized files that open directly in Foxglove. Type an instruction like "manipulate the blue tape rectangle," and in a few clicks you are looking at the camera feeds, gripper state, IMU signals, and end-effector path for that exact moment.
How the DecisionFacts annotation pipeline works
The pipeline takes a raw TRumi episode and produces a single, model-ready view of what happened. It does four things:
Annotates each episode with the natural-language instruction behind it and the subtasks inside it.
Links that language to the sensor data, tying each instruction to the matching camera views and telemetry.
Maps actions to end-effector trajectories, so every step of the task has a position and orientation path attached.
Exports files that open directly in Foxglove, with no custom viewer or glue code needed.
The result is that the instruction becomes the index. Instead of scrubbing through disconnected video files and time series to find a moment, you start from what the operator was trying to do and jump straight to it.
Multimodal robotics data in Foxglove: every signal, in sync
The video above walks through a single annotated episode in Foxglove. Everything sits on one shared timeline, so scrubbing to any moment updates every panel at once:
Two wide-angle camera feeds from the TRumi grippers, with annotation overlays drawn on the frames
The episode instruction and subtasks, broken out per gripper
A task and gripper-event timeline showing when each subtask starts and when the gripper opens or closes
Gripper width over time for each device
IMU linear acceleration on all three axes
End-effector position and orientation
An XY path plot tracing where the end effector traveled across the workspace
In the demo task, the operator works around a tabletop marked with blue tape and fiducial markers, picking up and moving a water bottle. Watching the camera view, the gripper width, and the XY path move together makes it easy to see not just what the operator did, but how.
Human-in-the-loop review for annotation quality
Automated annotation is only useful if you know when to trust it. The pipeline attaches a confidence score to each annotation, and low-confidence ones are flagged for a person to check. In the demo episode, the instruction carries a confidence of 0.3, so the episode is marked needs_review = true.
That flag matters. It lets a team spend review time where it counts instead of re-checking every episode by hand, and it keeps weak labels out of a training set until someone signs off.
DecisionFacts describes the whole process as a loop: annotate, correlate, visualize, understand, improve. Each pass through Foxglove surfaces something about the data, and that feeds back into better annotations and better collection on the next round.
Why searchable episode data matters for imitation learning and VLA training
Handheld, UMI-style collection has made gathering demonstrations far cheaper and faster than teleoperating a full robot. TRumi was built for exactly that: put a gripper in a person's hand, record what they do, and capture the cameras and telemetry along the way.
But more episodes also means more data to organize. Teams training vision-language-action models and other imitation learning policies tend to hit the same three pain points:
Organizing raw episodes so they can be found again
Grounding language annotations in actions, so an instruction lines up with what the gripper actually did
Adding the context models need to train reliably, without hand-labeling every frame
The DecisionFacts pipeline addresses all three on data captured with TRumi. It is a good example of what the ecosystem around a collection device should look like: capture with TRumi, structure with DecisionFacts, explore with Foxglove.
Frequently asked questions
What is TRumi?
TRumi is Trossen Robotics' handheld data collection device. An operator holds the gripper and performs a task by hand, while TRumi records camera views and telemetry for use in robot learning.
What is UMI-style data collection?
UMI-style collection uses a handheld gripper instead of a full robot to record demonstrations. It is faster and cheaper than teleoperation, which makes it practical to gather large imitation learning datasets.
What does DecisionFacts do with TRumi data?
DecisionFacts annotates TRumi episodes with language instructions and subtasks, links them to camera and telemetry data, maps actions to end-effector trajectories, and exports the result in a format that opens in Foxglove.
Can I view TRumi episodes in Foxglove?
Yes. Episodes processed by the DecisionFacts pipeline open directly in Foxglove, with camera feeds, gripper state, IMU signals, and end-effector paths synchronized on one timeline.
How are low-confidence annotations handled?
Each annotation gets a confidence score. Low-confidence annotations are flagged for human review, so teams can check uncertain labels before they reach a training set.
Get started
If you are collecting manipulation data and want it searchable from day one, TRumi and DecisionFacts are worth a look together.
Learn more about TRumi: TRumi product page
Talk to DecisionFacts about teleop and UMI training data in model-ready formats: info@decisionfacts.io
Comments