The MCAP Dilemma Because Your Robot Data Has Three Jobs
MCAP is not a bad training format. It is a recording format, and most of these arguments are two people describing different layers of the same stack.
I have some version of this argument about once a month. Someone tells me MCAP is the wrong way to store robotics data for foundation model training, usually with the air of correcting a common mistake.
They’re right, and they’re arguing past the person they’re arguing with.
Robot data gets asked to do three unrelated things. It has to survive being written on a machine that might lose power mid-session. It has to sit in cold storage long enough that the message definitions get refactored twice and still decode. And it has to keep eight GPUs busy with shuffled batches pulled out of object storage. Append-only and self-describing for the first. Immutable and cheap for the second. Columnar, sharded, randomly addressable for the third.
MCAP nails one of those, is defensible on another, and is the wrong shape for the last. Which one someone has in mind is almost never stated, and that’s the whole fight.
The part nobody actually disputes
Open an .mcap file and the layout announces its purpose. Magic bytes, header, then a data section of chunks, each holding a batch of messages under zstd or lz4, with a message index record written right after each chunk that maps timestamps to offsets inside it. Summary section at the end with the chunk index and statistics, then a summary offset section, footer, magic bytes again.
All of that is append-only. The writer never seeks backward. So when a robot browns out mid-recording, and it will, you lose the tail and nothing else; a reader scans forward through chunks and recovers every message up to the instant the write stopped. Formats that assemble their structure in memory and serialise on close hand you a zero-byte file instead.
People underrate the index. Chunk indexes and statistics get written last, on clean close, which buys you seeks by time and topic without scanning the file. Only on clean close, though. rosbag2_storage_mcap ships a fastwrite preset that skips index writing to cut recording overhead, and its own docs warn the output isn’t suitable for long-term storage: no index, no topic-subset reads, no seeking. Worth checking whether anyone on your team turned that on to save CPU and whether anything downstream quietly assumes it can seek.
Schemas live inside the file. That reads like a footnote and it’s the most valuable property MCAP has, because data outlives code by years. Whoever wrote MyCustomState.msg has left, the definition changed twice since, and the 2023 logs still decode.
It’s also heterogeneous. ROS 2 CDR, Protobuf, FlatBuffers, JSON, one timeline, one seek. Anyone who has tried to align five sensor streams sitting in five files against five clocks knows the value of that without being told.
ROS 2 made it the default bag format, NVIDIA Isaac ships it as the default log format, and writing your own binary capture format in 2026 is solving a problem that closed.
Flip the query
A dataloader never asks for every topic at instant t. It asks for one field, from a million random instants, ordered by a shuffle buffer, streamed out of S3, across four hundred workers.
Different question, different disk layout, and no amount of engineering makes one file answer both.
The mechanism deserves precision, because “row-oriented is slow” gets repeated by people who couldn’t say why.
Chunk-granularity compression is the culprit, and it’s also the feature. You’re not compressing per message, you’re amortising over a block, which is exactly what makes MCAP cheap to write. But zstd hands you the whole block or nothing. Your chunk holds six camera topics, lidar, IMU, CAN frames, and the 100 Hz control vector you actually wanted, and you decompress all of it to get the last one. Rough arithmetic on a modest rig: total log bitrate around 10 MB/s, control maybe 0.1 MB/s of it. Roughly a hundred bytes decompressed per byte used. Columnar layouts land somewhere near 1–3× on the same data because column chunks and dictionary encoding let you skip what you never read.
The index is optimised for the wrong axis, too. It answers “where is topic X at time T in this file.” Training asks for a random sample across forty thousand shards. Unrelated lookups.
Then there’s object storage, which punishes small random reads in a way that surprises people the first time. First-byte latency in the tens of milliseconds, per-request overhead dominating anything under a few megabytes. Training formats shard at roughly 100 MB to 1 GB specifically to amortise that: one request, then stream. WebDataset’s entire design is sequential reads inside a shard and randomness between shards, which approximates a global shuffle through shard ordering plus an in-memory buffer without ever issuing a small random GET. Point a naive reader at MCAP files in a bucket and the request overhead will find you before the decompression overhead does.
And the one that actually shows up in your utilisation graph, which almost nobody talks about: multi-rate alignment gets paid every epoch instead of once. Cameras at 30 Hz, proprioception at 200 Hz or 1 kHz, control at whatever the loop runs, GNSS at 10 Hz. Some of those timestamps are software-stamped at receipt rather than hardware-triggered or PTP-disciplined, if you’re honest about your stack. A log preserves that mess faithfully, which is correct; it’s the raw truth. A training format resamples and aligns it into fixed tensors once, at transcode time. Do that interpolation inside the dataloader instead and you burn it again on every epoch of every run, in Python, on the critical path.
What the field ships
If training off the log were workable, someone would be doing it.
Open X-Embodiment, DROID, LeRobot, Waymo, NVIDIA’s own robotics stack: five efforts, five different training formats, no MCAP. Every one has a conversion step, and several throw away fidelity on purpose. DROID’s RLDS training copy is downscaled, left-camera only, motor torques dropped, 1.7 TB sitting beside an 8.7 TB raw MP4 archive. Somebody decided a batch shouldn’t carry bytes the model never looks at.
NVIDIA’s version convinces me most. Isaac uses MCAP as its default logging format and emits LeRobot- and RLDS-compatible data for training. Same org, same stack, line drawn between layers without any fuss about it.
Worth admitting the training layer is unfinished rather than glossing over it. openpi falls back to RLDS for full DROID training because LeRobot’s format wasn’t scalable enough at that size, by Physical Intelligence’s own account. LeRobot v3 concatenates episodes per file to stop drowning in small files. Lance exists because Parquet’s random access is mediocre. Nobody has settled this. What is settled is that it’s a separate layer from the log.
The video sub-argument
Someone always says just use encoded video instead.
MCAP is a container, H.264 is a codec, and they aren’t alternatives. Encoded video goes inside MCAP.
The real argument underneath is older and correct: don’t put raw frames in a bag. 1920×1080 RGB8 at 30 fps runs about 6.2 MB a frame, 186 MB/s per camera. Six cameras clears a gigabyte per second, four terabytes an hour, per vehicle. The same footage at 10 Mbit/s H.264 is 1.25 MB/s. Call it 150× per camera. Teams who learned that in 2021 have the invoices.
The fix was never abandoning the container. It was CompressedVideo message types carrying H.264/H.265/VP9/AV1 frames inside MCAP, so you keep the codec’s compression and the cross-sensor alignment together. Writing sensor_msgs/Image to disk at fleet scale is the bug.
Bare .mp4 with a timestamp sidecar does genuinely win in one configuration: single camera, no other sensors, browser-first review UI. MCAP’s synchronisation is worth nothing there and you’re paying for a feature you don’t use. Though I’d note that describes a webcam more than a robot.
One constraint follows you into any container. Seek cost on video is governed by GOP structure, not by whatever wraps it — you can’t decode an arbitrary frame without walking back to the last IDR, so long GOPs save storage and make scrubbing miserable. Teams that need frame-accurate replay force short keyframe intervals and eat the bitrate. That’s an encoder decision, and it wants making before you’ve recorded a petabyte you can’t scrub.
The pipeline
Encode at the source, in hardware, before anything touches the bus. Record MCAP at the edge. Keep raw MCAP as the immutable cold tier, because being able to re-derive a dataset three years out beats the storage line item, particularly the first time a model does something inexplicable and you need the unmodified bytes.
Then curate before you transcode. Teams skip this step and it’s the expensive one. Egress and compute for a full-corpus transcode can rival what you pay to store the corpus: Robo-DM’s authors put an 8.9 TB Open-X slice at roughly $172/month on GCP against $172–$1,540 for a single full download. Index the MCAP, search it, work out which fraction deserves to go in front of a model, transcode that.
(For most fleets the fraction worth training on is small, and nobody enjoys saying so in the data meeting.)
Pick the training format from your model stack rather than from a blog post. WebDataset for streaming large image or video corpora sequentially. Parquet plus MP4 inside the LeRobot ecosystem. RLDS where you need Open X-Embodiment compatibility. Lance when random access and dataset versioning dominate.
Keep lineage from shard back to source file. The day someone asks what data trained a given checkpoint, and in anything safety-relevant someone will, you either have the answer or you have an incident.
What I’d measure instead
Read amplification first: bytes decompressed and read per byte the model consumes. Near 1 and you can stop reading. At 50× the transcode step has already paid for itself.
Then data-stall percentage, the wall-clock fraction where GPUs sit idle waiting on input. It’s the only number that decides whether your storage layer is a real problem, and most teams never instrument it. ⟨YOUR NUMBER GOES HERE — what yours was before and after. This is the single line that makes the post yours rather than a literature review.⟩
Time-to-first-batch on a cold cache is the third, and I care about it more than the other two combined. How long from wanting to train on a slice to the first gradient step. Iteration speed lives there, and it’s dominated by curation and transcode rather than by anything in your model code.
Everyone in Physical AI agrees data operations are the moat now. Fine. A moat is a set of unglamorous decisions about where bytes live and how fast you can reshape them into a batch. The advantage doesn’t go to whoever collects the most; it goes to whoever has the shortest path from a robot did something interesting to that experience is in a training batch. That path runs through the transcode step nobody wants to own.
So, is MCAP a bad format for foundation model training data? It’s the best format for recording robot data and the wrong shape for training on it, and those two claims were never in tension.
Whether that holds is the part I’m less sure about. Foxglove is pushing MCAP-native search at petabyte scale, and if curation moves into the log format itself the transcode step gets narrower and later. It doesn’t vanish, since the dataloader still wants columns and shards and physics doesn’t negotiate. But the boundary moves, and I don’t have a confident read on where it lands.