They’re right, and they’re arguing past the person they’re arguing with.

Robot data gets asked to do three unrelated things. It has to survive being written on a machine that might lose power mid-session. It has to sit in cold storage long enough that the message definitions get refactored twice and still decode. And it has to keep eight GPUs busy with shuffled batches pulled out of object storage. Append-only and self-describing for the first. Immutable and cheap for the second. Columnar, sharded, randomly addressable for the third.

MCAP nails one of those, is defensible on another, and is the wrong shape for the last. Which one someone has in mind is almost never stated, and that’s the whole fight.

The part nobody actually disputes

Open an .mcap file and the layout announces its purpose. Magic bytes, header, then a data section of chunks, each holding a batch of messages under zstd or lz4, with a message index record written right after each chunk that maps timestamps to offsets inside it. Summary section at the end with the chunk index and statistics, then a summary offset section, footer, magic bytes again.

All of that is append-only. The writer never seeks backward. So when a robot browns out mid-recording, and it will, you lose the tail and nothing else; a reader scans forward through chunks and recovers every message up to the instant the write stopped. Formats that assemble their structure in memory and serialise on close hand you a zero-byte file instead.

People underrate the index. Chunk indexes and statistics get written last, on clean close, which buys you seeks by time and topic without scanning the file. Only on clean close, though. rosbag2_storage_mcap ships a fastwrite preset that skips index writing to cut recording overhead, and its own docs warn the output isn’t suitable for long-term storage: no index, no topic-subset reads, no seeking. Worth checking whether anyone on your team turned that on to save CPU and whether anything downstream quietly assumes it can seek.

Schemas live inside the file. That reads like a footnote and it’s the most valuable property MCAP has, because data outlives code by years. Whoever wrote MyCustomState.msg has left, the definition changed twice since, and the 2023 logs still decode.

It’s also heterogeneous. ROS 2 CDR, Protobuf, FlatBuffers, JSON, one timeline, one seek. Anyone who has tried to align five sensor streams sitting in five files against five clocks knows the value of that without being told.

ROS 2 made it the default bag format, NVIDIA Isaac ships it as the default log format, and writing your own binary capture format in 2026 is solving a problem that closed.

Flip the query

A dataloader never asks for every topic at instant t. It asks for one field, from a million random instants, ordered by a shuffle buffer, streamed out of S3, across four hundred workers.

Different question, different disk layout, and no amount of engineering makes one file answer both.

The mechanism deserves precision, because “row-oriented is slow” gets repeated by people who couldn’t say why.

Chunk-granularity compression is the culprit, and it’s also the feature. You’re not compressing per message, you’re amortising over a block, which is exactly what makes MCAP cheap to write. But zstd hands you the whole block or nothing. Your chunk holds six camera topics, lidar, IMU, CAN frames, and the 100 Hz control vector you actually wanted, and you decompress all of it to get the last one. Rough arithmetic on a modest rig: total log bitrate around 10 MB/s, control maybe 0.1 MB/s of it. Roughly a hundred bytes decompressed per byte used. Columnar layouts land somewhere near 1–3× on the same data because column chunks and dictionary encoding let you skip what you never read.

The index is optimised for the wrong axis, too. It answers “where is topic X at time T in this file.” Training asks for a random sample across forty thousand shards. Unrelated lookups.

Then there’s object storage, which punishes small random reads in a way that surprises people the first time. First-byte latency in the tens of milliseconds, per-request overhead dominating anything under a few megabytes. Training formats shard at roughly 100 MB to 1 GB specifically to amortise that: one request, then stream. WebDataset’s entire design is sequential reads inside a shard and randomness between shards, which approximates a global shuffle through shard ordering plus an in-memory buffer without ever issuing a small random GET. Point a naive reader at MCAP files in a bucket and the request overhead will find you before the decompression overhead does.

And the one that actually shows up in your utilisation graph, which almost nobody talks about: multi-rate alignment gets paid every epoch instead of once. Cameras at 30 Hz, proprioception at 200 Hz or 1 kHz, control at whatever the loop runs, GNSS at 10 Hz. Some of those timestamps are software-stamped at receipt rather than hardware-triggered or PTP-disciplined, if you’re honest about your stack. A log preserves that mess faithfully, which is correct; it’s the raw truth. A training format resamples and aligns it into fixed tensors once, at transcode time. Do that interpolation inside the dataloader instead and you burn it again on every epoch of every run, in Python, on the critical path.

What the field ships

If training off the log were workable, someone would be doing it.

Open X-Embodiment, DROID, LeRobot, Waymo, NVIDIA’s own robotics stack: five efforts, five different training formats, no MCAP. Every one has a conversion step, and several throw away fidelity on purpose. DROID’s RLDS training copy is downscaled, left-camera only, motor torques dropped, 1.7 TB sitting beside an 8.7 TB raw MP4 archive. Somebody decided a batch shouldn’t carry bytes the model never looks at.

NVIDIA’s version convinces me most. Isaac uses MCAP as its default logging format and emits LeRobot- and RLDS-compatible data for training. Same org, same stack, line drawn between layers without any fuss about it.

Worth admitting the training layer is unfinished rather than glossing over it. openpi falls back to RLDS for full DROID training because LeRobot’s format wasn’t scalable enough at that size, by Physical Intelligence’s own account. LeRobot v3 concatenates episodes per file to stop drowning in small files. Lance exists because Parquet’s random access is mediocre. Nobody has settled this. What is settled is that it’s a separate layer from the log.

The video sub-argument

Someone always says just use encoded video instead.

MCAP is a container, H.264 is a codec, and they aren’t alternatives. Encoded video goes inside MCAP.

The real argument underneath is older and correct: don’t put raw frames in a bag. 1920×1080 RGB8 at 30 fps runs about 6.2 MB a frame, 186 MB/s per camera. Six cameras clears a gigabyte per second, four terabytes an hour, per vehicle. The same footage at 10 Mbit/s H.264 is 1.25 MB/s. Call it 150× per camera. Teams who learned that in 2021 have the invoices.

The fix was never abandoning the container. It was CompressedVideo message types carrying H.264/H.265/VP9/AV1 frames inside MCAP, so you keep the codec’s compression and the cross-sensor alignment together. Writing sensor_msgs/Image to disk at fleet scale is the bug.

Bare .mp4 with a timestamp sidecar does genuinely win in one configuration: single camera, no other sensors, browser-first review UI. MCAP’s synchronisation is worth nothing there and you’re paying for a feature you don’t use. Though I’d note that describes a webcam more than a robot.

One constraint follows you into any container. Seek cost on video is governed by GOP structure, not by whatever wraps it — you can’t decode an arbitrary frame without walking back to the last IDR, so long GOPs save storage and make scrubbing miserable. Teams that need frame-accurate replay force short keyframe intervals and eat the bitrate. That’s an encoder decision, and it wants making before you’ve recorded a petabyte you can’t scrub.

The pipeline

Encode at the source, in hardware, before anything touches the bus. Record MCAP at the edge. Keep raw MCAP as the immutable cold tier, because being able to re-derive a dataset three years out beats the storage line item, particularly the first time a model does something inexplicable and you need the unmodified bytes.

Then curate before you transcode. Teams skip this step and it’s the expensive one. Egress and compute for a full-corpus transcode can rival what you pay to store the corpus: Robo-DM’s authors put an 8.9 TB Open-X slice at roughly $172/month on GCP against $172–$1,540 for a single full download. Index the MCAP, search it, work out which fraction deserves to go in front of a model, transcode that.

(For most fleets the fraction worth training on is small, and nobody enjoys saying so in the data meeting.)

Pick the training format from your model stack rather than from a blog post. WebDataset for streaming large image or video corpora sequentially. Parquet plus MP4 inside the LeRobot ecosystem. RLDS where you need Open X-Embodiment compatibility. Lance when random access and dataset versioning dominate.

Keep lineage from shard back to source file. The day someone asks what data trained a given checkpoint, and in anything safety-relevant someone will, you either have the answer or you have an incident.

What I’d measure instead

Read amplification first: bytes decompressed and read per byte the model consumes. Near 1 and you can stop reading. At 50× the transcode step has already paid for itself.

Then data-stall percentage, the wall-clock fraction where GPUs sit idle waiting on input. It’s the only number that decides whether your storage layer is a real problem, and most teams never instrument it. ⟨YOUR NUMBER GOES HERE — what yours was before and after. This is the single line that makes the post yours rather than a literature review.⟩

Time-to-first-batch on a cold cache is the third, and I care about it more than the other two combined. How long from wanting to train on a slice to the first gradient step. Iteration speed lives there, and it’s dominated by curation and transcode rather than by anything in your model code.

Everyone in Physical AI agrees data operations are the moat now. Fine. A moat is a set of unglamorous decisions about where bytes live and how fast you can reshape them into a batch. The advantage doesn’t go to whoever collects the most; it goes to whoever has the shortest path from a robot did something interesting to that experience is in a training batch. That path runs through the transcode step nobody wants to own.

So, is MCAP a bad format for foundation model training data? It’s the best format for recording robot data and the wrong shape for training on it, and those two claims were never in tension.

Whether that holds is the part I’m less sure about. Foxglove is pushing MCAP-native search at petabyte scale, and if curation moves into the log format itself the transcode step gets narrower and later. It doesn’t vanish, since the dataloader still wants columns and shards and physics doesn’t negotiate. But the boundary moves, and I don’t have a confident read on where it lands.

TOPS is the most abused number in edge AI.

It looks objective. It looks comparable

TOPS isn't a lie. It answers a real but narrow question: how many low-precision operations an accelerator can execute per second if every arithmetic unit stays perfectly fed. That "if" is doing enormous work. It says nothing about whether your workload can actually feed those units, whether your model survives the compiler intact, whether memory movement swamps the math, or whether the rest of the system keeps pace.

Most edge workloads aren't matrix-multiply problems anyway. They're streaming pipelines: sensors, copies, preprocessing, quantization, scheduling, synchronization, post processing, all of it running against a control deadline. Peak compute touches one slice of that. So the equation people carry in their heads,

is really

and TOPS only bends one term.

Compute-bound is the exception, not the rule

The roofline model is the cleanest way to see why. Every workload has two numbers: operations it needs, and bytes it moves. Divide them and you get arithmetic intensity.

High intensity means you're compute-bound, and more TOPS can help. Low intensity means you're memory-bound, and TOPS barely moves the needle. Here's the trap in every TOPS pitch: it quietly assumes your model lives on the compute-bound side. A lot of real pipelines don't. A fat convolution might keep the array busy, but resize, normalization, layout conversion, feature-map shuffling, NMS, tracking, and the small dynamic ops mostly spend their time shoving bytes around, not multiplying them.

Your real ceiling is whichever wall you hit first:

A 100-TOPS part can behave like something a quarter its size the moment bandwidth, data layout, or host-device transfers become the binding constraint.

DGX Spark - an interesting analogy to understand this

The cleanest public proof of this isn’t an edge board at all. It’s NVIDIA’s DGX Spark, a desktop AI box, and it argues the point better than any robotics part because every number is out in the open.

The headline spec is 1,000 TOPS of FP4. A literal petaflop on your desk. Then you read the next line: 128 GB of unified LPDDR5x at 273 GB/s. If you do hands-on, you would only touch that petaflop in half-second bursts before memory bandwidth choked it off. Not that FP4 units running slow, but because the chip starving for data. That gap between the sticker number and the sustained number is the entire thesis of this essay, printed on one spec sheet.

Every independent review landed in the same spot. One of them called the unified bandwidth the key bottleneck and clocked Llama 3.1 70B decoding at under three tokens per second on a machine rated for a petaflop. Another split it more precisely: prefill, which chews through the whole prompt at once, stayed competitive because it’s compute-bound; decode, which emits one token at a time and re-streams the weights for each one, fell off a cliff because it’s memory-bound. Same silicon, same model, two phases, two completely different ceilings. That’s the roofline model doing its job in public.

Now shrink it to something that ships on a robot. The Spark runs in a ~200 W envelope with active cooling and no deadline. Your Jetson or Hailo has a fraction of that power budget, a fraction of that bandwidth, and a control loop that will not wait. If a plugged-in desktop holding a petaflop of FP4 gets pinned by memory movement, a battery-powered perception stack gets pinned harder and sooner. The physics doesn’t care that one sits on a desk and the other sits on an axle.

- Utilization is the number nobody prints

Peak TOPS assumes 100% utilization. You will not see 100% utilization.

Run a 100-TOPS accelerator at 20% and you have a 20-TOPS device wearing a bigger label. Utilization bleeds out through unsupported operators, small tensors, dynamic shapes, memory stalls, kernel-launch overhead, CPU fallback, sync gaps, thermal throttling, and batch-1 execution. This is the whole reason a smaller accelerator sometimes beats a larger one on a specific job: less peak compute, but far more of it actually used.

- The tensor doesn't just get multiplied. It gets moved.

Inference is as much data logistics as arithmetic. Trace one camera frame:

Every arrow is latency: copies, cache misses, DMA setup, driver overhead, synchronization, layout conversion. And the intermediate activations are often much larger than the input frame, so they, not the image, dominate memory traffic. A lot of the time the accelerator isn't waiting on math. It's waiting on data. Move your tensors too much and the TOPS number stops meaning anything.

- Batch size flatters the wrong systems

The most flattering benchmarks are throughput benchmarks, and throughput loves batching: more work per scheduling event, higher utilization, prettier FPS. But a robot, drone, camera, or inspection rig lives and dies on batch-1 latency. You can't wait to collect eight future frames. You need this frame now.

So the numbers that matter aren't max FPS. They're p50, p90, p99, jitter, dropped frames, and how latency holds once the part is hot. A device that screams at batch 8 can be useless to a robot that needs steady batch-1 timing.

- Operator coverage is a performance feature

Hardware doesn't run models. It runs supported graphs. Map every operator cleanly and things fly. Leave a few unsupported and the graph gets partitioned between accelerator and CPU, and that's where performance dies. The usual offenders: custom NMS, dynamic reshape, oddball interpolation, LayerNorm, attention blocks, deformable convolutions, unusual activations, quantization patterns the compiler won't take.

The damage isn't that one operator runs slow. It's fragmentation:

Every boundary is another round of copies and synchronization. One unsupported op can drag down the whole graph around it.

- Quantization isn't free

Almost every edge TOPS figure is an INT8 figure. But models are usually trained in FP32 or FP16, and dropping to INT8 trades accuracy for speed. "Can this chip run INT8?" is the wrong question. "Can my model run INT8 without losing accuracy I care about?" is the right one. Classifiers usually quantize cleanly. Detection can, with effort. Depth, segmentation, pose, transformers, and regression-heavy heads are touchier. A monster INT8 accelerator loses its shine if your model needs FP16 to stay correct.

- Post processing eats your gains

Detection benchmarks love to hide the tail. The network emitting tensors isn't the finish line. You still owe box decode, confidence filtering, NMS, class filtering, track association, coordinate transforms, and temporal smoothing. On a lot of platforms, inference is accelerated and all of that runs on the CPU. Cut inference from 10 ms to 5 ms and you've saved nothing if postprocessing still burns 8. The product feels total latency, not the slice you optimized.

Benchmark the pipeline, not the part

Stop asking how many TOPS. Ask what the sustained, end-to-end p99 latency of your actual workload is. A real answer looks like this:

Now the bottleneck is visible, and inference is 11 of 40. Double the accelerator's TOPS and, at best, you shave part of that one slice. You cannot double the application.

A benchmark worth trusting reports batch-1 latency, p99, power, temperature, dropped frames, CPU utilization, memory-bandwidth pressure, post-quantization accuracy, operator fallback count, and host-device transfer time.

TOPS is a real number. The inference people draw from it usually isn't. The fastest chip on paper loses, routinely, to a smaller one that keeps data local, maps the whole graph cleanly, avoids CPU fallback, and holds stable latency when it's hot.

So next time you choice a silicon for your robot, or are perplexed why you are not able to achieve advertised specs, remember this.

It doesn’t. The robot still reacts late, still misses fast-moving objects, still feels a beat behind in closed-loop operation.

TensorRT didn’t fail. Inference was never the biggest latency problem to begin with - it’s just the only one anyone bothered to measure.

That’s the strange thing about robotics perception pipelines in 2026: the neural network is usually the most heavily optimized component in the entire chain, and almost everything around it is still running on default settings from three years ago.

Here’s what actually happens between a photon hitting the sensor and a steering command reaching the wheels:

Most benchmark decks show one box from this chain. The other eleven don’t make it into the slide.

Take a real profile - an engineer reports 18ms inference and feels good about it. Then someone times the whole pipeline:

Camera DMA              8 ms
Image decode            4 ms
Color conversion        3 ms
Memory copies           5 ms
Resize & normalize      5 ms
Inference               18 ms
NMS                     7 ms
ROS publish             6 ms
Controller interface    4 ms
------------------------------
Total                   60 ms

Now the same pipeline after several weeks of TensorRT work:

Camera DMA               8 ms
Image decode             4 ms
Color conversion         3 ms
Memory copies            5 ms
Resize & normalize       5 ms
Inference                16 ms
NMS                      7 ms
ROS publish              6 ms
Controller interface     4 ms
--------------------------------------------
Total                    58 ms

Two milliseconds saved, on a pipeline that’s still burning over 40ms on everything that isn’t the model. Most of the engineering hours went into the smallest slice of the problem.

Why often teams still ignore it?

It’s not carelessness - it’s tooling. TensorRT ships with profilers. CUDA has timelines. Every framework spits out an inference number, and every paper leads with one. The rest of the pipeline is scattered across drivers, middleware, the OS scheduler, and hardware interfaces that were never built to be measured together. Getting an honest end-to-end number means stitching timestamps across four or five layers of the stack by hand, and almost nobody does that work.

So teams optimize what they can see. It’s a completely rational response to bad visibility - it’s just optimizing the wrong thing.

- Memory copies are where a lot of latency hides.

One of the least glamorous parts of a perception pipeline is how many times an image gets copied before the GPU ever sees it:

Every hop costs time and bandwidth, and none of it improves the prediction by a single percentage point. I’ve seen teams cut more latency by removing one unnecessary copy than by dropping from FP32 to FP16 - and that fix never shows up in a conference talk, because there’s no leaderboard for “deleted a memcpy.”

- Cameras aren’t really instant.

“Camera latency” gets treated as basically zero, which is wrong. Sensor exposure, readout, DMA transfer, driver buffering, synchronization - all of that happens before a single pixel reaches software. If exposure alone costs 10ms, no amount of TensorRT tuning gets those milliseconds back. You can’t infer on pixels that haven’t arrived yet.

Preprocessing eats time too - resizing, RGB conversion, normalization, undistortion, stereo rectification. None of it touches the model, all of it happens before the model runs, and on a lot of setups the GPU actually finishes inference before the CPU has fed it the next frame. Utilization looks low and everyone blames the model, when the model’s been sitting idle waiting on preprocessing the whole time.

- Middleware and the scheduler take a major cut.

Most robotics stacks aren’t one process - messages get serialized, copied, queued, deserialized, and rescheduled as they move through ROS or DDS. Each hop is a few milliseconds. Six here, four there, a context switch somewhere else, and the robot is reacting 30ms later than the raw compute numbers would suggest. No amount of model optimization touches any of that.

The OS scheduler adds its own tax. Perception finishes a frame, but the controller doesn’t necessarily run next - logging might be flushing, a background thread might wake up, a network interrupt might steal a few cycles. Average latency barely moves, but worst-case latency can blow out badly. And for a physical robot, worst case usually matters more than average: a car that responds in 20ms most of the time but occasionally stalls for 80ms is more dangerous than one that’s a steady, boring 40ms every time. Determinism beats raw speed.

Even after perception is done, the command still has to pass through safety checks, interface layers, the CAN bus, motor controllers, and firmware before anything physically moves. The robot is always acting on slightly stale information - the only question is how stale.

How should one approach benchmarking?

Instead of asking “how fast is our model?”, the better question to ask us is “how old is the information by the time the actuator receives it?”

That single number - call it perception-to-action latency - captures camera time, preprocessing, inference, middleware, scheduling, planning, and control all at once. It’s what the robot actually experiences. The robot has no idea whether 20ms disappeared inside TensorRT or inside a memcpy; it only knows when the command arrived.

The most useful exercise I’ve found is building a real waterfall chart per frame:

Camera Exposure        ██
DMA                    ███
Decode                 ██
Color Conversion       █
Memory Copies          ██
Preprocessing          ██
Inference              ███████
Post-processing        ███
ROS/DDS                ██
Planning               ███
Controller             ██

Once you actually look at the full bar, it’s obvious where the effort should go. Sometimes it really is the model. More often, it isn’t.

TensorRT is genuinely excellent engineering - but its own visibility has created a blind spot. Because inference is the easiest thing to profile, it becomes the thing everyone obsesses over, while camera pipelines, memory movement, scheduling, and controller interfaces quietly eat the rest of the budget with nobody watching.

The fastest robots don’t come from the teams with the fastest models. They come from the teams that’ve bothered to account for every millisecond between the sensor and the actuator.

Ask a senior engineer at Waymo, Tesla, or any Physical AI startup how they actually spend their time. The answer never matches the conference talks.

Control algorithms, perception architectures, planning systems - these are what get written about, presented, and funded. They’re maybe 15–20% of where engineering hours actually go. The rest disappears into data collection, labeling pipelines, dataset versioning, calibration tracking, validation scripts, evaluation infrastructure, and debugging corruptions that produce no error messages whatsoever.

Nobody talks about this at NeurIPS. It’s not glamorous. But it’s where the work is.

How complicated it can be?

To appreciate the hidden complexity, follow a single camera frame from capture to its eventual use in a training run six months later.

If you already know this, feel free to skip to the next section : )

═════════════════════════════
PHYSICAL LAYER
═════════════════════════════
1.  Image sensor captures photons (1/30 second exposure)
2.  Image Signal Processor: demosaicing, white balance, noise reduction
3.  Hardware timestamp generated by camera's internal oscillator
    ⚠ NOTE: This clock is NOT the system clock. It drifts 1–5ppm.
4.  Frame transmitted over GigE Vision to compute unit
5.  OS receives frame, applies software timestamp
    ⚠ NOTE: This is a SECOND timestamp. The two will diverge.
6.  DMA copy into shared memory ring buffer

MIDDLEWARE LAYER
═════════════════════════════
7.  ROS 2 node reads frame, wraps in sensor_msgs/Image
8.  Publishes to /camera/front_left/image_raw (~1MB per frame)
9.  Subscribers: perception node, recorder node, visualization node
10. Three independent subscribers receive the same message.
    ⚠ Each applies processing with unknown latency jitter.

RECORDING LAYER
══════════════════════════════
11. Recorder serializes to MCAP format with LZ4 compression
12. Writes to local NVMe in 30-second segments
13. Session metadata: robot_id, session_id, operator, timestamp range
14. Segment complete → MD5 checksum computed on in-memory buffer
    ⚠ NOTE: If the NVMe has a bad sector, the write is "successful"
    but the data is corrupted. The checksum passes.
15. Segment pushed to upload queue

SENSOR FUSION ASSOCIATION
═════════════════════════════
16. LiDAR arriving at 10Hz (different oscillator, different clock)
17. IMU at 200Hz (yet another clock)
18. GPS at 10Hz (UTC-synchronized via PPS signal — most accurate)
19. To fuse: what was the robot's pose at this exact camera timestamp?
    ⚠ Requires sub-10ms alignment. Uncompensated drift → wrong geometry.
20. Calibration lookup: where is this camera relative to LiDAR?
    ⚠ Is the calibration from this session? Or last week's? Did it drift?
    ⚠ Calibration changes with temperature. No one tracked today's temp.

UPLOAD AND INGEST
═════════════════════════════
21. Upload to object storage via chunked multipart upload
22. Upload job written to message queue (Kafka / SQS)
23. Ingest service parses MCAP, extracts metadata
24. Frame indexed: timestamp_ns, robot_id, session_id,
    scenario_type, GPS_bbox, weather_tag, calibration_id
    ⚠ NOTE: "weather_tag" was added to the schema in month 4.
    ⚠ All frames before month 4 have NULL weather_tag.
    ⚠ Training code does not handle NULL. Silent filter removes 40%
    ⚠ of your data and you don't notice.

VALIDATION PIPELINE
═════════════════════════════
25. Timestamp monotonicity check             [PASS]
26. Frame drop detection (>1% gap rate)      [PASS]
27. Calibration validity check               [FAIL — 3 frames]
    ⚠ Clock stutter caused those 3 frames to fall outside
    ⚠ the calibration validity window. Flagged for review.
28. Brightness range check                   [PASS]
29. Compression integrity check              [PASS — does not catch
    ⚠ bad-sector corruption, because checksum was wrong]

LABELING
═════════════════════════════
30. Frame sampled for annotation (diversity heuristics)
31. Sent to annotation pipeline: objects, lanes, drivable area
32. Human annotation: ~8 minutes per frame
33. QA review by senior annotator
34. Labels written to database with annotator_id, QA_status
35. Labels joined to frame via (robot_id, timestamp_ns)
    ⚠ NOTE: Labeling team changed in month 6.
    ⚠ New team uses slightly different bounding box convention.
    ⚠ Mixed conventions now exist in the training set.

DATASET VERSIONING
═════════════════════════════
36. Frame added to dataset v3.7.2 manifest
37. Train/val/test split assigned
    ⚠ Split is by session_id, not frame. Mostly prevents leakage.
    ⚠ But the robot drove the same route twice in different sessions.
    ⚠ Similar frames appear in both train and test.
38. Dataset committed to DVC remote with git hash

TRAINING
═════════════════════════════
39. Training job: 64 A100s, dataset v3.7.2
40. Frame decoded, augmented, normalized
41. Forward pass. Gradient. Weight update.
    ⚠ The bad-sector-corrupted frame was included.
    ⚠ So were the frames with wrong bounding box convention.
    ⚠ So were the test-set-leaked frames.
    ⚠ The model trains successfully.

EVALUATION AND REGRESSION
═════════════════════════════
42. mAP computed per class, per condition
43. Regression detected: pedestrian mAP dropped 2.3% in construction zones
44. Root cause hunt: Is it the model? The data? A labeling issue?
    ⚠ Debugging this takes 3 engineer-weeks.
    ⚠ The answer: bad-sector corruption + mixed bounding box convention
    ⚠ + 40% data loss from NULL weather_tag filter.
    ⚠ None of these produced a single error message.

SIX MONTHS LATER
═════════════════════════════
45. Engineer queries data lake for construction zone, night, rain frames
46. Returns 8,000 results
47. 340 have corrupted calibration (calibration service bug in month 2)
48. 200 have labels from deprecated tool with different class definitions
49. Engineer spends 3 days reconstructing which frames are usable
50. Eventual usable frames: ~7,460 of 8,000 — but which 7,460?
    ⚠ No provenance trail. Must re-validate manually.
═════════════════════════════
TOTAL TIME FROM CAPTURE TO USABLE TRAINING EXAMPLE: 6 months.
TOTAL ENGINEERING-DAYS LOST TO DATA ISSUES: ~20.
TOTAL ERROR MESSAGES PRODUCED BY ANY OF THESE ISSUES: 0.
═════════════════════════════

This isn’t a pathological story. This is Tuesday at a typical robotics startup.

So what poses as a problem?

These are the data problems that don’t produce stack traces. They produce models that fail in production in ways that take months to diagnose.

• Timestamp Drift - Camera hardware oscillators drift by 1–5 parts per million. After a 4-hour collection session, a sensor running at 2ppm drift is 29ms off from the system clock. During sensor fusion, your perception stack associates a LiDAR point cloud (captured at time T) with a camera frame (nominally at T, actually at T+29ms). At highway speeds (30 m/s), 29ms represents 87 centimeters of vehicle motion. You are training your model with a point cloud that doesn’t correspond to the image it thinks it does. This doesn’t crash anything. It silently corrupts your 3D bounding box training data, slightly but persistently - across every hour of data collected with that sensor.

• Schema Drift - You add a field to your custom protobuf message definition. You’re careful: it’s an optional field, backward compatible. But your MCAP recording tool hashes the message schema to identify the message type. Old bags reference the old hash. Your replay tool, written for the new schema, can’t find the message definition for the old hash. Three months of bags are now unreadable without a migration script. The migration script takes two weeks to write and test properly. During those two weeks, you discover that 4% of your bags also have a related issue with the companion metadata schema.

• Silent Disk Corruption - A frame is written to disk. The NVMe drive has a developing bad sector. The OS reports the write as successful. The checksum was computed on the in-memory buffer before the write. The corrupted frame sits on disk, looking exactly like a valid image, producing subtle artifacts that a labeler annotates as object detections. You’ve injected structured noise into your training set.

• Benchmark Leakage - Your evaluation set is sampled from the same routes as your training set, just at different timestamps. A rare sign type appears in both training and evaluation because you drove past it every morning during data collection. Your per-class mAP on that sign type looks excellent. Your production system fails at a different intersection with the same sign type, because it memorized the specific instance rather than the class.

• Calibration Drift - Camera intrinsics shift with temperature. LiDAR-camera extrinsic calibration can change after a minor collision that leaves no visible damage. If calibration is recorded once weekly, every datum collected between calibration sessions is tagged with potentially incorrect geometry. Models trained on this data learn the wrong spatial relationships - consistently, invisibly.

• Label Inconsistency - Labeling vendor A annotates occluded pedestrians with a partially_visible flag. Labeling vendor B, hired when the first vendor couldn’t scale, annotates only fully visible pedestrians. You merge their outputs without reconciling the conventions. Your model now receives the same visual stimulus with different ground truth depending on which batch the frame came from. You’ve permanently injected irreducible label noise.

• Duplicate Data - A recorder bug causes 2% of sessions to be written twice under different session IDs. Your training set contains tens of thousands of effectively duplicated frames. The model overfits to these examples. Evaluation performance looks inflated - the test set also contains duplicates. The effect is invisible until you deploy to a new environment and the model catastrophically fails on distributions it’s never actually generalized to.

  None of these produce error messages. They produce models with mysterious behavior, regressions that can’t be explained, and debugging sessions that last months.

Physical AI is in its Pre-Git Era

Consider where software engineering was in 1990. Code on network drives. No version history. “Works on my machine” was an acceptable bug report. Reproducing a build meant manually reconstructing the environment. Merging two developers’ work meant emailing patches.

That’s where robotics data is today.

Datasets live on shared NFS mounts without versioning. There’s no history of which data produced which model. Training runs aren’t reproducible because no one tracked which files were actually used. “The model trained on my workstation” is common. Merging two teams’ datasets means copying files and hoping for no conflicts.

The ML world partially solved this for static image data. Weights & Biases tracks experiments. DVC versions datasets. Hugging Face hosts models. But these tools were built for static datasets of labeled images or tabular rows. They don’t understand time-synchronized multi-modal sensor streams, calibration dependencies, coordinate frame transforms, or the need to replay a scenario to debug a failure. They’re the right shape of solution applied to the wrong problem structure.

Robotics needs purpose-built tooling. Not a reskin of MLflow. Something designed around what embodied physical data actually is.

MCAP - developed by Foxglove, now the default format in ROS 2 - is a good start: self-contained, corruption-resistant, schema-embedded, multi-language. But MCAP solves the storage format. It doesn’t solve versioning, provenance, calibration tracking, scenario search, or continuous evaluation. We have one good brick. The building doesn’t exist yet.

Why Foundation Models Make This Worse?

There’s a seductive belief that as models scale and become more capable, messy data becomes less of a problem - the model will figure it out. This is exactly backwards.

Physical Intelligence’s π0 is trained on data from seven different robot hardware platforms, across eight task types, mixed with internet-scale vision-language pretraining. The hard work isn’t the architecture - π0 builds on existing VLM foundations. The hard work is making data coherent across robot morphologies with different action spaces, different observation definitions, and different physical constraints. That is not a modeling problem. It is a data harmonization problem that precedes any modeling work.

Waymo’s recent scaling law research puts a sharper point on it. Their finding: for autonomous driving tasks, unlike language models, optimal systems tend to be “relatively smaller in size, while requiring significantly more data to train.” More data is the path to better systems - which means more data collection, more data operations, and more quality control, not less.

Scaling model capacity without scaling data infrastructure doesn’t plateau gracefully. It fails in ways that look like training instability, unexplained regressions, or evaluation results that don’t transfer to production. The root cause is usually in the data. The debugging path is usually weeks long.

What the Leaders Actually Built?

Tesla’s data engine - described by Andrej Karpathy at multiple public presentations - is the clearest public acknowledgment that data infrastructure, not model architecture, is the core competitive advantage in autonomous systems.

Trigger classifiers run on the production fleet, detecting situations where the current model and a candidate model diverge, or where the model’s output conflicts with the driver’s behavior. Those triggers route specific clips for annotation and retraining. New models run silently on production vehicles in “shadow mode” - outputs compared against the production model, never actually controlling the car. This is a continuous evaluation system running at fleet scale across millions of vehicles. At AI Day 2022, Tesla described datasets on the order of 1.5 petabytes for training their occupancy network alone.

None of that is model architecture. All of it is data infrastructure.

Waymo maintains 500,000+ hours of driving data. New software release candidates are automatically evaluated against millions of simulated miles. Their simulation system generates scenarios from real-world logs - which requires log management, scenario indexing, and closed-loop evaluation infrastructure of considerable sophistication. Dedicated teams work on sensor simulation, scenario search, and evaluation tooling. Not just perception and planning.

Karpathy, at a CVPR workshop, made it plain: “The only sure certain way I have seen of making progress on any task is, you curate the dataset that is clean and varied and you grow it and you pay the labeling cost.”

The algorithm is almost secondary. The machine that creates and validates training data is the real product. The robotics organizations that have achieved meaningful scale haven’t done it primarily through algorithmic novelty. They’ve done it by building industrial-grade data operations.

So how does it affect a typical robotics startup trajectory?

• Months 1–3: The robot moves. The demo works. The team is energized.

• Months 4–6: A model trains. It works in the lab. Investors are interested.

• Months 7–9: Real-world deployment. Performance degrades. Debug manually. Fix cases individually.

• Months 10–12: Retrain the model with new data. Performance is worse than before. The team doesn’t know why.

• Months 13–18: A senior engineer joins. She discovers: no dataset versioning, no reproducible training, evaluation set contaminated with training data, calibration not tracked, labeling conventions inconsistent across vendors. Six months of reconstruction work begins.

The reason founders underestimate data operations is that early success doesn’t require them. Two engineers can build a demo that impresses investors with zero data infrastructure. The failure modes only surface when you need to reproduce a training run, when a calibration change corrupts months of data you didn’t know was invalid, or when your evaluation set has silently leaked into your training set and you’ve been measuring the wrong thing for half a year.

The first dedicated data infrastructure engineer - someone who designs the schema before you have data, builds the validation pipeline before you have regressions, creates the reproducible evaluation framework before you establish a baseline - often has more total leverage than the third ML researcher. The ML researcher improves the current model. The infrastructure engineer makes all future models improvable faster. The second is harder to justify to a board. It’s more important.

But if we do X then it’s not a problem?

“Simulation solves the data problem.”

Simulation reduces the need for some real-world data collection. It doesn’t eliminate the data problem - it adds a parallel one. Simulation asset management requires versioning. Domain randomization parameters require tracking. Sim-to-real gap requires characterization against real-world benchmarks. Synthetic data generation pipelines fail in ways that look exactly like real data pipeline failures: schema drift, coverage gaps, distribution mismatch, evaluation leakage. Waymo runs millions of simulated miles per software release candidate. That infrastructure - scenario management, parameter tracking, results storage, regression detection - is a data operations problem of comparable complexity to the real-data problem.

“Foundation models trained on internet data will generalize to robotics.”

Internet-scale pretraining provides genuine value for visual recognition and language grounding. It doesn’t help with calibration tracking, multi-modal timestamp synchronization, or the physical geometry of sensor fusion. The “last mile” of embodied AI - grounding general knowledge in precise physical sensor data - requires exactly the infrastructure this article describes. Physical Intelligence’s π0, which makes aggressive use of VLM pretraining, still required building a multi-robot data collection and harmonization infrastructure as the core technical work.

“This is just MLOps with extra steps.”

MLOps tools were designed for static datasets. Robotics data is time-synchronized, calibration-dependent, multi-modal, spatially grounded, and replay-dependent. Applying MLflow to manage a dataset that includes MCAP files with associated per-session calibration metadata, LiDAR-camera extrinsics, GPS trajectories, and scenario tags reveals immediately that the abstractions don’t transfer. The tooling gap is real and purpose-built solutions are needed.

“Big companies have this problem. Startups should focus on shipping product.”

This is precisely backwards. Big companies can absorb three months of debugging a data corruption issue. Startups cannot. The startup that establishes data discipline in months 1–3 compounds its learning rate for the entire subsequent trajectory. The one that defers it will spend months 12–18 in reconstruction mode while competitors iterate. Early investment in data infrastructure is not a distraction from building the product - it is the infrastructure on which the product’s learning rate depends.

“Better sensors and hardware will reduce calibration and sync problems.”

Better hardware reduces but doesn’t eliminate calibration drift. More importantly, tracking calibration state over time - associating every data frame with the calibration that was valid at that moment, flagging data when calibration may have changed, managing calibration as a versioned artifact - is not a sensor problem. It’s a data management problem. Even perfect sensors require a calibration management system.

How should we solve it?

DevOps emerged when shipping code without operational discipline stopped scaling. MLOps emerged when deploying models without infrastructure discipline stopped scaling. RobotOps is next - and the path is legible.

It all starts with changing how we approach a robotics problem - and try giving a serious thought through the ‘DevOps for Robotics’ perspective.

Standardize before you tool up. MCAP is a good start, but the next step is a robotics dataset format analogous to Parquet - purpose-built for time-synchronized multi-modal sensor data, with versioning and calibration baked in. The community needs to converge on this before every company keeps reinventing it privately. At the team level, this means defining canonical sensor schemas, a timestamp policy (hardware timestamps only, sync mechanism documented), calibration tracking (every file references a calibration session ID with a validity window), and a session metadata standard - written down before you have data. Two weeks of work that saves six months.

Build the platform layer, not five duct-taped tools. Early entrants - Foxglove, Rerun, Scale AI, HuggingFace via LeRobot - are solving pieces of the puzzle. The gap is integration: schema management, calibration tracking, dataset lineage, and scenario search in one coherent platform.

Treat data like code: validate, checksum, test. Every model commit should automatically validate datasets, check timestamp and calibration integrity, run scenario regression benchmarks, and compare against historical results - the same discipline software teams built into CI/CD two decades ago. In practice, this starts small: checksum every recorded file in-memory before upload and verify after (one engineering-day, catches silent corruption before it reaches training), and run a validation script before every training job - timestamp monotonicity, frame drop rate, calibration validity, label schema compliance, duplicate detection. Fail loudly, automatically.

Make scenario search the primary interface. Manual log inspection should be as archaic as grepping server logs. The target: query petabyte-scale sensor archives the way you’d query a database - by scenario type, condition, failure mode, time window. The scenario library becomes more valuable than any individual model checkpoint. Build your evaluation set first as part of this discipline: sample deliberately, freeze it, version it, never train on it, and store every result against it with model version, dataset version, and timestamp.

Formalize the discipline and hire for it. Robot Data Engineer needs to be a real job title with a real career path, not a responsibility absorbed by whoever has bandwidth. Counterintuitively, the first data infrastructure hire often has more leverage than the third ML engineer - and is the recommendation most consistently ignored despite having the highest ROI.

A better model architecture can be published and reproduced in six months. A decade of well-organized operational data cannot.

References and Resources

^{Foundational Papers}

• ^{Open X-Embodiment Collaboration (2024). Open X-Embodiment: Robotic Learning Datasets and RT-X Models.ICRA 2024. https://robotics-transformer-x.github.io/}

• ^{Black et al. (Physical Intelligence, 2024). π0: A Vision-Language-Action Flow Model for General Robot Control.https://physicalintelligence.company/download/pi0.pdf}

• ^{Brohan et al. (Google, 2022). RT-1: Robotics Transformer for Real-World Control at Scale. arXiv:2212.06817}

• ^{Chi et al. (TRI / Columbia, 2023). Diffusion Policy: Visuomotor Policy Learning via Action Diffusion.arXiv:2303.04137}

• ^{Sun et al. (Waymo, 2020). Scalability in Perception for Autonomous Driving: Waymo Open Dataset. CVPR 2020.}

• ^{Waymo (2026). Scaling Laws Research: Data-Driven Autonomous Driving.https://datacenterdynamics.com/en/news/waymo-research-confirms-self-driving-scaling-laws/}

^{Engineering Blogs and Talks}

• ^{Karpathy, A. (Tesla Autonomy Day, 2019). Tesla Data Engine.}

• ^{Karpathy, A. (Tesla AI Day, 2021). Autopilot and Auto-Labeling.}

• ^{Karpathy, A. (CVPR 2021 Workshop on Autonomous Driving). Lessons learned from deploying neural networks at Tesla.}

• ^{Waymo Engineering. Industry Best Practices in Robotics Software Engineering. arXiv:2212.04877}

• ^{Foxglove Blog. Introducing the MCAP File Format. https://foxglove.dev/blog/introducing-the-mcap-file-format}

• ^{Rerun Blog. Introducing Experimental MCAP Support. https://rerun.io/blog/introducing-experimental-support-for-mcap-file-format}

• ^{Segments.ai. MCAP vs ROS bag: Simplifying Multi-Modal Sensor Data. https://segments.ai/blog/mcap-vs-ros-bag-simplifying-multi-modal-sensor-data-in-robotics/}

^{Community and Discussion}

• ^{ROS Discourse: https://discourse.ros.org/ (Active community discussions on data management, MCAP, and tooling evolution)}

• ^{Papers With Code — Robot Learning: https://paperswithcode.com/task/robot-learning}

• ^{ScenarioNet (open-source scenario management): arXiv:2306.12241}

_{Technical claims in this article about engineering time allocation represent informed analysis based on public statements from industry engineers, published engineering blogs, and the author’s experience in Physical AI development. Where specific figures are cited (e.g., Waymo’s 500,000 hours of driving data, Tesla’s 1.5PB dataset scale), sources are linked above. Claims presented as analysis are clearly identified as such.}

_{The argument that data infrastructure determines competitive outcomes in Physical AI is not a prediction about the future. It is an observation about the present, backed by every major organization that has achieved meaningful scale in autonomous systems.}