TOPS Is a Vanity Metric
Peak compute tells you almost nothing about how your robot will actually perform
Walk into any edge AI product launch and one specification dominates the marketing slide: TOPS
TOPS is the most abused number in edge AI.
It looks objective. It looks comparable
TOPS isn't a lie. It answers a real but narrow question: how many low-precision operations an accelerator can execute per second if every arithmetic unit stays perfectly fed. That "if" is doing enormous work. It says nothing about whether your workload can actually feed those units, whether your model survives the compiler intact, whether memory movement swamps the math, or whether the rest of the system keeps pace.
Most edge workloads aren't matrix-multiply problems anyway. They're streaming pipelines: sensors, copies, preprocessing, quantization, scheduling, synchronization, post processing, all of it running against a control deadline. Peak compute touches one slice of that. So the equation people carry in their heads,
is really
and TOPS only bends one term.
Compute-bound is the exception, not the rule
The roofline model is the cleanest way to see why. Every workload has two numbers: operations it needs, and bytes it moves. Divide them and you get arithmetic intensity.
High intensity means you're compute-bound, and more TOPS can help. Low intensity means you're memory-bound, and TOPS barely moves the needle. Here's the trap in every TOPS pitch: it quietly assumes your model lives on the compute-bound side. A lot of real pipelines don't. A fat convolution might keep the array busy, but resize, normalization, layout conversion, feature-map shuffling, NMS, tracking, and the small dynamic ops mostly spend their time shoving bytes around, not multiplying them.
Your real ceiling is whichever wall you hit first:
A 100-TOPS part can behave like something a quarter its size the moment bandwidth, data layout, or host-device transfers become the binding constraint.
DGX Spark - an interesting analogy to understand this
The cleanest public proof of this isn’t an edge board at all. It’s NVIDIA’s DGX Spark, a desktop AI box, and it argues the point better than any robotics part because every number is out in the open.
The headline spec is 1,000 TOPS of FP4. A literal petaflop on your desk. Then you read the next line: 128 GB of unified LPDDR5x at 273 GB/s. If you do hands-on, you would only touch that petaflop in half-second bursts before memory bandwidth choked it off. Not that FP4 units running slow, but because the chip starving for data. That gap between the sticker number and the sustained number is the entire thesis of this essay, printed on one spec sheet.
Every independent review landed in the same spot. One of them called the unified bandwidth the key bottleneck and clocked Llama 3.1 70B decoding at under three tokens per second on a machine rated for a petaflop. Another split it more precisely: prefill, which chews through the whole prompt at once, stayed competitive because it’s compute-bound; decode, which emits one token at a time and re-streams the weights for each one, fell off a cliff because it’s memory-bound. Same silicon, same model, two phases, two completely different ceilings. That’s the roofline model doing its job in public.
Now shrink it to something that ships on a robot. The Spark runs in a ~200 W envelope with active cooling and no deadline. Your Jetson or Hailo has a fraction of that power budget, a fraction of that bandwidth, and a control loop that will not wait. If a plugged-in desktop holding a petaflop of FP4 gets pinned by memory movement, a battery-powered perception stack gets pinned harder and sooner. The physics doesn’t care that one sits on a desk and the other sits on an axle.
- Utilization is the number nobody prints
Peak TOPS assumes 100% utilization. You will not see 100% utilization.
Run a 100-TOPS accelerator at 20% and you have a 20-TOPS device wearing a bigger label. Utilization bleeds out through unsupported operators, small tensors, dynamic shapes, memory stalls, kernel-launch overhead, CPU fallback, sync gaps, thermal throttling, and batch-1 execution. This is the whole reason a smaller accelerator sometimes beats a larger one on a specific job: less peak compute, but far more of it actually used.
- The tensor doesn't just get multiplied. It gets moved.
Inference is as much data logistics as arithmetic. Trace one camera frame:
Every arrow is latency: copies, cache misses, DMA setup, driver overhead, synchronization, layout conversion. And the intermediate activations are often much larger than the input frame, so they, not the image, dominate memory traffic. A lot of the time the accelerator isn't waiting on math. It's waiting on data. Move your tensors too much and the TOPS number stops meaning anything.
- Batch size flatters the wrong systems
The most flattering benchmarks are throughput benchmarks, and throughput loves batching: more work per scheduling event, higher utilization, prettier FPS. But a robot, drone, camera, or inspection rig lives and dies on batch-1 latency. You can't wait to collect eight future frames. You need this frame now.
So the numbers that matter aren't max FPS. They're p50, p90, p99, jitter, dropped frames, and how latency holds once the part is hot. A device that screams at batch 8 can be useless to a robot that needs steady batch-1 timing.
- Operator coverage is a performance feature
Hardware doesn't run models. It runs supported graphs. Map every operator cleanly and things fly. Leave a few unsupported and the graph gets partitioned between accelerator and CPU, and that's where performance dies. The usual offenders: custom NMS, dynamic reshape, oddball interpolation, LayerNorm, attention blocks, deformable convolutions, unusual activations, quantization patterns the compiler won't take.
The damage isn't that one operator runs slow. It's fragmentation:
accelerator → CPU → accelerator → CPU
Every boundary is another round of copies and synchronization. One unsupported op can drag down the whole graph around it.
- Quantization isn't free
Almost every edge TOPS figure is an INT8 figure. But models are usually trained in FP32 or FP16, and dropping to INT8 trades accuracy for speed. "Can this chip run INT8?" is the wrong question. "Can my model run INT8 without losing accuracy I care about?" is the right one. Classifiers usually quantize cleanly. Detection can, with effort. Depth, segmentation, pose, transformers, and regression-heavy heads are touchier. A monster INT8 accelerator loses its shine if your model needs FP16 to stay correct.
- Post processing eats your gains
Detection benchmarks love to hide the tail. The network emitting tensors isn't the finish line. You still owe box decode, confidence filtering, NMS, class filtering, track association, coordinate transforms, and temporal smoothing. On a lot of platforms, inference is accelerated and all of that runs on the CPU. Cut inference from 10 ms to 5 ms and you've saved nothing if postprocessing still burns 8. The product feels total latency, not the slice you optimized.
Benchmark the pipeline, not the part
Stop asking how many TOPS. Ask what the sustained, end-to-end p99 latency of your actual workload is. A real answer looks like this:
Now the bottleneck is visible, and inference is 11 of 40. Double the accelerator's TOPS and, at best, you shave part of that one slice. You cannot double the application.
A benchmark worth trusting reports batch-1 latency, p99, power, temperature, dropped frames, CPU utilization, memory-bandwidth pressure, post-quantization accuracy, operator fallback count, and host-device transfer time.
TOPS is a real number. The inference people draw from it usually isn't. The fastest chip on paper loses, routinely, to a smaller one that keeps data local, maps the whole graph cleanly, avoids CPU fallback, and holds stable latency when it's hot.
So next time you choice a silicon for your robot, or are perplexed why you are not able to achieve advertised specs, remember this.