# Does INT8 quantization hurt small object detection? The damage mAP does not show

Edge deployment | Research | Published 2026-10-01

Quantizing a detector to INT8 costs small objects 1.4 to 3 times more than the overall metric, and below 12 pixels the published loss reaches a third of the accuracy; what is known, the fixes, and what nobody has measured yet.

Concepts: quantization, int8, post-training-quantization, small-object-detection, edge-inference, thermal-imaging, anti-uav, evaluation

## The problem

To run a detector on a small computer next to the camera, you compress it: weights and activations go from 32-bit floats to 8-bit integers (INT8), which made a small detector about twice as fast on a Raspberry Pi 5 CPU in one community measurement. The usual check is the overall accuracy (mAP) before and after, and it usually moves by a point or two. But the overall number averages over every object in the test set, and most of them are large. When the target you care about is a few pixels wide (a drone at a kilometre in a thermal camera, a bird, a person in an aerial image), the same compression can cost it far more, and the overall metric hides it. This article collects what is published about that effect, the fixes that exist, and what is still unmeasured.

## The usual approach

- Train in FP32, then apply **post-training quantization (PTQ)**: run a few hundred calibration images, record the range of every activation tensor, and map that range onto 256 integer levels.
- One **scale per tensor** for activations (per channel for weights), chosen by the maximum, a percentile, or the KL divergence between the float and integer histograms.
- Accept the model if overall mAP drops by less than about one point; switch to **quantization-aware training (QAT)** if it drops more.
- Report accuracy on COCO-style averages. COCO's "small" bin is everything under 32 x 32 pixels, which is large compared with a 4-pixel target.

## Where it fails, with numbers

**Small objects lose more, consistently.** Relative drops, computed from each paper's own tables (AP is the overall metric, AP_S the small-object one, AP_L the large-object one):

| paper | setting | AP | AP_S | AP_L |
|---|---|---|---|---|
| [FQN, CVPR 2019](https://openaccess.thecvf.com/content_CVPR_2019/papers/Li_Fully_Quantized_Network_for_Object_Detection_CVPR_2019_paper.pdf) | RetinaNet-R50, 4-bit QAT | -8.7 % | -14.8 % | -8.4 % |
| FQN, CVPR 2019 | Faster R-CNN-R50, 4-bit | -12.2 % | -17.3 % | -10.8 % |
| [AQD, CVPR 2021](https://arxiv.org/abs/2007.06919) | FCOS-R50, 4-bit | -2.3 % | -7.2 % | -2.2 % |
| [Q-DETR, CVPR 2023](https://arxiv.org/abs/2304.00253) | DETR-R50, 4-bit (attention 8-bit) | -6.2 % | -13.7 % | -2.0 % |
| Q-DETR, a baseline in its tables | SMCA-DETR, **8-bit PTQ**, percentile calibration | -8.5 % | -19.6 % | -5.4 % |
| [Q-YOLO](https://arxiv.org/abs/2307.04816) | YOLOv5s / v7, 8-bit weights and activations, the Q-YOLO method | 0 to -0.2 % | -0.3 to -0.9 % | -0.2 % |

The pattern: under moderate damage, small objects lose 1.4 to 3 times more than the overall number and 1.6 to 7 times more than large ones; in the CNN rows the overall number sits close to the large-object one, because large objects dominate the count. The last row matters too: for a CNN YOLO, a careful 8-bit method costs COCO-small objects under one percent. Transformer detectors lose more: the best 8-bit method in Q-DETR's tables (VT-PTQ) still costs small objects 7.3 %. At COCO scale and with a good recipe, INT8 is close to safe for a CNN.

**Below 16 pixels it is no longer small.** Two measurements, neither peer-reviewed, point the same way:

- A bachelor thesis at Leiden (LIACS, 2026, [PDF](https://theses.liacs.nl/pdf/2025-2026-JohannsenLLenard.pdf)) quantized YOLOv11n to INT8 with ONNX Runtime PTQ (min-max calibration) on XS-VID, a dataset whose mean object is 10.6 pixels, with a ByteTrack tracker on top. Overall AP fell 10.5 %, but **35 % for objects of 0 to 12 pixels**, 22 % for 12 to 20 and 15 % for 20 to 32. At track level, MOTA went from 0.247 to 0.235 and IDF1 from 0.404 to 0.376. Its QAT came out worse than PTQ and its proposed fix did not work, so read it as an indication, not a proof. It is the only track-level data we found on a dataset whose mean object is about 11 pixels (the tracking numbers are for the whole dataset, not per size).
- A community measurement ([hailo-uav-tracker](https://github.com/pierrosimonestd-cpu/hailo-uav-tracker), DUT Anti-UAV, YOLOv8n, ONNX Runtime INT8 as a stand-in for a hardware quantizer): FP32 to INT8 cost 6.8 % of overall AP, 14.4 % of small-object AP and 1.1 % of large-object AP. On a Raspberry Pi 5 CPU the INT8 model ran in 75.0 ms against 153.6 ms.

**The mechanism.** One activation scale per tensor is set by its largest values. In a thermal or aerial scene those are background outliers: sun glint, a hot roof, a cloud edge. The faint response of a tiny target in the high-resolution feature maps (the P2 and P3 levels of a feature pyramid) then spans one or two quantization steps, and rounding merges it into the background. A large target spans many steps and survives. A [2026 ISPRS paper](https://www.bgd.ed.tum.de/pdf/2026_full_ISPRSCongress_QATDet.pdf) argues, assuming the error averages over an object's pixels, that the signal-to-quantization-noise ratio per object falls as its pixel count falls. The figure on this page is a simplified illustration of this mechanism, not a measurement.

## Known fixes

| fix | what it does | where it was tested | limits |
|---|---|---|---|
| Better calibration (percentile, KL) and per-channel scales | stops one outlier from setting the step for the whole tensor | standard in TensorRT, ONNX Runtime, ncnn | helps at COCO scale (Q-YOLO above); not shown to be enough below 12 px |
| [OSA regularization, ISPRS Annals 2026](https://www.bgd.ed.tum.de/pdf/2026_full_ISPRSCongress_QATDet.pdf) | 8-bit QAT with a feature-alignment loss on P2 / P3, weighted by 1 / object area | YOLOX-Nano and Tiny, small birds and aerial buildings, on a Kria KV260 FPGA | needs training, not calibration; reports overall AP only, no size bins |
| MLFQ, IET Image Processing 2026 ([Wiley](https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/ipr2.70476)) | PTQ block reconstruction with a spatial weight map from object sizes | VisDrone, UAVDT | we read the abstract only: gains at 4 bits and below, mixed at 8; numbers left out |
| [LiDAR-PTQ, ICLR 2024](https://arxiv.org/abs/2401.15865) | a foreground-weighted PTQ loss | 3D detection | another modality |
| Mixed precision | keeps the detection head or the feature pyramid at higher precision | various detectors | not aimed at tiny objects on a CPU |
| [ScaleKD, CVPR 2023](https://openaccess.thecvf.com/content/CVPR2023/papers/Zhu_ScaleKD_Distilling_Scale-Aware_Knowledge_in_Small_Object_Detector_CVPR_2023_paper.pdf) | scale-aware distillation into a small detector | COCO, VisDrone | about model size, not quantization |
| [CSCWD, 2026](https://arxiv.org/abs/2609.30395) | distils a P2-head YOLO11m into YOLO11n for anti-UAV, runs it on a Pi 5 through ncnn | Drone-vs-Bird; DUT Anti-UAV zero-shot | FP16 only, no INT8, no size study |

A related lever sits before the network. Thermal cameras deliver 14 or 16 bits per pixel, and the usual automatic gain control squeezes that into 8 bits before the detector sees it, which is a quantization step of its own. Task-adaptive tone mapping for thermal images exists ([Thermal Chameleon, RA-L 2024](https://arxiv.org/abs/2410.18340)), and so does adaptive 14-to-8-bit rescaling ([Fieldscale, RA-L 2024](https://arxiv.org/abs/2405.15395)). Nobody we found has studied it together with INT8 inference.

## What we measured

Nothing yet. A measurement on a Raspberry Pi 5 is planned, and its rules are fixed before it runs:

- **Data**: the thermal anti-UAV benchmark (Anti-UAV410 and CST Anti-UAV), single-object tracking, where a target with a box diagonal of 10 pixels or less is "tiny".
- **What is reported**: per target-size bin and per contrast bin, recall at a matched false-alarm rate, the benchmark's state accuracy (SA), lock losses per minute and time to first detection; per layer, the target-to-background activation ratio in FP32 and INT8; latency for every configuration, with the device card.
- **Hypothesis**: at 8 bits, recall on targets of 10 pixels or less falls at least twice as much as overall AP, and the loss shows up at track level as more lock losses and a later first detection.
- **What would prove it wrong**: tiny-target recall falls no more than overall AP. Then the effect is a COCO-scale artefact, not a property of this setting, and this article will say so.

The result, either way, will be added here with its numbers.

## Open questions

1. **Thermal, 3 to 16 pixels.** No study of quantization damage on thermal anti-UAV data at these sizes.
2. **Single-object tracking.** Lock loss, re-acquisition time and time to first detection under INT8 are unmeasured; the only track-level study is RGB and multi-object.
3. **ARM CPU.** Prior work runs on GPUs (TensorRT) and FPGAs; the CPU path through ncnn, the common one on a Raspberry Pi, is unstudied.
4. **Contrast, not only size.** Results are binned by size; in thermal data a 6-pixel target can be easy or invisible. Is contrast the better predictor?
5. **Raw bit depth.** How does 14 or 16-bit input interact with an INT8 first layer, which quantizes the input anyway?
6. **A recipe at 8 bits.** Is there a cheap way to recover tiny-target recall at 8 bits, at near-INT8 latency? Published size-aware fixes either need training or pay off only at 4 bits and below.
7. **Which layers to protect.** No paper keeps exactly the early, high-resolution layers at FP16 for tiny targets on a CPU, and no rule says which layers to protect from target size and layer stride.

## Sources

- Li et al., [Fully Quantized Network for Object Detection](https://openaccess.thecvf.com/content_CVPR_2019/papers/Li_Fully_Quantized_Network_for_Object_Detection_CVPR_2019_paper.pdf), CVPR 2019.
- Chen et al., [AQD: Towards Accurate Quantized Object Detection](https://arxiv.org/abs/2007.06919), CVPR 2021.
- Xu et al., [Q-DETR: An Efficient Low-Bit Quantized Detection Transformer](https://arxiv.org/abs/2304.00253), CVPR 2023.
- Wang et al., [Q-YOLO: Efficient Inference for Real-time Object Detection](https://arxiv.org/abs/2307.04816), 2023.
- L. Johannsen, [Efficient small-object video detection via quantization-aware training](https://theses.liacs.nl/pdf/2025-2026-JohannsenLLenard.pdf), bachelor thesis, LIACS, Leiden University, 2026.
- [hailo-uav-tracker](https://github.com/pierrosimonestd-cpu/hailo-uav-tracker), community measurements (accuracy with ONNX Runtime, latency on a Raspberry Pi 5).
- Luo et al., [Quantization-Aware Training for Efficient Object Detection on FPGAs: Case Studies](https://www.bgd.ed.tum.de/pdf/2026_full_ISPRSCongress_QATDet.pdf), ISPRS Annals 2026.
- Dong et al., [Mask-Guided Local Feature Quantization for Small Object Detection (MLFQ)](https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/ipr2.70476), IET Image Processing 2026 (abstract only).
- Zhou et al., [LiDAR-PTQ](https://arxiv.org/abs/2401.15865), ICLR 2024.
- Zhu et al., [ScaleKD](https://openaccess.thecvf.com/content/CVPR2023/papers/Zhu_ScaleKD_Distilling_Scale-Aware_Knowledge_in_Small_Object_Detector_CVPR_2023_paper.pdf), CVPR 2023.
- Zamani et al., [CSCWD: Cross-Scale Channel-wise Knowledge Distillation for Lightweight Tiny Object Detection on Edge Devices](https://arxiv.org/abs/2609.30395), 2026.
- Lee et al., [Thermal Chameleon](https://arxiv.org/abs/2410.18340), RA-L 2024; Gil et al., [Fieldscale](https://arxiv.org/abs/2405.15395), RA-L 2024.

