Model quantization
also: quantization · INT8 · 4-bit models
Running an AI model at lower numeric precision, such as 8 or 4 bits instead of 16 or 32, to cut memory and speed up inference at some cost in accuracy.
Neural network weights are normally stored as 16- or 32-bit floating-point numbers. Quantization maps them to fewer bits, shrinking memory roughly in proportion (a 4-bit model needs about a quarter of the memory of the same model at 16 bits) and often speeding up inference on hardware with low-precision math support. It is how large language models fit on single GPUs and workstations, and how neural networks run on microcontrollers.
Accuracy usually drops somewhat, more at lower bit widths and for smaller models; careful methods such as calibration and quantization-aware training reduce the loss.
Founder takeaway
Benchmark quantized models on your own tasks before deploying; how much quality loss is acceptable depends on the application.
engineer review pending