Local LLM

Open-weight language models can run on a workstation, an on-site server or, in smaller versions, edge devices. Data never leaves your infrastructure, the system works without internet access, per-query costs become fixed hardware costs, and you control exactly which model version is in use.

The trade-offs: smaller local models are generally less capable than the largest hosted models, you need GPUs or other accelerators with enough memory, and you take on serving, updates and monitoring. Quantization reduces memory needs at some cost in quality. Running a trained model (inference) is far cheaper than training one, and most local deployments only do inference, sometimes with light fine-tuning.

For hardware companies, local models suit engineering knowledge assistants over confidential designs, factory and field-service assistants, and products that must work offline.