Microsoft Expands Windows ML for Local GGUF Model Execution

Microsoft added experimental llama.cpp support to Windows ML, allowing developers to run supported GGUF language models locally. The Text Generation API can handle GGUF and ONNX models through one interface, using different inference backends. Microsoft also offers an OpenAI-compatible local endpoint, but the APIs remain experimental and not production-ready.

Microsoft’s update lets Windows ML accept GGUF models through an experimental llama.cpp path while keeping ONNX Runtime for compatible ONNX models. The Text Generation API selects a backend by model format, giving developers one interface for supported text-generation workloads. ONNX models must expose expected token inputs, logits outputs, and the required fixed-capacity state or KV cache arrangement.

The local OpenAI-style endpoint is aimed at quick trials: existing OpenAI SDK clients can target localhost instead of a cloud service. Current limits include decoding only through greedy methods and no controls for sampling or structured output. Native APIs support C++ and Python, not C#, and Microsoft warns against shipping Store apps that rely on these experimental interfaces.