BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal

The Native Metal Breakthrough: BaseRT Redefines Apple Silicon Inference

A new research paper from arXiv introduces BaseRT, a runtime specifically engineered for LLM inference on Apple Silicon using Apple’s native Metal graphics and compute framework. The authors claim it achieves the highest inference throughput ever recorded on this hardware, outperforming established solutions like llama.cpp and MLX-based frameworks.

The core innovation lies in BaseRT’s deep integration with Metal Performance Shaders (MPS) and careful memory management tailored to the unified memory architecture of Apple’s M-series chips. Unlike cross-platform runtimes that rely on generic GPU backends, BaseRT leverages Metal’s low-level control over shader cores, tensor operations, and memory bandwidth. This allows it to minimize kernel launch overhead and maximize utilization of the Neural Engine and GPU simultaneously.

Why This Matters

Apple Silicon has long been a paradox for AI practitioners: the hardware is remarkably powerful, with massive unified memory (up to 192GB on M2 Ultra) and impressive raw compute, yet software support has lagged. Existing runtimes treat Apple GPUs as second-class citizens, often falling back to CPU inference or using suboptimal Metal abstractions. BaseRT changes this by proving that native Metal optimization can close—or even eliminate—the performance gap with NVIDIA GPUs for local inference.

For the broader AI ecosystem, this signals a maturation of Apple’s platform as a viable inference target. The unified memory architecture is particularly advantageous for large models, as it avoids the PCIe bandwidth bottleneck that plagues discrete GPU setups. BaseRT’s approach could make running 70B-parameter models feasible on a single Mac Studio, a task that currently requires multiple high-end NVIDIA GPUs.

Implications for AI Practitioners

First, local inference on Apple hardware becomes dramatically more practical. Developers building privacy-sensitive applications or offline tools can now expect near-datacenter throughput on consumer hardware. This is especially relevant for edge AI, where latency and data sovereignty are critical.

Second, the research highlights the importance of platform-specific optimization. The gains reported by BaseRT suggest that generic frameworks like llama.cpp leave significant performance on the table. Practitioners evaluating inference solutions should consider whether “good enough” cross-platform support is worth the throughput penalty.

Third, this development may accelerate Apple’s push into AI infrastructure. If third-party runtimes can achieve best-in-class performance, Apple has strong incentive to formalize and support such efforts—potentially through official Metal AI libraries or tighter integration with Xcode.

However, caution is warranted. The paper’s claims are based on specific benchmarks and model sizes; real-world performance may vary with model architecture, quantization level, and batch size. Additionally, BaseRT is currently a research prototype, not a production-ready tool. Adoption depends on whether the authors release it as open-source software and whether the community can replicate the results.

Key Takeaways

• BaseRT achieves the highest reported LLM inference throughput on Apple Silicon by using native Metal optimizations, outperforming llama.cpp and MLX-based runtimes.
• This demonstrates that Apple’s unified memory architecture can compete with discrete GPU setups for local inference, especially for large models.
• Practitioners should evaluate platform-specific runtimes for Apple hardware rather than relying solely on cross-platform frameworks.
• The research is promising but remains a prototype; real-world adoption requires open-source release and community validation.