Paddock: Truespar's Open-Source Rust/C++ Inference Engine with Custom CUDA Kernels

Native Implementation and Kernel-Level Optimization

At 32 concurrent clients, Paddock delivers a median 0.697-second time to first token, versus 2.5 seconds for vLLM and 6.9 seconds for llama.cpp. That gap between raw computational power and the user’s perception of responsiveness defines the stakes of inference. The industry has long accepted a trade-off: ease of deployment for moderate efficiency, or complex optimization for narrow hardware support. Truespar’s release of Paddock challenges this binary by open-sourcing a native Rust and C++ inference engine equipped with custom CUDA kernels, prioritizing low-level hardware optimization over generalized abstraction to eliminate the latency that plagues current high-throughput environments [1]kblip.comPaddock: Rust inference engine claims 697 ms TTFT on Qwen3.6-27B, beating vLLM and llama.cppOpen the source to inspect the supporting evidence.Open source ↗[4]reddit.comWe open-sourced Paddock, our Rust/C++ inference engine with its own CUDA kernels (MIT/Apache-2.0)Open the source to inspect the supporting evidence.Open source ↗.

Paddock distinguishes itself from contemporary inference engines through its choice of substrate, built as a native Rust inference server for open models on NVIDIA GPUs [5]github.comGitHub - truespar/paddock: Native Rust inference server for open models on NVIDIA GPUsOpen the source to inspect the supporting evidence.Open source ↗. This is a structural necessity for performance, leveraging Rust’s memory safety and concurrency primitives to avoid the garbage collection pauses inherent in Python or the manual memory management risks of C++. The engine is released under dual MIT and Apache-2.0 licenses, a strategic decision that lowers the barrier for enterprise adoption while allowing for commercial flexibility [4]reddit.comWe open-sourced Paddock, our Rust/C++ inference engine with its own CUDA kernels (MIT/Apache-2.0)Open the source to inspect the supporting evidence.Open source ↗. This licensing structure signals that the project is designed for broad integration into both open-source and proprietary software stacks, removing the legal friction that often stalls new infrastructure.

The core differentiator lies in the custom CUDA kernels, which Truespar emphasizes are not generic wrappers around existing libraries but are written in-house and tuned per NVIDIA GPU generation [2]truespar.comPaddock - High-Throughput LLM Inference for NVIDIA GPUsOpen the source to inspect the supporting evidence.Open source ↗. This granular approach to hardware optimization is essential for maximizing throughput, with documentation highlighting support for Blackwell's block-scaled FP4 tensor cores to leverage the latest hardware capabilities from inception. By tuning these kernels specifically for each generation, Truespar aims to eliminate the overhead associated with generalized kernel implementations, extracting performance that generalized frameworks might miss. The GitHub repository hosting the code provides technical documentation that supports these claims, offering a transparent view of the native Rust implementation and its integration with NVIDIA’s compute capabilities [5]github.comGitHub - truespar/paddock: Native Rust inference server for open models on NVIDIA GPUsOpen the source to inspect the supporting evidence.Open source ↗.

Performance Claims and Benchmarking Analysis

The performance metrics released alongside Paddock are bold, directly challenging the dominance of existing leaders with a Time-To-First-Token (TTFT) claim of 697 milliseconds for the Qwen3.6-27B model with 32 concurrent clients [1]kblip.comPaddock: Rust inference engine claims 697 ms TTFT on Qwen3.6-27B, beating vLLM and llama.cppOpen the source to inspect the supporting evidence.Open source ↗. This metric determines the perceived responsiveness of the system, and independent coverage by KBlip reported this specific benchmark condition, providing external validation of the claim [1]kblip.comPaddock: Rust inference engine claims 697 ms TTFT on Qwen3.6-27B, beating vLLM and llama.cppOpen the source to inspect the supporting evidence.Open source ↗. To put this in perspective, the same workload on vLLM reportedly took 2.5 seconds, while llama.cpp required 6.9 seconds. The drop from 6.9 seconds to under 700 milliseconds represents a profound shift in how quickly interactive applications can begin responding to user input, transforming usability by enabling immediate feedback loops previously impossible with standard inference stacks.

Throughput, or the number of tokens generated per second over time, is equally important for batch processing and high-load scenarios. Truespar claims that Paddock is up to 37 times faster than llama.cpp on GGUF files, a figure supported by evidence showing that 13 of 13 scenarios tested were faster than llama.cpp when using identical GGUF files [8]truespar.com13 of 13 Scenarios faster than llama.cpp Identical GGUF fileOpen the source to inspect the supporting evidence.Open source ↗. The speedup ranged from 1.5x to 37.5x, indicating that the advantage is consistent across various workloads but varies in magnitude depending on the specific configuration. The significant gains over llama.cpp are likely attributable to Paddock’s native CUDA kernel optimization versus llama.cpp’s more generalized approach, which must accommodate a wider variety of hardware and formats. This suggests that for users strictly on NVIDIA hardware, the abstraction layer of llama.cpp introduces significant overhead that Paddock bypasses entirely.

When compared to vLLM, the gains are more modest but still notable. Truespar claims that 13 of 13 scenarios are faster than vLLM using identical FP8 checkpoints, with speedups ranging from 1.02x to 1.19x [6]truespar.com13 of 13 Scenarios faster than vLLM Identical FP8 checkpointOpen the source to inspect the supporting evidence.Open source ↗. While a 19% increase in speed might seem small, in the context of high-throughput server inference, even single-digit percentage improvements can translate to significant cost savings or increased capacity. Similarly, Paddock is reported to be faster than SGLang in 10 of 13 scenarios with identical FP8 checkpoints, achieving up to a 9.4% speedup [7]truespar.com10 of 13 Scenarios faster than SGLang Identical FP8 checkpointOpen the source to inspect the supporting evidence.Open source ↗. These comparisons suggest that Paddock’s scheduler and key-value cache management offer consistent marginal gains over vLLM in specific workloads, challenging the assumption that vLLM is the undisputed king of throughput. The claim of being faster than SGLang in the majority of tested scenarios further validates the engine’s efficiency in handling complex tensor operations [7]truespar.com10 of 13 Scenarios faster than SGLang Identical FP8 checkpointOpen the source to inspect the supporting evidence.Open source ↗.

API Compatibility and Ecosystem Integration

A common critique of new inference engines is their lack of compatibility with existing tools and workflows, as developers often hesitate to adopt new software if it requires rewriting application code or integrating with proprietary APIs. Paddock addresses this concern head-on by serving both OpenAI and Anthropic-compatible API endpoints [5]github.comGitHub - truespar/paddock: Native Rust inference server for open models on NVIDIA GPUsOpen the source to inspect the supporting evidence.Open source ↗. This dual compatibility is a strategic move to reduce friction for users migrating from other platforms, allowing applications built for OpenAI or Anthropic to run on Paddock with minimal to no code changes. The GitHub repository confirms the implementation of these compatible endpoints, ensuring that the engine can plug directly into the broader ecosystem of LLM tools.

This compatibility layer is essential for Paddock’s adoption, as the local LLM community is fragmented with many tools and interfaces built around the OpenAI API standard. By adhering to these standards, Paddock ensures that it can be used with a wide range of front-end applications, chat interfaces, and development libraries. The inclusion of Anthropic-compatible endpoints further broadens its appeal, as some advanced prompting techniques and model features are specific to Anthropic’s API structure. This approach positions Paddock not just as a raw inference engine, but as a drop-in replacement for existing infrastructure, lowering the adoption barrier significantly. The engine’s ability to function as a native Rust server while maintaining this high level of API compatibility demonstrates a sophisticated engineering effort to balance performance with usability [5]github.comGitHub - truespar/paddock: Native Rust inference server for open models on NVIDIA GPUsOpen the source to inspect the supporting evidence.Open source ↗.

Market Positioning and Future Implications

The release of Paddock occurs at a time when the demand for efficient, local inference is growing rapidly, as models become larger and more complex, increasing the computational requirements for running them locally or in private cloud environments. Truespar’s positioning of Paddock targets high-throughput inference for NVIDIA GPUs, directly competing with established players like vLLM and llama.cpp. The emphasis on a native Rust implementation and custom hardware-specific kernels is a clear signal that Truespar believes the current frameworks have hit a performance ceiling and that a new architectural approach is necessary to unlock further gains. The blog post introducing Paddock highlights these high-performance capabilities as the primary value proposition [3]truespar.comPaddock: High-Performance Inference - Truespar BlogOpen the source to inspect the supporting evidence.Open source ↗.

The market implications of this release are significant. If Paddock’s performance claims hold up under independent, rigorous testing, it could force a reevaluation of the industry standards. The 37x speedup over llama.cpp alone is a disruptive claim that could drive a migration of users who prioritize latency and throughput over the broader hardware support that llama.cpp offers. Similarly, the consistent outperformance of vLLM in FP8 scenarios suggests that there is still room for optimization in the high-throughput space. This competition is healthy for the ecosystem, as it drives innovation and pushes developers to explore new languages and hardware optimizations.

However, skepticism is warranted. The performance claims are attributed to Truespar and require independent verification across a wider range of models and hardware configurations. The marginal gains over vLLM, while consistent, may not justify the effort required to migrate to a new engine for all users. The significant gains over llama.cpp are impressive but may be less relevant for users who rely on CPU inference or non-NVIDIA hardware. The success of Paddock will depend on its ability to maintain performance advantages as models evolve and as the community contributes to its development. The dual MIT/Apache-2.0 licensing facilitates this, allowing for broad community involvement and commercial integration [4]reddit.comWe open-sourced Paddock, our Rust/C++ inference engine with its own CUDA kernels (MIT/Apache-2.0)Open the source to inspect the supporting evidence.Open source ↗.

Conclusion

The open-sourcing of Paddock represents a significant event in the ongoing evolution of local LLM inference, combining a native Rust foundation with custom, per-generation CUDA kernels to challenge the performance boundaries set by vLLM and llama.cpp. The claims of 697 ms TTFT and up to 37x speedup over llama.cpp are bold and, if verified, could reshape the landscape of high-performance inference. The engine’s compatibility with OpenAI and Anthropic APIs ensures that it can integrate seamlessly into existing workflows, lowering the barrier to adoption.

As the community reviews the code and conducts independent benchmarks, the true value of Paddock will become clear. If it delivers on its promises, it will stand as a testament to the power of specialized, native implementations in the age of large language models. If it falls short, it will still have contributed to the discourse on hardware optimization and language choice in inference engines. Regardless of its ultimate success, Paddock’s release marks a decisive moment where the industry is forced to look beyond the established frameworks and consider new architectural possibilities. The race for inference efficiency is far from over, and Paddock is a formidable new contender in that race.