AI Infra Summit – Tokenomics has a new name: Inferra by Lightbits™- an intelligent KV cache orchestration engine that dramatically improves GPU utilization, enabling long-context and multi-session AI workloads, and making its public debut at the AI Infra Summit on September 15th. Built by Lightbits Labs®, inventors of the NVMe over TCP storage protocol, Inferra™ tackles one of AI inference’s most pertinent challenges — the memory bottlenecks created by exploding KV caches with long context lengths in LLMs — enabling Neoclouds and Enterprise organizations to run Inference faster, scale bigger, and spend less money on AI infrastructure.

“The AI inference market will top $117 billion this year, forcing legacy vendors designed for model training and ingest to scramble and re-architect,” said Avigdor Willenz, Co-Founder and Chairman of Lightbits Labs. “Inference is a completely different paradigm – retrofitting legacy training and storage systems to solve the KV cache bottleneck simply doesn’t work. It was built for a different era. We engineered Inferra from the ground up specifically to solve the GPU efficiency problem, and it has already proven successful in customer beta programs. Disrupting the status quo with tech that raises the bar for data infrastructure performance and efficiency is in our DNA. Applying those same principles to Inference that we applied when inventing NVMe over TCP makes Inferra uniquely positioned to deliver significant value for this market.”

Inferra maximizes hardware utilization, eliminating idle GPU cycles and redundant context recomputation, enabling organizations to run multi-session clusters with strict SLA guarantees. By virtualizing GPU memory across memory and storage tiers and transforming the KV cache into an intelligent, persistent data layer, Inferra fundamentally changes the economics of inference.

“Our collaboration with Lightbits has demonstrated a clear path to maximizing AI infrastructure efficiency,” added Yaniv Fdida, Chief Product & Technology Officer at OVH. “With Inferra’s intelligent KV cache tiering, we were able to demonstrate substantial GPU utilization gains, paving the way to providing our customers a more scalable and cost-effective foundation for their AI agent and RAG workloads.”

Key AI inference workload breakthroughs on display:

  • Infinite HBM Virtualization & 16x Session Density: Extends and shares the KV cache beyond limited GPU memory, enabling organizations to run multi-session clusters with strict SLA guarantees. Serve up to 16x more concurrent inference sessions on existing GPU infrastructure – without costly hardware upgrades.

  • >100x Latency Improvement & 10M-Token Context Windows: Drastically lowers Time-to-First-Token (TTFT) and Time Per Output Token (TPOT) by proactively and predictively pre-fetching attention states from storage rather than recomputing them, enabling next-gen AI assistants and multi-session agents that require context windows extending well beyond GPU memory limitations.

  • AI-Native Security and Isolation: Delivers consistent SLAs with secure tenant isolation and intelligent KV cache management across shared inference infrastructure.

AI infrastructure teams, hyperscalers, and GPU cloud providers are invited to visit Booth 219 at the AI Infra Summit for live demonstrations and to learn how Inferra delivers superior token economics and unbounded LLM scale. Book a product demonstration with the onsite team of engineers and experts today.

Rewrite your own tokenomics! See the difference Inferra can make to your top line and profit margins; try the Pod Efficiency Analyzer today.

Supporting Quotes

Avi Shetty, VP of Ecosystem, Solutions & Market Enablement, Solidigm

“Scaling long-context inference workloads requires rethinking how data moves between the GPU, high-performance memory and storage. Results from testing Lightbits’ Inferra engine in the Solidigm AI Central Lab, leveraging Solidigm’s D7-PS1010 enterprise NVMe SSDs, showcased how network attached storage with intelligent software layer virtualization can break through the memory wall for large context, agentic AI. This collaboration empowers hyperscalers and cloud providers to achieve superior tokenomics while running multi-session workloads with uncompromising SLAs.”

Alexey Stolyar, Chief Technology Officer, ICC

“We’re seeing strong momentum as enterprises and AI infrastructure providers move beyond model training and focus on the economics of inference. Organizations need technologies that maximize utilization of large-scale AI inference environments. We see Inferra as an innovative solution that provides inference and token efficiency. We look forward to helping customers evaluate its potential.”

Scott Sinclair, Practice Director, Infrastructure, Cloud & DevOps, OMDIA

“The current surge in AI inference workloads necessitates a strategic rethinking in infrastructure design. Inferra’s predictive prefetching across memory tiers more effectively optimizes GPU investments by helping eliminate KV cache bottlenecks and subsequent GPU stalls. It’s an innovative approach to virtualizing memory that directly targets the real cost and performance drivers for Neocloud providers.”

Steven Dickens, CEO and Principal Analyst, HyperFRAME Research

State of the Enterprise AI Stack

“Our research indicates that while AI is strategically critical, execution is often bottlenecked by underlying data architectures. Accelerating LLM workloads at scale requires intelligent orchestration at the inference software layer. Lightbits’ Inferra addresses the KV cache memory bottleneck, maximizing hardware investments into much higher-yielding assets. By drastically lowering latency and increasing session density on existing hardware, Inferra gives Neoclouds a measurable path to improving AI infrastructure ROI.”

Matt Kimball, VP & Principal Analyst, Moor Insights & Strategy

“Inference economics increasingly come down to how efficiently memory and GPU capacity get used, not how much compute is deployed. Inferra is going after that directly, orchestrating across memory tiers to raise utilization and support longer-context sessions. That’s the right constraint to be attacking as inference scales.”

Ramesh Chettuvetty, Senior Vice President of AI Product and Business, Lightbits Labs

“We’re reinventing inference by shipping the world’s first KV cache acceleration engine that eliminates idle GPUs and power context windows of up to 10M tokens on commodity hardware. By delivering Smart Infinite HBM Virtualization across memory tiers, Inferra enables significant latency improvements with no hardware upgrades. Inferra delivers instant payback and generates net positive savings from day one. It eliminates the stalls that traditionally limit long-context performance, so Neoclouds and enterprises can serve larger models and longer conversations at lower infrastructure cost.”

About Lightbits Labs

Lightbits Labs (Lightbits®) is the inventor of the NVMe over TCP storage protocol, which is natively built into its industry-leading block storage, and the first KV cache prefetch engine for AI acceleration. Lightbits data infrastructure solutions are engineered to deliver unmatched high performance and maximum hardware efficiency for LLM inference, real-time analytics, and transactional workloads at scale. Lightbits is backed by enterprise technology leaders [Cisco Investments, Dell Technologies Capital, Intel Capital, J.P. Morgan, Lenovo, and Micron] and is on a mission to deliver best-in-class, cost-efficient software solutions for performance-sensitive workloads at scale.

To learn more about Lightbits Labs, visit https://www.lightbitslabs.com/ and follow Lightbits Labs on Linkedin, X, Facebook, Instagram, and YouTube.

Lightbits, Lightbits Labs, Lightbits SDS, and Inferra are registered trademarks of Lightbits Labs, Ltd.

Media gallery

About The Author
Business Wire