- Kove:SDM has created software that virtualizes memory and creates a shared memory pool
- It says its tech can deliver performance without compromising latency across distances of 150 meters
- Analysts said its tech could be useful for hyperscalers, but weren't convinced its right for the telco edge
Virtualization has slowly come for every bit of hardware that rules the world. First, it washed over CPUs and local area networks. Then it swept across the GPU landscape. Memory is now the latest target of virtualization efforts from companies like Kove:SDM and others and for good reason: the shift could help tear down a major wall holing back AI performance.
In a nutshell, Kove has developed software that can be deployed across servers in a data center to pool their memory and dynamically allocate it where it is needed without compromising latency. Its technology has already been adopted by Swift for its global payments processing system and trialed by the likes of Red Hat and Supermicro.
The company’s work has major implications for hyperscalers and data center design but also – if CEO John Overton is right – for server vendors and even telcos looking to run AI workloads at the edge of the network.
Why? Well, something called the “memory wall’ is currently holding back AI performance.
What’s the problem?
Memory is required to feed data into the processors that run AI workloads. The term “memory wall” refers to the fact that while processing power has increased dramatically memory hasn’t scaled in parallel, making memory bandwidth and latency limiting factors for performance.
To cope, some underutilize available memory – hoping to avoid a memory shortage during processing that could cause a workload to crash. Others overbuy compute in order to gain access to additional memory. The industry has also created high-bandwidth memory (HBM) as a solution, but this is both very costly and doesn’t completely solve the problem.
So, a number of companies are looking at other ways to tackle the memory wall. For instance, Majestic Labs – founded by the executives that built Google and Meta’s silicon divisions – is building a new class of server architecture focused squarely on memory.
But Kove thinks the answer is virtualization.
New approach
According to Overton, virtualization solves a number of issues. First, it obviously helps break down the wall hindering performance. But he said it also helps increase memory utilization and can help lower energy costs by putting memory in the virtualized pool into a low-power mode when it’s not being used.
One of the primary concerns associated with virtualization is the fact that utilizing memory from outside of a system on chip tends to come with a latency tax. That is, the father the connection between the memory and the processor, the greater the latency.
Overton, however, said Kove has found a way to hide the latency associated with its pooled memory up to a distance of 150 meters and enable processing performance to occur at the same speed as if it were using local memory.
“This isn’t magic, it’s science,” Overton told Fierce.
Why it matters
If Kove can actually deliver on what it is promising, Gartner Research VP Joseph Unsworth said its technology would have a “pretty profound” impact on the market. After all, lends itself to efficiency and reduced cost in a world where “cost is a big deal.” Plus, he pointed out, there’s currently a massive memory shortage squeezing the market.
The ability to pool memory could also have implications for data center design and if it was very broadly adopted, it could even start to impact the compute and server vendors who have been benefitting from those who have been over-purchasing compute to get extra memory.
But Unsworth said he still has questions around both total cost of ownership and performance, and doesn’t think broad adoption really makes sense.
“You’re not going to use this for Tier 2 business applications. You’re going to use this where you’re absolutely pushing the boundaries,” he told Fierce. Of course, in that case “any degradation in performance would be problematic” and Unsworth said he’d like to see more evidence that Kove can deliver sustained performance.
“When you virtualize there can be a memory tax, especially if you go farther and farther away, there’s got to be some latency issues,” Unsworth said. “If you’re dealing with those very largest, most demanding customers…They cannot tolerate any downtime at all or any disruption in service.”
J.Gold Associates Founder and Principal Jack Gold offered a similar take, telling Fierce Kove’s technology makes the most sense for Amazon, Google and Microsoft, which both have scale and are already running other virtualized elements.
What is the telco angle?
Then there’s the telco angle. Though it doesn’t yet have any telco customers, Overton said the company is part of bids for 10,000 to 30,000 server installations for unnamed telcos.
He argued Kove’s technology stands to benefit operators looking to run AI inferencing workloads more efficiently on the network edge – without using pricey GPUs. In essence, he argued that instead of buying small clusters of pricey servers for each tower, telcos could buy more inexpensive servers and pair them with a memory pool. This, he said, would allow them to cut costs while still delivering “superior performance” for inferencing workloads on the edge.
He didn’t name names, but anyone who has been paying attention to AI-RAN and Nvidia’s attempts to convince operators to deploy GPUs at the edge can see why this might derail those plans and why AMD could benefit.
But Gold was skeptical – Kove’s software doesn’t make much sense for small on-prem deployments and operators aren’t going to share compute between towers, he argued. Gold added it might make sense to deploy Kove’s technology at a more centralized network location, but then telcos wouldn’t want to process inferencing workloads there.