TORONTO — As businesses clamour for the compute to run artificial intelligence systems, startups selling software to squeeze every drop of power from that hardware are generating plenty of interest of their own. 

Sandwiched between an AI application and chips are several layers of programs that translate machine-learning models into instructions for the processors. Startups CentML and Lemurian say they can help customers get better performance by optimizing what happens between AI applications and the hardware they run on.

“We make AI go fast and run more cheaply on different types of hardware,” said Jay Dawani, Lemurian’s CEO and co-founder. The startup, based in Menlo Park, Calif. and with an office in Toronto, claims its technology can help developers roll out large language and other machine-learning models faster by rewriting the code that tells the chips that power them what to do.

Making more kinds of AI hardware systems more efficient could also reduce the industry’s reliance on Nvidia’s graphics processing units (GPUs), which are both extremely expensive and incredibly difficult to find as demand for them soars.

Companies making AI models don’t have the time or resources to tune each one for all the possible chips on which they could run, said Gennady Pekhimenko, CEO and co-founder of CentML, a Toronto-based startup. “The way we write applications, it’s never going to be fully efficient.” His company’s technology automates the work of optimizing clients’ models for the chips they choose by tapping under-used hardware capacity and using other tricks.

The type of business interested in this kind of boost has evolved over time. 

Pekhimenko, a University of Toronto researcher, started CentML in March 2022, months before OpenAI launched ChatGPT. At the time, CentML focused on firms training AI systems from scratch—an expensive and time-consuming process. But the market became more limited as OpenAI, Cohere, Meta and others began offering access to high-quality models for a fee or free.

“There are very few customers in the world [that] can afford pre-training right now,” said Pekhimenko. Instead, CentML is targeting clients that are tweaking existing models for their own needs, or building them into applications. For example, CentML claims EquoAI, a Toronto startup, greatly sped up its development of large language models and cut the cost of deploying them by $250,000 annually using its CServe product.

CentML has raised US$30.9 million in venture funding to date, from backers including Nvidia, Radical Ventures and Google’s Gradient Ventures. The firm has 46 employees, mostly in its hometown of Toronto. 

Lemurian, headquartered in Menlo Park, Calif., also has a large Toronto presence, with the city home to about a third of its 27 staff, including CEO Dawani. The startup has raised US$14 million from investors including Vancouver’s Elevation Capital.

As well as selling software directly to businesses, Lemurian and CentML are working with cloud service providers to make the technology available to their customers. Compute providers have gone big on generative AI, launching their own models, chips and tools to help clients adopt the technology. 

Neither Pekhimenko nor Dawani are worried that cloud or semiconductor giants will try to replicate what their startups do and cut them out of the market, arguing it’s simply too difficult or costly for larger companies to optimize customers’ AI for so many different hardware combinations and use cases.

Lemurian and CentML are also signing up more chipmakers, who in return get another place to show off their products. “The software stack is mostly a burden that they’ve had to take on,” said Dawani. 

Both firms also expect the market for their software to keep growing as AI developers launch more and different kinds of models, and businesses use them more.“We are getting good traction,” said CentML’s Pekhimenko, but “it’s still relatively early.”