TORONTO — Toronto has emerged as a hotbed for the software that semiconductor startups and technology giants need to compete for the business of supplying artificial intelligence computing capacity.

Modern chips are more than just wafers of silicon studded with transistors. Firms selling semiconductors, or access to them, also write code that lets developers easily and efficiently deploy their machine-learning (ML) models and applications on those systems. “Software is critical in any of these chip companies—it’s the window into the hardware,” said Alex Grbic, vice-president of software engineering at Untether AI. 

Headquartered just off Toronto’s high-rise financial district, Untether makes semiconductors specialized for running deep-learning systems. More than half of the startup’s 170-person team of staff and contractors works on software. It is one of several firms—new entrants and established majors, homegrown and international—with teams in the city writing critical code for AI chips.

Companies are drawn by the city’s base of skilled workers. Toronto was “a very good place to find people who worked on very hard software problems, who were unafraid of software that controlled hardware,” said Andrew Feldman, CEO of Cerebras. The Sunnyvale, Calif., startup plans to double the 100-person workforce of its office here, which it opened in March 2020. 

Firms that recently formed or arrived in the city have recruited graduates from nearby universities, and more experienced staff from the longer-established Toronto offices of tech multinationals. Advanced Micro Devices (AMD) has maintained a major presence in the suburban city of Markham, Ont., since buying ATI Technologies in July 2006. Intel’s Canadian operations include a sizable Toronto technology centre, bolstered by its June 2015 acquisition of Altera. 

Executives say Toronto researchers and labs have helped advance technologies that are used in today’s AI chip architectures, or are closely related to them. That includes systems that run multiple processors in parallel; compilers, which translate applications into code that hardware can run; and field-programmable gate arrays, which allow circuits to be adjusted without a soldering iron. 

The city also has engineers versed in the AI powered by the chips, with Toronto researchers key to the development of deep learning; Untether co-founder Martin Snelgrove worked down the hall from field pioneer Geoffrey Hinton when both were University of Toronto professors in the late 1980s.

Like Cerebras, fellow Silicon Valley startup Groq’s compiler team is based in Toronto. The outpost, opened in October 2020, is led by vice-president of software engineering Andrew Ling, another Altera-Intel alumnus. Taalas and Tenstorrent—two venture-backed AI chip startups founded by a former AMD executive—are also hiring software engineers in Toronto.

Cloud giants, to whose business AI has given a major boost, are benefiting from the city’s talent pool, too. Toronto houses the Amazon Web Services (AWS) team working on the software development kit for its in-house chips for training and running deep-learning models. “It’s a critical piece of the stack,” said Matt Wood, AWS vice-president for AI. 

The software ensures that popular models are compatible with the cloud provider’s chips. It has proved so capable and efficient, Wood said, that clients selling generative tools no longer need to use more expensive graphics processing units (GPUs), the traditional workhorses of deep learning. 

“The kind of stuff that I do is definitely more in demand now,” said Antonio Kim, an ML infrastructure engineer at Cerebras. He works on a team that translates users’ programs like large language models (LLMs) into code that’s understandable to the firm’s compiler.

Kim first joined Cerebras in September 2020 as a co-op student from the University of Waterloo, returning full time the following May. He was drawn by the performance of its chip and the chance to work on ML infrastructure. Few of his classmates followed him into the space, with the LLM-driven AI boom still to come. Today, “a lot more people are trying to tackle this from a hardware perspective,” Kim said.

Competition for Toronto workers capable of writing good chip code has increased as other AI hardware firms set up in the city, acknowledged Nish Sinnadurai, vice-president of software engineering and Canadian general manager at Cerebras. But more firms beget more talent. “We’ve created a stronger community,” said Sinnadurai, who previously worked at Altera and Intel. Untether’s Grbic noted that salaries for workers with semiconductor software skills have been rising steadily.

The work done by these Toronto chip software teams could prove very valuable as that AI boom drives up demand for computing capacity, and the companies employing them try to meet it.

Nvidia currently dominates the lucrative market. The Santa Clara, Calif.-headquartered firm’s sales and stock have skyrocketed as cloud services and developers clamour for its GPUs. Nvidia’s “hardware is very good, but their biggest barrier to entry, at this point, is their software stack,” said Vaughn Betz, a U of T professor who’s worked with Intel and Cerebras. (Another faculty member at the university, Sanja Fidler, leads Nvidia’s Toronto AI lab). 

To break that hold, Intel and AMD as well as model-makers OpenAI, Google, Microsoft and Meta are contributing to an open-source alternative to Nvidia’s software. Startups are focusing on their own code, with the same objective: “We want to present an easy-to-use, unified interface that allows a customer to switch … to our architecture, without having to relearn exactly how to use this device,” said Grbic.

To do that, Untether will lean on hometown Toronto’s chip software expertise. “These are world-class researchers and developers,” Gbric said. “There really is a draw here.”