Skip to main content

Nvidia’s banking on TensorRT to expand its generative AI dominance

Nvidia’s banking on TensorRT to expand its generative AI dominance


The company is adding its TensorRT-LLM to Windows in order to play a bigger role in the inference side of AI.

Share this story

Illustration of an Nvidia logo
Illustration by Alex Castro / The Verge

Nvidia announced that it’s adding support for its TensorRT-LLM SDK to Windows and models like Stable Diffusion as it aims to make large language models (LLMs) and related tools run faster. TensorRT speeds up inference, the process of going through pretrained information and calculating probabilities to come up with a result — like a newly generated Stable Diffusion image. With this software, Nvidia wants to play a bigger part on that side of generative AI.

TensorRT-LLM breaks down LLMs like Meta’s Llama 2 and other AI models like Stability AI’s Stable Diffusion to let them run faster on Nvidia’s H100 GPUs. The company said that by running LLMs through TensorRT-LLM, “this acceleration significantly improves the experience for more sophisticated LLM use — like writing and coding assistants.”

This way Nvidia can not only provide the GPUs that train and run LLMs but also provide the software that allows models to run and work faster so users don’t seek other ways to make generative AI cost-efficient. The company said TensorRT-LLM will be “available publicly to anyone who wants to use or integrate it” and can access the SDK on its site.

Nvidia already has a near monopoly on the powerful chips that train LLMs like GPT-4 — and to train and run one, you typically need a lot of GPUs. Demand has skyrocketed for its H100 GPUs; estimated prices have reached $40,000 per chip. The company announced a newer version of its GPU, the GH200, coming next year. No wonder Nvidia’s revenues increased to $13.5 billion in the second quarter.

But the world of generative AI moves fast, and new methods to run LLMs without needing a lot of expensive GPUs have come out. Companies like Microsoft and AMD announced they’ll make their own chips to lessen the reliance on Nvidia. 

And companies have set their sights on the inference side of AI development. AMD plans to buy software company to help LLMs specifically run on AMD chips, while companies like SambaNova already offer services that make it easier to run models as well. 

Nvidia, for now, remains the hardware leader in generative AI, but it already looks like it’s angling for a future where people don’t have to depend on buying huge numbers of its GPUs.