Using these new TensorRT-LLM optimizations, NVIDIA has pulled out a huge 2.4x performance leap with its current H100 AI GPU in MLPerf Inference 3.1 to 4.0 with GPT-J tests using an offline scenario.

The AI chip giant says the open-source software library, TensorRT-LLM, will double the H100’s performance for running inference on leading large language models when it comes out next month. Nvidia ...

Nvidia plans to release an open-source software library that it claims will double the speed of inferencing large language models (LLMs) on its H100 GPUs. TensorRT-LLM will be integrated into Nvidia's ...

The company is adding its TensorRT-LLM to Windows in order to play a bigger role in the inference side of AI. The company is adding its TensorRT-LLM to Windows in order to play a bigger role in the ...

Rival GPU vendors Intel and Nvidia both support the latest large language models from Meta, Llama 3. According to Intel VP and GM of AI Software Engineering Wei Li, “Meta Llama 3 represents the next ...