It started in 2007 (CUDA)

Transformers, the architecture behind ChatGPT, require billions of mathematical computations, giant matrix multiplying each other at lightning speed that must be performed simultaneously in parallel, particularly for the attention mechanism.

CUDA is the software layer that unlocks Nvidia GPUs to execute these billions of operations simultaneously