It started in 2007 (CUDA)

Transformers, the architecture behind ChatGPT, require billions of mathematical computations (matrix multiplications) that must be performed simultaneously (in parallel), particularly for the attention mechanism.

CUDA is the software layer that unlocks Nvidia GPUs to execute these billions of operations simultaneously.