Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:
- On a reasonably "mature" ubuntu22 machine, I ran `./deploy/bin/install-k3s-nvidia.sh'. It spun for a long time, but the GPU capacity never came online - timed out after 10 minutes. So the setup failed.
- I could see that the various pods in the
gpu-operator space were failing to come online.
Expected behavior
I'd like if the install magically worked every time, but I recognize that's a tall order.
Additional context
The apt upgrade -y step took a very long time, because apparently this machine hadn't been upgraded in a while. I could see it was churning through many many gigabytes of cuda stuff in the process. After the setup script timed out, nvidia-smi failed saying the drivers were (something) which I recognize as commonly requiring a reboot.
The workaround was to reboot the machine. The nvidia drivers came back. I ran the install script again, and it finished quite quickly, but complained (incorrectly this time) that the GPU capacity wasn't online. So that seems like a bug in the test.
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:
gpu-operatorspace were failing to come online.Expected behavior
I'd like if the install magically worked every time, but I recognize that's a tall order.
Additional context
The
apt upgrade -ystep took a very long time, because apparently this machine hadn't been upgraded in a while. I could see it was churning through many many gigabytes of cuda stuff in the process. After the setup script timed out,nvidia-smifailed saying the drivers were (something) which I recognize as commonly requiring a reboot.The workaround was to reboot the machine. The nvidia drivers came back. I ran the install script again, and it finished quite quickly, but complained (incorrectly this time) that the GPU capacity wasn't online. So that seems like a bug in the test.