docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

[niktheod]

I have docker and nvidia-container-toolkit installed on my system (Ubuntu 24.04) but I keep getting the error on the title when I try to run any image that needs access to gpu.

Here are some more details for my setup:

(base) nikostheodoridis@Aurora1:\~$ docker --version
Docker version 27.4.1, build b9d17ea

(base) nikostheodoridis@Aurora1:\~$ dpkg -l | grep nvidia-container-toolkit
ii nvidia-container-toolkit 1.17.3-1 amd64 NVIDIA Container toolkit
ii nvidia-container-toolkit-base 1.17.3-1 amd64 NVIDIA Container Toolkit Base

(base) nikostheodoridis@Aurora1:\~$ cat /etc/docker/daemon.json
{
"runtimes": {
"nvidia": {
"path": "nvidia-container-runtime",
"runtimeArgs": []
}
}
}

(base) nikostheodoridis@Aurora1:\~$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nvidia/cuda 12.6.3-cudnn-devel-ubuntu24.04 50efab398f76 6 weeks ago 12.9GB
airsplay/bottom-up-attention latest 66d17015f76c 5 years ago 5.92GB

(base) nikostheodoridis@Aurora1:~$ docker run --rm --gpus all nvidia/cuda:12.6.3-cudnn-devel-ubuntu24.04 nvidia-smi
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].`

Any idea what could be wrong?