guide : setting up NVIDIA DGX Spark with ggml #16514
ggerganov
started this conversation in
Show and tell
Replies: 3 comments 3 replies
-
|
This is an outstanding guide and I was able to get everything up and running on the Spark with minimal fuss. I did run into an interesting issue. Right now out of the box with the latest updates, it looks like my nvcc version was reported as this and my gcc version was Apparently, there were some changes made to the gcc vector type definitions starting with version 13, so I had to revert and rebuild with gcc 12 by appending these headers to your shell script ...{everything before line 48}
printf "[I] Installing llama.cpp\n"
git clone https://github.com/ggml-org/llama.cpp ~/ggml-org/llama.cpp
cd ~/ggml-org/llama.cpp
cmake -B build-cuda -DCMAKE_C_COMPILER=/usr/bin/gcc-12 -DCMAKE_CXX_COMPILER=/usr/bin/g++-12 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-12 -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc -DGGML_CUDA=ON
cmake --build build-cuda -j
printf "[I] Installing whisper.cpp\n"
git clone https://github.com/ggml-org/whisper.cpp ~/ggml-org/whisper.cpp
cd ~/ggml-org/whisper.cpp
cmake -B build-cuda -DCMAKE_C_COMPILER=/usr/bin/gcc-12 -DCMAKE_CXX_COMPILER=/usr/bin/g++-12 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/g++-12 -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc -DGGML_CUDA=ON
cmake --build build-cuda -j
...{everything after line 58} |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
Thanks for the feedback. That's weird - I have the same config as you and there are no issues with the build: ggml@spark-17ed:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Wed_Aug_20_01:57:39_PM_PDT_2025
Cuda compilation tools, release 13.0, V13.0.88
Build cuda_13.0.r13.0/compiler.36424714_0
ggml@spark-17ed:~$ gcc --version
gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
ggml@spark-17ed:~$ bash <(curl -s https://ggml.ai/dgx-spark.sh)
[I] Setting up NVIDIA DGX Spark for local AI with ggml
[W] The directory '~/ggml-org' already exists and will be deleted. Continue? (y/N) y
[I] Proceeding...
[I] Installing llama.cpp
Cloning into '/home/ggml/ggml-org/llama.cpp'...
remote: Enumerating objects: 65253, done.
remote: Counting objects: 100% (104/104), done.
remote: Compressing objects: 100% (86/86), done.
remote: Total 65253 (delta 72), reused 18 (delta 18), pack-reused 65149 (from 4)
Receiving objects: 100% (65253/65253), 178.28 MiB | 20.42 MiB/s, done.
Resolving deltas: 100% (47464/47464), done.
-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
...
[I] Downloading Whisper model...
[P] Starting service on port 8021 (embd) ...
[P] Starting service on port 8022 (fim) ...
[P] Starting service on port 8023 (chat, tools) ...
[P] Starting service on port 8024 (vision) ...
[P] Starting service on port 8025 (stt) ...
[I] Downloading models and waiting for services to become healthy - please wait ... (this can take a long time)
[P] Service on port 8021 is ready (waiting for 4 services to initialize ...)
[P] Service on port 8022 is loading model ... (waiting for 4 services to initialize ...)
[P] Service on port 8024 is loading model ... (waiting for 4 services to initialize ...)
[P] Service on port 8025 is ready (waiting for 3 services to initialize ...)
[P] Service on port 8023 is loading model ... (waiting for 3 services to initialize ...)
[P] Service on port 8024 is ready (waiting for 2 services to initialize ...)
[P] Service on port 8022 is ready (waiting for 1 services to initialize ...)
[P] Service on port 8023 is ready (waiting for 0 services to initialize ...)
[I] All ggml services are up and ready - your NVIDIA DGX Spark is ready to use!
[I] Entering monitoring loop (Ctrl-C to stop) |
Beta Was this translation helpful? Give feedback.
-
|
It was a weird issue for sure and the build environments should be identical since I'd have to assume you're running the same Ubuntu variant that comes with all DGX Sparks, but I couldn't for the life of me get it to compile. But the workaround sorted it out whereas before I was getting this. |
Beta Was this translation helpful? Give feedback.
-
|
just got my spark, had to install the libcurl dev package so cmake could find curl but otherwise no issues |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Thanks for the report - fixed it with ggml-org/ggml-org.github.io@c13a770. Any additional feedback is highly appreciated. |
Beta Was this translation helpful? Give feedback.
-
|
Why is the model LLama 3.3 70B (4 bits quantization) so slow on the DGX Spark, using the latest version of llama.cpp (checkout 25 october 2025) ? On other consumer architectures this model is very fast with llama.cpp. Can you give me some suggestions ? Thank you command line (the model was quantized 2 days ago with the latest version of llama.cpp): In the following a video. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Overview
In this guide we will configure the NVIDIA DGX™ Spark as a local and private AI assistant using the ggml software stack. The guide is geared towards developers and builders. We are going to setup the following AI capabilities:
These features will run simultaneously, in your local network, allowing you to fully utilize the power of your device at home or in the office.
Software
We are going to use the following open-source software:
Setup
Simply run the following command in a terminal on your NVIDIA DGX™ Spark:
bash <(curl -s https://ggml.ai/dgx-spark.sh)Note
The
dgx-spark.shscript above is quite basic and is merely one of the many possible ways you can configure your device for AI use cases. It is provided here mainly for convenience and as an example. Feel free to inspect it and adjust it for your needs.The command downloads and builds the latest version of the
ggmlsoftware stack and starts multiple HTTP REST services as shown in the following table:http://localhost:8021http://localhost:8022http://localhost:8023http://localhost:8024http://localhost:8025The first time running the command can take a few minutes to download the model weights. If everything goes OK, you should see the following output:
At this point, the machine is fully configured and ready to be used. Internet connection is no longer necessary.
Here's sample output of
nvidia-smiwhile theggmlservices are running:Use cases
Here is a small fraction of the AI use cases that are possible with this configuration.
Basic chat
Simply point your browser to the chat endpoint
http://localhost:8023:Inline code completions (FIM)
Install the llama.vim plugin in your Vim/Neovim editor and configure it to use the FIM endpoint
http://localhost:8022:In VSCode, install the llama.vscode extension and configure it in a similar way to use the FIM endpoint:
Coding agent
In VSCode, configure the llama.vscode extension to use the endpoints for completions, chat, embeddings and tools:
Document and image processing
Submit PDFs and image documents in the WebUI to analyze them with a multimodal LLM. For visuals, use the vision endpoint
http://localhost:8024:Audio transcription
Use the speech-to-text endpoint at
http://localhost:8025to quickly transcribe audio files:Performance
For performance numbers, see Performance of llama.cpp on NVIDIA DGX Spark
Conclusion
The new NVIDIA DGX Spark is a great choice for serving the latest AI models locally and privately. With 128GB of unified system memory it has the capacity to host multiple AI services simultaneously. And the
ggmlsoftware stack is the best way to do that.Beta Was this translation helpful? Give feedback.
All reactions