Open-weight model installation

Model weights are not part of this repository or its releases. Downloading a model means accepting its publisher's license and disk/security implications.

Before installing any model

Read the model card and license.
Confirm the files are from the publisher's repository.
Check free disk space before starting.
Keep model files outside Git.
Do not run two GPU-heavy runtimes at the same time.
Treat an “abliterated” or security-tuned model like any powerful local tool: keep the endpoint private and follow applicable law/policy.

GGUF / CyberStrike 35B

Hardware choice

For an RTX 4090 24 GB, use the publisher's Q4_K build. The full BF16 checkpoint is about 66–70 GB and is not a practical all-GPU fit. Q8 also exceeds 24 GB and requires substantial CPU/RAM offload.

Download

Use the exact GGUF companion repository, not the original safetensors repository:

https://huggingface.co/huihui-ai/Huihui-CyberStrike-OffSec-35B-abliterated-GGUF

Download:

Huihui-CyberStrike-OffSec-35B-abliterated-Q4_K.gguf (the publisher may label its internal quantization Q4_K_M)
mmproj-model-bf16.gguf for vision, if desired

The original Huihui-CyberStrike-OffSec-35B-abliterated repository contains Transformers safetensors and is not directly loadable by llama.cpp/Ollama.

Place files

KimiRoot\models\CyberStrike-OffSec-35B\
  CyberStrike-OffSec-35B-Q4_K_M.gguf
  CyberStrike-OffSec-35B-mmproj-bf16.gguf

Friendly filenames are recommended. Other .gguf files are discovered recursively; projector pairing is conservative and only attaches an unambiguous neighboring mmproj/projector file.

Install llama.cpp

Download a current Windows x64 build from https://github.com/ggml-org/llama.cpp/releases.
Choose the GPU backend compatible with your machine.
Extract the complete runtime into:
```
KimiRoot\runtime\llama.cpp\bin\
```

Verify this file exists:

KimiRoot\runtime\llama.cpp\bin\llama-server.exe

Do not copy only llama-server.exe; keep the release's required DLLs beside it.

Launch

Open Runtime Center and select the discovered CyberStrike 35B entry. The app:

validates required file sizes/GGUF headers for the bundled profile;
starts llama-server on a random private port with a random bearer token;
waits for /v1/models readiness;
runs an actual structured-tool-call protocol test;
starts Kimi Web only after validation passes.

The local model does not require or contact Ollama. Kimi search/fetch tools remain normal Kimi services and may use the internet.

Kimi K3 (2.8T) / AirLLM 3.1

Reality check

This is the complete 1.56 TB, 96-shard moonshotai/Kimi-K3 checkpoint. AirLLM streams layers/experts so it can execute on a single CUDA GPU, but disk bandwidth makes it extremely slow.

Validated local result on RTX 4090/64 GB Windows:

real initialization: about 28.5 minutes;
one-token app response: about 14.5 minutes;
text-only native-Windows compatibility path;
desktop cap: 2,048 context and one output token.

AirLLM's own upstream RTX 6000 Ada result reports roughly 900 seconds initialization and 292 seconds/token. Do not expect identical performance on different storage/GPU/platform combinations.

Required storage and filesystem

At least 1.70 TiB free before download.
Additional headroom for cache/temp/logs.
Put the checkpoint and layer-shards on the same NTFS volume so hard links work.
Never place the model inside the Git checkout if that checkout is synced/backed up automatically.

Install dependencies only

From the repository root:

powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\Install-AirLlmK3.ps1 `
  -KimiRoot "$env:USERPROFILE\Desktop\kimi"

This creates:

KimiRoot\airllm-v3.1.0\
  .venv\
  server\
  models\
  model-cache\
  layer-shards\

It installs the pinned CUDA/Python packages and runs verify_install.py. It does not download a model without the explicit switch.

Download the exact checkpoint

powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\Install-AirLlmK3.ps1 `
  -KimiRoot "$env:USERPROFILE\Desktop\kimi" `
  -DownloadCheckpoint

The downloader pins revision:

9f62e4e9fffbd0a83ddd60e1c209d828994b3569

It is resumable. Rerun the same command after a network or power interruption.

Verification and patching

The installer verifies:

required K3 metadata files;
exactly 96 shard filenames from model.safetensors.index.json;
all files exist and are non-empty;
total tensor bytes equal 1,560,936,091,448;
safetensor headers can be opened;
pinned AirLLM/CUDA packages and GPU visibility.

It then applies apply_windows_k3_compat.py. The patch is exact/idempotent and preserves upstream copies. If the upstream source no longer matches AirLLM 3.1.0/K3's pinned code, it fails instead of applying an unsafe fuzzy edit.

First launch and layer cache

Choose Kimi K3 (2.8T) in Runtime Center. The app requires the existing local checkpoint and sets the AirLLM child to offline mode. On the first run AirLLM creates/resumes layer-shards\splitted_model; the desktop accepts a clean or internally consistent partial cache but rejects empty tensors/orphan completion markers.

Do not close the app during initial preparation unless necessary. If interrupted, the next launch resumes completed layers using .done markers.

Internet behavior

The model child receives:

HF_HUB_OFFLINE=1
TRANSFORMERS_OFFLINE=1
HF_DATASETS_OFFLINE=1
HF_HUB_DISABLE_TELEMETRY=1

KIMI_AIRLLM_MODEL_PATH must resolve to an existing local directory. The Python adapter repeats this check before importing AirLLM. This prevents silent checkpoint downloads during normal app launch.

Those variables are not applied to Kimi Web. Search, FetchURL, OAuth refresh, and other Kimi internet services remain available if the account/network permits them.

License

Kimi K3 uses the publisher's Kimi K3 License, including commercial service/revenue and attribution conditions. Read the current license before using or redistributing the model:

https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE

This project does not grant additional rights to the checkpoint.

FilesExpand file tree

MODELS.md

Latest commit

History