Instructions to use unsloth/Kimi-K2-Instruct-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use unsloth/Kimi-K2-Instruct-GGUF with Transformers:

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("unsloth/Kimi-K2-Instruct-GGUF", dtype="auto")

llama-cpp-python

How to use unsloth/Kimi-K2-Instruct-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="unsloth/Kimi-K2-Instruct-GGUF",
	filename="BF16/Kimi-K2-Instruct-BF16-00001-of-00045.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use unsloth/Kimi-K2-Instruct-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf unsloth/Kimi-K2-Instruct-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama-cli -hf unsloth/Kimi-K2-Instruct-GGUF:UD-Q4_K_XL

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf unsloth/Kimi-K2-Instruct-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama-cli -hf unsloth/Kimi-K2-Instruct-GGUF:UD-Q4_K_XL

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf unsloth/Kimi-K2-Instruct-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./llama-cli -hf unsloth/Kimi-K2-Instruct-GGUF:UD-Q4_K_XL

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf unsloth/Kimi-K2-Instruct-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./build/bin/llama-cli -hf unsloth/Kimi-K2-Instruct-GGUF:UD-Q4_K_XL

Use Docker

docker model run hf.co/unsloth/Kimi-K2-Instruct-GGUF:UD-Q4_K_XL

LM Studio
Jan
Ollama
How to use unsloth/Kimi-K2-Instruct-GGUF with Ollama:
```
ollama run hf.co/unsloth/Kimi-K2-Instruct-GGUF:UD-Q4_K_XL
```

Unsloth Studio

How to use unsloth/Kimi-K2-Instruct-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/Kimi-K2-Instruct-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for unsloth/Kimi-K2-Instruct-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for unsloth/Kimi-K2-Instruct-GGUF to start chatting

How to use unsloth/Kimi-K2-Instruct-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf unsloth/Kimi-K2-Instruct-GGUF:UD-Q4_K_XL

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "unsloth/Kimi-K2-Instruct-GGUF:UD-Q4_K_XL"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use unsloth/Kimi-K2-Instruct-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf unsloth/Kimi-K2-Instruct-GGUF:UD-Q4_K_XL

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default unsloth/Kimi-K2-Instruct-GGUF:UD-Q4_K_XL

Run Hermes

hermes

Docker Model Runner
How to use unsloth/Kimi-K2-Instruct-GGUF with Docker Model Runner:
```
docker model run hf.co/unsloth/Kimi-K2-Instruct-GGUF:UD-Q4_K_XL
```

Lemonade

How to use unsloth/Kimi-K2-Instruct-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull unsloth/Kimi-K2-Instruct-GGUF:UD-Q4_K_XL

Run and chat with the model

lemonade run user.Kimi-K2-Instruct-GGUF-UD-Q4_K_XL

List all available models

lemonade list

danielhanchen commited on Jul 17, 2025

Commit

4051186

verified ·

1 Parent(s): a15af49

Upload folder using huggingface_hub

Browse files

Files changed (23) hide show

Q8_0/Kimi-K2-Instruct-Q8_0-00001-of-00023.gguf +2 -2
Q8_0/Kimi-K2-Instruct-Q8_0-00002-of-00023.gguf +2 -2
Q8_0/Kimi-K2-Instruct-Q8_0-00003-of-00023.gguf +2 -2
Q8_0/Kimi-K2-Instruct-Q8_0-00004-of-00023.gguf +2 -2
Q8_0/Kimi-K2-Instruct-Q8_0-00005-of-00023.gguf +2 -2
Q8_0/Kimi-K2-Instruct-Q8_0-00006-of-00023.gguf +2 -2
Q8_0/Kimi-K2-Instruct-Q8_0-00007-of-00023.gguf +2 -2
Q8_0/Kimi-K2-Instruct-Q8_0-00008-of-00023.gguf +2 -2
Q8_0/Kimi-K2-Instruct-Q8_0-00009-of-00023.gguf +2 -2
Q8_0/Kimi-K2-Instruct-Q8_0-00010-of-00023.gguf +2 -2
Q8_0/Kimi-K2-Instruct-Q8_0-00011-of-00023.gguf +2 -2
Q8_0/Kimi-K2-Instruct-Q8_0-00012-of-00023.gguf +2 -2
Q8_0/Kimi-K2-Instruct-Q8_0-00013-of-00023.gguf +2 -2
Q8_0/Kimi-K2-Instruct-Q8_0-00014-of-00023.gguf +2 -2
Q8_0/Kimi-K2-Instruct-Q8_0-00015-of-00023.gguf +2 -2
Q8_0/Kimi-K2-Instruct-Q8_0-00016-of-00023.gguf +2 -2
Q8_0/Kimi-K2-Instruct-Q8_0-00017-of-00023.gguf +2 -2
Q8_0/Kimi-K2-Instruct-Q8_0-00018-of-00023.gguf +2 -2
Q8_0/Kimi-K2-Instruct-Q8_0-00019-of-00023.gguf +2 -2
Q8_0/Kimi-K2-Instruct-Q8_0-00020-of-00023.gguf +2 -2
Q8_0/Kimi-K2-Instruct-Q8_0-00021-of-00023.gguf +2 -2
Q8_0/Kimi-K2-Instruct-Q8_0-00022-of-00023.gguf +2 -2
Q8_0/Kimi-K2-Instruct-Q8_0-00023-of-00023.gguf +2 -2

Q8_0/Kimi-K2-Instruct-Q8_0-00001-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d46e150aa9fdeb2ff2586425e668dc6736b1d7d4514ea59ba544c8289ea4f0b6
-size 44040276896

 version https://git-lfs.github.com/spec/v1
+oid sha256:536d2f032cfb034c2f44207e6958ddbef582e9c4d43bda91666b5082bcaebbee
+size 45411161440

Q8_0/Kimi-K2-Instruct-Q8_0-00002-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9af945c670d35ce49a628f61117f1c05aa8b01a703cfe0c0a4874766234d113d
-size 48411668448

 version https://git-lfs.github.com/spec/v1
+oid sha256:4fa917009be30b128ae994ae3edbc26004d5dc970856c59ba85d55d600c17cf5
+size 48396070752

Q8_0/Kimi-K2-Instruct-Q8_0-00003-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8946f88c09db88440b1961133d740b56bc395ef61acc4654aadf88d46b90146f
-size 48246355520

 version https://git-lfs.github.com/spec/v1
+oid sha256:35aa821b58c390122ab32fe266889e746810173e0c874888c0c05fc6d0bb1b64
+size 48288589568

Q8_0/Kimi-K2-Instruct-Q8_0-00004-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:903153a7ed518c029fe57517a3acaf36f35bb42120a464cbaf5747bdf621d8a9
-size 48411668448

 version https://git-lfs.github.com/spec/v1
+oid sha256:83d8df9776c1c7f55c20ae2c4b57fc67138711de6103d5e7b41ffa4b62cd5c59
+size 48385031968

Q8_0/Kimi-K2-Instruct-Q8_0-00005-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:898077a78faf55057e1e6ae2f4578f8db1b70f123c53bb66cc95848f92452b45
-size 48411668448

 version https://git-lfs.github.com/spec/v1
+oid sha256:8cc06dd510a34dfbadcc65c082c60b86953f5dbb5e85fc59df2a422661f25c61
+size 48396070816

Q8_0/Kimi-K2-Instruct-Q8_0-00006-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e628dbfe474e7d43b7aa707fa93f377ca565f9e70e03853d8be5ff45b0a9d7d2
-size 48246355520

 version https://git-lfs.github.com/spec/v1
+oid sha256:f3b026d78b99d1d41f05b90290a6f71e7188e3a46b5ab6997e8aca7c5fc70dda
+size 48288589632

Q8_0/Kimi-K2-Instruct-Q8_0-00007-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1851492c8d083f09a56895a4c749c34f0397f77524ae3d856bcf3ea6c32fc08f
-size 48411668448

 version https://git-lfs.github.com/spec/v1
+oid sha256:62e4e88ca3ad7c47eaf5f4a34d0d2f4dc74d32abbf97b8edc01c91b16d8f5a6b
+size 48385031968

Q8_0/Kimi-K2-Instruct-Q8_0-00008-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9bd819572f551cef3da65689750ffca86ed114463b2aa9761cdbdf55a6c6da4c
-size 48411668448

 version https://git-lfs.github.com/spec/v1
+oid sha256:c07cc209989f1f8af2a1f878d0364c3019f6ab9079faab9f587d477c738263e4
+size 48396070816

Q8_0/Kimi-K2-Instruct-Q8_0-00009-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7ab8210d355ac32e801b428ad5dc09216402f3fc23cddc81881c4ca742de98da
-size 48246355488

 version https://git-lfs.github.com/spec/v1
+oid sha256:1c5a4d4fd9b07ed48064aac415530f4b33fafa9339a062752c102abc654acde7
+size 48288589632

Q8_0/Kimi-K2-Instruct-Q8_0-00010-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:aa98a0ff3607709c815530becd173bd1041ca73523378ff5077c6e92d6d6004b
-size 48411668448

 version https://git-lfs.github.com/spec/v1
+oid sha256:b5fcdfbd2c9840c0023bbd799112caedf2e62c74132174f49a8cf9194954361d
+size 48385031968

Q8_0/Kimi-K2-Instruct-Q8_0-00011-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:74d6e455d78d75e52e805cd6231832dc54cd68d16c6a88a3bb29c7e6dc04ea58
-size 48411668448

 version https://git-lfs.github.com/spec/v1
+oid sha256:6ca56c93f59a2330da0d5549b69e0c588af1d8848c4e77b7010c069011e26682
+size 48396070816

Q8_0/Kimi-K2-Instruct-Q8_0-00012-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bedca0e24882c1e2fc5155d5bc1d55716081f46859c8932d9f7c880fda4a73e9
-size 48246355520

 version https://git-lfs.github.com/spec/v1
+oid sha256:16d57f22c8f928e5ea765c309b8944a2da2daf919243a4411b68e46ee2f66780
+size 48288589632

Q8_0/Kimi-K2-Instruct-Q8_0-00013-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8d619287fd4f916590cd39f72f3de4d6f2112cf7026afca3ce89bfd7147a032e
-size 48411668448

 version https://git-lfs.github.com/spec/v1
+oid sha256:2fc31d4d289921a0f1ffb37a49131ee0dfec07386b54b86fc12c265f9a6ce73c
+size 48385031968

Q8_0/Kimi-K2-Instruct-Q8_0-00014-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:605af9360dc50571f224a5d534149e57661fa75086ad834acd4b03a63f497a9b
-size 48411668448

 version https://git-lfs.github.com/spec/v1
+oid sha256:7fb13f2801434fb718e417373f2a0d5c441bbbef2f9d9dcf0363371b35858d08
+size 48396070816

Q8_0/Kimi-K2-Instruct-Q8_0-00015-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a4b9c694161d659c38d83c58deb71f20c36776f6340664ff73ac6c9700abfb50
-size 48246355520

 version https://git-lfs.github.com/spec/v1
+oid sha256:ef7b2bbaf538cf148aca847c02ceb6c1ac4ea94dd9536d58bd424db65b098913
+size 48288589632

Q8_0/Kimi-K2-Instruct-Q8_0-00016-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:005bc053a59d4222a62bbc9ba11457e52d719a4fee5dab046d51927703fd9d2f
-size 48411668448

 version https://git-lfs.github.com/spec/v1
+oid sha256:e14c8574023d807d4d543e7b2c8568c221028879ec4614be46f918e3a0ac0a94
+size 48385031968

Q8_0/Kimi-K2-Instruct-Q8_0-00017-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bc5c2e622a6def0eac71a0d752dda8418a7753bf68e9dafe1b572e0d61e084fb
-size 48411668448

 version https://git-lfs.github.com/spec/v1
+oid sha256:2e9f8876c753c5fd0d2e747c76a696389c77407dc9b883ab93ca389fe6257450
+size 48396070816

Q8_0/Kimi-K2-Instruct-Q8_0-00018-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a01a837e06cffc74f1e828cfe56fe0e4fa042bfce07e586088cc833e4198eb37
-size 48246355520

 version https://git-lfs.github.com/spec/v1
+oid sha256:a6fb74354f1fe2e219d4f368aedf6fb8875c6f38c5be1986ede0a09c27af704e
+size 48288589632

Q8_0/Kimi-K2-Instruct-Q8_0-00019-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a5f96d4c39d4ca93f5f2e5cf041afe4fd74a1f6cd7a28d04d704826686b29908
-size 48411668448

 version https://git-lfs.github.com/spec/v1
+oid sha256:a8f6056daad8419a69ad42f2a2cfae26c608cb7ef7681c5f656fb7252565a1fa
+size 48385031968

Q8_0/Kimi-K2-Instruct-Q8_0-00020-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7a6b7cbaf17100271779e176cdea6ceb6ff030ce30a06cd175d767a7a3ef83ad
-size 48411668448

 version https://git-lfs.github.com/spec/v1
+oid sha256:79a8cbd4f2599689a3d0515c4c8ab952346b243cb7e3f9452f1d56b429c625fc
+size 48396070816

Q8_0/Kimi-K2-Instruct-Q8_0-00021-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8128be9e8d96d0a9387feb90eed0e618dfecad8501341e9473e48dbd8b6b3d9e
-size 48246355488

 version https://git-lfs.github.com/spec/v1
+oid sha256:b1311264d2f0a2ef458346d8f1b3b2c39fc14c690e5463a94535ab86d663951a
+size 48288589632

Q8_0/Kimi-K2-Instruct-Q8_0-00022-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c4805238a1d44ebecdc7703d9155ec36949707bb8fb1a9bdcceee2fbb84597be
-size 49659502656

 version https://git-lfs.github.com/spec/v1
+oid sha256:0022b25eb9b2199aec4fd7ac21f68ea89b32cdd9103937784e742a11958aee23
+size 48385031968

Q8_0/Kimi-K2-Instruct-Q8_0-00023-of-00023.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:35ffd7f9d59dadc86776d5952f2b10060421bdb4b3e449c8aa43a03e0ecba82d
-size 30277928192

 version https://git-lfs.github.com/spec/v1
+oid sha256:7536608e33a65ac02c18d108b176b80323669449bcb0be6bdaa243352c96e72c
+size 30154878112