Deploying cyankiwi Models

All cyankiwi quantized models are open source on Hugging Face and deploy out of the box as OpenAI-compatible servers with vLLM or SGLang. They are 4-bit AWQ checkpoints, up to 75% smaller than the BF16 originals, so you can serve leading LLMs on your own smaller hardware with no code changes.

This guide uses cyankiwi/Llama-3.1-8B-Instruct-AWQ-INT4 as the example. Replace it with any model id from the cyankiwi collection. Both engines expose the same OpenAI-compatible API at http://localhost:8000/v1, so the usage examples below work unchanged whichever engine you pick.

Deploy with vLLM

Python (NVIDIA GPU)

pip install vllm
vllm serve cyankiwi/Llama-3.1-8B-Instruct-AWQ-INT4

Docker

docker pull vllm/vllm-openai:latest
docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 --ipc=host \
  vllm/vllm-openai:latest \
  --model cyankiwi/Llama-3.1-8B-Instruct-AWQ-INT4

The container’s entrypoint is vllm serve, so anything after the image name is forwarded as flags. The complete vLLM user guide.

Deploy with SGLang

Python (NVIDIA GPU)

pip install "sglang[all]"
python3 -m sglang.launch_server \
  --model-path cyankiwi/Llama-3.1-8B-Instruct-AWQ-INT4 \
  --host 0.0.0.0 --port 8000

Docker

docker pull lmsysorg/sglang:latest
docker run --gpus all --shm-size 32g \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 --ipc=host \
  lmsysorg/sglang:latest \
  python3 -m sglang.launch_server \
    --model-path cyankiwi/Llama-3.1-8B-Instruct-AWQ-INT4 \
    --host 0.0.0.0 --port 8000

SGLang’s default port is 30000; the --port 8000 flag aligns it with vLLM so the examples below work for both. The complete SGLang documentation.

Using the API

Whichever engine you deployed, you now have an OpenAI-compatible server: any OpenAI client (SDK, curl, LangChain, etc.) works by pointing it at your base URL.

curl

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "cyankiwi/Llama-3.1-8B-Instruct-AWQ-INT4",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user",   "content": "Explain PagedAttention in one sentence."}
    ]
  }'

Add "stream": true for Server-Sent Events streaming.

Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="cyankiwi/Llama-3.1-8B-Instruct-AWQ-INT4",
    messages=[{"role": "user", "content": "Explain PagedAttention in one sentence."}],
)
print(resp.choices[0].message.content)

# streaming
for chunk in client.chat.completions.create(
    model="cyankiwi/Llama-3.1-8B-Instruct-AWQ-INT4",
    messages=[{"role": "user", "content": "Count to five."}],
    stream=True,
):
    print(chunk.choices[0].delta.content or "", end="")

Prefer not to deploy on your own GPUs? The hosted cyankiwi Inference API serves models optimized using our proprietary methods.