Instructions to use pipecat-ai/phonellm-alpha-1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use pipecat-ai/phonellm-alpha-1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="pipecat-ai/phonellm-alpha-1", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("pipecat-ai/phonellm-alpha-1", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("pipecat-ai/phonellm-alpha-1", trust_remote_code=True, device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use pipecat-ai/phonellm-alpha-1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "pipecat-ai/phonellm-alpha-1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pipecat-ai/phonellm-alpha-1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/pipecat-ai/phonellm-alpha-1

SGLang

How to use pipecat-ai/phonellm-alpha-1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "pipecat-ai/phonellm-alpha-1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pipecat-ai/phonellm-alpha-1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "pipecat-ai/phonellm-alpha-1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pipecat-ai/phonellm-alpha-1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use pipecat-ai/phonellm-alpha-1 with Docker Model Runner:
```
docker model run hf.co/pipecat-ai/phonellm-alpha-1
```

vLLM prefix caching for phonellm (nemotron_h): bug report + first fix upstream

by kushalpatil - opened Aug 31

Discussion

kushalpatil

Aug 31

Heads-up for the phonellm team: we benchmarked phonellm-alpha-1 for a multi-turn voice workload and found vLLM's prefix caching is a no-op for nemotron_h hybrids — a byte-identical resend re-prefills the whole transcript every turn (v0.28.0 and nightly, all cache modes/KV dtypes), which capped concurrency ~5x below an equivalent-activation attention model on one L40S. We root-caused the CPU-backend half and sent a fix upstream, with the details and a no-GPU repro here:

Bug report: https://github.com/vllm-project/vllm/issues/54504
First fix (CPU wrappers, greedy warm==cold verified): https://github.com/vllm-project/vllm/pull/54505

The GPU-path no-op on the v0.28 images is still open in the issue. Since turn-2+ TTFT for phone calls rides on prefix reuse, you probably want this fixed more than anyone — a nudge/review from your side on the issue would help. Happy to share the benchmark harness details.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment