Instructions to use pipecat-ai/phonellm-alpha-1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use pipecat-ai/phonellm-alpha-1 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="pipecat-ai/phonellm-alpha-1", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("pipecat-ai/phonellm-alpha-1", trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained("pipecat-ai/phonellm-alpha-1", trust_remote_code=True, device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use pipecat-ai/phonellm-alpha-1 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "pipecat-ai/phonellm-alpha-1" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "pipecat-ai/phonellm-alpha-1", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/pipecat-ai/phonellm-alpha-1
- SGLang
How to use pipecat-ai/phonellm-alpha-1 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "pipecat-ai/phonellm-alpha-1" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "pipecat-ai/phonellm-alpha-1", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "pipecat-ai/phonellm-alpha-1" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "pipecat-ai/phonellm-alpha-1", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use pipecat-ai/phonellm-alpha-1 with Docker Model Runner:
docker model run hf.co/pipecat-ai/phonellm-alpha-1
Model confirms details back instead of calling the tool, even when told not to
Title
Model confirms details back instead of calling the tool, even when told not to
Body
Sharing a reproducible behavior we hit while evaluating alpha-1 for a phone agent. On flows where a tool call must happen before the conversation can proceed, PhoneLLM reliably spends the turn reading the caller's details back and asking for confirmation, instead of calling the tool.
Minimal reproduction
Three-line system prompt, exactly one tool, an explicit instruction to call it immediately, temperature=0, thinking disabled.
System prompt
You are a phone assistant for a medical clinic.
To look up a patient record you must first verify identity.
When the caller has given first name, last name and date of birth, call the
verify_identity tool immediately. Do not ask for anything else.
Tool — a single verify_identity function taking first_name, last_name, date_of_birth.
User turn
Hi, I'd like to look at my account. My name is Jordan Avery, date of birth March 20, 1985.
All three required arguments are present in that one utterance.
PhoneLLM Alpha 1 — no tool call, 3/3 samples:
"I can certainly help you with that. Just to confirm, that's Jordan Avery,
date of birth March 20, 1985. Is that correct?"
gpt-4.1-mini, same prompt, same tool, same user turn — tool call, 3/3 samples:
tool_calls: [verify_identity]
content: ""
The behavior is unchanged at temperature=0 and 0.1, with 1 tool or 27, and with a short prompt or a large production one. Adding the anti-narration rules from the pipecat-examples/phonellm bot ("don't narrate your own tool use", "Either answer, or call the tool and then say what it did") removed the trailing question mark and nothing else.
Why it matters for gated flows
The example bot is a reservation flow, where confirming before booking is natural and the tool fires at the end. Our flow inverts that: identity verification is a gate — nothing can proceed until the tool returns. "Speak first, act second" costs a turn every time, and in a scripted evaluation it fails the turn outright.
Questions
- Is confirm-before-acting intended? The training description mentions calling "the right tools at the right time," which we read as the opposite of this.
- Is there a prompt pattern that makes it act on the same turn? We could not find one.
- Would
tool_choice: "required"work here, or does that fight the fine-tune?
Happy to share more traces. Serving on Modal AutoEndpoints, vLLM, bf16, 65K context.