Calling Model APIs from Python
Send messages to a hosted model or a local one, read the reply, stream long answers, count tokens and estimate cost, handle rate limits and errors with retries, and keep API keys safe.
Hosted APIs and local models
There are two ways to use an LLM from code:
| Hosted API (Anthropic, OpenAI, Google, …) | Local model (Ollama, llama.cpp, LM Studio) | |
|---|---|---|
| Quality | the strongest models | smaller open-weight models: good for many tasks, weaker at hard ones |
| Cost | pay per token | free after you own the hardware |
| Privacy | data goes to the provider (read their data-use terms) | data never leaves your machine |
| Setup | an API key and one pip install | a capable computer; a GPU or Apple silicon helps |
The code looks almost the same either way. Many local tools, including Ollama, also offer an OpenAI-compatible endpoint, so the same client code can talk to a local model by changing one address.
Your first API call
First, set your key as an environment variable in the terminal (not in the code): export ANTHROPIC_API_KEY=... on macOS or Linux, or setx ANTHROPIC_API_KEY "..." on Windows (then open a new terminal). The client library reads it automatically. Model names change often: copy a current one from your provider's documentation.
# pip install anthropic (reads ANTHROPIC_API_KEY from the environment)
from anthropic import Anthropic
client = Anthropic()
reply = client.messages.create(
model="MODEL_NAME", # copy a current model name from the provider's docs
max_tokens=300, # the most tokens the answer may use
system="You explain machine learning to beginners in plain English.",
messages=[{"role": "user", "content": "What is overfitting, in two sentences?"}],
)
print(reply.content[0].text)
print(reply.usage) # input and output token counts, for cost tracking# pip install openai (reads OPENAI_API_KEY from the environment)
from openai import OpenAI
client = OpenAI()
# for a local Ollama model instead:
# client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
reply = client.chat.completions.create(
model="MODEL_NAME", # e.g. a model you pulled with: ollama pull <name>
messages=[
{"role": "system", "content": "You explain machine learning to beginners in plain English."},
{"role": "user", "content": "What is overfitting, in two sentences?"},
],
temperature=0.2,
)
print(reply.choices[0].message.content)Streaming, conversations and limits
- Streaming sends the answer token by token as it is generated, so users see text appear immediately instead of waiting many seconds. Every major SDK supports it (look for
stream=Trueor astream()method). - Conversations: append each user message and assistant reply to the messages list and send the whole list each time. Long chats grow expensive; summarise or trim old turns.
- max_tokens caps the length of the answer (and your cost). If the answer stops mid-sentence, the response tells you it hit this limit.
- Rate limits: providers limit requests and tokens per minute. When you exceed them you get an HTTP 429 error. Servers also occasionally fail with 5xx errors. The fix for both is to retry with exponential backoff: wait 1 s, then 2 s, then 4 s, with a little randomness, up to a limit.
import random, time
class RateLimited(Exception):
pass
calls = {"n": 0}
def flaky_model_call(prompt):
# pretend API: fails with "rate limited" the first two times
calls["n"] += 1
if calls["n"] <= 2:
raise RateLimited("429 Too Many Requests")
return f"answer to: {prompt}"
def with_retries(fn, *args, tries=5, base=0.5):
for attempt in range(tries):
try:
return fn(*args)
except RateLimited as e:
if attempt == tries - 1:
raise
wait = base * 2 ** attempt + random.uniform(0, 0.1) # exponential backoff + jitter
print(f"attempt {attempt + 1} failed ({e}); waiting {wait:.2f}s")
time.sleep(wait)
print(with_retries(flaky_model_call, "What is a token?"))Try it: tokens and cost
Paste a typical prompt, set how long answers are and how many requests you expect, and enter your provider's prices (they change often, so copy them from the pricing page). The token count is the rough 4-characters-per-token estimate.
Interactive: this part runs in your browser.
Cutting cost and waiting time
- Pick the smallest model that passes your tests. Small models are often many times cheaper and faster.
- Keep prompts short and do not resend huge documents on every turn; retrieve only the relevant parts (RAG).
- Output tokens usually cost several times more than input tokens: ask for concise answers.
- Prompt caching: many providers charge much less for a long prompt prefix (system prompt, documents) that repeats across calls.
- Batch APIs process many requests within hours instead of seconds, usually at a large discount, which is ideal for labelling datasets.
- Cache answers to repeated identical questions in your own database.
Quiz
1. Your app gets HTTP 429 errors at busy times. What is the standard fix?
2. Why does a chat app send the whole conversation on every call?
3. A hospital cannot send patient notes to an outside company. Which option fits?
4. Which usually costs more per token?
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Ollama ↗ run open-weight LLMs on your own computer
- Anthropic: Prompt engineering overview ↗ practical, well-organised prompting guide from a model provider