Generative AI & LLM Apps
Topic 4 · Building LLM apps · ⏱ about 3 h

Calling Model APIs from Python

Send messages to a hosted model or a local one, read the reply, stream long answers, count tokens and estimate cost, handle rate limits and errors with retries, and keep API keys safe.

Show:
In this topic
Beginner

Hosted APIs and local models

There are two ways to use an LLM from code:

Hosted API (Anthropic, OpenAI, Google, …)Local model (Ollama, llama.cpp, LM Studio)
Qualitythe strongest modelssmaller open-weight models: good for many tasks, weaker at hard ones
Costpay per tokenfree after you own the hardware
Privacydata goes to the provider (read their data-use terms)data never leaves your machine
Setupan API key and one pip installa capable computer; a GPU or Apple silicon helps

The code looks almost the same either way. Many local tools, including Ollama, also offer an OpenAI-compatible endpoint, so the same client code can talk to a local model by changing one address.

Beginner

Your first API call

First, set your key as an environment variable in the terminal (not in the code): export ANTHROPIC_API_KEY=... on macOS or Linux, or setx ANTHROPIC_API_KEY "..." on Windows (then open a new terminal). The client library reads it automatically. Model names change often: copy a current one from your provider's documentation.

Python (Anthropic)
# pip install anthropic   (reads ANTHROPIC_API_KEY from the environment)
from anthropic import Anthropic

client = Anthropic()
reply = client.messages.create(
    model="MODEL_NAME",          # copy a current model name from the provider's docs
    max_tokens=300,              # the most tokens the answer may use
    system="You explain machine learning to beginners in plain English.",
    messages=[{"role": "user", "content": "What is overfitting, in two sentences?"}],
)
print(reply.content[0].text)
print(reply.usage)               # input and output token counts, for cost tracking
This code calls a hosted model, so it needs the internet and an API key. On your computer or in Colab: pip install anthropic, set your key as the environment variable ANTHROPIC_API_KEY (never paste keys into code), then run it. Press Copy to take it with you.
Python (OpenAI-compatible, also works with Ollama)
# pip install openai   (reads OPENAI_API_KEY from the environment)
from openai import OpenAI

client = OpenAI()
# for a local Ollama model instead:
# client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

reply = client.chat.completions.create(
    model="MODEL_NAME",          # e.g. a model you pulled with: ollama pull <name>
    messages=[
        {"role": "system", "content": "You explain machine learning to beginners in plain English."},
        {"role": "user", "content": "What is overfitting, in two sentences?"},
    ],
    temperature=0.2,
)
print(reply.choices[0].message.content)
This code calls a hosted model, so it needs the internet and an API key. On your computer or in Colab: pip install openai, set your key as the environment variable OPENAI_API_KEY (never paste keys into code), then run it. Press Copy to take it with you.
Beginner

Streaming, conversations and limits

  • Streaming sends the answer token by token as it is generated, so users see text appear immediately instead of waiting many seconds. Every major SDK supports it (look for stream=True or a stream() method).
  • Conversations: append each user message and assistant reply to the messages list and send the whole list each time. Long chats grow expensive; summarise or trim old turns.
  • max_tokens caps the length of the answer (and your cost). If the answer stops mid-sentence, the response tells you it hit this limit.
  • Rate limits: providers limit requests and tokens per minute. When you exceed them you get an HTTP 429 error. Servers also occasionally fail with 5xx errors. The fix for both is to retry with exponential backoff: wait 1 s, then 2 s, then 4 s, with a little randomness, up to a limit.
Python
import random, time

class RateLimited(Exception):
    pass

calls = {"n": 0}
def flaky_model_call(prompt):
    # pretend API: fails with "rate limited" the first two times
    calls["n"] += 1
    if calls["n"] <= 2:
        raise RateLimited("429 Too Many Requests")
    return f"answer to: {prompt}"

def with_retries(fn, *args, tries=5, base=0.5):
    for attempt in range(tries):
        try:
            return fn(*args)
        except RateLimited as e:
            if attempt == tries - 1:
                raise
            wait = base * 2 ** attempt + random.uniform(0, 0.1)   # exponential backoff + jitter
            print(f"attempt {attempt + 1} failed ({e}); waiting {wait:.2f}s")
            time.sleep(wait)

print(with_retries(flaky_model_call, "What is a token?"))
Open in playground ↗
Try it

Try it: tokens and cost

Paste a typical prompt, set how long answers are and how many requests you expect, and enter your provider's prices (they change often, so copy them from the pricing page). The token count is the rough 4-characters-per-token estimate.

Interactive: this part runs in your browser.

Beginner

Cutting cost and waiting time

  • Pick the smallest model that passes your tests. Small models are often many times cheaper and faster.
  • Keep prompts short and do not resend huge documents on every turn; retrieve only the relevant parts (RAG).
  • Output tokens usually cost several times more than input tokens: ask for concise answers.
  • Prompt caching: many providers charge much less for a long prompt prefix (system prompt, documents) that repeats across calls.
  • Batch APIs process many requests within hours instead of seconds, usually at a large discount, which is ideal for labelling datasets.
  • Cache answers to repeated identical questions in your own database.
Check yourself

Quiz

1. Your app gets HTTP 429 errors at busy times. What is the standard fix?

2. Why does a chat app send the whole conversation on every call?

3. A hospital cannot send patient notes to an outside company. Which option fits?

4. Which usually costs more per token?

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready