Ollama API in Python, Then a Hosted Endpoint

Set up the Python client

Install Ollama itself first, start it, and pull a model from a terminal: ollama pull llama3. The server listens on http://localhost:11434. Then install the Python package in a virtual environment:

Copypython -m venv .venv
source .venv/bin/activate
pip install ollama openai

The ollama package talks to Ollama's native routes under /api. The openai package is for the second half of this tutorial, where the target is a hosted endpoint. If you prefer a visual tool before coding, see the notes on using Ollama with a GUI.

Check the install by running ollama list in the terminal. If your model appears there, Python will find it. If you get a connection error later, the server is probably not running.

Your first chat call

This is the ollama api python basics in one snippet: a system message, a user message and two options.

Copyimport ollama

resp = ollama.chat(
    model="llama3",
    messages=[
        {"role": "system", "content": "Answer in one short paragraph."},
        {"role": "user", "content": "What is a context window?"},
    ],
    options={"temperature": 0.4, "num_ctx": 8192},
)
print(resp["message"]["content"])

The reply is a dictionary-like object; the text lives in message.content. In options you can set generation parameters. temperature controls randomness and num_ctx sets the context length for that call. Ollama defaults to a fairly small context unless you raise it, which surprises people who paste long documents and get answers that ignore the start.

For multi-turn chat, keep a list of messages and append each reply yourself. Ollama, like most chat APIs, is stateless between calls.

Streaming responses

Without streaming you wait for the full answer. With stream=True the function returns a generator and you print pieces as they arrive:

Copyimport ollama

for part in ollama.chat(
    model="llama3",
    messages=[{"role": "user", "content": "Explain recursion with a cooking analogy."}],
    stream=True,
):
    print(part["message"]["content"], end="", flush=True)

Each part carries a small message.content. The last part has done set to true and includes timing and token counts. For web apps, forward these pieces to the browser over server-sent events or a websocket.

Errors you will actually see

Two failures cover most beginner problems.

Copyimport ollama

try:
    ollama.chat(model="llama3", messages=[{"role": "user", "content": "hi"}])
except ollama.ResponseError as exc:
    if exc.status_code == 404:
        print("Model missing. Run: ollama pull llama3")
    else:
        raise
except ConnectionError:
    print("Cannot reach Ollama. Is the server running on port 11434?")

A 404 from ResponseError means the model name does not match something you pulled. Names include tags, so llama3 and llama3:70b are different models. A connection error means the server is not up, or you are inside a container where localhost points at the container, not the host. In that case pass a client with the right host: ollama.Client(host="http://host.docker.internal:11434").

More patterns are collected in common Ollama API mistakes.

Same program, hosted endpoint

Local models are limited by your memory and speed. When a script has to run on a server, handle many parallel jobs, or use a larger window, a hosted endpoint is easier. The trick is to write the program against the OpenAI SDK and treat base URL, key and model as configuration:

Copyimport os
from openai import OpenAI

client = OpenAI(
    base_url=os.environ.get("LLM_BASE_URL", "https://api.ollamansfw.com/v1"),
    api_key=os.environ["LLM_API_KEY"],
)
MODEL = os.environ.get("LLM_MODEL", "uncensored")

def ask(prompt, history=None):
    messages = (history or []) + [{"role": "user", "content": prompt}]
    out = ""
    for chunk in client.chat.completions.create(model=MODEL, messages=messages, stream=True):
        if chunk.choices and chunk.choices[0].delta.content:
            piece = chunk.choices[0].delta.content
            out += piece
            print(piece, end="", flush=True)
    return out

ask("Explain recursion with a cooking analogy.")

For local Ollama, set LLM_BASE_URL=http://localhost:11434/v1, LLM_API_KEY=ollama (any non-empty string is accepted there) and LLM_MODEL=llama3. For the hosted API, use the defaults above, your real key and model uncensored. Nothing else in the code changes.

Get a key on the API key page. New accounts can claim a $0.50 trial in one click, with no payment, valid for 7 days. The model is one open-weight model from an independent provider, tuned to answer without content refusals for lawful adult use. It is text only and not an official API of any big lab.

Running many prompts in parallel

A local model on one machine usually handles one request at a time, so loops are sequential. A hosted endpoint allows concurrency, which makes batch work much faster. Keep the worker count under your key's limit: 8 concurrent requests for a full key, 2 for a trial key.

Copyfrom concurrent.futures import ThreadPoolExecutor
from openai import OpenAI

client = OpenAI(base_url="https://api.ollamansfw.com/v1", api_key="YOUR_KEY")

def summarize(text):
    r = client.chat.completions.create(
        model="uncensored",
        messages=[{"role": "user", "content": "Summarize in one line: " + text}],
        max_tokens=60,
    )
    return r.choices[0].message.content

docs = ["first text", "second text", "third text"]
with ThreadPoolExecutor(max_workers=6) as pool:
    results = list(pool.map(summarize, docs))
print(results)

Six workers leave headroom below the limit of 8. If a worker gets a 429, the OpenAI SDK retries a couple of times by default, but add your own backoff for long runs. Also remember that every streamed request holds its slot until the stream ends, so do not mix a long streaming chat and a big batch on the same trial key.

Differences to plan for

• No local-only options. num_ctx and keep_alive have no meaning on the hosted side. The window is 100,000 tokens, prompt and completion together.
• Rate limits. A full key allows 8 concurrent requests and 300 requests per minute; a trial key allows 2 and 60. Past that you get 429 with rate_limited or concurrency. Retry after a pause; on temporary overload the response carries Retry-After: 5.
• Cost. Input is $0.15 per 1M tokens and output is $0.50 per 1M. A script that sends 2,000 prompts of 600 input and 400 output tokens spends 2,000 x (600 x $0.15 + 400 x $0.50) / 1M = 2,000 x $0.00029 = $0.58.
• Billing. Prepaid credit, from $20 to $500, paid with crypto (USDT on TRC20 or USDC on Base). Failed or refused requests are not charged.

For a side-by-side look at endpoints, read our API pricing note or the docs.

More guides

• Ollama API vs OpenAI API: Endpoints and Code
• How to Use Ollama with a GUI (and a Hosted Fallback)