Step 9 of 31
Engineer context, model access, routing, structured responses, streaming, and evaluation.
Representative options for this chapter. Compare them against workload, policy, cost, and operating capability.
A language model predicts tokens from context. The application around it determines access, structure, safety, latency, and usefulness.
Text is tokenized, represented as vectors, processed through transformer layers, and decoded one token at a time. The context window is finite working space, not durable memory. Temperature and sampling settings change output distribution; they do not repair weak evidence or unclear instructions.
Hallucination is a system property to manage, not a switch to disable. Reduce unsupported output with grounded context, constrained tasks, structured responses, verification, and abstention.
Tokenization maps text to model-specific identifiers. The same sentence can use different token counts across models, which changes cost and context use. Inputs are converted to embeddings and combined with positional information so attention can relate tokens across the sequence.
Self-attention computes relationships among tokens. Transformer layers combine attention and feed-forward computation to produce a probability distribution for the next token. Decoding samples or selects a token, appends it, and repeats.
The model does not retrieve a stored paragraph from its weights. It generates from learned statistical structure and the supplied context. This explains why fluent output can be unsupported.
Temperature reshapes the distribution. Top-k limits candidates to a fixed number; top-p limits them to a cumulative probability mass. Lower randomness can improve repeatability but does not guarantee truth or identical output.
Context includes system instructions, conversation state, retrieved evidence, tool descriptions, tool results, examples, and user input. Every item competes for a finite budget and can influence behavior.
Prioritize authoritative instructions and relevant evidence. Remove duplicated history and stale tool output. Summaries reduce size but are lossy derived data. Preserve source records when later verification matters.
Long context does not remove retrieval design. More tokens increase latency and cost and can reduce attention to critical evidence. Measure quality across realistic context sizes.
Loading diagram...
Put provider SDKs behind a typed gateway contract. Record model revision, parameters, token use, latency, request class, and policy result. Support timeouts, cancellation, rate limits, and streaming without exposing provider-specific objects to product code.
Route by task requirements:
Evaluate routing on real tasks. A cheaper model that causes more retries or human corrections can cost more per successful outcome.
Loading diagram...
Start with one direct SDK call so the network boundary is visible before introducing a gateway. This example uses the OpenAI Python SDK because the same client can call OpenAI-compatible endpoints exposed by Ollama and vLLM. Compatibility is an interface claim, not a guarantee that every provider supports every parameter.
Create an isolated environment:
python -m venv .venv
source .venv/bin/activate
python -m pip install openai
Set the endpoint, credential, and model outside the program. Choose one configuration:
# Hosted provider
export MODEL_API_KEY='replace-with-a-development-key'
export MODEL_NAME='replace-with-an-available-model'
# Local Ollama alternative
export MODEL_BASE_URL='http://localhost:11434/v1'
export MODEL_API_KEY='ollama'
export MODEL_NAME='replace-with-a-pulled-model'
# Local vLLM alternative
export MODEL_BASE_URL='http://localhost:8000/v1'
export MODEL_API_KEY='local-development'
export MODEL_NAME='replace-with-the-served-model'
Only set MODEL_BASE_URL when using a compatible endpoint. Create sdk_call.py:
import asyncio
import os
from time import monotonic
from openai import AsyncOpenAI
def create_client() -> AsyncOpenAI:
options = {"api_key": os.environ["MODEL_API_KEY"]}
if base_url := os.getenv("MODEL_BASE_URL"):
options["base_url"] = base_url
return AsyncOpenAI(**options)
async def main() -> None:
client = create_client()
started = monotonic()
response = await client.chat.completions.create(
model=os.environ["MODEL_NAME"],
messages=[
{
"role": "user",
"content": "Return one sentence explaining request backpressure.",
}
],
temperature=0,
max_tokens=80,
)
elapsed_ms = round((monotonic() - started) * 1_000)
usage = response.usage
print(response.choices[0].message.content)
print(
{
"model": response.model,
"latency_ms": elapsed_ms,
"input_tokens": usage.prompt_tokens if usage else None,
"output_tokens": usage.completion_tokens if usage else None,
}
)
await client.close()
asyncio.run(main())
Run python sdk_call.py. Confirm that the answer and operational metadata are separate. Do not log the API key, full prompt, or generated text as telemetry.
Streaming changes the response lifecycle. Replace the request block with a streaming call and measure time to the first content token separately from total duration:
started = monotonic()
first_token_ms = None
stream = await client.chat.completions.create(
model=os.environ["MODEL_NAME"],
messages=[{"role": "user", "content": "Explain request backpressure."}],
stream=True,
max_tokens=120,
)
async for event in stream:
if not event.choices:
continue
content = event.choices[0].delta.content
if content:
if first_token_ms is None:
first_token_ms = round((monotonic() - started) * 1_000)
print(content, end="", flush=True)
print()
print({"first_token_ms": first_token_ms})
Interrupt the client during generation. The exercise is incomplete if the application continues unnecessary work without detecting cancellation.
Reuse GenerateRequest, GenerateResult, and TextGenerator from the Python service chapter. Provider-specific objects remain inside this adapter:
from openai import AsyncOpenAI
from domain import GenerateRequest, GenerateResult
class OpenAICompatibleGenerator:
def __init__(self, client: AsyncOpenAI, model: str) -> None:
self._client = client
self._model = model
async def generate(self, request: GenerateRequest) -> GenerateResult:
response = await self._client.chat.completions.create(
model=self._model,
messages=[{"role": "user", "content": request.text}],
max_tokens=request.max_tokens,
)
usage = response.usage
return GenerateResult(
text=response.choices[0].message.content or "",
model=response.model,
input_tokens=usage.prompt_tokens if usage else 0,
output_tokens=usage.completion_tokens if usage else 0,
)
Replace StubGenerator in the FastAPI exercise with this adapter. Run the same API tests with a fake TextGenerator; unit tests should not spend provider quota or require network access.
Finally, run the same prompt against a hosted model and either Ollama or vLLM. Record status, time to first token, total latency, token reporting, and output validity. If the compatible endpoint ignores a parameter or omits usage, capture that difference in the adapter rather than leaking it into the API response contract.
Managed APIs reduce infrastructure work and provide rapid access to capable models. They introduce provider quotas, policy, data-processing terms, regional availability, and model lifecycle dependencies.
Self-hosting provides control over weights, runtime, locality, and capacity. It introduces accelerator procurement, serving optimization, security patching, scaling, and on-call ownership. Open weights do not automatically grant unrestricted licensing or safe use.
Use a decision record covering quality, latency, throughput, context, modality, data policy, availability, cost, and operational capability. Avoid choosing based on a single public benchmark.
Streaming reduces time to first token but not total generation time. The application must handle partial UTF-8 content, disconnects, provider errors after output starts, and validation that may finish only at the end.
Rate limits can apply to requests, input tokens, output tokens, or concurrent work. Centralize quota accounting and apply backpressure. Independent service retries can multiply traffic during a provider incident.
Treat prompts as versioned program inputs. Separate trusted instructions from untrusted user content and retrieved data. Define the expected output schema and validate it in code. Few-shot examples should represent difficult boundary cases, not only ideal outputs.
A prompt template should define variables and escaping rules. Validate required inputs before rendering. Record the template revision, model, parameters, tool set, and output schema in evaluation and production traces.
Use zero-shot instructions for simple, well-defined tasks. Add examples when format or decision boundaries need demonstration. Use decomposition when subtasks can be verified independently. Use tool calls for current data and deterministic actions.
Do not request verbose hidden reasoning as proof. Ask for observable evidence: selected category, cited source, calculation result, tool trace, or concise rationale permitted by the product.
Schema-constrained generation narrows syntax, not truth. Validate types, ranges, identifiers, authorization, and cross-field rules after generation. For safety-critical actions, treat model output as a proposal evaluated by deterministic policy.
Tool descriptions should be precise and non-overlapping. Return bounded structured results and explicit errors. Tool output is untrusted content even when the tool itself is trusted, because it may contain external text.
Build a task dataset with normal, boundary, adversarial, and unanswerable cases. Measure correctness, instruction following, schema validity, safety, latency, and cost. Evaluate important user and language slices.
Compare models under the same prompt and then allow model-specific optimization as a second experiment. Otherwise a prompt tuned for one model can create an unfair comparison.
Pin the evaluated revision where the provider supports it. When only moving aliases exist, run continuous regression tests and maintain a tested fallback.
Reasoning techniques are task-dependent. Do not require hidden reasoning text or expose it as evidence. Ask for concise answers, structured decisions, citations, or verifiable intermediate artifacts instead.
Design a provider-neutral model gateway with a typed request, structured response, routing policy, quota, telemetry, and fallback rules. Create an evaluation set before comparing models.
Completion means model choice is an evidence-based deployment decision, not a hard-coded SDK call.