Quickstart: route your first AI request in 5 minutes

Inference Labs gives you one endpoint and one billing surface for OpenAI / Azure, Anthropic, Google, AWS Bedrock, and RunwayML. This page walks you from zero to a working request with policy-based routing.

1

Sign up & get an API key

Create a free account — 500 credits, no card required. After verifying your email you'll land on the dashboard; head to API Keys › Create and copy the il_live_… value.

Create account →

2

Install the client

# Install from our self-hosted PEP 503 index (no PyPI account anywhere in the loop):
pip install --extra-index-url https://inference-labs.com/pypi/simple/ inference-labs

# Or pin an exact release straight from GitHub:
pip install https://github.com/bosslesss/inference-labs-python/releases/download/v0.1.0/inference_labs-0.1.0-py3-none-any.whl
# No SDK needed -- the platform speaks plain HTTP.
# Any HTTP client works. Examples below use fetch and curl.
pip install --extra-index-url https://inference-labs.com/pypi/simple/ "inference-labs[langchain]"
3

Send your first request

from inference_labs import InferenceLabs

client = InferenceLabs(api_key="il_live_...")  # or set INFERENCE_LABS_API_KEY

out = client.generate(
    prompt="Summarize: 'The cache hit rate determines spend more than model choice.'",
    strategy="cost-first",
    max_cost_usd=0.01,
)

print(out.text)
print(f"routed via {out.provider}/{out.model}  -- ${out.cost_usd:.5f}, {out.latency_ms} ms")
const resp = await fetch("https://app.inference-labs.com/api/v1/generate", {
  method: "POST",
  headers: {
    "Authorization": "Bearer il_live_...",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    modality: "text",
    prompt: "Summarize: 'The cache hit rate determines spend more than model choice.'",
    constraints: { strategy: "cost-first", max_cost_usd: 0.01 },
  }),
});
const { data } = await resp.json();
console.log(data.text, data.model, data.cost_usd);
curl -sS https://app.inference-labs.com/api/v1/generate \
  -H "Authorization: Bearer il_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "modality": "text",
    "prompt": "Hello!",
    "constraints": { "strategy": "cost-first", "max_cost_usd": 0.01 }
  }' | jq .data
from inference_labs.langchain import ChatInferenceLabs
from langchain_core.messages import HumanMessage, SystemMessage

llm = ChatInferenceLabs(
    api_key="il_live_...",
    strategy="balanced",
    max_cost_usd=0.01,
)

resp = llm.invoke([
    SystemMessage(content="You are a terse SRE."),
    HumanMessage(content="What does workers=1 threads=8 mean for SQLite?"),
])
print(resp.content)
print(resp.additional_kwargs)   # model, provider, cost_usd, latency_ms, cached, trace_id
4

Set routing policy per request

The router supports five strategies. Pass strategy in constraints (HTTP) or as a keyword (SDK):

  • balanced — default; price-for-quality optimum.
  • cost-first — cheapest model that satisfies max_cost_usd.
  • quality-first — best quality regardless of cost.
  • latency-first — fastest model in your allowlist.
  • judge — run two models, keep the better output (uses 2× tokens).

Add allow_models / deny_models arrays to lock the choice to a subset.

5

Stream long responses

for chunk in client.stream(prompt="Write a 300-word essay on caching."):
    print(chunk, end="", flush=True)

Or hit /api/v1/generate/stream directly — it returns Server-Sent Events.

6

Check the dashboard

Open app.inference-labs.com › Traces. Every request you've made appears with model, cost, latency, cache-hit flag, and the routing decision. Use the filter to see how a strategy change shifts cost.

That's it. Common next steps: grab your referral link (200 credits per referral), benchmark your real workload across models, or read the blog for routing patterns in production.