People call that a scaling problem. Usually it's a path problem: the request was designed for an audience of one.

I lead development on Rawk.ai, a live voice agent builder, where a latency pass cut end-to-end response time from about 900ms to about 320ms. None of that came from a better prompt. It came from the pipeline, caching, and where services ran. Here's the approach, with the patterns in code. The snippets are simplified illustrations, not production source.

Load is a tail, not an average

A product can look healthy on average and still fail the people who hit the slow path. Voice callers hang up. Checkout users retry and double-charge themselves. Staff refresh a dashboard and act on stale data.

Three things matter, in this order:

• Speed: time to the first useful response, not time until a spinner disappears.

• Correctness: a fast stale cache is just a bug with better graphs.

• Cost: a path that calls a model or vendor API on every keystroke gets expensive before it gets slow.

Before optimizing anything, name the exact request you're timing. "The app feels slow" is a complaint. "Time to first audio on a live call" is a job.

Measure the tail first

Averages hide the problem, so measure percentiles on production-like data:

import statistics
import time

async def time_request(fn, runs: int = 200) -> dict:
    samples = []
    for _ in range(runs):
        start = time.perf_counter()
        await fn()
        samples.append((time.perf_counter() - start) * 1000)  # ms
    q = statistics.quantiles(samples, n=100)
    return {"p50": q[49], "p95": q[94], "p99": q[98]}

If p50 looks fine and p95 doesn't, you've found where users are actually suffering. Keep this number, because you'll rerun the exact same measurement after every change.

Where the time actually goes

Most slow products are a chain of reasonable steps that were never allowed to overlap.

1. Sequential I/O

The page waits for the user, then permissions, then the list, then the counts. Each call is fine on its own. Together, they add up:

import asyncio

# Before: each await waits for the previous one to finish
async def dashboard_slow(user_id: str):
    user = await get_user(user_id)
    perms = await get_permissions(user_id)
    items = await list_items(user_id)
    counts = await get_counts(user_id)
    return user, perms, items, counts

# After: independent calls run concurrently
async def dashboard(user_id: str):
    return await asyncio.gather(
        get_user(user_id),
        get_permissions(user_id),
        list_items(user_id),
        get_counts(user_id),
    )

The total time drops from the sum of all four calls to roughly the slowest one. The catch: only run calls together when they're truly independent. If list_items depends on the permissions result, it has to wait.

2. A cache that's missing, or lying

Postgres stays the system of record. Redis sits in front of reads that are hot, safe to serve more than once, and expensive to recompute. The real design work is the boundary: which keys exist, how long they live, and which writes delete them.

import json
import redis.asyncio as redis

r = redis.Redis()
TTL_SECONDS = 300

async def get_plan_prices(account_id: str) -> dict:
    key = f"plan_prices:{account_id}"
    cached = await r.get(key)
    if cached is not None:
        return json.loads(cached)
    prices = await db.fetch_plan_prices(account_id)  # Postgres is the source of truth
    await r.set(key, json.dumps(prices), ex=TTL_SECONDS)
    return prices

async def update_plan_prices(account_id: str, prices: dict) -> None:
    await db.save_plan_prices(account_id, prices)
    await r.delete(f"plan_prices:{account_id}")  # invalidate on write

The TTL is a safety net. The delete on write is what keeps the cache honest. A cache without an invalidation rule is how a paid invoice keeps showing as "new lead" in the dashboard your team trusts.

3. Queries that scan

If the access pattern was never the index, Postgres will still answer. It'll just answer late. Run EXPLAIN ANALYZE on the slow request's queries and look for sequential scans on large tables.

4. Retrieval on every request

Embeddings, vector search, and a model call for a question the last hundred users already asked. Pinecone or pgvector belongs in the path only when the answer actually depends on search. If the fact is already a column, read the column.

5. Cross-region hops

When each vendor defaults to a different cloud region, every request picks up tens of milliseconds per hop. No amount of code optimization fixes that. Colocate the services that talk on every request.

What changed on Rawk AI

For a voice agent, a 900ms gap after every sentence breaks the conversation. Callers talk over the agent, the agent talks over them, and the call falls apart. Sub-400ms is roughly where a call starts to feel natural.

Getting there was pipeline work:

• Streaming speech-to-text instead of waiting for a complete transcript

• Streaming the model's output instead of buffering a full paragraph

• Starting audio on the first clause, so the caller hears a response while the rest is still generating

• Prefetching tool results the agent was likely to need

• Colocating services so no extra regions were added to every turn

Each change shaved time off the same measured request. None of them involved the prompt.

A checklist for your own slow path

1. Name the one request users feel, and time it on production-like data, including p95 and p99.

2. List every downstream call on that request, in order, with what each returns.

3. Run independent calls concurrently.

4. Mark which results are identical across requests and which must never be shared.

5. Cache only the safe, identical reads, with an explicit delete on write.

6. Colocate the services that talk on every request.

7. Re-measure the same request. If the tail didn't move, that wasn't the bottleneck.

And write down what you changed. Undocumented speed is a demo you won't be able to repeat.

The original version, including how we scope latency audits, is on keencraft.tech.