Proxying LLM Streaming Responses through Edge Workers

Introduction to LLM Latency Challenges

As generative AI features become standard in web applications, optimizing the latency of Large Language Model (LLM) responses is a key challenge. Traditional LLM API requests can take several seconds to complete, which degrades user experience. To mitigate this, modern AI apps stream responses block-by-block using Server-Sent Events (SSE).

However, connecting client browsers directly to upstream providers like OpenAI or Anthropic is unsafe. It exposes API keys, prevents server-side rate-limiting, and bypasses audit logging.

To secure these interactions, developers must deploy a backend proxy. By building this proxy at the network edge using Cloudflare Workers, we can stream responses with low latency while enforcing token counting, request validation, and caching.

Architecting an Edge AI Gateway

An edge AI gateway acts as a proxy between your client application and the LLM API. The gateway intercepts the client's request, appends API keys, applies rate limits, inspects the request payload, forwards it to the provider, and streams the response back.

An edge-based gateway provides several critical security and operational benefits:

• API Key Protection: Prevents exposure of upstream access credentials to client-side code.
• Distributed Rate Limiting: Enforces global and IP-level execution limits to prevent API abuse.
• Input/Output Auditing: Sanitizes payloads and logs usage telemetry for compliance and billing.
• Staged Stream Transforms: Alters or decorates stream blocks in real-time as they cross the network edge.

To perform this processing efficiently, the edge worker must parse the SSE stream in real time without buffering the response. If the proxy buffers the output, streaming benefits are lost, and Time to First Token (TTFT) rises. By utilizing standard ReadableStream and TransformStream interfaces, Cloudflare Workers process stream chunks as they arrive.

Real-Time SSE Stream Processing

An LLM stream returns text fragments structured as SSE data blocks, typically prefixed with data: . Our proxy needs to decode these byte chunks, inspect the JSON content, count the generated tokens, and forward the data to the client.

// Transform stream handler to inspect LLM chunks
class SSETokenCounterTransform {
  constructor() {
    this.decoder = new TextDecoder();
    this.encoder = new TextEncoder();
    this.totalTokens = 0;
  }

  transform(chunk, controller) {
    const text = this.decoder.decode(chunk, { stream: true });
    
    // Parse individual lines from the SSE chunk
    const lines = text.split('\n');
    for (const line of lines) {
      if (line.startsWith('data: ')) {
        const payload = line.slice(6).trim();
        if (payload === '[DONE]') {
          controller.enqueue(this.encoder.encode(line + '\n'));
          continue;
        }
        try {
          const json = JSON.parse(payload);
          const token = json.choices?.[0]?.delta?.content;
          if (token) {
            // Count word/token heuristics
            this.totalTokens += 1;
          }
        } catch (e) {
          // Ignore parsing errors on partial chunks
        }
      }
    }
    // Forward original chunk to client
    controller.enqueue(chunk);
  }

  flush(controller) {
    console.log(`Stream complete. Total tokens: ${this.totalTokens}`);
  }
}

By feeding the upstream response through a TransformStream containing this transformer, we can run logging and tracking tasks asynchronously while maintaining active stream delivery to the client.

Token Rate Limiting, Counting Engines, and Semantic Caching

Beyond proxying, edge gateways help protect upstream infrastructure from abuse. By running rate limiting on Cloudflare Edge, requests that exceed quotas are rejected before reaching the upstream provider, avoiding API costs.

Counting tokens precisely at the edge is historically difficult because libraries like Tiktoken require large dictionary assets. To resolve this inside the V8 isolate's 128MB memory budget, developers often run lightweight byte-pair encoding (BPE) implementations or fetch token estimates dynamically.

Additionally, you can implement a semantic caching layer. By generating vector embeddings of incoming prompts using edge-native models and performing a cosine-similarity search against an edge vector database like Cloudflare Vectorize, the proxy can retrieve cached answers for equivalent prompts, completely bypassing upstream LLM calls and cutting response times to under 50ms.

For validation, the gateway can run prompt injection scanners directly at the edge. The worker evaluates the incoming prompt against custom regex matches or vector search caches before dispatching the request.

Additionally, to avoid redundant computation, we can cache common prompts and their completions using the Cache API. Although natural language has high variance, template requests can benefit from caching, reducing downstream API costs.

Handling Client Aborts and Network Cleanups

In production environments, users frequently cancel LLM operations mid-generation by closing the browser tab or hitting a "Stop" button. When this happens, client connections are terminated abruptly.

An optimized edge gateway must detect these abort signals and clean up upstream connections immediately. If the worker does not handle client aborts, it will continue to read from the LLM provider, consuming costly tokens. By binding the request's signal event listener to the outbound fetch request, the Edge Workers runtime halts processing the moment the client disconnects, saving significant token costs.

Furthermore, comparing edge architectures to typical serverless environments (like AWS Lambda) highlights the advantages of workers. Edge Workers maintain connection pools to upstream endpoints globally, allowing TCP and TLS handshakes to be reused, reducing Time to First Token (TTFT) from several hundred milliseconds to single digits.

LLM Gateway Optimization at the Edge with Bramsley

Scaling secure, low-latency AI integrations requires moving the core validation and translation logic to the edge. Bramsley optimizes LLM gateways with the following features:

• Custom Token Rate-Limiting: Enforcing request quotas at the rate-limiting layer before hitting upstream services.
• Secure Key Rotations: Managing API access keys securely on edge runtimes without client exposure.
• Real-Time SSE Transformations: Injecting metadata and monitoring Server-Sent Events stream backpressure.
• Semantic Cache Integration: Caching common translations and vector search results to lower operational costs.