Building Multi-Tenant Analytics on Cloudflare's Edge

Why Cloudflare's Edge

ClickStream processes behavioral data in under 3ms. That's not a rounding error -- it's a hard requirement. Behavioral scores need to be available in the same page load to enable real-time personalization and intervention. Round-tripping to an origin server in us-east-1 adds 50-200ms of latency. Edge computing eliminates that entirely.

Cloudflare's developer platform provides the building blocks: Workers for compute, Analytics Engine for time-series storage, Durable Objects for stateful coordination, KV for configuration, R2 for object storage, and D1 for relational data. Each has tradeoffs. This post covers how we use each one and the gotchas we've encountered.

Analytics Engine: The Write Path

Analytics Engine (AE) is Cloudflare's purpose-built time-series analytics store. It's designed for high-volume write ingestion with SQL-based querying. ClickStream uses AE as the primary write path for all behavioral events.

The Schema Constraint: 20 Fields

Analytics Engine has a hard schema limit: 20 blob fields (strings, sharing a fixed byte budget per event) and 20 double fields (64-bit floats). That's it. No arrays, no nested objects, no variable-length schemas.

For an analytics platform that tracks 26 behavioral scores plus identity signals, UTM parameters, device info, and page metadata, 20 fields is not enough -- unless you get creative.

Pipe-Delimited Encoding

ClickStream packs multiple values into single blob fields using pipe-delimited encoding:

// ClickStream's AE layout: two datasets, each with a documented
// 20-blob / 20-double allocation.
//
// clickstream_events -- event facts. The tenant client ID is the
// dataset's index field (scoped into every query); identity hashes
// live in dedicated blobs; pipe-delimited blobs pack UTM, device,
// and geo context into single fields:
//
//   utmParams:  "google|cpc|brand|identity|ad1"
//   deviceInfo: "desktop|Chrome|macOS|1920x1080"
//   geoInfo:    "US|CA|San Francisco"
//
// clickstream_scores -- the 26-model output. A sample of the
// documented double allocation:
{
    double2:  intent,                // 0-100
    double3:  frustration,           // 0-100
    double7:  sessionMomentum,       // -100..100
    double8:  navigationEntropy,     // page-transition entropy
    double9:  attention,             // engaged seconds
    double15: churn,                 // 0-100
    double17: ltv                    // tiered lifetime value
    // ...plus engagement, value, abandonment, conversion
    // readiness, confusion, emotional state, decision stage,
    // purchase timing, content affinity, form friction,
    // next action, and the rest of the 26-model inventory
}

SQL Gotchas

Querying pipe-delimited fields in AE's SQL requires string functions. Some gotchas we've learned:

-- Extracting UTM source from a pipe-delimited blob
-- Format: "source|medium|campaign|term|content"
-- (blob numbers illustrative -- the allocation is documented per dataset)
SELECT
    index1 AS client_id,
    -- Extract first segment (utm_source)
    SUBSTRING(blob8, 1, POSITION('|' IN blob8) - 1) AS utm_source,
    COUNT(*) AS events
FROM clickstream_events
WHERE
    index1 = 'client_abc123'
    AND timestamp >= NOW() - INTERVAL '7' DAY
GROUP BY client_id, utm_source
ORDER BY events DESC

-- GOTCHA: Analytics Engine SQL is NOT full SQL.
-- No JOINs. No subqueries. No CTEs.
-- Aggregations only: COUNT, SUM, AVG, MIN, MAX, QUANTILE
-- String functions are limited: SUBSTRING, POSITION, LENGTH

Analytics Engine is a write-optimized store with read-side limitations. Cloudflare designed it to ingest millions of writes per second with zero configuration. But if you need JOINs or complex queries, you need to export to a real analytical database.

Multi-Tenant Isolation

ClickStream is a multi-tenant platform. Each customer's data must be isolated. In Analytics Engine, tenant isolation is achieved through the dataset's index field — the client ID — which is scoped into every query server-side.

// Worker: enforce tenant isolation on every AE query
async function queryAnalytics(clientId, query) {
    // CRITICAL: Always scope to the tenant's index field
    // This prevents cross-tenant data leakage
    const safeQuery = query.replace(
        'FROM clickstream_events',
        `FROM clickstream_events WHERE index1 = '${sanitize(clientId)}'`
    );

    return await env.ANALYTICS_ENGINE.query(safeQuery);
}

At the API layer, every request is authenticated with a site-specific API key. The API extracts the site ID from the key and injects it into every query. There's no way for a customer to query another customer's data because the site ID filter is enforced server-side, not client-side.

Durable Objects: Real-Time Streaming

Durable Objects (DOs) are Cloudflare's solution for stateful edge compute. Each DO instance is a single-threaded JavaScript actor with persistent storage, automatically colocated with its most recent caller.

ClickStream uses Durable Objects for two things: real-time streaming dashboards and session aggregation.

Real-Time Dashboard Streaming

// Durable Object: Real-time event stream per site
export class SiteEventStream {
    constructor(state, env) {
        this.state = state;
        this.sessions = new Map(); // WebSocket connections
    }

    async fetch(request) {
        if (request.headers.get('Upgrade') === 'websocket') {
            const pair = new WebSocketPair();
            const [client, server] = Object.values(pair);

            this.state.acceptWebSocket(server);
            this.sessions.set(server, { connectedAt: Date.now() });

            return new Response(null, { status: 101, webSocket: client });
        }

        // Incoming event from the tracking worker
        if (request.method === 'POST') {
            const event = await request.json();

            // Broadcast to all connected dashboard clients
            for (const ws of this.state.getWebSockets()) {
                try {
                    ws.send(JSON.stringify(event));
                } catch (e) {
                    // Client disconnected, will be cleaned up
                }
            }

            return new Response('OK');
        }
    }

    // Hibernation API: DO sleeps when no connections
    async webSocketClose(ws) {
        this.sessions.delete(ws);
    }
}

The Hibernation API

Durable Objects are billed for active wall-clock time. Without hibernation, a DO serving real-time dashboards would be billed 24/7 even when no events are flowing. The Hibernation API lets the DO sleep between events, waking only when a WebSocket message arrives or a new connection is made.

This reduces DO costs from tens of dollars per month per always-on instance to pennies per actual usage. For a platform serving thousands of sites, hibernation would be the difference between viable and prohibitively expensive.

KV: Configuration Caching

Workers KV stores site configuration: which behavioral models are enabled, consent settings, SDK version, custom event definitions, and API keys.

// KV structure for site config
// Key: "config:site_abc123"
// Value:
{
    "site_id": "site_abc123",
    "domain": "example.com",
    "cname": "t.example.com",
    "models_enabled": [
        "intent", "frustration", "engagement", "value",
        "anomaly", "confusion", "emotional_state",
        "decision_confidence", "regret", "churn",
        "purchase_timing", "ltv", "abandonment",
        "content_affinity", "form_friction", "next_action"
    ],
    "consent_required": true,
    "cookie_max_age": 34560000,
    "sdk_version": "1.4.0"
}

The Consistency Model

KV is eventually consistent with a propagation delay of up to 60 seconds globally. This is fine for configuration (you don't need instant config updates) but would be terrible for real-time event data. That's why events go to Analytics Engine (immediate write availability) and not KV.

KV is a read-optimized store with weak write consistency. Use it for data that changes rarely and is read on every request (config, API keys, feature flags). Never use it for event streams or counters.

R2: The Data Lake

R2 is Cloudflare's S3-compatible object storage with zero egress fees. ClickStream uses R2 as the managed data lake: a Parquet export pipeline (Scale+) writes Snappy-compressed, Hive-partitioned files encrypted per tenant with AES-256-GCM. Exports currently run on request while the self-serve trigger is finished; files are retained for 90 days.

Parquet Export Pipeline

// Export pipeline: AE data to R2 as Parquet
// (ops-triggered today -- hourly scheduled() automation ships later)
export default {
    async scheduled(event, env) {
        // Query last hour of events from Analytics Engine
        const events = await env.ANALYTICS_ENGINE.query(`
            SELECT *
            FROM clickstream_events
            WHERE timestamp >= NOW() - INTERVAL '1' HOUR
        `);

        // Group by site_id for multi-tenant isolation
        const bySite = groupBySiteId(events);

        for (const [siteId, siteEvents] of Object.entries(bySite)) {
            // Convert to Parquet using parquet-wasm
            const parquetBuffer = await toParquet(siteEvents);

            // Write to ClickStream's R2 data lake
            const key = `${siteId}/${formatDate()}/events_${Date.now()}.parquet`;
            await env.R2_DATA_LAKE.put(key, parquetBuffer, {
                httpMetadata: { contentType: 'application/octet-stream' },
                customMetadata: {
                    siteId,
                    eventCount: String(siteEvents.length),
                    exportedAt: new Date().toISOString()
                }
            });
        }
    }
};

D1: Relational Data

D1 is Cloudflare's SQLite-at-the-edge database. ClickStream uses D1 for relational data that doesn't fit AE or KV: customer accounts, billing records, site registrations, API key management, and schema migrations.

Migration Strategy

-- Migration: 001_create_sites.sql
CREATE TABLE IF NOT EXISTS sites (
    id TEXT PRIMARY KEY,
    domain TEXT NOT NULL UNIQUE,
    cname TEXT,
    owner_email TEXT NOT NULL,
    plan TEXT DEFAULT 'free',
    created_at TEXT DEFAULT (datetime('now')),
    updated_at TEXT DEFAULT (datetime('now'))
);

CREATE TABLE IF NOT EXISTS api_keys (
    key_hash TEXT PRIMARY KEY,
    site_id TEXT NOT NULL REFERENCES sites(id),
    name TEXT,
    permissions TEXT DEFAULT 'read',
    created_at TEXT DEFAULT (datetime('now')),
    last_used_at TEXT,
    FOREIGN KEY (site_id) REFERENCES sites(id)
);

CREATE INDEX idx_api_keys_site ON api_keys(site_id);

Local vs. Remote D1

D1's local development experience uses a local SQLite file. Remote D1 runs on Cloudflare's infrastructure. The schemas are the same, but the data is separate. Migrations must be applied to both:

# Apply migration locally
wrangler d1 execute clickstream-db --local --file=migrations/001_create_sites.sql

# Apply migration to production
wrangler d1 execute clickstream-db --remote --file=migrations/001_create_sites.sql

# GOTCHA: --local and --remote are COMPLETELY separate databases
# There is no automatic sync between them

Cloudflare for SaaS: Custom Hostnames

Cloudflare for SaaS (CF4SaaS) is the infrastructure that makes the CNAME cookie architecture work. It allows ClickStream to provision SSL certificates for customer subdomains automatically.

How Custom Hostnames Work

1. Customer creates CNAME: t.example.com → feynman.clickstream.com
2. ClickStream API calls CF4SaaS to register t.example.com as a custom hostname
3. Cloudflare automatically provisions a DCV (Domain Control Validation) and SSL certificate
4. Traffic to t.example.com hits ClickStream's Worker, which reads the Host header to identify the customer

// Worker: Route requests by Host header
export default {
    async fetch(request, env) {
        const host = request.headers.get('Host');
        // host = "t.example.com" (customer's CNAME subdomain)

        // Look up site config by custom hostname
        const siteConfig = await env.KV.get(`hostname:${host}`, 'json');

        if (!siteConfig) {
            return new Response('Unknown hostname', { status: 404 });
        }

        // Now we know which customer this request belongs to
        const siteId = siteConfig.site_id;

        // Handle tracking request, set cookie, etc.
        return handleTrackingRequest(request, siteId, siteConfig, env);
    }
};

The Host Header Pattern

The Host header is the key to multi-tenancy with custom hostnames. When t.example.com CNAMEs to feynman.clickstream.com, the HTTP request arrives at ClickStream's Worker with Host: t.example.com. This lets the Worker identify the customer without any path prefix or query parameter convention.

This is cleaner than alternatives like feynman.clickstream.com/site/abc123 (which leaks the site ID in URLs) or abc123.feynman.clickstream.com (which requires wildcard DNS).

Edge Caching Strategy

Not everything needs to be computed fresh on every request. ClickStream caches aggressively at the edge:

The Cache API (distinct from the CDN cache) is the design pattern here: caching AE query results at each Cloudflare PoP means a dashboard that refreshes every 30 seconds hits AE once per 30 seconds per PoP, not once per viewer. For a dashboard with 50 concurrent viewers across 20 PoPs, that's 20 AE queries/30s instead of 50.

Architecture Summary

Here's how all the pieces fit together:

Every component runs at the edge. There is no origin server. There is no us-east-1. A visitor in Tokyo hits a Cloudflare PoP in Tokyo, their behavioral scores are computed in Tokyo, and the event is written from Tokyo. The 3ms latency target is achievable precisely because there's no cross-continent round-trip.

The edge isn't where you cache things. The edge is where you compute things. When your entire analytics pipeline runs at the edge, latency becomes a rounding error.