AI-Powered Experimentation Platform

Enabling non-technical stakeholders to design, launch, and analyze A/B tests through natural language interfaces.

Next.js App Router · Middleware · AI SDK · PostgreSQL · Edge Functions

Context

Most experimentation tooling assumes engineering involvement at every stage — writing the split logic, instrumenting events, querying results. Product and growth teams end up bottlenecked on eng bandwidth to run even simple tests. The opportunity: collapse the distance between a hypothesis and a result by making natural language the interface layer, and pushing the infrastructure to the edge where it belongs.

How It Works

Traffic splitting happens in middleware.ts, which runs at the edge before any page is rendered. On an incoming request, the middleware checks for an existing experiment cookie. If none exists, it buckets the user into variant A or B using a deterministic assignment (a hash of the user ID or a random roll, depending on whether sticky sessions are required), sets the cookie, and rewrites the request internally to the correct variant route — without a visible URL change. This means zero client-side flicker and no round-trip to a server.

Running this logic at the edge via Vercel Middleware means the split decision is made in the same network location as the request origin — typically under 50ms globally. There's no lambda cold start, no database lookup on the critical path, and no additional latency added to the user experience. The experiment is invisible to the end user.

Conversion events are written to Postgres via a lightweight API route. Results are aggregated server-side using React Server Components with ISR — the dashboard refreshes on a defined revalidation interval rather than polling, keeping read costs low while keeping data reasonably fresh.

// middleware.ts
import { NextRequest, NextResponse } from 'next/server'

const EXPERIMENT_COOKIE = 'experiment_variant'
const VARIANTS = ['control', 'treatment'] as const

export function middleware(request: NextRequest) {
  const { pathname } = request.nextUrl

  // Only run on experiment-enrolled routes
  if (!pathname.startsWith('/app')) return NextResponse.next()

  const existingVariant = request.cookies.get(EXPERIMENT_COOKIE)?.value
  const variant = existingVariant ?? VARIANTS[Math.random() < 0.5 ? 0 : 1]

  // Rewrite internally — URL stays clean for the user
  const url = request.nextUrl.clone()
  url.pathname = `/app/${variant}${pathname.replace('/app', '')}`

  const response = NextResponse.rewrite(url)

  // Persist the bucket across sessions
  if (!existingVariant) {
    response.cookies.set(EXPERIMENT_COOKIE, variant, {
      maxAge: 60 * 60 * 24 * 30, // 30 days
      httpOnly: true,
      sameSite: 'lax',
    })
  }

  return response
}

export const config = {
  matcher: ['/app/:path*'],
}

AI Analysis

Once enough conversion data is collected, the platform passes experiment results to Claude via Vercel's AI SDK. The prompt includes the hypothesis, variant definitions, sample sizes, and conversion rates. The model returns a plain-language recommendation — whether to ship the treatment, revert to control, or continue running — along with a confidence assessment and suggested next steps.

Using streamText from the AI SDK makes the response feel immediate. The recommendation streams token-by-token into the UI, which matters perceptually. A result that arrives progressively feels faster and more alive than one that appears all at once after a delay.

// app/api/analyze/route.ts
import { streamText } from 'ai'
import { anthropic } from '@ai-sdk/anthropic'

export async function POST(req: Request) {
  const { hypothesis, control, treatment } = await req.json()

  const result = streamText({
    model: anthropic('claude-sonnet-4-6'),
    system: `You are an experimentation analyst. Given A/B test results,
provide a clear recommendation: ship the treatment, revert to control,
or continue running. Be direct. Explain your reasoning in 2–3 sentences.
Suggest one follow-up test if relevant.`,
    prompt: `
Hypothesis: ${hypothesis}

Control: ${control.conversions} conversions from ${control.visitors} visitors
(${((control.conversions / control.visitors) * 100).toFixed(1)}% CVR)

Treatment: ${treatment.conversions} conversions from ${treatment.visitors} visitors
(${((treatment.conversions / treatment.visitors) * 100).toFixed(1)}% CVR)
    `,
  })

  return result.toDataStreamResponse()
}

Sample Output

AI-generated recommendation based on real experiment results.

Based on the results, I'd recommend shipping the treatment. The 3-step onboarding flow drove a 13% lift in completion rate (73.8% vs 65.0%) across comparable sample sizes — enough to be confident this isn't noise. The delta is meaningful for a change this low-risk to revert if needed. One follow-up worth running: test whether the removed steps can be reintroduced later in the user journey without hurting retention.

Reflections

The main tradeoff in this architecture is freshness vs. cost. ISR keeps Postgres read volume low, but it means dashboards lag behind real-time by the revalidation window. For most experimentation use cases this is acceptable — statistical significance doesn't shift meaningfully minute-to-minute. For high-velocity tests, you'd swap ISR for on-demand revalidation triggered by the event pipeline.

At scale, the cookie-based bucketing approach breaks down for logged-in users who switch devices. The production solution is to move bucket assignment to the user record in Postgres and read it in middleware via a lightweight edge-compatible query or a KV store like Vercel KV. The middleware stays fast; the source of truth moves server-side.

This architecture is particularly well-suited to Vercel because Middleware, Edge Functions, ISR, and KV are first-class platform primitives — not bolted-on integrations. The experimentation layer lives entirely at the edge with no additional infrastructure to manage. That's the bet: when the platform handles the hard parts, product teams can focus on the hypothesis.