ab-test-setup-and-analysis

--- name: ab-test-setup-and-analysis description: Designs statistically valid split tests for ads, audiences, landing pages, or bid strategies. Calculates required sample sizes before you start, monitors results during the test, and calls winners when statistical significance is actually reached — not when you feel like one is winning. Platform: Google and Meta. metadata: platform: Google and Meta --- # 28/ A/B Test Setup and Analysis — Google + Meta ## What it does Designs statistically valid split tests for ads, audiences, landing pages, or bid strategies. Calculates required sample sizes before you start, monitors results during the test, and calls winners when statistical significance is actually reached — not when you feel like one is winning. ## How it works Claude takes your test hypothesis, current baseline metrics, and the minimum detectable effect you care about, then calculates how much traffic and time the test needs to produce a reliable result. During the test, it tracks results and tells you whether differences are statistically significant or just noise. It prevents premature test calls that waste the effort. ## Practical example You want to test a new headline variant against your current best performer. Your current ad gets a 2.8% CTR with 1,200 daily impressions. Claude calculates that to detect a 15% improvement (3.22% CTR) with 95% confidence, you need approximately 12,400 impressions per variant — about 10 days at current traffic levels. After 7 days, the new variant shows 3.1% CTR vs 2.7% for control. Claude tells you it's trending positive but not yet significant (p=0.14, need p<0.05) — keep running for 4 more days before making a call. ## What you get back - Test design with clear hypothesis, control, variant, and success metric - Required sample size and estimated test duration before you start - Daily monitoring with current results, confidence level, and days remaining - Winner call only when significance is reached, with effect size and confidence interval - Post-test recommendations: roll out winner, iterate further, or test something else ## When to use it - Before launching any creative, audience, or landing page test - When someone on the team wants to call a winner after 2 days of data - For structured testing programs where you need a pipeline of tests running sequentially - When clients ask "is this result real or just noise" and you need a definitive answer ## Data access (Ryze MCP) This skill works best with live account data. Connect the free Ryze MCP once and Claude reads your Google Ads, Meta Ads, GA4 and Search Console directly: - claude.ai / Claude Desktop: Settings → Connectors → Add custom connector → `https://connector.get-ryze.ai/mcp` - Claude Code: `claude mcp add ryze --transport http https://connector.get-ryze.ai/mcp` - Cursor: Settings → MCP → add the same URL Setup guide: https://www.get-ryze.ai/how-to-connect-claude-to-google-meta-ads-mcp

[{"step":"Define your objective and baseline metrics.","action":"Gather historical data (conversion rate, traffic volume, cost per action) for the campaign you want to test. Use this to set a realistic minimum detectable effect (e.g., 10-20% lift). Tools: Google Analytics, Meta Ads Manager, or your CRM.","tip":"Avoid testing too many variables at once. Focus on one element (e.g., ad creative, landing page, or audience) to isolate the impact."},{"step":"Calculate sample size and test duration.","action":"Plug your baseline metrics and desired lift into the prompt template. Adjust confidence level (90-99%) and power (70-90%) based on your risk tolerance. Tools: Use an A/B test calculator (e.g., Evan’s Awesome A/B Tools) to verify the sample size.","tip":"If traffic is low, extend the test duration or reduce the minimum detectable effect. Avoid running tests for <7 days (weekend effects can skew results)."},{"step":"Set up the test in your ad platform.","action":"Create two identical campaigns (or ad sets) with the only difference being the variable you’re testing (e.g., ad copy, image, or audience). Use the platform’s built-in A/B testing tool (Meta’s Advantage+ or Google’s Drafts & Experiments) to ensure clean splits.","tip":"Exclude overlapping audiences to prevent contamination. For example, if testing two landing pages, ensure the ad sets point to separate URLs."},{"step":"Monitor daily and act on results.","action":"Use the prompt to generate daily updates. Check for statistical significance, traffic distribution, and cost parity. Pause the test if p-values fluctuate wildly or CPL spikes unexpectedly.","tip":"Set up automated alerts in your ad platform (e.g., Meta’s “Significant Results” notifications) to avoid manual checks. Export data to Google Sheets or Looker Studio for deeper analysis."},{"step":"Scale the winner and document lessons.","action":"Once a winner is declared, scale the winning variant to 100% of traffic and archive the test results. Update your campaign playbook with the new best practices and plan the next test.","tip":"Run a “holdout test” (e.g., 10% of traffic sees the old variant) for 1-2 weeks post-test to confirm the lift is sustainable."}]