How to Find the Right Ideas for A/B Testing (Stop Guessing)

“The biggest bottleneck in most experimentation programs is not the testing tool, the traffic volume, or the engineering capacity. It is the quality of the test ideas. Teams that run dozens of tests on button colors and headline variations wonder why their conversion rate has barely moved.”

Teams that run fewer, better-informed tests see compounding gains quarter after quarter. The difference is not luck. It is process. High-impact test ideas come from systematic analysis of data, user feedback, competitive landscape, and usability principles. This guide walks through each source, shows you how to extract testable ideas from it, and provides a framework for prioritizing those ideas so you always work on the highest-leverage tests first.

Why Test Ideas Matter More Than Test Volume

There is a widespread belief that successful experimentation is a numbers game: run enough tests and some will win. While there is a grain of truth here - you do need to run many tests to accumulate meaningful gains - the quality of your ideas determines the magnitude of your wins and the efficiency of your testing program.

A study by Experimentation Hub analyzed over 28,000 A/B tests across dozens of companies and found that only 10% to 20% of tests produce statistically significant positive results.That means 80% to 90% of tests end in no change or a negative result. If your test ideas are random, you are spending most of your testing capacity learning nothing actionable.

But the 10% to 20% win rate is an average. Companies with mature experimentation programs that use data-driven ideation methods report win rates of 30% to 40%. The difference is not better tools or more traffic. It is better hypotheses rooted in deeper understanding of user behavior and motivation.

The goal is not to test everything. It is to test the right things. The following methods will help you identify those right things.

Using Analytics to Find Opportunities

Your existing analytics data is the richest source of test ideas, because it tells you exactly where users are struggling, dropping off, or behaving differently than expected. The key is knowing where to look.

Funnel Analysis

Start by examining your key conversion funnels: sign-up, onboarding, purchase, and upgrade. Identify the step with the largest absolute drop-off. This is usually your highest-leverage testing opportunity because even a small percentage improvement at a high-volume drop-off point translates to significant gains downstream.

For example, if 10,000 users start your sign-up flow each month and 4,000 drop off at step two, a 10% improvement at that step recovers 400 additional users every month. Those 400 users then flow through the rest of your funnel, amplifying the impact at every subsequent stage.

Use your funnel reports to segment drop-offs by device type, traffic source, and user properties. You will often find that the overall drop-off rate masks dramatically different experiences. Mobile users might drop off at 60% while desktop users drop off at 20%. That tells you the problem is mobile-specific and narrows your investigation.

Page-Level Performance

Look at exit rates and bounce rates for key pages. A pricing page with a 70% bounce rate is a clear testing candidate. So is a feature page with high traffic but low click-through to the sign-up form. Pages where users spend an unusually long time may indicate confusion rather than engagement, especially if the page is supposed to drive a quick decision.

Segment Comparisons

Compare conversion rates across user segments. If organic search visitors convert at 4% but paid search visitors convert at 1.5%, investigate why. Are the paid visitors landing on the wrong page? Does the landing page messaging match the ad copy? Is the paid audience fundamentally different in their needs? Each explanation suggests a different test.

User Feedback Analysis

Quantitative data tells you where the problems are. User feedback tells you what the problems are. Together, they give you specific, testable hypotheses that are far more likely to produce positive results than guesses based on analytics alone.

Survey Responses

Review responses from on-site surveys, post-purchase surveys, and NPS follow-ups. Look for recurring themes, not individual comments. If twelve out of fifty survey respondents mention that pricing was confusing, that is a strong signal that a pricing page redesign test could produce significant results.

Pay special attention to the language users employ. If they say “I did not understand the difference between plans,” that suggests a comparison clarity test. If they say “It seemed too expensive for what I get,” that suggests a value communication test. Different user language points to different solutions.

Support Tickets and Chat Logs

Support interactions are gold mines for test ideas because they represent moments of failure.Every ticket about “How do I do X?” is evidence that the interface for X is not intuitive enough. Categorize support tickets by theme and map the highest-volume themes to specific pages or flows in your product.

User Interviews and Session Recordings

Interviews reveal the motivations and mental models behind user behavior. Session recordings reveal the mechanics of how users interact with your interface. Both generate test ideas that are grounded in observed behavior rather than assumptions.

When reviewing session recordings, focus on moments of hesitation, misclicks, and rage clicks (rapid repeated clicking on an element that is not responding as expected). These behavioral signals are reliable indicators of UX friction that can be addressed through testing.

Competitive Analysis

Your competitors are running their own experiments, and their current designs represent the accumulated results of those experiments. Studying competitor approaches can generate test ideas that you might not have considered based on your own data alone.

What to Analyze

Focus on the conversion-critical pages: homepage, pricing page, sign-up flow, and key landing pages. Note specific design patterns: How do they structure their pricing comparison? What information do they include on their sign-up form? What social proof elements do they use? How do they handle objections?

Do not copy competitors blindly. What works for their audience may not work for yours. Instead, treat competitive observations as hypotheses to test. If three competitors all include a live chat widget on their pricing page and you do not, that is worth testing - not because they are doing it, but because it suggests that their users (who may be similar to yours) respond to real-time support during the purchase decision.

Monitor Over Time

Take screenshots of competitor pages monthly. When a competitor changes a significant element and keeps the change for several months, they likely tested it and it won. These persistent changes are stronger signals than temporary variations that revert after a few weeks.

Tools like the Wayback Machine, VisualPing, or simple periodic screenshots can help you track these changes systematically. Look for patterns across multiple competitors - if three competitors independently adopt a similar approach, that convergence is a strong signal.

Heuristic Evaluation

Heuristic evaluation is a structured method for identifying usability problems by evaluating your interface against established design principles. Unlike analytics and user feedback, which reflect actual user behavior, heuristic evaluation applies expert knowledge to predict where users are likely to struggle.

Key Heuristics for Conversion Optimization

Several usability heuristics are particularly relevant for identifying high-impact test opportunities:

• Clarity of value proposition. Can a first-time visitor understand what your product does and why they should care within five seconds? If the answer is no, your above-the-fold content is your highest-priority test.
• Visual hierarchy. Does the page guide the user’s eye toward the primary action? If competing elements (navigation, sidebars, banners) distract from the main CTA, simplification is a strong test candidate.
• Friction in forms. Every field in a form is an opportunity for drop-off. Are you asking for information that is not essential at this stage? Can you use progressive disclosure to collect additional data after the initial conversion?
• Trust signals. Does the page include sufficient social proof (testimonials, logos, case studies, review scores) to support the conversion decision? Insufficient trust signals are a common cause of high-intent, low-conversion pages.
• Cognitive load. Is the user being asked to process too much information at once? Pricing pages are common offenders, presenting dozens of feature comparisons in a dense table that overwhelms rather than informs.

Conducting the Evaluation

Walk through your key pages and flows as if you were a first-time visitor. For each page, ask: What is the single action I want the user to take? Is it obvious? What might prevent them from taking it? Score each issue by severity (how much it is likely to hurt conversion) and document it as a potential test idea.

For best results, have two or three people evaluate independently and then compare notes. Different evaluators catch different issues, and the combination produces a more comprehensive list than any individual could generate alone.

The ICE Scoring Framework

By this point, you should have a list of potential test ideas from analytics, user feedback, competitive analysis, and heuristic evaluation. The challenge now is prioritization. You cannot run every test at once, so you need a systematic way to decide which tests to run first.

The ICE scoring framework, developed by Sean Ellis, is one of the most practical prioritization methods for experimentation. Each test idea is scored on three dimensions:

• Impact: If this test wins, how much will it move the needle? Score from 1 (minimal impact) to 10 (transformative impact). Base this on the volume of users affected and the potential magnitude of improvement.
• Confidence: How confident are you that this change will produce the expected result? Score from 1 (pure guess) to 10 (strong evidence from data and research). Ideas backed by analytics data, user feedback, and successful precedents score higher than ideas based on opinion alone.
• Ease: How easy is this test to implement? Score from 1 (requires major engineering work) to 10 (can be deployed in hours with front-end changes only). Ease matters because faster implementation means faster learning.

Calculate the ICE score by averaging the three dimensions: (Impact + Confidence + Ease) / 3. Rank your test ideas by ICE score and work from the top down. This ensures that you prioritize tests that combine high potential impact, strong supporting evidence, and practical feasibility.

Common ICE Scoring Mistakes

The most common mistake is inflating Impact scores. Every test idea feels like it could be a big win when you first conceive it. Discipline yourself to calibrate Impact against the actual data. A test on a page that receives 500 visits per month cannot have the same Impact score as a test on a page that receives 50,000.

Another mistake is undervaluing Confidence. A high-Impact, low-Confidence test is a gamble. A moderate-Impact, high-Confidence test is a reliable improvement. Over time, reliable improvements compound into significant results, while gambles average out to zero.

Building a Test Backlog

A test backlog is a prioritized list of test ideas that serves as the pipeline for your experimentation program. Without one, teams fall into reactive testing - running whatever test the loudest voice in the room suggests this week - instead of systematically working through their highest-value opportunities.

Backlog Structure

Each entry in your test backlog should include:

• Hypothesis - Using the if/then/because format (if we change X, then metric Y will improve by Z, because evidence suggests W).
• Primary metric - The single metric you will use to evaluate the test.
• Source of idea - Where the idea came from (analytics, user feedback, competitive analysis, heuristic evaluation). Tracking sources helps you evaluate which ideation methods produce the best results over time.
• ICE score - The prioritization score calculated as described above.
• Estimated sample size and duration - How long the test will need to run based on the page’s traffic and the expected effect size.
• Status - Backlog, in design, live, analyzing, or completed.

Keeping the Backlog Healthy

A healthy backlog has 15 to 30 ideas at any given time, with a mix of quick wins (high Ease) and bigger bets (high Impact). Review and re-score the backlog monthly. Ideas that have been sitting at the bottom for three months should be removed - they are consuming mental overhead without contributing value.

Feed the backlog continuously. Set up a recurring monthly meeting where product, marketing, and engineering review new data from your analytics platform, recent user feedback, and competitive changes. Each meeting should generate three to five new test ideas to replace completed and retired tests.

Maintaining Testing Momentum

The hardest part of experimentation is not running a single test. It is maintaining a consistent testing cadence over months and years. Programs stall when test ideas dry up, when inconclusive results are discouraging, or when competing priorities squeeze out testing capacity.

The ideation methods described in this guide are the antidote to idea drought. If you systematically review your analytics, user feedback, competitive landscape, and usability heuristics on a monthly basis, you will never lack for high-quality test ideas.

For discouraging results, reframe the goal of experimentation. The goal is not to produce wins. It is to produce learning. A test that fails tells you something valuable: your hypothesis was wrong, which means your understanding of the user was incomplete. That learning improves your next hypothesis, which improves your next test, which gradually increases your win rate.

For competing priorities, protect testing capacity by embedding it into your sprint planning. Allocate a fixed percentage of engineering capacity (10% to 20% is common) to experimentation. Treat it as infrastructure investment, not a discretionary activity. The companies that build the strongest testing cultures are the ones that treat experimentation as a core capability rather than a side project.

Start with the method that feels most accessible given your current data and resources. If you have good analytics, start with funnel analysis. If you have active customer support, start with ticket analysis. As you build competence in one method, layer on the others. Within six months, you will have a robust ideation engine that consistently generates high-impact test ideas and keeps your experimentation program moving forward.

Key Takeaways

The quality of your test ideas determines the ROI of your entire experimentation program. Here is how to consistently generate better hypotheses: