Most A/B tests fail for a boring reason: the idea behind them was weak. Someone suggested changing a button colour, the test ran for a couple of weeks, the result was inconclusive, and everyone moved on having learned nothing. Good testing depends on good hypotheses, and good hypotheses depend on understanding why visitors do or do not act. That is a thinking problem, which is partly where AI can help and partly where it cannot.
AI tools are very good at producing lots of variations and at spotting patterns in feedback. They are poor at knowing your audience, your traffic levels or the real reason a page underperforms. This guide shows how to use AI to generate and prioritise test ideas without falling into the trap of testing whatever the model happens to suggest. You will learn how to frame a hypothesis, where to find evidence, what to ask the AI for, how to prioritise, how to design a fair test, and how to record what you learn.
Key takeaways
- A test idea is a hypothesis with evidence behind it: we believe changing X will affect Y for this audience, because of this observation.
- Feed the AI your own evidence, such as analytics notes, reviews, support questions and session observations, so suggestions are grounded.
- Prioritise with a simple scoring method, and be honest about whether you have enough traffic to detect a difference.
- Test one meaningful change at a time, decide the success measure and duration before you start, and do not stop early because a result looks good.
- Record every result, including the failures. A learning log is worth more than any single winning variant.
What an A/B test is, in plain terms
In an A/B test, you show two versions of something to similar groups of people at the same time, and compare how each group behaves. Version A is usually the current one. Version B has one change. If B produces a better result than random variation would explain, you have evidence the change helped. It might be a web page, an email subject line, an advert, a signup form or a button.
The key phrases are “at the same time” and “similar groups”. Comparing this month with last month is not an A/B test, because the season, the news and the traffic mix may all have changed. And the phrase “better than random variation would explain” matters because small samples produce big swings by chance. A test with only a few conversions per version tells you almost nothing.
Because most tools handle the splitting and the statistics, you do not need advanced maths. You do need patience and a clear plan. Everything in this guide is general and conceptual, so check your testing tool’s documentation for how it calculates results.
| Term | Plain meaning |
|---|---|
| Control | The existing version you compare against |
| Variant | The version with your change |
| Conversion | The action you want, such as a signup or purchase |
| Hypothesis | A reasoned prediction about what will happen and why |
| Significance | How unlikely the difference is to be pure chance, as reported by your tool |
| Sample size | How many visitors or recipients you need for a reliable result |
Start with a hypothesis, not a hunch
A good hypothesis has four parts: the change, the expected effect, the audience, and the reason. For example: “Because customer reviews show people are unsure what is included, adding a short list of inclusions near the button will increase the number of visitors who start an enquiry.” The reason is the part most people skip, and it is the part that makes a result informative. If the test wins or loses, you learn whether your understanding of visitors was right.
Compare that with “Let us try a green button.” There is no reason, so a win teaches nothing you can reuse and a loss gives you no direction.
A hypothesis template
- Because we observed [evidence],
- we believe that changing [element] to [new version]
- will cause [audience] to [behaviour change],
- which we will measure by [metric] over [time or sample].
Ask the AI to rewrite weak ideas into this format, but insist that every “because” cites evidence you supplied. Otherwise it will invent plausible reasons, which is the same habit described in our article on AI hallucinations.
Gather evidence the AI can work with
AI cannot see your analytics, your customers or your market unless you tell it. The better the evidence you supply, the better the ideas. Collect a handful of sources and summarise them in plain text.
| Evidence source | What it can show | How to give it to the AI |
|---|---|---|
| Analytics | Where visitors drop off, which pages underperform | Describe the numbers in your own words, never paste private data you should not share |
| Customer reviews and survey comments | Reasons for hesitation, confusing points | Paste anonymised excerpts |
| Support questions and sales objections | Common doubts at decision time | Paste a list of the top ten |
| Session recordings or user testing notes | Where people hesitate or click the wrong thing | Write short observations |
| Previous test results | What has and has not worked here | Share the learning log |
Always remove personal data first, and follow your organisation’s rules on what can go into a tool. Our guide on protecting company data when using AI tools explains how to decide. For reviews in particular, grouping feedback into themes is a great source of test ideas, since each recurring doubt suggests something to address on the page.
Prompts for generating test ideas
Once the evidence is organised, ask for ideas in a structured way. If you want a refresher on structuring requests, see our guide to writing better prompts, but the essentials are the context, the evidence and the output format.
An idea generation prompt
“Here is the page’s goal, the audience, and a summary of evidence: [evidence]. Suggest eight A/B test ideas. For each one, give: the hypothesis in the template format citing which piece of evidence supports it, the single element to change, the metric to measure, and any risk. Do not invent data. If the evidence does not support an idea, say so.”
Prompts for variations
When you have chosen a hypothesis, ask for three to five variations of the specific element, such as headlines, descriptions or calls to action. Give the voice guidelines and the facts that must stay true. Remove anything that makes a claim you cannot support. Variants that promise things the product does not deliver may win a test and lose you trust, so keep honesty in the guard rails.
A critique prompt
Ask the tool to challenge each idea: “What could make this test misleading? What else might explain a difference?” The answers often remind you of practical issues such as overlapping campaigns, tracking gaps or tiny sample sizes.
What to test, by area
Different parts of your marketing offer different kinds of test. Match the idea type to the stage where visitors struggle, and avoid cosmetic changes unless the evidence points to a visual problem.
| Area | Example elements | Typical hypothesis theme |
|---|---|---|
| Landing pages | Headline, offer clarity, proof, form length, call to action wording | Visitors do not understand the offer or doubt it |
| Emails | Subject line, preview text, send time, length, single versus multiple links | The message is not noticed or the ask is unclear |
| Adverts | Message, image concept, audience angle | The promise does not match what people want |
| Pricing and plans pages | Order of options, wording, explanation of what is included | Visitors cannot compare options easily |
| Forms and checkout | Number of fields, labels, error messages, reassurance text | Friction or uncertainty at the final step |
For email-specific testing, our guide to using AI for email marketing covers practical ideas. Bigger, evidence-led changes such as clarifying an offer usually teach more than small tweaks such as changing a shade of colour.
Prioritise ideas with a simple score
You will usually end up with more ideas than you can test. Rank them with a lightweight scoring method. Three questions are enough: How strong is the evidence? How big could the effect be? How easy is it to run? Score each from one to five, and multiply or add the scores. Use the AI to help fill in a first pass, but make the final scores a team conversation.
Be honest about traffic. Tests need enough visitors or recipients, and enough actions, to detect a difference. If your page gets few visits, you may only be able to detect large changes, and small tweaks will be a waste of time. In low traffic situations, consider bolder changes, longer test periods, or using other methods such as customer interviews and simple before-and-after observation, while remembering their weaknesses.
| Idea | Evidence (1 to 5) | Potential impact (1 to 5) | Ease (1 to 5) | Total |
|---|---|---|---|---|
| Add an inclusions list near the button | 4 | 4 | 5 | 13 |
| Shorten the form from seven fields to four | 3 | 4 | 4 | 11 |
| Change the button colour | 1 | 1 | 5 | 7 |
| Rewrite the headline to name the audience | 3 | 3 | 5 | 11 |
The numbers above are only a worked illustration of the method, not real results. The point is to make trade-offs visible and to discourage tests whose only merit is that they are easy.
Design a fair test
A good idea can be ruined by a poor test. Before starting, decide on the single primary metric, the audience, the length of the test and the minimum sample you need. Write these down. Change only one meaningful thing at a time, so you know what caused the difference. If you change the headline, the image and the button together, you cannot tell which mattered, though sometimes testing a whole redesign as a package is a legitimate choice if you accept that you will learn less about the cause.
Run the test for full weekly cycles, since behaviour differs by day of the week, and avoid periods with unusual events such as sales or holidays where possible. Make sure the split is random and that the same visitor sees the same version each time. Check that tracking works on both versions before you begin.
Rules to avoid fooling yourself
- Decide the success measure and duration before you start.
- Do not stop the test early just because one version looks ahead. Early swings are often noise.
- Do not keep testing many metrics until one looks good.
- Do not run overlapping tests that affect the same visitors, unless your tool handles it.
- Do not change the test mid-way. If you must, restart.
- Check for technical problems, such as one version loading more slowly.
Read the results honestly
When the test ends, look at the result in context. A clear win on the primary metric is good news, but check that secondary effects have not gone the wrong way, for example more signups but lower quality leads. A flat result is also information: it may mean the change did not matter, or that the test lacked the power to detect it.
AI can help you write up the results and suggest follow-up hypotheses, but do not ask it to interpret numbers it cannot see, and be sceptical of explanations it supplies for why something worked. Treat them as hypotheses for the next test. Our guide on checking AI output for mistakes applies to any analysis summary the tool produces.
| Result | What it might mean | Next step |
|---|---|---|
| Clear win | The change helped this audience | Roll out, then test a follow-up idea |
| Clear loss | The hypothesis was wrong, or the change created friction | Learn from it, revisit the evidence |
| No clear difference | Change did not matter, or not enough data | Consider a bolder change or a longer test |
| Mixed metrics | Trade-offs between outcomes | Decide which metric matters most and why |
Honesty and customer trust
Testing should never involve deceiving customers. Do not test fake scarcity, invented testimonials, false countdown timers or misleading claims, however well they might perform. They damage trust, and in many places advertising and consumer protection rules prohibit them. This is general information, not legal advice. Test clarity, relevance and ease, not manipulation.
Also consider privacy. If your testing tool uses cookies or collects data, follow the rules of your region on consent and disclosure, and be careful with any personal data you place into AI tools while preparing tests.
Keep a learning log
The long-term value of testing is accumulated knowledge. Keep a simple log with one row per test: the hypothesis, the evidence behind it, what changed, dates, sample size, primary metric, result, and what you learned. Review it before planning new tests, so you do not repeat past failures or forget past wins.
AI can help you search and summarise the log, and spot patterns such as “tests that clarified the offer tend to win”. Keep the underlying records accurate, though, and make sure patterns are based on enough tests to mean something. A handful of results is a hint, not a law.
| Log field | Example content |
|---|---|
| Hypothesis | The full because, change, audience, metric statement |
| Evidence | Where the idea came from |
| Setup | Dates, split, sample size, tool |
| Result | Outcome on the primary and key secondary metrics |
| Learning | What we now believe, and the next question |
A/B test idea checklist
Run through this list before you launch any test.
- The idea is written as a hypothesis with a reason and evidence.
- The evidence comes from real data, feedback or observation.
- One meaningful change is being tested, with a single primary metric.
- You have enough traffic to detect a plausible difference, or you have chosen a bolder change.
- The duration is set in advance and covers full weekly cycles.
- Variants make no claims you cannot support.
- Tracking and rendering have been checked on both versions.
- Personal data was handled in line with your rules.
- The result and learning will be recorded in the log.
Frequently asked questions
Can AI tell me which version will win?
No. It can suggest ideas and variations, but only a properly run test with real visitors can show which version performs better. Treat any prediction as a guess.
How long should I run an A/B test?
Long enough to reach the sample you planned and to cover full weekly cycles, often at least a couple of weeks. Decide this in advance and avoid stopping early because one version looks ahead.
What if I do not have much traffic?
Test bigger changes, run tests for longer, and combine testing with other methods such as customer interviews and surveys. Tiny tweaks need large samples to show anything.
Should I test more than one change at once?
Testing one meaningful change makes results easier to interpret. Testing a full redesign as a package can be valid, but you will not know which part made the difference.
How many ideas should I generate?
Generate more than you need, then score and select a few. A small number of well-founded tests beats a long list of random ones.
Is it okay to test urgency or scarcity messages?
Only if they are true. Fake timers or invented stock levels mislead customers and may break consumer protection rules. This is general information, not legal advice.
Conclusion
AI is a useful partner for A/B testing when you use it for what it does well: turning evidence into structured hypotheses, generating variations, and challenging your assumptions. It is not a substitute for evidence, for sufficient traffic or for a fair test design. The best results come from a clear hypothesis, a single meaningful change, a pre-set measure and an honest reading of what happened.
Pick one page or email with a known problem, gather the evidence, and write three hypotheses in the template format. Score them, run the best one properly, and record what you learn. If you want help deciding which assistant to use in this work, our guide on how to choose an AI tool is a good starting point, and your log will make every later test smarter.
