Experimentation on Low-Traffic Websites

Modified on Fri, 04 Aug 2023 at 04:07 PM

Running an A/B test on a website with low traffic can be a challenge due to the limited volume that hinders achieving the sample size required by the test. 


However, this hurdle can be surmounted by designing tests that don't need large sample sizes. 


Here's a detailed guide to help with that process:


1. Testing on Upstream Metrics:


Don't aim at the conversion rate, but target an upstream event with a higher percentage.


For instance, test the copy on a Call-to-action (CTA) button on a landing page and use its click-through-rate (CTR) as the experiment's primary goal. 


This approach draws focus on a user's immediate behavior. However, be aware that a click does not necessarily translate to a product purchase. 



2. Prioritize Research and Analytics:


Get closer to your customers by understanding their behavior, wants, and needs. Ideas based on customer research are likely to have a more substantial effect and are more likely to succeed. 


Employ customer research to make informed decisions on the changes to be made to your website.



3. Be Bold and Radical with Changes:


Maximize the chance of significant positive uplift by testing clear and straightforward changes and evading subtle modifications like colors or layouts. 


Opt for more noticeable changes like altering your main value proposition, pricing, discounts, offers, and product merchandising. 


As a startup, being experimental and open to radical changes is key to growth and development.



4. Accept Lower Confidence Levels:


Contrary to the common belief in the Conversion Rate Optimization (CRO) community, an A/B test doesn't strictly require 95% statistical significance to be declared a winner.


Confidence levels reflect your risk level which can be adjusted based on your appetite for risk. You can leverage non-inferiority testing which aims to demonstrate that variant B is not worse than variant A. 


However, remember that a result with a lower confidence level may not be true and may be purely due to error.



5. Broaden Your Concept of Experimentation to Validation:


When traffic volume isn't sufficient for a statistical test, the "validation" approach can be helpful. Don't discard the scientific method of creating and testing hypotheses entirely just because the testing capacity is contrained. 


Consider other types of validation like user testing, click testing, online card sorting and more. While these won't provide statistically significant samples, their qualitative nature can provide invaluable insights into your site's performance and inform your decision-making. 


Remember, the primary goal for these experiments is to learn about user behavior and adapt your site to meet their unique needs.



How to conduct website experimentation on low traffic websites:


Step 1: Identify and understand your customer pain points and behaviors. This could be done through on-site surveys, customer feedback, or by utilizing analytical tools to track user behavior on your website.



Step 2: Come up with ideas on what to test based on the data collected. Remember, these ideas should aim at making significant changes that can potentially have high impact.



Step 3: Prioritize the ideas and decide which one to test. A good approach is to focus on one idea at a time, monitoring the impact before moving on to the next one.



Step 4: Use Howuku Optimize to set up the experiment. Make sure to properly set up the control (A) and variant (B).



Step 5: Run the test on an upstream metric with a higher percentage. For example, if you're testing a CTA button, choose the CTA’s click-through-rate as your main goal. 



Step 6: Monitor your experiment. See how users interact with the changes on your site and record these observations.



Step 7: Carry out an analysis of the results. Once the test is completed, analyze the data collected. Be prepared that due to the nature of low traffic websites, you may need to accept lower confidence levels.



Step 8: Use the results to decide if the change should be permanently implemented, tweaked and retested, or abandoned. If the change yields positive results, it may be a good idea to implement it permanently. 



Step 9: Use the current test results to inform future tests. The valuable learnings you gain from each test, regardless of the outcome, should help refine subsequent tests.



Step 10: Lastly, consider other forms of validation such as user testing, click testing, online card sorting, etc., for a richer qualitative understanding of your users.


Conclusion


Always remember that conducting website experiments, even with low site traffic, is better than simply making guesses or decisions based on opinion. 


The key thing is not to get disheartened if the tests do not yield significant results immediately. Persistence and continuous testing can lead to greater insights over time, greatly enriching your user experience and boosting your brand.

Was this article helpful?

That’s Great!

Thank you for your feedback

Sorry! We couldn't be helpful

Thank you for your feedback

Let us know how can we improve this article!

Select atleast one of the reasons

Feedback sent

We appreciate your effort and will try to fix the article