Split Testing with a Genetic Algorithm

Posted by

This is the featured advertiser on ShoeMoney. Put your company here!


I’ve got some interesting data following the ShoeMoney System launch that I want to share.

This time around I used genetify to do multi-variate testing with an optimizing algorithm. Let me show you why this is important.

Let’s say we got 50,000 unique visitors to our landing page.

3.0% conversion rate

5.0% conversion rate

So you say, great, orange converted better. So what? Everyone knows orange buttons convert better. Well you’re right, but we also tested some much more interesting things that I’m not willing to share. But, let me show you something more important.

Assume we had 50,000 unique visitors, with a $200 sale price.

50,000 * 3% = 1500 = $300,000
50,000 * 5% = 2500 = $500,000

If we had simply 50/50 split tested to SEE which result was better:

25,000 * 3% + 25,000 * 5% = $150,000 + $250,000 = $400,000

But using a genetic algorithm that optimized in real-time for the best conversion rate:

10,000 * 3% + 40,000 * 5% = $60,000 + $400,000 = $460,000

Because our algorithm optimized itself for the best conversion, we made an extra 15% in revenue.

The coolest thing about MVT with a genetic algorithm is that it will find combinations that work the best that you would never have dreamed of.

If you aren’t using the tools available, you are leaving money on the table.

72 thoughts on “Split Testing with a Genetic Algorithm

  1. Jeff Dempey

    This is a great post to show clients. Some people hate testing because they would rather have their site look one way instead of converting better. Personally I’ll take 5% and a non matching button.

  2. Greg

    It’s interesting how some inane little thing can sometimes make such a big difference.

    I use green buttons. Would orange buttons work better on my site? How about red ones.

    You never know until you test.

    Sometimes one word, or even one period, can make a difference.

    1. PyrogenicMedia

      Actually not a bad question. I would think that you would want to go further then just the buy button, maybe images, video, header text etc. How many combination’s can it handle?

    2. ddn Post author

      Tim that was an oops in the CSS. Genetify itself however can handle basically an unlimited number of variables.

      1. Andi | WebMarketerDepot

        Yeah, I’ve re-read it. And I still will ask my question above.

        I understand that you are testing a lot of other things than this orange button. But, as you said, orange button is proven to convert better (esp. in internet marketing niche, as many other marketing gurus have tested).

        Thus, there’s no point of testing the thing that have been proven to work again and again…but then Shoemoney have the money and resources to do it…so, I guess, why not?

        1. Justin Goff

          Just because orange converts better most of the time, doesnt mean it always will – thats why theyre testing…

          along with testing numerous other things on the page…

  3. Mike Chiasson

    Wow that is a pretty neat idea to split testing. It really takes away from the traditional idea of split testing. I’m curious as to how the algorithm accept a decent sample size. As everyone knows there is a ton of numerous variables that could affect the effectiveness of the variations (color blindness, time of day, etc.) that really have no correlation on their general converting process.

    This is typically why most people would allow a split test to run over the course of say an entire day. However when you weigh the two out I think I will take the extra money in the shorter time anyways. I’ve never been fortunate enough to have an offer get 50k views in a day anyways lol.

  4. ChrisGuthrie

    Thanks for the article Dave. Very interesting stuff, I had never heard of an algorithm that could self optimize.

    Any chance you’ll do a follow up post to share the other info you learned? If not no biggie I understand.

    Either way very interesting indeed.

    Chris

  5. Maximus

    More outside the box thinking from Dave… this is great. Just goes to show the simplest things make all the difference.

  6. Damien Anderson

    That is all well and good but the part regards MVT that I am as yet sold on is really that in order to test these variations to achieve an enlightened UX and conversion rate pairing, there is a need to serve a great amount of garbage to users. I would be interested to learn more on your thoughts about structuring MVT to minimise this potential negative audience impact.

    Thanks

    Damien

  7. ZK@Web Marketing

    Great stuff Shoe, the landing page itself was very interesting…I actually did not see the changing colour of the tab ..I guess that why you are a millionaire and I am just another struggling blogger :)

  8. Dom A

    Great post David – This is really useful info as we have been doing split testing and multivariate testing on our site sevenfigureblueprints.com.

    You can do lots of different variations but ultimately you want the best combination that will make the most money – and this seems to do that. The fact that is open source makes this all the more exciting!

    I’m off to check it out.

    Thanks again.

  9. Chris Pontine

    I read an article on the color orange and they make a good point that we associate it with important, such as life vests and hunting gear, so from a social perspective many of us probably connect it in this way.

  10. Andrew

    The tool is cool, but if simply use a human mind, I think a sample size of 100 will already be enough to tell orange is better, then you will have 49900 * 5% instead… Isn’t it?

    Of course, it may useful when involve more combinations….

  11. Eric Itzkowitz

    @DDN

    WOW! This looks freaking fantastic! I am immediately sending this to our dev team and CEO.

    We, like most, use G’s Optimizer, but wow this looks good.

    Thanks for sharing such a nice resource.

    You guys rock!

  12. jtGraphic

    So, should one keep Genetify on all the time, or eventually move towards a split that works? I’d assume you’d always keep it on, but change the tests often?

  13. Anderson-Paulson-Williams

    After reviewing the new Shoemoney system we have found it to be a chock full of copyright infringements. Get a good lawyer now Sparky, we are going to be opening a big can of whoop ass on you that could last for years.

    David Paulson

    1. Jeremy Schoemaker

      Dear Anonymous jackass.

      You come here every day with a different name and ip alleging random crap. You have threaten the lives of myself and my family. You think you have your computer to hide behind and I can’t find out who you are. You could not be more wrong. I am have spent a couple thousand dollars finding out who you are and what you do for a living. I look forward to sharing it with the world very soon.

  14. krillz

    It’s like anything in life. We are built in such a way that we react to variations strongly as the brain picks that up to do calculations whether this new change is a threat to me or not, call it primal instinct or something left in our reptile part of the brain but it’s a fact.

    It’s proven that we react more to the colors orange/yellow than for example blue and green. And it has a very logical explanation, the first bunch of colors are not that common to our brain than the green and blue that the brain registers continously throughout the day and years and it has become a constant thing for us where the orange in this case is a variation from what the brain is used to registrating.

    Same thing with sounds, constant sound pretty much never grabs our attention, but put a variation to that and the brains starts letting you know about it straight away, could be something left from back in the days where variation in sound could be a animal sneaking up on you, today we see it used in alarms, police sirens etc to attract attention.

    Same thing with visual stimuli, you will notice something moving sideways instantly whereas something that approaches you or moves away from you is a lot harder to notice.

    Taking that behavioral science class really was worth it, great fun if you have the oppertunity to take one then think it through as I don’t regret it at all and you will never see the world in the same way again.

  15. Build Web Traffic

    I think that orange is more inviting as a color. It sounds funny but this is why you are who you are. People do the same things, but those that make the small, yet huge discoveries from testing and trials are the guys that stand above the rest with proven results. Thanks for sharing as always.

  16. Alex

    Love the way he says –

    “but we also tested some much more interesting things that I’m not willing to share.”

    Translation – You’re not IN with the in-crowd so crawl back under the rock you came from :)

    Apart from that a good article.

  17. Malpractice

    You *assume* 50,000 unique visitors, so how many visitors was it real-life?

    Also, was this the exact button you used?
    Interested to know …

  18. Udi

    Awww man, I didn’t know this exists. I wanted to invent it on my own. I have a Ph.D. in genetic algorithms (more specifically, the broader field: Evolutionary Computation) and this is one of the first implementations of affiliate marketing and genetic algorithms I wanted to do – just didn’t have the time to work on it, currently focusing on plain affiliate marketing.

    Oh well. Got plenty more of ideas ;) Nice to see that it works – not that I had any doubt.

    1. Andy

      @Udi

      I have the same interest as you, and was thinking like you that it was my idea.

      It’s certainly a fascinating subject, the idea of automatically optimizing elements of web sites.

      1. Sean

        Haha, scientists unite on this post!

        In my field, we perform a function called simulated annealing to avoid local minimums, which should be applicable in this scenario.

        The idea is that if you only change one variable at a time such as in this case color. You can only maximize your results on that one particular variable. In other words, you have found a local minimum. I am new to this, but currently it seems the method to attain the global minimum (max conversion) seems to be adjusting one variable, track it, repeat – this post is cool because it adjusts in real time.

        Perhaps the next level is to allow ‘all’ variables to adjust simultaneously (widget position, color, pictures, text, copy, etc…) have variables weighted so that you don’t get complete nonsense displayed.

        The website would automatically adjust as mentioned in this post to maximize conversion.

  19. FM

    Good stuff. Orange color usually make better conversion but not always. It depends on the target audience and the product or service you offer. Yes, Amazon, ebay, they use orange but iTune doesn’t. Why? I believe that’s all about design.

  20. World Travel

    Yes, this research is quite interesting. I know that psychology is quite important is shopping, advertising and other internet actions. If you know how to make people clicking more, this will be great

  21. Jason

    Interesting study yeah it shows anything that is not real bland will get more clicks banners, sign up now buttons, whatever it may be!

  22. Eric Bridges

    I really liked this post. Your formula will help articulate the importance of what clients see as the “small” things into real time results. There was a good post on contact forms and how the “Mad Libs” version of the contact form converted at a much better rate. Imagine if they had evaluated conversion with your formula! http://www.lukew.com/ff/entry.asp?1007

  23. multivariate testing and digital optimization insider

    The benefits of a more holistic optimization and marketing approach can be extended beyond content used to attract leads and sales. There other types of search that can drive or benefit business including customer service related content, job listings and news content. Each has it�s own audience to consider and therefore, a different context for optimization.

  24. Walker Upton

    I just identified your blog, I bookmarked it and i ‘m reading the actual blogposts. I already enjoy it. Interesting issue in any event you actually look on this. I come as a result of this standpoint that notice remarks as akin of attentive.

  25. Bari Bynum

    Usefull points made on your blog, most I agree with. Remember seeing a similar blog which I will look to post. Definately will bookmark in any case I await your next thought provoking blog post

  26. wanda

    so could you show me on how to really make money from home?because every time i try to apply for working @ home there’s always a scam and it makes me not want to do it.but then they want to charge me alot of money and im looking for work to me if someone looking for work why we have to pay to get a job what’s the point?when people look for work and get hired then they get paid for there work.that’s my point.why i have to pay if i want to work from home?i dont trust noone with my creditcard.when i try to apply for it but when i see they asking for creditcards i delete it quick.

  27. Haluk Akin

    Genetic algorithm is an overkill for this case.
    Much simpler heuristics can do the job easily.
    This is like using a sword to spread butter.

  28. Jamie Hudson

    Mind boggling number! Question: when does the split testing stop? Seriously tough to figure out, but you definitely put things into perspective on split testing. Appreciate it!

  29. Rea

    i want to understand something. how does the algorithm know how to split the traffic? the whole point is to test for better conversion and such isn’t it?
    how can you tell where do you the best conversions before you test it?

    tnx

Comments are closed.