Post image for Split Testing with a Genetic Algorithm

Split Testing with a Genetic Algorithm

by ddn on March 3, 2010 · 72 comments

I’ve got some interesting data following the ShoeMoney System launch that I want to share.

This time around I used genetify to do multi-variate testing with an optimizing algorithm. Let me show you why this is important.

Let’s say we got 50,000 unique visitors to our landing page.

3.0% conversion rate

5.0% conversion rate

So you say, great, orange converted better. So what? Everyone knows orange buttons convert better. Well you’re right, but we also tested some much more interesting things that I’m not willing to share. But, let me show you something more important.

Assume we had 50,000 unique visitors, with a $200 sale price.

50,000 * 3% = 1500 = $300,000
50,000 * 5% = 2500 = $500,000

If we had simply 50/50 split tested to SEE which result was better:

25,000 * 3% + 25,000 * 5% = $150,000 + $250,000 = $400,000

But using a genetic algorithm that optimized in real-time for the best conversion rate:

10,000 * 3% + 40,000 * 5% = $60,000 + $400,000 = $460,000

Because our algorithm optimized itself for the best conversion, we made an extra 15% in revenue.

The coolest thing about MVT with a genetic algorithm is that it will find combinations that work the best that you would never have dreamed of.

If you aren’t using the tools available, you are leaving money on the table.

full disclosure

About the author...

– who has written 9 posts on ShoeMoney.com.

David is the CTO of ShoeMoney Media Group. He makes the gears turn in opposite directions.


Jeremy recommends you check out these amazing posts:

  1. trump-youre-fired How To Fire Someone
  2. Shoemoney Featured Post How to out your competitors and frenemies for spamming Google
  3. cash How To Make $100 A Day With An Info Product (Part 4)

{ 72 comments }

1 Tom | Build That List

You definitely want to be optimizing your landing pages for maximum profits! So is orange working better for your really or was that just an example?

2 jtGraphic

Heeeeyyyy I just read this on Dave’s blog… oh well…

3 sherlythomas

But this has more pratical information . Split testing with various factors will bing up more convertions

4 Joey Smith

I would love to see more data on this type of optimization.

Pretty kewl stuff. :)

–Joey

5 Jeff Dempey

This is a great post to show clients. Some people hate testing because they would rather have their site look one way instead of converting better. Personally I’ll take 5% and a non matching button.

6 Veles2003_2

Great article. This is why I love ShoeMoney.com they are not afraid to tell some of the secrets that make them so successful !

7 Jack

Very interesting test. I knew colors made a difference but never knew the difference in percentage.

8 Greg

It’s interesting how some inane little thing can sometimes make such a big difference.

I use green buttons. Would orange buttons work better on my site? How about red ones.

You never know until you test.

Sometimes one word, or even one period, can make a difference.

9 S Ahsan

Shoemoney’s success speaks for itself and the system..need not say much

10 Tim Paulino

On the version I saw, the button was orange, then green on the mouseover. Did the algorithm goes as far to find combinations like that?

11 PyrogenicMedia

Actually not a bad question. I would think that you would want to go further then just the buy button, maybe images, video, header text etc. How many combination’s can it handle?

12 ddn

Tim that was an oops in the CSS. Genetify itself however can handle basically an unlimited number of variables.

13 Rick

that’s an awesome tool! i mean look how much extra money this thing makes… def worth looking into.

14 Andi | WebMarketerDepot

Woah, that automatic optimizer is awesome…but if you know that orange button converts the best, what’s the point of testing it again and earn less than what you could have?

15 ddn

Please read the article again.

16 Andi | WebMarketerDepot

Yeah, I’ve re-read it. And I still will ask my question above.

I understand that you are testing a lot of other things than this orange button. But, as you said, orange button is proven to convert better (esp. in internet marketing niche, as many other marketing gurus have tested).

Thus, there’s no point of testing the thing that have been proven to work again and again…but then Shoemoney have the money and resources to do it…so, I guess, why not?

17 Justin Goff

Just because orange converts better most of the time, doesnt mean it always will – thats why theyre testing…

along with testing numerous other things on the page…

18 Jeremy Schoemaker

The point is you dont know… and just because it might convert better from one traffic source doesnt mean it will for all.

19 Mike Chiasson

Wow that is a pretty neat idea to split testing. It really takes away from the traditional idea of split testing. I’m curious as to how the algorithm accept a decent sample size. As everyone knows there is a ton of numerous variables that could affect the effectiveness of the variations (color blindness, time of day, etc.) that really have no correlation on their general converting process.

This is typically why most people would allow a split test to run over the course of say an entire day. However when you weigh the two out I think I will take the extra money in the shorter time anyways. I’ve never been fortunate enough to have an offer get 50k views in a day anyways lol.

20 Jeremy Schoemaker

You don’t need 50k its really all relative

21 PPC Icon

Yup thats a sick bit of kit.

22 Veles2003_2

Great stuff David

23 ChrisGuthrie

Thanks for the article Dave. Very interesting stuff, I had never heard of an algorithm that could self optimize.

Any chance you’ll do a follow up post to share the other info you learned? If not no biggie I understand.

Either way very interesting indeed.

Chris

24 Jeremy Schoemaker

As we get more data we will have some more interesting stuff.

25 Jonathan Volk

I love this type of stuff. Did you use google website optimizer or something different?

Thx

26 Jeremy Schoemaker

Its a open source program called Genetify – http://wiki.github.com/gregdingle/genetify/

27 Wynne

That tool looks sweet. And the pricing is not so bad either.

28 Veles2003_2

Great Stuff David

29 neil802

Is that algorithm to become one the Shoemoney tools?

30 Maximus

More outside the box thinking from Dave… this is great. Just goes to show the simplest things make all the difference.

31 Damien Anderson

That is all well and good but the part regards MVT that I am as yet sold on is really that in order to test these variations to achieve an enlightened UX and conversion rate pairing, there is a need to serve a great amount of garbage to users. I would be interested to learn more on your thoughts about structuring MVT to minimise this potential negative audience impact.

Thanks

Damien

32 ZK@Web Marketing

Great stuff Shoe, the landing page itself was very interesting…I actually did not see the changing colour of the tab ..I guess that why you are a millionaire and I am just another struggling blogger :)

33 Dom A

Great post David – This is really useful info as we have been doing split testing and multivariate testing on our site sevenfigureblueprints.com.

You can do lots of different variations but ultimately you want the best combination that will make the most money – and this seems to do that. The fact that is open source makes this all the more exciting!

I’m off to check it out.

Thanks again.

34 Chris Pontine

I read an article on the color orange and they make a good point that we associate it with important, such as life vests and hunting gear, so from a social perspective many of us probably connect it in this way.

35 fas

Oh ok, so this was a teaser?

36 Andrew

The tool is cool, but if simply use a human mind, I think a sample size of 100 will already be enough to tell orange is better, then you will have 49900 * 5% instead… Isn’t it?

Of course, it may useful when involve more combinations….

37 Eric Itzkowitz

@DDN

WOW! This looks freaking fantastic! I am immediately sending this to our dev team and CEO.

We, like most, use G’s Optimizer, but wow this looks good.

Thanks for sharing such a nice resource.

You guys rock!

38 sean

does this essientially help create and run a design of experiment?

39 jtGraphic

How did that make it past moderation?

40 jtGraphic

So, should one keep Genetify on all the time, or eventually move towards a split that works? I’d assume you’d always keep it on, but change the tests often?

41 Anderson-Paulson-Williams

After reviewing the new Shoemoney system we have found it to be a chock full of copyright infringements. Get a good lawyer now Sparky, we are going to be opening a big can of whoop ass on you that could last for years.

David Paulson

42 Jeremy Schoemaker

Dear Anonymous jackass.

You come here every day with a different name and ip alleging random crap. You have threaten the lives of myself and my family. You think you have your computer to hide behind and I can’t find out who you are. You could not be more wrong. I am have spent a couple thousand dollars finding out who you are and what you do for a living. I look forward to sharing it with the world very soon.

43 Wynne

lol. lame.

44 krillz

It’s like anything in life. We are built in such a way that we react to variations strongly as the brain picks that up to do calculations whether this new change is a threat to me or not, call it primal instinct or something left in our reptile part of the brain but it’s a fact.

It’s proven that we react more to the colors orange/yellow than for example blue and green. And it has a very logical explanation, the first bunch of colors are not that common to our brain than the green and blue that the brain registers continously throughout the day and years and it has become a constant thing for us where the orange in this case is a variation from what the brain is used to registrating.

Same thing with sounds, constant sound pretty much never grabs our attention, but put a variation to that and the brains starts letting you know about it straight away, could be something left from back in the days where variation in sound could be a animal sneaking up on you, today we see it used in alarms, police sirens etc to attract attention.

Same thing with visual stimuli, you will notice something moving sideways instantly whereas something that approaches you or moves away from you is a lot harder to notice.

Taking that behavioral science class really was worth it, great fun if you have the oppertunity to take one then think it through as I don’t regret it at all and you will never see the world in the same way again.

45 satilik

Is that algorithm to become one the Shoemoney tools

46 Jeremy Schoemaker

its open source code. free to everyone

47 Build Web Traffic

I think that orange is more inviting as a color. It sounds funny but this is why you are who you are. People do the same things, but those that make the small, yet huge discoveries from testing and trials are the guys that stand above the rest with proven results. Thanks for sharing as always.

48 Alex

Love the way he says -

“but we also tested some much more interesting things that I’m not willing to share.”

Translation – You’re not IN with the in-crowd so crawl back under the rock you came from :)

Apart from that a good article.

49 Malpractice

You *assume* 50,000 unique visitors, so how many visitors was it real-life?

Also, was this the exact button you used?
Interested to know …

50 Udi

Awww man, I didn’t know this exists. I wanted to invent it on my own. I have a Ph.D. in genetic algorithms (more specifically, the broader field: Evolutionary Computation) and this is one of the first implementations of affiliate marketing and genetic algorithms I wanted to do – just didn’t have the time to work on it, currently focusing on plain affiliate marketing.

Oh well. Got plenty more of ideas ;) Nice to see that it works – not that I had any doubt.

51 Andy

@Udi

I have the same interest as you, and was thinking like you that it was my idea.

It’s certainly a fascinating subject, the idea of automatically optimizing elements of web sites.

52 Sean

Haha, scientists unite on this post!

In my field, we perform a function called simulated annealing to avoid local minimums, which should be applicable in this scenario.

The idea is that if you only change one variable at a time such as in this case color. You can only maximize your results on that one particular variable. In other words, you have found a local minimum. I am new to this, but currently it seems the method to attain the global minimum (max conversion) seems to be adjusting one variable, track it, repeat – this post is cool because it adjusts in real time.

Perhaps the next level is to allow ‘all’ variables to adjust simultaneously (widget position, color, pictures, text, copy, etc…) have variables weighted so that you don’t get complete nonsense displayed.

The website would automatically adjust as mentioned in this post to maximize conversion.

53 FM

Good stuff. Orange color usually make better conversion but not always. It depends on the target audience and the product or service you offer. Yes, Amazon, ebay, they use orange but iTune doesn’t. Why? I believe that’s all about design.

54 scheng1

That’s interesting. I wonder what is the conversion rate for using a bloody-red button?

55 World Travel

Yes, this research is quite interesting. I know that psychology is quite important is shopping, advertising and other internet actions. If you know how to make people clicking more, this will be great

56 a web developer

When I go to the ShoeMoney System link I see the less converting green Add to Cart button? Are you still doing a/b testing with it?

57 Jason

Interesting study yeah it shows anything that is not real bland will get more clicks banners, sign up now buttons, whatever it may be!

58 adsense alternatives

Thanks. I’m gonna have to try this.

59 Eric Bridges

I really liked this post. Your formula will help articulate the importance of what clients see as the “small” things into real time results. There was a good post on contact forms and how the “Mad Libs” version of the contact form converted at a much better rate. Imagine if they had evaluated conversion with your formula! http://www.lukew.com/ff/entry.asp?1007

60 Farinha

This looks like powerful stuff. Will certainly consider it for my future projects.

61 multivariate testing and digital optimization insider

The benefits of a more holistic optimization and marketing approach can be extended beyond content used to attract leads and sales. There other types of search that can drive or benefit business including customer service related content, job listings and news content. Each has it�s own audience to consider and therefore, a different context for optimization.

62 Walker Upton

I just identified your blog, I bookmarked it and i ‘m reading the actual blogposts. I already enjoy it. Interesting issue in any event you actually look on this. I come as a result of this standpoint that notice remarks as akin of attentive.

63 Bari Bynum

Usefull points made on your blog, most I agree with. Remember seeing a similar blog which I will look to post. Definately will bookmark in any case I await your next thought provoking blog post

64 SEO Simon

This tool is great when you have many stakeholders who all think they know how best to design a website.
The proof is in the stats!

65 ACME Squares

I’m not a fan of multi-variate testing. Google chronically overused it, and went from web-design hero to zero, in just a few years.

66 wanda

so could you show me on how to really make money from home?because every time i try to apply for working @ home there’s always a scam and it makes me not want to do it.but then they want to charge me alot of money and im looking for work to me if someone looking for work why we have to pay to get a job what’s the point?when people look for work and get hired then they get paid for there work.that’s my point.why i have to pay if i want to work from home?i dont trust noone with my creditcard.when i try to apply for it but when i see they asking for creditcards i delete it quick.

67 Haluk Akin

Genetic algorithm is an overkill for this case.
Much simpler heuristics can do the job easily.
This is like using a sword to spread butter.

68 Jamie Hudson

Mind boggling number! Question: when does the split testing stop? Seriously tough to figure out, but you definitely put things into perspective on split testing. Appreciate it!

69 SportsNub

Surely great stuff, I would love to optimize like that if it increase conversation like 2% then it is like a bomb for me.

70 CoeurdAlenechiropractor

stats can lie. stats can be twisted into what u want them to be. i trust shoe!

71 Rea

i want to understand something. how does the algorithm know how to split the traffic? the whole point is to test for better conversion and such isn’t it?
how can you tell where do you the best conversions before you test it?

tnx

72 Everette

Great to know this.The color orange does seem to be more magnetic than green. Oh well , go figure.

Previous post:

Next post: