In my previous entry I introduced Selenium and a setup for running automated builds in the browser.

How can you achieve the multi-browser multi-environment testing?
There is a subproject of Selenium called Selenium-Grid that works in a heterogeneous cluster of machines. Install a selenium hub in one server and start the selenium remote clients in the machines with the environments you want to test. For instance you can have a windows machine with IE 7 and Firefox 3, another one with IE 6, a linux machine with Firefox 2 and a OS X with Safari. There’s no limit in the number of machines that you can attach to the hub, and the hub redirects the tests to the machine that fulfills the environment you request in your tests.

At this point you can have a very nice integration testing environment, with the continuous integration server deploying your app every day, and running the Selenium tests in several machines with different operating systems and browsers.

Now the bad news, you may have noticed already that having a bunch of machines dedicated to QA just to run once a day can be quite expensive. Another issue is that Selenium tests, as they run in the browser, are not particularly fast.

But it’s a great use case for cloud computing. Using Amazon EC2 you can have machines with different environments (Linux, Windows, Solaris,…) and browsers (IE, Firefox,…) and pay just for the time they are running. Let’s say you need 5 combinations of OS/browser and your tests take 2 hours in each of them. If you run the tests once a day you could end paying

5 machines * 30 days * 2 hours * 0.10 $/hour (approx) = 30$ month, results in 2 hours

compare it with the cost of 5 machines, power, space, hardware maintenance,…

Not only that, but you could parallelize your tests, using Selenium with TestNG for instance, and get the results faster. Particularly, using EC2, if you start twice as many machines to get the results in one hour instead of two, you would pay

10 machines * 30 days * 1 hours * 0.10 $/hour (approx) = 30$ month, results in 1 hour

exactly the same!

And if you need to really really get those results fast, you are not constrained by your hardware

100 machines * 1 day * 1 hours (the minimum charge is one hour) * 0.10 $/hour (approx) = 10$ day, but with results in 6 minutes

assuming that your tests are split and can be run properly in parallel.

You can see a demo of how does it work in our Maestro S+ page. Obviously you can set this environment up yourself, but if you need help we can assist you setting the environment up, creating an outsourced on-demand QA team, or subscribe to a fully hosted service where we will handle all your continuous integration and testing needs using open source, Apache licensed, software, with a continuous integration server, Apache Continuum, artifact repository, Apache Archiva, and a list of browsers and environments to test on that we’ll maintain for you, and paying just for the time you need them.