Cloud Computing opportunities in the development (build-test-deploy) space

You heard the word “cloud” everywhere, running applications on the cloud, scaling with the cloud,… but not so often from the development lifecycle perspective: code, commit, test, deploy to QA, release, etc but it brings fundamental changes to this aspect too.

The scenario

If you belong to, or manage, a group of developers, you are doing at least some sort of automated builds with continuous integration. You have continuous integration servers, building your code on specific schedules, not as often as you would like, when developers commit changes. The number of projects grow and you add more servers for the new projects, mixing and matching environments for different needs (linux, windows, os x,…)

The problem and the solution

The architecture we use for our Maestro 3 product is composed of one server that handles all the development lifecycle assets. Behind the scenes we use proven open source projects: Apache Continuum for distributed builds, Apache Archiva for repository management, Sonar for reporting, Selenium for multienvironment integration and load testing. And we add Morph mCloud private cloud solution, which is also based on open source projects such as Eucalyptus or Puppet.

We have multiple Continuum build agents doing builds, and multiple Selenium agents for webapp integration testing, as well as several application servers for continuous deployment and testing.

  • limited capacity
    • problem: your hardware is limited, and provision and setup of new servers requires a considerable amount of time
    • solution: assets are dynamic, you can spin off new virtual machines when you need them, and shuffle infrastructure around in a matter of minutes with just few clicks from the web interface. The hybrid cloud approach means you can start new servers in a public cloud, Amazon EC2, if you really need to
  • capacity utilization
    • problem: you need to setup less active projects in the same server as more active ones to make sure servers are not under/over-utilized
    • solution: infrastructure is shared across all projects. If a project needs it more often than another then it’s there to be used
  • scheduling conflicts
    • problem: at specific times, i.e. releases, you need to stop automatic jobs to ensure resources are available for those builds
    • solution: a smart queue management can differentiate between different build types (ie. continuous builds, release builds) and prioritize
  • location dependence
    • problem: you need to manage the infrastructure, knowing where each server is and what is it building
    • solution: a central view of all the development assets for easier management: build agents, test agents or application servers
  • continuous growing
    • problem: new projects are being added while you are trying to manage and optimize your current setup
    • solution: because infrastructure is shared adding new projects is just a matter of scaling wide the cloud, without assigning infrastructure to specific projects
  • complexity in process
    • problem: multiply that for the number of different stages in your promotion process: development environment, QA, Staging, Production
    • solution: you can keep independent networks in the cloud while sharing resources like virtual machine templates for your stack for instance
  • long time-to-market
    • problem: transition from development to QA to production is a pain point because releases and promotion is not automated
    • solution: compositions (workflows) allow to design and automate the steps from development to production, including manual approval
  • complexity in organization:
    • problem: in large organizations, multiply that for the number of separate entities, departments or groups that have their own separate structure
    • solution: enabling self provisioning you can assign quotas to developers or groups to start servers as they need them in a matter of minutes from prebuilt templates

Why a private cloud?

  • cost effectiveness: development infrastructure is running continuously. Global development teams make use of it 24×7
  • bandwidth usage: the traffic between your source control system and the cloud can be expensive, because it’s continuously getting the code for building
  • security restrictions: most companies don’t like their code being exported anywhere outside their firewall. Companies that need to comply with regulations (ie. PCI) have strong requirements on external networks
  • performance: in a private cloud you can optimize the hardware for your specific scenario, reducing the number of VMs needed for the same job compared to public cloud providers
  • heterogeneous environments: if you need to develop in different environments then there are chances that the public cloud service won’t be able to provide them

The new challenges

  • parallelism, you need to know the dependencies between components to know what needs to be built before what
  • stickyness, or how to take advantage of the state of the agents to start builds in the same ones if possible, ie. agents that built a project before can do a source control update instead of a checkout, or have the dependencies already in the filesystem
  • asset management, when you have an increasing number of services running, stoping and starting as needed, you need to know what’s running and where, not only at hardware level but at service level: build agents, test agents and deployment servers.

The new vision

  • you can improve continuous integration as developers checkin code because the barrier to add new infrastructure is minimal, given you have enough hardware in your cloud or if you use external cloud services, which means reduced time to find problems
  • developers have access to infrastructure they need to do their jobs, for instance start an exact copy of the production environment to fix an issue by using a cloud template that they can get up and running in minutes and tear down at the end, not incurring in high hardware costs
  • less friction and easier interaction with IT people as developers can self provision infrastructure, if necessary shuffling virtual machines that they no longer need for the ones they needed

By leveraging the cloud you can solve existing problems in your development lifecycle and at the same time you will be able to do things that you would not even consider because the technology made it complicated or impossible to do. Definitely something worth checking out for large development teams.

Maestro 3 is going to be released this week at InterOp New York (come over and say hi if you are around) but we are already demoing the development snapshots to clients and at conferences like JavaOne.

Maven, Amazon EC2 and SpringSource Cloud Foundry

You may have heard about the just announced SpringSource Cloud Foundry and how it is based on the CloudTools project, that includes a Maven plugin to deploy Java EE applications to Amazon EC2, starting the images as part of the build process.

Some time ago I started another Maven plugin, the Amazon EC2 Maven plugin, which allows you to start and stop EC2 AMIs as part of your build process. Unlike CloudTools, it’s a lower level plugin that can start any AMI, a very different goal.

My use case? starting Selenium Grid Remote Control images for different environments and browsers before the integration tests start, wait for the images to be online, run the integration tests, and shutdown the images. Check my previous Enterprise Build and Test in the Cloud entry for more details.

You could also have your AMIs with your webserver, db,… pre-installed, start it, deploy using the Maven Cargo plugin to any container of your choice, and shutdown the image at the end of the tests.

The plugin allows all the configuration options than the EC2 API does, because it’s based on the Typica EC2 library. Start any number of images, associate elastic IPs, choose availability zones,…

Hope you find it useful.

JavaOne slides: Enterprise Build and Test in the Cloud

I have uploaded the slides from my talk Enterprise Build and Test in the Cloud at JavaOne in San Francisco.

You can check also the code, and an introduction in previous posts

Enterprise build and Test in the Cloud with Selenium I
and
Enterprise build and Test in the Cloud with Selenium II.

Follow me on twitter

JavaOne talk: Enterprise Build and Test in the Cloud

I’ll be presenting Enterprise Build and Test in the Cloud at JavaOne in San Francisco, Wednesday June 3rd 11:05am Esplanade 301 and will be around the whole week.

You can check the slides from the previous talk at ApacheCON, the code, and an introduction in previous posts

Enterprise build and Test in the Cloud with Selenium I
and
Enterprise build and Test in the Cloud with Selenium II.

Follow me on twitter

Enterprise Build and Test in the Cloud code available

The code accompanying the slides Enterprise Build and Test in the Cloud is available at the appfuse-selenium github page.

Provides a Selenium test environment for Maven projects, Appfuse as an example. Allows to run Selenium tests as part of the Maven build, either in an specific container and browser or launching the tests in parallel in several browsers at the same time.

For more information check my slides on Enterprise Build and Test in the Cloud and the blog entries Enterprise Build and Test in the Cloud with Selenium I and Enterprise Build and Test in the Cloud with Selenium II.

By default it’s configured to launch 3 browsers in parallel, Internet Explorer, Firefox 2 and 3

Check src/test/resources/testng.xml for the configuration.

In the single browser option you could do

  • Testing in Jetty 6 and Firefox

    • mvn install

  • Testing in Internet Explorer

    • mvn install -Pjetty6x,iexplore

  • Testing with any browser

    • mvn install
      -Pjetty6x,otherbrowser -DbrowserPath=path/to/browser/executable

  • Start the server (no tests
    running, good for recording tests)

    • mvn package cargo:start

ApacheCON slides: “Enterprise Build and Test in the Cloud” and “Eclipse IAM, Maven integration for Eclipse”

Here you have the slides from my talks at ApacheCON

Enterprise Build and Test in the Cloud

Building and testing software can be a time and resource consuming task. Cloud computing / on demand services like Amazon EC2 allow a cost-effective way to scale applications, and applied to building and testing software can reduce the time needed to find and correct problems, meaning a reduction also in time and costs. Properly configuring your build tools (Maven, Ant,…), continuous integration servers (Continuum, Cruise Control,…), and testing tools (TestNG, Selenium,…) can allow you to run all the build/testing process in a cloud environment, simulating high load environments, distributing long running tests to reduce their execution time, using different environments for client or server applications,… and in the case of on-demand services like Amazon EC2, pay only for the time you use it.
In this presentation we will introduce a development process and architecture using popular open source tools for the build and test process such as Apache Maven or Ant for building, Apache Continuum as continuous integration server, TestNG and Selenium for testing, and how to configure them to achieve the best results and performance in several typical use cases (long running testing processes, different client platforms,…) by using he Amazon Elastic Computing Cloud EC2, and therefore reducing time and costs compared to other solutions.

Download PDF

Eclipse IAM, Maven integration for Eclipse

Eclipse IAM (Eclipse Integration for Apache Maven), formerly “Q for Eclipse”, is an Open Source project that integrates Apache Maven and the Eclipse IDE for faster, more agile, and more productive development. The plugin allows you to run Maven from the IDE, import existing Maven projects without intermediate steps, create new projects using Maven archetypes, synchronize dependency management, search artifact repositories for dependencies that are automatically downloaded, view a graph of dependencies and more! Join us to discover how to take advantage of all these features, as well as how they can help you to improve your development process.

Download PDF

Enterprise build and Test in the Cloud with Selenium II

In my previous entry I introduced Selenium and a setup for running automated builds in the browser.

How can you achieve the multi-browser multi-environment testing?
There is a subproject of Selenium called Selenium-Grid that works in a heterogeneous cluster of machines. Install a selenium hub in one server and start the selenium remote clients in the machines with the environments you want to test. For instance you can have a windows machine with IE 7 and Firefox 3, another one with IE 6, a linux machine with Firefox 2 and a OS X with Safari. There’s no limit in the number of machines that you can attach to the hub, and the hub redirects the tests to the machine that fulfills the environment you request in your tests.

At this point you can have a very nice integration testing environment, with the continuous integration server deploying your app every day, and running the Selenium tests in several machines with different operating systems and browsers.

Now the bad news, you may have noticed already that having a bunch of machines dedicated to QA just to run once a day can be quite expensive. Another issue is that Selenium tests, as they run in the browser, are not particularly fast.

But it’s a great use case for cloud computing. Using Amazon EC2 you can have machines with different environments (Linux, Windows, Solaris,…) and browsers (IE, Firefox,…) and pay just for the time they are running. Let’s say you need 5 combinations of OS/browser and your tests take 2 hours in each of them. If you run the tests once a day you could end paying

5 machines * 30 days * 2 hours * 0.10 $/hour (approx) = 30$ month, results in 2 hours

compare it with the cost of 5 machines, power, space, hardware maintenance,…

Not only that, but you could parallelize your tests, using Selenium with TestNG for instance, and get the results faster. Particularly, using EC2, if you start twice as many machines to get the results in one hour instead of two, you would pay

10 machines * 30 days * 1 hours * 0.10 $/hour (approx) = 30$ month, results in 1 hour

exactly the same!

And if you need to really really get those results fast, you are not constrained by your hardware

100 machines * 1 day * 1 hours (the minimum charge is one hour) * 0.10 $/hour (approx) = 10$ day, but with results in 6 minutes

assuming that your tests are split and can be run properly in parallel.

You can see a demo of how does it work in our Maestro S+ page. Obviously you can set this environment up yourself, but if you need help we can assist you setting the environment up, creating an outsourced on-demand QA team, or subscribe to a fully hosted service where we will handle all your continuous integration and testing needs using open source, Apache licensed, software, with a continuous integration server, Apache Continuum, artifact repository, Apache Archiva, and a list of browsers and environments to test on that we’ll maintain for you, and paying just for the time you need them.

Enterprise build and Test in the Cloud with Selenium I

I introduced Amazon Web Services and Cloud computing in previous posts (Amazon Web Services: an introduction, habemus Windows, tools), now let’s get full advantage of them in a practical application.

For those not familiar with Selenium, it’s a testing library that allows you to write integration tests to run in the browser, and when I say in the browser I mean it, you can tests how web applications behave when using different browsers: Internet Explorer, Firefox, Safari,… and we all know how painful cross-browser compatibility can be.

With Selenium you can write the tests in different languages: Java, Ruby, Groovy,… but, and this is a huge plus, you can also record them in Firefox using the Selenium IDE AddOn, and then run them with playback or export them to any of the languages supported.

Now follow me to this scenario. You have a team of developers writing a webapp. You can take anybody with a minimum experience (an user!) and tell them to use the site while SeleniumIDE records what they are doing. Then export the recording and add it to your integration builds to make sure it doesn’t break in future developments. Imagine also what you could achieve with experienced QA people, that don’t need to be developers at all nor know the internals of your app, and that can apply QA practices, trying to make your webapp break in any possible way. Sounds interesting? Keep reading

Let’s say that the site needs to be supported in different browsers too, Internet Explorer, Firefox and Safari. With Selenium you can run the same tests previously recorded against any browser.

At this point you have a development practice where developers develop, and a QA team that can be just test experts, not developers, either inside or outside your company or department, recording and running tests using only the requirements. Not a bad thing to have, but let’s make it better using automation.

Let’s assume you already have a working automated build and continuous integration, I know for many people is a big assumption, but let’s have hope ;), and assume you are using Maven and Continuum, not necessary, but that’s what I’m more familiar with. It is pretty easy to add Selenium testing to run as part of the build. With the Selenium Maven plugin you can start the Selenium server, launch the tests that will run the browser and execute the in-browser testing, and stop Selenium server at the end. When you have that running on Maven is straightforward to get it working in Continuum. For the gory technical details you can read my previous post Functional testing with Maven, Cargo and Selenium.

Will continue in my next entry

Amazon Web Services: tools

Second entry in the series: Amazon Web Services: an introduction

Amazon provides some tools to interact with its Web Services, and many others exist provided by 3rd parties. Here is my selection

S3 tools

JetS3t

JetS3t is a free, open-source Java toolkit and application suite for the Amazon Simple Storage Service (Amazon S3). The toolkit provides Java programmers with a powerful yet simple API for interacting with S3 and managing data stored there, while the applications allow anyone to easily manage and interact with their S3 accounts.

JetS3t is composed by a java library to access S3, a command line tool to synchronize local folders to S3 (s3sync) and a GUI to upload/download files (cockpit)

EC2 tools

ElasticFox

A graphical interface to Amazon EC2 web service as a Firefox addOn, provided by Amazon

You can do pretty much everything with it, register instances, start/stop them, manage access keys, security groups, elastic ips,
EBS volumes,…

Amazon Management Console

A web based interface to Amazon EC2 launched recently by Amazon. Same functionality as ElasticFox

Amazon EC2 API tools

The command line version, you won’t need it if using ElasticFox, unless the command line is your thing

Amazon EC2 AMI tools

The Amazon EC2 AMI Tools are command-line utilities to help bundle an Amazon Machine Image (AMI), create an AMI from an existing machine or installed volume, and upload a bundled AMI to Amazon S3.

Even using ElasticFox you’ll need it in your running instances to create new images from them (only for *nix, Windows images are bundled from Elasticfox or other GUI)

EC2 services

Morph

Morph AppSpace allows you to run webapps with no effort. You provide them with a war (for Java) or your Rails, Grails or PHP app and they’ll run it for you in the cloud, with a DB (postgres or mysql) with warm backups, a webserver, continuosly monitored and a two machine cluster for load balancing and high availability, all from 1$ a day. There’s a development version to try it for free.

How it works for Java apps? You just need to access your database connection pool using JNDI. They will provide you the resource name to use.

Morph AppCloud is another product that adds more flexibility, providing a web based cluster editor where you can design your cloud just by dragging and connecting the modules provided (ie. Jetty web server, mysql master, mysql slave, load balancer,…), then just click to run it.

Disclaimer: I know the guys behind Morph and have done some work for them

Rightscale

It’s a lower level service where you can interact with all the EC2 operations. You can create images, start/stop them, health montirs,… all through their web interface. You have a developer edition to try it for free

CohesiveFT

It’s more of a provisioning system where you select components and create an image (Elastic Server) to run it in EC2, VMware, and other virtualization frameworks, and with a management web interface to start/stop the components, monitor the services running,…

They have a list of components and templates, mostly open source projects, and you can add your own too.

Amazon Web Services: habemus Windows

This is going to be a little bit out of order with my other posts about Amazon AWS, but it’s worth doing it. Amazon has launched finally the ability to run Windows images in EC2. Microsoft Windows Server 2003 actually, from 0.125$ per hour, compared to the 0.10$ for the unix/linux instances.

The instances are accessible through Windows Remote Desktop or the open source multiplatform rdesktop client. The first time you launch an image the password is generated, but you can change it for new images you make based on that one.

A nice feature that will save a lot of time is the ability of creating new Windows AMIs from the ElasticFox interface with a couple of clicks, without needing to mess with software installation in the image (more on image creation in next posts) which is way easier than the process for linux images.

You can also run Windows Server with SQL Server Standard, but watch out for the 1.10$ per hour, which makes 792$ per month that you can compare to the 6000$ processor license fee.

Previous entry: Amazon Web Services: an introduction

Next: Amazon Web Services: tools