Tag Archives: devops

From Dev to DevOps slides from ApacheCON NA Vancouver 2011

Posted on by Carlos Sanchez
1

ApacheCON NA 2011The slides from my From Dev to DevOps talk at ApacheCON NA 2011 Vancouver. Thanks to all the attendees for coming!

The audio recording seems that it’s going to be uploaded to FeatherCast.

UPDATE: the Lanyrd official page is up too

The code for the Maven-Puppet module is avalable in GitHub, and I’ll write some posts about it in the coming weeks.

The DevOps movement aims to improve communication between developers and operations teams to solve critical issues such as fear of change and risky deployments. But the same way that Agile development would likely fail without continuous integration tools, the DevOps principles need tools to make them real, and provide the automation required to actually be implemented. Most of the so called DevOps tools focus on the operations side, and there should be more than that, the automation must cover the full process, Dev to QA to Ops and be as automated and agile as possible. Tools in each part of the workflow have evolved in their own silos, and with the support of their own target teams. But a true DevOps mentality requires a seamless process from the start of development to the end in production deployments and maintenance, and for a process to be successful there must be tools that take the burden out of humans.

Apache Maven has arguably been the most successful tool for development, project standardization and automation introduced in the last years. On the operations side we have open source tools like Puppet or Chef that are becoming increasingly popular to automate infrastructure maintenance and server provisioning.

In this presentation we will introduce an end-to-end development-to-production process that will take advantage of Maven and Puppet, each of them at their strong points, and open source tools to automate the handover between them, automating continuous build and deployment, continuous delivery, from source code to any number of application servers managed with Puppet, running either in physical hardware or the cloud, handling new continuous integration builds and releases automatically through several stages and environments such as development, QA, and production.

Vancouver harbour panorama

ApacheCON NA Vancouver 2011

Posted on by Carlos Sanchez
Reply

ApacheCON NA 2011I’m in Vancouver for ApacheCON NA 2011, where I’ll be speaking on Friday 14:30 about DevOps, From Dev to DevOps, my take on DevOps for people like me interested in DevOps and automation coming from the dev side, and expanding the dev lifecycle all the way to deployment to production. I have previously posted the slides from a another event and will post the updated ones after the talk.

If you are at the conference, come over and say hi!

The DevOps movement aims to improve communication between developers and operations teams to solve critical issues such as fear of change and risky deployments. But the same way that Agile development would likely fail without continuous integration tools, the DevOps principles need tools to make them real, and provide the automation required to actually be implemented. Most of the so called DevOps tools focus on the operations side, and there should be more than that, the automation must cover the full process, Dev to QA to Ops and be as automated and agile as possible. Tools in each part of the workflow have evolved in their own silos, and with the support of their own target teams. But a true DevOps mentality requires a seamless process from the start of development to the end in production deployments and maintenance, and for a process to be successful there must be tools that take the burden out of humans.

Apache Maven has arguably been the most successful tool for development, project standardization and automation introduced in the last years. On the operations side we have open source tools like Puppet or Chef that are becoming increasingly popular to automate infrastructure maintenance and server provisioning.

In this presentation we will introduce an end-to-end development-to-production process that will take advantage of Maven and Puppet, each of them at their strong points, and open source tools to automate the handover between them, automating continuous build and deployment, continuous delivery, from source code to any number of application servers managed with Puppet, running either in physical hardware or the cloud, handling new continuous integration builds and releases automatically through several stages and environments such as development, QA, and production.

From Dev to DevOps slides from Agile Spain

Posted on by Carlos Sanchez
4

Conferencia Agile Spain 2011Updated slides from my “From Dev to DevOps” presentation at the Agile Spain conference in Castellon on October 20th. Thanks to all the attendees for the questions and feedback!

Just some updates on Vagrant, VeeWee, Geppetto, and Puppet-Maven. Next stop, ApacheCON Vancouver!

UPDATE The video is also available (in Spanish) at the UJI web server as Flash and WMV.

From Dev to DevOps slides from Apache Barcamp Spain

Posted on by Carlos Sanchez
Reply

Here are the slides from my “From Dev to DevOps” presentation at the Apache Barcamp Spain in Seville on October 8th. Not all that useful without the talking and hand waving 🙂

I’ll be presenting it too at the Agile Spain conference on Thursday, with new slides, and adding some more info on Vagrant, VeeWee, Geppetto, and Puppet-Maven, just ten days after, things evolve really fast! Then, on to present at ApacheCON in Vancouver.

I’ll hopefully find the time to publish here at some point, in the meantime, there’s a good summary about the tools, Setup your devops playground with Puppet, Vagrant & co by Arnaud Heritier.

New challenges from DevOps: development cycle for your infrastructure

Posted on by Carlos Sanchez
1

One of the main ideas behind DevOps adoption is the concept of  “infrastructure as code”. Tools like Puppet or Chef allow you to programmatically define your infrastructure, the provisioning of your servers: what packages are installed, what is the content of files,…

If server provisioning is a key point in operations, then code management becomes key too once you start coding your servers. You need source control for your infrastructure, you need tags, versioning, dependencies between components,… You need development, testing, QA, release,… for your infrastructure!

Imagine a environment where you have some server stack running in production using Puppet, with a manifest that defines packages and files in that server, and many servers running the same configuration. That Puppet definition must be in source control.

Now a security fix or new version of package must be installed in all the servers, do you just want to change the manifest and push it out to all the running servers? doesn’t sound like a great idea, does it? Hey, we have been tuning development best practices over the years for use cases just like this one.

What you want to do is create a new branch where you can do that change, and test it in some server that is not in production, let’s call it development environment, original isn’t it?
The change works as expected, your app still works, great! now you can probably merge that branch of the Puppet manifests into trunk, with possibly other changes made by other people, that at some point you will want to test together, in a production-like environment, maybe with several servers in a cluster, load balancing, etc… and very importantly, with the next version of the application that is going to be deployed. You create a new tag and version to be able to identify it later and deploy to that environment, let’s call it QA or staging.

What all this cycle allows you to do is clearly define what is running in each environment, using versions, and easily find issues between deployments, using source control, being able to roll back to known working configuration if needed.

After all, if you deal with infrastructure as code you should use code development best practices, and you’ll get the same benefits.

Cloud Computing opportunities in the development (build-test-deploy) space

Posted on by Carlos Sanchez
Reply

You heard the word “cloud” everywhere, running applications on the cloud, scaling with the cloud,… but not so often from the development lifecycle perspective: code, commit, test, deploy to QA, release, etc but it brings fundamental changes to this aspect too.

The scenario

If you belong to, or manage, a group of developers, you are doing at least some sort of automated builds with continuous integration. You have continuous integration servers, building your code on specific schedules, not as often as you would like, when developers commit changes. The number of projects grow and you add more servers for the new projects, mixing and matching environments for different needs (linux, windows, os x,…)

The problem and the solution

The architecture we use for our Maestro 3 product is composed of one server that handles all the development lifecycle assets. Behind the scenes we use proven open source projects: Apache Continuum for distributed builds, Apache Archiva for repository management, Sonar for reporting, Selenium for multienvironment integration and load testing. And we add Morph mCloud private cloud solution, which is also based on open source projects such as Eucalyptus or Puppet.

We have multiple Continuum build agents doing builds, and multiple Selenium agents for webapp integration testing, as well as several application servers for continuous deployment and testing.

  • limited capacity
    • problem: your hardware is limited, and provision and setup of new servers requires a considerable amount of time
    • solution: assets are dynamic, you can spin off new virtual machines when you need them, and shuffle infrastructure around in a matter of minutes with just few clicks from the web interface. The hybrid cloud approach means you can start new servers in a public cloud, Amazon EC2, if you really need to
  • capacity utilization
    • problem: you need to setup less active projects in the same server as more active ones to make sure servers are not under/over-utilized
    • solution: infrastructure is shared across all projects. If a project needs it more often than another then it’s there to be used
  • scheduling conflicts
    • problem: at specific times, i.e. releases, you need to stop automatic jobs to ensure resources are available for those builds
    • solution: a smart queue management can differentiate between different build types (ie. continuous builds, release builds) and prioritize
  • location dependence
    • problem: you need to manage the infrastructure, knowing where each server is and what is it building
    • solution: a central view of all the development assets for easier management: build agents, test agents or application servers
  • continuous growing
    • problem: new projects are being added while you are trying to manage and optimize your current setup
    • solution: because infrastructure is shared adding new projects is just a matter of scaling wide the cloud, without assigning infrastructure to specific projects
  • complexity in process
    • problem: multiply that for the number of different stages in your promotion process: development environment, QA, Staging, Production
    • solution: you can keep independent networks in the cloud while sharing resources like virtual machine templates for your stack for instance
  • long time-to-market
    • problem: transition from development to QA to production is a pain point because releases and promotion is not automated
    • solution: compositions (workflows) allow to design and automate the steps from development to production, including manual approval
  • complexity in organization:
    • problem: in large organizations, multiply that for the number of separate entities, departments or groups that have their own separate structure
    • solution: enabling self provisioning you can assign quotas to developers or groups to start servers as they need them in a matter of minutes from prebuilt templates

Why a private cloud?

  • cost effectiveness: development infrastructure is running continuously. Global development teams make use of it 24×7
  • bandwidth usage: the traffic between your source control system and the cloud can be expensive, because it’s continuously getting the code for building
  • security restrictions: most companies don’t like their code being exported anywhere outside their firewall. Companies that need to comply with regulations (ie. PCI) have strong requirements on external networks
  • performance: in a private cloud you can optimize the hardware for your specific scenario, reducing the number of VMs needed for the same job compared to public cloud providers
  • heterogeneous environments: if you need to develop in different environments then there are chances that the public cloud service won’t be able to provide them

The new challenges

  • parallelism, you need to know the dependencies between components to know what needs to be built before what
  • stickyness, or how to take advantage of the state of the agents to start builds in the same ones if possible, ie. agents that built a project before can do a source control update instead of a checkout, or have the dependencies already in the filesystem
  • asset management, when you have an increasing number of services running, stoping and starting as needed, you need to know what’s running and where, not only at hardware level but at service level: build agents, test agents and deployment servers.

The new vision

  • you can improve continuous integration as developers checkin code because the barrier to add new infrastructure is minimal, given you have enough hardware in your cloud or if you use external cloud services, which means reduced time to find problems
  • developers have access to infrastructure they need to do their jobs, for instance start an exact copy of the production environment to fix an issue by using a cloud template that they can get up and running in minutes and tear down at the end, not incurring in high hardware costs
  • less friction and easier interaction with IT people as developers can self provision infrastructure, if necessary shuffling virtual machines that they no longer need for the ones they needed

By leveraging the cloud you can solve existing problems in your development lifecycle and at the same time you will be able to do things that you would not even consider because the technology made it complicated or impossible to do. Definitely something worth checking out for large development teams.

Maestro 3 is going to be released this week at InterOp New York (come over and say hi if you are around) but we are already demoing the development snapshots to clients and at conferences like JavaOne.