FOSDEM

The slides from my From Dev to DevOps talk at FOSDEM 2012 Brussels are up in Slideshare. The material is also in the Lanyrd page.

The conference was huge, I’ve heard that over 4000 people showed up this year, for this free (as in both software and beer) event, organized on several tracks. Not bad considering that the temperature was between -15 and -6C (5 and 23 Fahrenheit) and got a lot of snow.

I spoke on the main auditorium with capacity for 1400 people, there was plenty of space. Unfortunately (for me, but great for the other speakers and the organization) some other devrooms got filled quickly, and couldn’t get into theĀ Configuration and Systems Management Devroom, which should have been really good as nobody left as I was waiting outside between talks šŸ˜¦

A lot of care by the organization, including a note with a 24/7 phone number in case I got “lost/confused/arrested” (which makes me think it must have happened before), and a full commitment to promote belgian beer šŸ˜‰

And thanks to the people that attended my talk and engaged through twitter, got great feedback and hints for improvements on the future of development and infrastructure automation!

Grand Place 360 panorama

“From Dev to DevOps” at the Paris JUG next week, FOSDEM Brussels in February

The year starts with two speaking engagements, about development and DevOps

SoirĆ©e Devops @Ā Paris JUG

Next week, Tuesday January 10thĀ at theĀ I.S.E.P. Paris

An afternoon about DevOps, withĀ Henri GomezĀ also speaking, where I’ll be finally able to put faces to some long time online friends and collaborators on different open source projects.

FOSDEM

Sunday February 5th at theĀ ULB Campus Solbosch in Brussels.

the biggest free and non-commercial event organized by and for the community

Interesting conference concept with no registration necessary, around 4000 attendees in 4-6 tracks, completely free, including a Friday Belgium beer event, as they sayĀ don’t miss this great opportunity to taste some of the finestĀ beer in BelgiumĀ (and, hence, in the world). And they even include spouse/partner tours!

Are you going to any of them? say hi!

About: DevOps

This is the start of a (hopefully) series of posts about DevOps, based on my presentation From Dev to DevOps.

TheĀ Agile movement stablished a series of development practices quite common nowadays, or at least highly desired:

  • planning
  • iterative development
  • continuous integration
  • release soon, release often

But what happens after the development cycle? The process of building software or products does not stop there.Ā Even if you are an agile shop, you may hit a wall when trying to move between Development, QA and Operations.

DevOps is a door in that wall.

DevOps is the intersection between Development, QA and Operations. You can also think of it asĀ DevQAOps.

Old issues fixed, new issues come up

As some issues were solved by Agile methodologies, new issues arose. Agile development generates a higher number of releases (release early/release often) than waterfall methodologies, and that pushes the problem from development teams to operations, that have to deal with more frequent deliveries to production.

DevOps addresses

Fear of change

Once a product is working, changes are seen as a risk by business, developers, QA, operations,… Reducing the time to make (and revert!) changes increases the assurance to do more frequent changes.

Risky deployments

Automation, automation, automation! Not only the development needs to be automated. No more docs with instructions, manual steps, individual knowledge, “deployment master” or hacks. All the steps up to production need to be automated too, including deployments, to reduce risk of manual mistakes.

It works on my machine!

With the proliferation of virtualization, cloud, PaaS, stacks,… is absolutely necessary to be able to replicate environments for development, QA, pre-production,… Ensuring that proper level of testing is conducted as the product (software + environment) progresses through the lifecycle.

Siloisation

Teams split in Dev, QA, Ops,… cause walls to stand between them, and unnecessary friction, with each team pushing for its own benefit and requiring escalation of issues to account for the overall business good. Getting all the teams on the same table more often the friction is reduced.

Dev Change vs. Ops stability

Developers are usually measured by the number of features implemented, while operation teams are measured by the performance and stability on production. Two different goals that make both teams clash in manyĀ occasions. Bring issues early on and involve all the teams in the decisions.

Read more atĀ What Is This Devops Thing, Anyway?

DevOps is a extension of Agile

Does this sound familiar?

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

Exactly, that’s the Agile manifesto, which is still perfectly valid, with some clarifications, if we get into detail:

  • Our highest priority is to satisfy the customer through early and continuous delivery of valuable softwareĀ products.
  • Welcome changing requirements, even late in development any part of the DEV/QA/OPs cycle. Ā Agile processes harness change for the customer’s competitive advantage.
  • Deliver working software products frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
  • Business people and developers development/QA/Operations teams must work together daily throughout the project.
  • The most efficient and effective method of conveying information to and within a development DEV/QA/OPs team is face-to-face conversation.
  • Agile processes promote sustainable development. The sponsors, developers, QA and OPs teams, and users should be able to maintain a constant pace indefinitely.

What does that exactly mean

  • software/working software. The concept of product, meaning software running, in production
  • late in development. Late in the development, QA or operations phases, not just development
  • developers. Don’t consider only developers, QA and operations teams need to be involved too

The key part is the involvement of not just the development team but the QA and Operations teams.

Game theory

Game theory is defined as the method for analyzing calculated circumstances, such as in games, where a personā€™s success is based upon the choices of others. If you have watched A beautiful mind, you’ll remember the scene where he tries to pick up a girl at a bar.

What does it have to do with DevOps? The success of the Dev team greatly depends on actions done by the Ops team, good software deployed incorrectly will transmit the wrong impression. Likewise, the success of the Ops team definitely depends on the work of the Dev team, as poorly developed software will cause all sort of performance issues.

If all teams try to achieve their own benefit, it’s not necessarily a good thing for the overall business. Imagine the Ops team not pushing new features because they don’t want to risk the production stability. Or the Dev team pushing new features to production without proper testing because it’s a QA problem, not theirs, and are only measured on the amount of features pushed.

From Dev to DevOps slides from ApacheCON NA Vancouver 2011

ApacheCON NA 2011The slides from my From Dev to DevOps talk at ApacheCON NA 2011 Vancouver. Thanks to all the attendees for coming!

The audio recording seems that it’s going to be uploaded to FeatherCast.

UPDATE: the Lanyrd official page is up too

The code for the Maven-Puppet module is avalable in GitHub, and I’ll write some posts about it in the coming weeks.

The DevOps movement aims to improve communication between developers and operations teams to solve critical issues such as fear of change and risky deployments. But the same way that Agile development would likely fail without continuous integration tools, the DevOps principles need tools to make them real, and provide the automation required to actually be implemented. Most of the so called DevOps tools focus on the operations side, and there should be more than that, the automation must cover the full process, Dev to QA to Ops and be as automated and agile as possible. Tools in each part of the workflow have evolved in their own silos, and with the support of their own target teams. But a true DevOps mentality requires a seamless process from the start of development to the end in production deployments and maintenance, and for a process to be successful there must be tools that take the burden out of humans.

Apache Maven has arguably been the most successful tool for development, project standardization and automation introduced in the last years. On the operations side we have open source tools like Puppet or Chef that are becoming increasingly popular to automate infrastructure maintenance and server provisioning.

In this presentation we will introduce an end-to-end development-to-production process that will take advantage of Maven and Puppet, each of them at their strong points, and open source tools to automate the handover between them, automating continuous build and deployment, continuous delivery, from source code to any number of application servers managed with Puppet, running either in physical hardware or the cloud, handling new continuous integration builds and releases automatically through several stages and environments such as development, QA, and production.

Vancouver harbour panorama

ApacheCON NA Vancouver 2011

ApacheCON NA 2011I’m in Vancouver for ApacheCON NA 2011, where I’ll be speaking on Friday 14:30 about DevOps, From Dev to DevOps, my take on DevOps for people like me interested in DevOps and automation coming from the dev side, and expanding the dev lifecycle all the way to deployment to production. I have previously posted the slides from a another event and will post the updated ones after the talk.

If you are at the conference, come over and say hi!

The DevOps movement aims to improve communication between developers and operations teams to solve critical issues such as fear of change and risky deployments. But the same way that Agile development would likely fail without continuous integration tools, the DevOps principles need tools to make them real, and provide the automation required to actually be implemented. Most of the so called DevOps tools focus on the operations side, and there should be more than that, the automation must cover the full process, Dev to QA to Ops and be as automated and agile as possible. Tools in each part of the workflow have evolved in their own silos, and with the support of their own target teams. But a true DevOps mentality requires a seamless process from the start of development to the end in production deployments and maintenance, and for a process to be successful there must be tools that take the burden out of humans.

Apache Maven has arguably been the most successful tool for development, project standardization and automation introduced in the last years. On the operations side we have open source tools like Puppet or Chef that are becoming increasingly popular to automate infrastructure maintenance and server provisioning.

In this presentation we will introduce an end-to-end development-to-production process that will take advantage of Maven and Puppet, each of them at their strong points, and open source tools to automate the handover between them, automating continuous build and deployment, continuous delivery, from source code to any number of application servers managed with Puppet, running either in physical hardware or the cloud, handling new continuous integration builds and releases automatically through several stages and environments such as development, QA, and production.

Headless VirtualBox with Windows 7 guest

Virtualbox logoContinuing my VirtualBox series, let’s talk about headless VirtualBox.

You can run any VBox machine on headless mode, without the GUI. This is particularly interesting for *nix servers where you don’t want to install the X components, but it’s also interesting for Windows boxes that you can use as servers, in order to save some resources (memory, cpu,…) while it’s running.

Use case

I have a Windows 7 box running VPN software (that only runs in Windows), but I don’t want to use the Windows GUI and keep switching from the Mac to the Windows UI do do some things.

Solution

Configuring VirtualBox

Set an IP for your Windows guest, for easier configuration. You could use port forwarding too, but that’d be more cumbersome.

Install VirtualBox extension pack to enable remote display (VRDP support). In the host install a RDP viewer, for instance, for OS X, Microsoft Remote Desktop client.

Configure the guest vbox, enabling in settings Display -> Remote Display and set authentication method to null for no authentication. There are other options for authentication, if you want to check.

Manage the VM headless with VBoxHeadless and VBoxManage commands


# start the VM

VBoxHeadless -s "Windows 7"

# save VM state

VBoxManage controlvm "Windows 7" savestate

Connect to the running VM with your RDP client as you would do with any Windows box.

Web browsing from the host

Install a proxy server on Windows, FreeProxy for instance, but there should be better options out there.

Use FoxyProxy on Firefox on the host, configuring all the remote VPN urls to go through the Windows server. Then you can connect from your host’s Firefox to any url in the VPN.

SSH and other ports from the host

On the Windows guest, connect with Putty to a SSH server on the VPN side, and configure port forwarding. Then you can connect to your VM ip on the ports you forward and end up in the remote server. You could set a server name on /etc/hosts with the VM ip so it’s easier to remember.

With this setup I can use my Mac Firefox, iTerm and other tools without needing to use the Windows interface.

From Dev to DevOps slides from Agile Spain

Conferencia Agile Spain 2011Updated slides from my “From Dev to DevOps” presentation at the Agile Spain conference in Castellon on October 20th. Thanks to all the attendees for the questions and feedback!

Just some updates on Vagrant, VeeWee, Geppetto, and Puppet-Maven. Next stop, ApacheCON Vancouver!

UPDATE The video is also available (in Spanish) at the UJI web server as Flash and WMV.

From Dev to DevOps slides from Apache Barcamp Spain

Here are the slides from my “From Dev to DevOps” presentation at the Apache Barcamp Spain in Seville on October 8th. Not all that useful without the talking and hand waving šŸ™‚

I’ll be presenting it too at the Agile Spain conference on Thursday, with new slides, and adding some more info on Vagrant, VeeWee, Geppetto, and Puppet-Maven, just ten days after, things evolve really fast! Then, on to present at ApacheCON in Vancouver.

I’ll hopefully find the time to publish here at some point, in the meantime, there’s a good summary about the tools, Setup your devops playground with Puppet, Vagrant & co by Arnaud Heritier.

Migrating from VMWare Fusion to VirtualBox: networking

Virtualbox logoIn my previous post on migrating Windows 7 to VirtualBox there were a couple things missing on the networking configuration to make VirtualBox work the same way VMWare does by default.

In VMWare you can just connect to the ip of the guest, the host is by default configured with new virtual network adapters that will route the packages to the guest.

In VirtualBox however by default guests are configured with NAT to allow internet access from the guest through the host, but doesn’t allow connecting from the host to the guest directly, only assigning each port in the host that you want redirected to the guest (which is a PITA).

There are two options to achieve the same result that you had in VMWare:

  1. Change the Network Adapter from NAT to Bridged Adapter. The drawback is that it needs to bridge to a specific hardware adapter (ethernet or airport), so it won’t work if you switch connections.
  2. Add a new Host-Only Network Adapter. First, under VirtualBox global preferences – Network, add a Host-Only network. That will create a virtual interfaceĀ vboxnet0 in your host machine, and you can customize the ip ranges and DHCP server. Then in the VM settings, under Network, enable Adapter 2 attached to Host-Only adapter and choose the virtual adapter, vboxnet0.

I chose option 2 and works as expected, I just need to connect to 192.168.56.101 (the default assigned ip for the first VM) to get to the host.

Issues

The Windows 7 guest gets the ip from DHCP but for some reason it does not get the default gateway. Windows won’t let you assign the network to a Home or Work group without a default gateway, and therefore Windows Firewall will block incoming connections. The default gateway needs to be added by hand to the tcp/ip network configuration by hand (ie. 192.168.56.100) and then you can assign the network to Home/Work and the firewall will allow traffic.

Migrating Windows 7 from VMWare Fusion to VirtualBox

Virtualbox logoI recently had to start using Windows to connect to a client’s VPN network. Their VPN solution only works (or is supported) on Windows, plus the need to test a few things on Internet Explorer šŸ˜¦

Previously I had used Amazon EC2 whenever I needed to use windows, to avoid license costs, 20GB of my drive wasted and the CPU/RAM overhead, but this time there was no way out, didn’t seem cool to store the VPN credentials on a public cloud instance, although it’s probably as safe. So I bought Windows 7 online and downloaded the 3GB iso image and VMWare Fusion for OS X which happens to have a 30-day trial. Got Win 7 working there with no problems, which is not what I can say about the VPN setup.

Then I got good recommendations to try VirtualBox, which is both open source and free as in beer, and was glad to see there are ways to easily move your VMWare images to VirtualBox.

Step by step instructions

To move the Windows 7 image to VirtualBox 4.1 just needed to

  1. Uninstall the VMWare tools and shutdown windows
  2. Copy the disk files from VMWare image to a new folder
    1. In Documents/Virtual Machines right click on the image, Show package contents
    2. Copy all *.vmdk files to a new folder
  3. Create a new VirtualBox machine with the same characteristics
    1. Make sure you choose Windows 7 64 bits if that’s what you used
    2. On Virtual Hard Disk, choose the main vmdk file you copied in previous step (although you’ll need to change the default storage config later)
    3. Customize created VM settings:
      1. General: Windows 7 64 bits
      2. System/Motherboard: set the same amount of base memory as the VMWare one
      3. System/Motherboard: Enable IO APIC
      4. Storage: By default VirtualBox adds it to SATA controller but you have to remove the SATA controller, and use IDE PIIX4 without host I/O cache, attaching vmdk to primary master IDE, leave CD/DVD drive
  4. Boot the VirtualBox VM and follow any prompt to restart Windows to install new devices
  5. On the VirtualBox Devices menu, click on install guest additions

Troubleshooting

Several things went wrong before I got it working, so just in case you have the same problems

  • Stuck in “Windows is loading files” black screen, rebooting continuously: make sure you have selected 64 bits Windows and enabled IO APIC
  • Blue screen of death: Remove SATA, SCSI and any other non-IDE controllers from Storage, attach vmdk to IDE and make sure PIIX4 is selected
  • Windows needs to be repaired / Cannot repair windows: same as previous one, make sure disk is attached to IDE