Eclipse IAM WTP support, now EARs too

I recently had some time to spend in Eclipse IAM, working on improving the WTP support.

Version 0.11.0 already had good support for WAR projects, including war overlays (which was a bit tricky to implement in Eclipse). Now the last builds of the coming 0.12.0 version have EAR support.

You can import your Maven EAR projects and Eclipse will recognize the Maven-generated application.xml and configure automatically the dependencies to the other WAR projects opened in the workspace, with no extra configuration from you. And from the usual WTP "Run in Server" wizard you can run the EAR project and all associated WAR files in your favorite application server.

You can install the development builds of 0.12 from http://q4e.googlecode.com/svn/trunk/updatesite-dev/ until it’s released, and check the installation instructions for requirements or if you have issues. For help and feedback, we have a newsgroup at Eclipse.

Eclipse IAM 0.11.0, Archiva 1.3, Continuum 1.3.5

This is definitely release week! After Archiva 1.3 and Continuum 1.3.5 beta, I’ve just pushed the new release of Eclipse IAM 0.11.0:

This new version includes most notably

P2 Update site is published at http://q4e.googlecode.com/svn/trunk/updatesite-iam/

Ganymede users (Eclipse 3.4) should make sure they have added all the update sites listed in the installation instructions. If P2 complains about missing dependencies, check the update sites again.

Adopters of the latest and greatest Eclipse Galileo can install from the update site as usual.

If upgrading from Q4E 0.8.1 or earlier, some extra steps must be followed

The list of changes is available on the eclipse wiki.

Note that this is not an official Eclipse IAM release to allow our users to enjoy the progress made until we complete the move to the foundation and clear all the IP issues involving the maven embedder.

Continuum-ruby

continuum-ruby is a Ruby library to interact with Apache Continuum, using the XML-RPC interface and enabling access to the working copy directories. continuum-ruby is now available in the Continuum Sandbox.

More info on the Continuum XML-RPC interface:

Example

continuum = Continuum::Continuum.new("my.continuum.host", 8080, "admin", "password", "/continuum"

# xml-rpc interface
xml_rpc = Continuum::XmlRpc.new(continuum)
ok, result = xml_rpc.build_project(1)
error = Continuum.parse_error(result) if !ok

# getting working copy files
working_copy = Continuum::WorkingCopy.new(continuum)
test_results = working_copy.get(1, "target/surefire-reports", "emailable-report.html")
files = working_copy.dir(1, "target")
files.each do |file|
file_content = working_copy.get(1, "target", file)
end

Maven, Amazon EC2 and SpringSource Cloud Foundry

You may have heard about the just announced SpringSource Cloud Foundry and how it is based on the CloudTools project, that includes a Maven plugin to deploy Java EE applications to Amazon EC2, starting the images as part of the build process.

Some time ago I started another Maven plugin, the Amazon EC2 Maven plugin, which allows you to start and stop EC2 AMIs as part of your build process. Unlike CloudTools, it’s a lower level plugin that can start any AMI, a very different goal.

My use case? starting Selenium Grid Remote Control images for different environments and browsers before the integration tests start, wait for the images to be online, run the integration tests, and shutdown the images. Check my previous Enterprise Build and Test in the Cloud entry for more details.

You could also have your AMIs with your webserver, db,… pre-installed, start it, deploy using the Maven Cargo plugin to any container of your choice, and shutdown the image at the end of the tests.

The plugin allows all the configuration options than the EC2 API does, because it’s based on the Typica EC2 library. Start any number of images, associate elastic IPs, choose availability zones,…

Hope you find it useful.

JavaOne slides: Enterprise Build and Test in the Cloud

I have uploaded the slides from my talk Enterprise Build and Test in the Cloud at JavaOne in San Francisco.

You can check also the code, and an introduction in previous posts

Enterprise build and Test in the Cloud with Selenium I
and
Enterprise build and Test in the Cloud with Selenium II.

Follow me on twitter

JavaOne talk: Enterprise Build and Test in the Cloud

I’ll be presenting Enterprise Build and Test in the Cloud at JavaOne in San Francisco, Wednesday June 3rd 11:05am Esplanade 301 and will be around the whole week.

You can check the slides from the previous talk at ApacheCON, the code, and an introduction in previous posts

Enterprise build and Test in the Cloud with Selenium I
and
Enterprise build and Test in the Cloud with Selenium II.

Follow me on twitter

Enterprise Build and Test in the Cloud code available

The code accompanying the slides Enterprise Build and Test in the Cloud is available at the appfuse-selenium github page.

Provides a Selenium test environment for Maven projects, Appfuse as an example. Allows to run Selenium tests as part of the Maven build, either in an specific container and browser or launching the tests in parallel in several browsers at the same time.

For more information check my slides on Enterprise Build and Test in the Cloud and the blog entries Enterprise Build and Test in the Cloud with Selenium I and Enterprise Build and Test in the Cloud with Selenium II.

By default it’s configured to launch 3 browsers in parallel, Internet Explorer, Firefox 2 and 3

Check src/test/resources/testng.xml for the configuration.

In the single browser option you could do

  • Testing in Jetty 6 and Firefox

    • mvn install

  • Testing in Internet Explorer

    • mvn install -Pjetty6x,iexplore

  • Testing with any browser

    • mvn install
      -Pjetty6x,otherbrowser -DbrowserPath=path/to/browser/executable

  • Start the server (no tests
    running, good for recording tests)

    • mvn package cargo:start

ApacheCON slides: “Enterprise Build and Test in the Cloud” and “Eclipse IAM, Maven integration for Eclipse”

Here you have the slides from my talks at ApacheCON

Enterprise Build and Test in the Cloud

Building and testing software can be a time and resource consuming task. Cloud computing / on demand services like Amazon EC2 allow a cost-effective way to scale applications, and applied to building and testing software can reduce the time needed to find and correct problems, meaning a reduction also in time and costs. Properly configuring your build tools (Maven, Ant,…), continuous integration servers (Continuum, Cruise Control,…), and testing tools (TestNG, Selenium,…) can allow you to run all the build/testing process in a cloud environment, simulating high load environments, distributing long running tests to reduce their execution time, using different environments for client or server applications,… and in the case of on-demand services like Amazon EC2, pay only for the time you use it.
In this presentation we will introduce a development process and architecture using popular open source tools for the build and test process such as Apache Maven or Ant for building, Apache Continuum as continuous integration server, TestNG and Selenium for testing, and how to configure them to achieve the best results and performance in several typical use cases (long running testing processes, different client platforms,…) by using he Amazon Elastic Computing Cloud EC2, and therefore reducing time and costs compared to other solutions.

Download PDF

Eclipse IAM, Maven integration for Eclipse

Eclipse IAM (Eclipse Integration for Apache Maven), formerly “Q for Eclipse”, is an Open Source project that integrates Apache Maven and the Eclipse IDE for faster, more agile, and more productive development. The plugin allows you to run Maven from the IDE, import existing Maven projects without intermediate steps, create new projects using Maven archetypes, synchronize dependency management, search artifact repositories for dependencies that are automatically downloaded, view a graph of dependencies and more! Join us to discover how to take advantage of all these features, as well as how they can help you to improve your development process.

Download PDF

Using Lava Lamps for Continuous Integration build status notifications

Taking the idea of using Lava Lamps as notification tools for your Continuous Integration status, green lamp for success, red lamp for failure, the so called eXtreme Feedback Devices, and using as a guide the instructions in Pragmatic Automation, I have made some improvements to use remote notification, meaning that lamps don’t need to be connected to the build server, and a more automated process, that will turn off the lamps out of business hours.

The code is in a new project, Continuous Lava, in case it’s useful for somebody else. There are two parts, one for Apache Continuum and one for CruiseControl.

Actually, it is not necessary to use lava lamps but you can turn on and off any device you plug to the X10 power adapters. Imagine a loud siren, or my favorite, electric discharges to developer’s chairs!!! (we’ll leave this as an idea for future projects)

UPDATE: fixed the link to Pragmatic Automation

Functional testing with Maven, Cargo and Selenium

Setting up automated functional integration tests is not too hard if you have the right tools. It can take you a bit of time to setup but in the long run you’ll benefit from reduced QA times, reduced risks, a more confident development team, the ability to do safe refactorings, and many more advantages.

I’m going to explain how Maven, Selenium, Cargo and JBoss 4.2 can be setup to run automatically in a continuous integration server such as Continuum customizing the server configuration as needed and deploying any webapp automatically. Every time the webapp is changed the CI server will execute the tests against the latest version ensuring you are always in a safe state.

The biggest difference with other tutorials I’ve found is that most of them cover just Jetty and are not updated to the latest versions of libraries and tools, so here it is my contribution.

Architecture

  • A new project is setup with dependencies to the war project to be tested. Also required a dependency to selenium java client.
  • Cargo will download and install the application server (JBoss)
  • We will copy any required configuration and libraries (ie. jdbc driver)
  • Cargo will start the application server
  • The Selenium server is started
  • Surefire executes the junit tests that interact with the selenium server and test the running app
  • Cargo will stop the app server

We use profiles to enable different combination of browser/application server. By default cargo uses jetty.

Config Profiles
JBoss 4.2 and Firefox (default) -Pjboss42x,firefox
JBoss 4.2 and Internet Explorer -Pjboss42x,iexplore
Jetty and Firefox -Pfirefox
Jetty and Internet Explorer -Piexplore

The POM

Dependencies

<dependencies>
    <dependency>
      <groupId>com.acme</groupId>
      <artifactId>mywebapp</artifactId>
      <version>${project.version}</version>
      <type>war</type>
    </dependency>
    <!-- the jdbc driver we need to copy to the appserver -->
    <dependency>
      <groupId>mysql</groupId>
      <artifactId>mysql-connector-java</artifactId>
    </dependency>
    <dependency>
      <groupId>org.openqa.selenium.client-drivers</groupId>
      <artifactId>selenium-java-client-driver</artifactId>
      <version>1.0-SNAPSHOT</version> <!-- required for firefox 3 else use 1.0-beta-1 -->
      <scope>test</scope>
    </dependency>
  </dependencies>

Properties used in several places

Ports, where to uncompress the application server,…

<properties>
    <cargo.install.directory>${project.build.directory}/installs</cargo.install.directory>
    <selenium.port>14444</selenium.port>
    <servlet.port>18880</servlet.port>
    <selenium.background>true</selenium.background>
  </properties>

Plugin configuration

JDBC driver

Copy mysql jdbc driver to the app server lib folder

<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-dependency-plugin</artifactId>
        <executions>
          <execution>
            <id>copy-jdbc-lib</id>
            <phase>generate-resources</phase>
            <goals>
              <goal>copy-dependencies</goal>
            </goals>
            <configuration>
              <includeGroupIds>mysql</includeGroupIds>
              <outputDirectory>${lib.target}</outputDirectory>
            </configuration>
          </execution>
        </executions>
      </plugin>

Cargo

Install the application server in an early phase so we can customize it with our configuration files (see profiles). Then start before integration tests and stop afterwards. Parameters are used so different profiles can use different application servers.

      <plugin>
        <groupId>org.codehaus.cargo</groupId>
        <artifactId>cargo-maven2-plugin</artifactId>
        <executions>
          <execution>
            <id>install</id>
            <phase>generate-resources</phase>
            <goals>
              <goal>install</goal>
            </goals>
          </execution>
          <execution>
            <id>start-container</id>
            <phase>pre-integration-test</phase>
            <goals>
              <goal>start</goal>
            </goals>
            <configuration>
              <wait>false</wait>
            </configuration>
          </execution>
          <execution>
            <id>stop-container</id>
            <phase>post-integration-test</phase>
            <goals>
              <goal>stop</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <container>
            <containerId>${container.name}</containerId>
            <zipUrlInstaller>
              <url>${container.url}</url>
              <installDir>${cargo.install.directory}/${container.name}</installDir>
            </zipUrlInstaller>
            <log>${project.build.directory}/logs/${container.name}.log</log>
            <output>${project.build.directory}/logs/${container.name}.out</output>
            <timeout>600000</timeout>
          </container>
          <configuration>
            <!--
            <home>${project.build.directory}/${container.name}conf</home>
            <type>existing</type>
            -->
            <properties>
              <cargo.servlet.port>${servlet.port}</cargo.servlet.port>
              <cargo.jboss.configuration>default</cargo.jboss.configuration>
              <cargo.rmi.port>1099</cargo.rmi.port>
            </properties>

            <deployables>
              <!-- application to deploy -->
              <deployable>
                <groupId>com.acme</groupId>
                <artifactId>mywebapp</artifactId>
                <type>war</type>
                <properties>
                  <context>acontext</context>
                </properties>
              </deployable>
            </deployables>
          </configuration>
        </configuration>
      </plugin>

Selenium

Make surefire skip tests during test phase and run them in the integration-test phase. Pass some properties as system properties so they are accessible from the junit test case.

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
          <!-- Skip the normal tests, we'll run them in the integration-test phase -->
          <skip>true</skip>
          <systemProperties>
            <property>
              <name>browser</name>
              <value>${browser}</value>
            </property>
            <property>
              <name>servlet.port</name>
              <value>${servlet.port}</value>
            </property>
            <property>
              <name>selenium.port</name>
              <value>${selenium.port}</value>
            </property>
          </systemProperties>
        </configuration>
        <executions>
          <execution>
            <phase>integration-test</phase>
            <goals>
              <goal>test</goal>
            </goals>
            <configuration>
              <skip>false</skip>
            </configuration>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>selenium-maven-plugin</artifactId>
        <!-- to run headless in a Unix server with a virtual framebuffer X server Xvfb
             you need to call first the goal selenium:xvfb ie. "mvn clean selenium:xvfb install"
             see http://mojo.codehaus.org/selenium-maven-plugin/examples/headless-with-xvfb.html -->
        <executions>
          <execution>
            <id>start-selenium</id>
            <phase>pre-integration-test</phase>
            <goals>
              <goal>start-server</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <background>${selenium.background}</background>
          <port>${selenium.port}</port>
          <logOutput>true</logOutput>
        </configuration>
      </plugin>

Application server profiles

We can configure a different profile for each application server and set some specific application server configuration.

<profiles>
    <profile>
      <id>jboss42x</id>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>
      <properties>
        <container.name>jboss42x</container.name>
        <container.url>http://internap.dl.sourceforge.net/sourceforge/jboss/jboss-4.2.1.GA.zip</container.url>
        <jboss.version>4.2.1.GA</jboss.version>
        <jboss.conf.directory>${cargo.install.directory}/${container.name}/jboss-${jboss.version}/jboss-${jboss.version}/server/default</jboss.conf.directory>
        <lib.target>${jboss.conf.directory}/deploy/lib</lib.target>
        <war.target>${jboss.conf.directory}/deploy</war.target>
      </properties>

      <dependencies>
        <dependency>
          <groupId>org.jboss</groupId>
          <artifactId>jboss</artifactId>
          <version>${jboss.version}</version>
          <type>zip</type>
          <scope>test</scope>
        </dependency>
      </dependencies>
      <build>
        <plugins>
          <!-- copy to the application server directory any customized configuration files that we need -->
          <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-antrun-plugin</artifactId>
            <executions>
              <execution>
                <phase>process-resources</phase>
                <configuration>
                  <tasks>
                    <copy todir="${jboss.conf.directory}" overwrite="true">
                      <fileset dir="${basedir}/src/test/${container.name}"/>
                    </copy>
                  </tasks>
                </configuration>
                <goals>
                  <goal>run</goal>
                </goals>
              </execution>
            </executions>
          </plugin>
        </plugins>
      </build>
    </profile>

Browser profiles

As with the application servers we have a profile for each browser

<profile>
      <id>firefox</id>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>
      <properties>
        <browser>*firefox</browser>
      </properties>
    </profile>
    <profile>
      <id>iexplore</id>
      <properties>
        <browser>*iexplore</browser>
      </properties>
    </profile>
    <profile>
      <id>otherbrowser</id>
      <properties>
        <browser>*custom ${browserPath}</browser>
      </properties>
    </profile>

Enabling testing during development

Make selenium not to run in the background so we can execute tests from the IDE

    <profile>
      <id>dev</id>
      <properties>
        <selenium.background>false</selenium.background>
      </properties>
    </profile>

Repositories

Required for Selenium dependencies

<repositories>
    <repository>
      <id>openqa.org</id>
      <name>OpenQA Repository</name>
      <url>http://archiva.openqa.org/repository/releases</url>
      <snapshots>
        <enabled>false</enabled>
      </snapshots>
      <releases>
        <enabled>true</enabled>
      </releases>
    </repository>
    <!-- for selenium 1.0-SNAPSHOT -->
    <repository>
      <id>snapshots.openqa.org</id>
      <name>OpenQA Sanpshots Repository</name>
      <url>http://archiva.openqa.org/repository/snapshots</url>
      <snapshots>
        <enabled>true</enabled>
      </snapshots>
      <releases>
        <enabled>false</enabled>
      </releases>
    </repository>
  </repositories>

Running in the build server

In an Unix server without X running you can still run Selenium tests using Xvfb (virtual framebuffer X server) by calling selenium:xvfb provided it’s properly configured.

Also you can pass the path to the browser binary if not in the PATH

mvn clean selenium:xvfb install -Dbrowser="*firefox /usr/lib64/firefox-1.5.0.12/firefox-bin"

The JUnit test

public class SeleniumHelloWorldTest
    extends TestCase
{
    private DefaultSelenium selenium;

    private String baseUrl;

    @Override
    public void setUp()
        throws Exception
    {
        super.setUp();
        String port = System.getProperty( "servlet.port" );
        baseUrl = "http://localhost:" + port;
        selenium = createSeleniumClient( baseUrl );
        selenium.start();
    }

    @Override
    public void tearDown()
        throws Exception
    {
        selenium.stop();
        super.tearDown();
    }

    protected DefaultSelenium createSeleniumClient( String url )
        throws Exception
    {
        String browser = System.getProperty( "browser" );
        String port = System.getProperty( "selenium.port" );
        return new DefaultSelenium( "localhost", Integer.parseInt( port ), browser, url );
    }

    public void testHelloWorld()
        throws Exception
    {
        selenium.open( baseUrl + "/mycontext/" );
        assertTrue( selenium.isTextPresent( "acme" ) );
    }
}

Debugging and troubleshooting (update)

You can check JBoss logs in target/logs/jboss42x.out and Selenium server logs in target/selenium/server.log

References

Other wiki entries and blogs