Google Cloud Next Recap

Posted on August 2, 2018 by Carlos Sanchez

Several interesting announcements from last week Google Next conference.

Knative, a new OSS project built by Google, Red Hat, IBM,… to build, deploy, and manage modern serverless workloads on Kubernetes. Built upon Istio, with 1.0 coming soon and managed Istio on GCP. It includes a build primitive to manage source to kubernetes flows, that can be used independently. Maybe it is the new standard to define sources and builds in Kubernetes. Read more from Mark Chmarny.

GKE on premise, a Google-configured version of Kubernetes with multi-cluster management, running on top of VMware’s vSphere.

Another Kubernetes related mention was the gVisor pod sandbox, with experimental support for Kubernetes, to allow running sandboxed containers in a Kubernetes cluster. Very interesting for multi-tenant clusters and docker image builds.

Cloud Functions are now Generally Available, and more serverless features are launched:

Serverless containers allow you to run container-based workloads in a fully managed environment and still only pay for what you use. Sign up for an early preview of serverless containers on Cloud Functions to run your own containerized functions on GCP with all the benefits of serverless.

A new GKE serverless add-on lets you run serverless workloads on Kubernetes Engine with a one-step deploy. You can go from source to containers instantaneously, auto-scale your stateless container-based workloads, and even scale down to zero.

Cloud Build, a fully-managed CI/CD platform that lets you build and test applications in the cloud. With an interesting approach where all the pipeline steps are containers themselves so it is reasonably easy to extend. It integrates with GitHub for repos with a Dockerfile (let’s see if it lasts long after Microsoft acquisition).

Other interesting announcements include:

Edge TPU, a tiny ASIC chip designed to run TensorFlow Lite ML models at the edge.
Shielded VMs – untampered virtual machines
Titan Security Key, a FIDO security key with firmware developed by Google. Google security was giving away at the conference both NFC and bluetooth keys, a good replacement for the yubikeys specially for mobile devices.

Introduction to Amazon Web Services Identity and Access Management

Posted on May 4, 2011 by Carlos Sanchez

Using AWS Identity and Access Management you can create separate users and permissions to use any AWS service, for instance EC2, and avoid giving other people your Amazon username, password or private key.

You can set very granular permissions, on users, groups, specific resources, and a combination of them. It will become really complex soon! But there are several very common use cases, that IAM is useful for. For instance having a AWS account for a team of developers.

Getting started

You can go through the Getting Started Guide, but I’ll save you some time:

Download IAM command line tools

Store your AWS credentials in a file, ie. ~/account-key

AWSAccessKeyId=AKIAIOSFODNN7EXAMPLE
AWSSecretKey=wJalrXUtnFEMI/K7MDENG/bPxRfiCYzEXAMPLEKEY

Configure environment variables

export AWS_IAM_HOME=<path_to_cli>
export PATH=$AWS_IAM_HOME/bin:$PATH
export AWS_CREDENTIAL_FILE=~/account-key

Creating an admin group

When you have IAM setup, the next step is to create an Admins group where you can add yourself

iam-groupcreate -g Admins

Create a policy in a file, ie. MyPolicy.txt

{
   "Statement":[{
      "Effect":"Allow",
      "Action":"*",
      "Resource":"*"
      }
   ]
}

Upload the policy

iam-groupuploadpolicy -g Admins -p AdminsGroupPolicy -f MyPolicy.txt

Creating an admin user

Create an admin user with

iam-usercreate -u YOUR_NAME -g Admins -k -v

The response looks similar to this:

AKIAIOSFODNN7EXAMPLE
wJalrXUtnFEMI/K7MDENG/bPxRfiCYzEXAMPLEKEY
arn:aws:iam::123456789012:user/YOUR_NAME
AIDACKCEVSQ6C2EXAMPLE

The first line is your Access Key ID; the second line is your Secret Access Key. You need to save these IDs.

Save your Access Key ID and your Secret Access Key to a file called for instance ~/YOUR_NAME_cred.txt. You can use those credentials from now on instead of the global AWS credentials for the whole account.

export AWS_CREDENTIAL_FILE=~/YOUR_NAME_cred.txt

Creating a dev group

Let’s create an example dev group where the users will have only read access to EC2 operations.

 iam-groupcreate -g dev

Now we need to set the group policy to allow all EC2 Describe* actions, which are the ones that allow users to see data, but not to change it. Create a file MyPolicy.txt with these contents

{
  "Statement": [
     {
       "Sid": "EC2AllowDescribe",
       "Action": [
         "ec2:Describe*"
       ],
       "Effect": "Allow",
       "Resource": "*"
     }
   ]
 }

Now upload the policy

iam-groupuploadpolicy -g dev -p devGroupPolicy -f MyPolicy.txt

Creating dev users

To create a new AWS user under the dev group

iam-usercreate -u username -g dev -k -v

Create a login profile for the user to log into the web console

iam-useraddloginprofile -u username -p password

The user can now access the AWS console at

https://your_AWS_Account_ID.signin.aws.amazon.com/console/ec2

Or you can make life easier by creating an alias

 iam-accountaliascreate -a maestrodev

and now the console is available at

https://maestrodev.signin.aws.amazon.com/console/ec2

About Policies

AWS policy files can be really complex. The AWS Policy Generator will help as a start point and see what actions can be used, but it won’t help you making them easier to read (using wildcards) or applying them to specific resources. Amazon could have provided a better generator tool allowing you to choose your own resources (users, groups, S3 buckets,…) from a easy to use interface and not having to lookup all sorts of crazy AWS identifiers. Hopefully they will be provide a comprehensive tool as part of the AWS Console.

There is more information available at the IAM User Guide.

Update

Just after I wrote this post Amazon has made IAM available in the AWS management console, which makes using IAM way easier.

Javagruppen 2011: Build and test in the cloud slides

Posted on February 21, 2011 by Carlos Sanchez

Last week spent some good days in Denmark for Javagruppen annual conference as I mentioned in a previous post. It’s a small conference that allows you to cover any question that the attendees have and be able to select what you talk about based on their specific interests.

I talked about creating an Apache Continuum + Selenium grid on EC2 for massively multi-environment and parallel build and test. You can find the slides below, although it’s mostly a talk/visual presentation.

The location was great, in a hotel with spa in Jutland and very nice people and the other speakers too. My advice, go to Denmark, but try to do it in summer 🙂 I’m sure it makes a difference – although it’s pretty cool to be on a hot tub outside at 0C (32F)

And you can find some trip pictures in flickr.

Nyhavn panorama

Cloud Computing opportunities in the development (build-test-deploy) space

Posted on October 18, 2010 by Carlos Sanchez

You heard the word “cloud” everywhere, running applications on the cloud, scaling with the cloud,… but not so often from the development lifecycle perspective: code, commit, test, deploy to QA, release, etc but it brings fundamental changes to this aspect too.

The scenario

If you belong to, or manage, a group of developers, you are doing at least some sort of automated builds with continuous integration. You have continuous integration servers, building your code on specific schedules, not as often as you would like, when developers commit changes. The number of projects grow and you add more servers for the new projects, mixing and matching environments for different needs (linux, windows, os x,…)

The problem and the solution

The architecture we use for our Maestro 3 product is composed of one server that handles all the development lifecycle assets. Behind the scenes we use proven open source projects: Apache Continuum for distributed builds, Apache Archiva for repository management, Sonar for reporting, Selenium for multienvironment integration and load testing. And we add Morph mCloud private cloud solution, which is also based on open source projects such as Eucalyptus or Puppet.

We have multiple Continuum build agents doing builds, and multiple Selenium agents for webapp integration testing, as well as several application servers for continuous deployment and testing.

limited capacity
- problem: your hardware is limited, and provision and setup of new servers requires a considerable amount of time
- solution: assets are dynamic, you can spin off new virtual machines when you need them, and shuffle infrastructure around in a matter of minutes with just few clicks from the web interface. The hybrid cloud approach means you can start new servers in a public cloud, Amazon EC2, if you really need to
capacity utilization
- problem: you need to setup less active projects in the same server as more active ones to make sure servers are not under/over-utilized
- solution: infrastructure is shared across all projects. If a project needs it more often than another then it’s there to be used
scheduling conflicts
- problem: at specific times, i.e. releases, you need to stop automatic jobs to ensure resources are available for those builds
- solution: a smart queue management can differentiate between different build types (ie. continuous builds, release builds) and prioritize
location dependence
- problem: you need to manage the infrastructure, knowing where each server is and what is it building
- solution: a central view of all the development assets for easier management: build agents, test agents or application servers
continuous growing
- problem: new projects are being added while you are trying to manage and optimize your current setup
- solution: because infrastructure is shared adding new projects is just a matter of scaling wide the cloud, without assigning infrastructure to specific projects
complexity in process
- problem: multiply that for the number of different stages in your promotion process: development environment, QA, Staging, Production
- solution: you can keep independent networks in the cloud while sharing resources like virtual machine templates for your stack for instance
long time-to-market
- problem: transition from development to QA to production is a pain point because releases and promotion is not automated
- solution: compositions (workflows) allow to design and automate the steps from development to production, including manual approval
complexity in organization:
- problem: in large organizations, multiply that for the number of separate entities, departments or groups that have their own separate structure
- solution: enabling self provisioning you can assign quotas to developers or groups to start servers as they need them in a matter of minutes from prebuilt templates

Why a private cloud?

cost effectiveness: development infrastructure is running continuously. Global development teams make use of it 24×7
bandwidth usage: the traffic between your source control system and the cloud can be expensive, because it’s continuously getting the code for building
security restrictions: most companies don’t like their code being exported anywhere outside their firewall. Companies that need to comply with regulations (ie. PCI) have strong requirements on external networks
performance: in a private cloud you can optimize the hardware for your specific scenario, reducing the number of VMs needed for the same job compared to public cloud providers
heterogeneous environments: if you need to develop in different environments then there are chances that the public cloud service won’t be able to provide them

The new challenges

parallelism, you need to know the dependencies between components to know what needs to be built before what
stickyness, or how to take advantage of the state of the agents to start builds in the same ones if possible, ie. agents that built a project before can do a source control update instead of a checkout, or have the dependencies already in the filesystem
asset management, when you have an increasing number of services running, stoping and starting as needed, you need to know what’s running and where, not only at hardware level but at service level: build agents, test agents and deployment servers.

The new vision

you can improve continuous integration as developers checkin code because the barrier to add new infrastructure is minimal, given you have enough hardware in your cloud or if you use external cloud services, which means reduced time to find problems
developers have access to infrastructure they need to do their jobs, for instance start an exact copy of the production environment to fix an issue by using a cloud template that they can get up and running in minutes and tear down at the end, not incurring in high hardware costs
less friction and easier interaction with IT people as developers can self provision infrastructure, if necessary shuffling virtual machines that they no longer need for the ones they needed

By leveraging the cloud you can solve existing problems in your development lifecycle and at the same time you will be able to do things that you would not even consider because the technology made it complicated or impossible to do. Definitely something worth checking out for large development teams.

Maestro 3 is going to be released this week at InterOp New York (come over and say hi if you are around) but we are already demoing the development snapshots to clients and at conferences like JavaOne.

JavaOne slides: Enterprise Build and Test in the Cloud

Posted on July 2, 2009 by Carlos Sanchez

I have uploaded the slides from my talk Enterprise Build and Test in the Cloud at JavaOne in San Francisco.

You can check also the code, and an introduction in previous posts

Enterprise build and Test in the Cloud with Selenium I
and
Enterprise build and Test in the Cloud with Selenium II.

Follow me on twitter

JavaOne talk: Enterprise Build and Test in the Cloud

Posted on May 20, 2009 by Carlos Sanchez

I’ll be presenting Enterprise Build and Test in the Cloud at JavaOne in San Francisco, Wednesday June 3rd 11:05am Esplanade 301 and will be around the whole week.

You can check the slides from the previous talk at ApacheCON, the code, and an introduction in previous posts

Enterprise build and Test in the Cloud with Selenium I
and
Enterprise build and Test in the Cloud with Selenium II.

Follow me on twitter

ApacheCON slides: “Enterprise Build and Test in the Cloud” and “Eclipse IAM, Maven integration for Eclipse”

Posted on April 8, 2009 by Carlos Sanchez

Here you have the slides from my talks at ApacheCON

Enterprise Build and Test in the Cloud

Building and testing software can be a time and resource consuming task. Cloud computing / on demand services like Amazon EC2 allow a cost-effective way to scale applications, and applied to building and testing software can reduce the time needed to find and correct problems, meaning a reduction also in time and costs. Properly configuring your build tools (Maven, Ant,…), continuous integration servers (Continuum, Cruise Control,…), and testing tools (TestNG, Selenium,…) can allow you to run all the build/testing process in a cloud environment, simulating high load environments, distributing long running tests to reduce their execution time, using different environments for client or server applications,… and in the case of on-demand services like Amazon EC2, pay only for the time you use it.
In this presentation we will introduce a development process and architecture using popular open source tools for the build and test process such as Apache Maven or Ant for building, Apache Continuum as continuous integration server, TestNG and Selenium for testing, and how to configure them to achieve the best results and performance in several typical use cases (long running testing processes, different client platforms,…) by using he Amazon Elastic Computing Cloud EC2, and therefore reducing time and costs compared to other solutions.

Download PDF

Eclipse IAM, Maven integration for Eclipse

Eclipse IAM (Eclipse Integration for Apache Maven), formerly “Q for Eclipse”, is an Open Source project that integrates Apache Maven and the Eclipse IDE for faster, more agile, and more productive development. The plugin allows you to run Maven from the IDE, import existing Maven projects without intermediate steps, create new projects using Maven archetypes, synchronize dependency management, search artifact repositories for dependencies that are automatically downloaded, view a graph of dependencies and more! Join us to discover how to take advantage of all these features, as well as how they can help you to improve your development process.

Download PDF

	sadsadsd on Building Docker Images with Ka…
	Carlos Sanchez on Running a JVM in a Container W…
	Salaikumar on Running a JVM in a Container W…
	Ford Lady on Como enviar un coche de USA a…
	José Luis on Como enviar un coche de USA a…

Share this:

Like this:

Getting started

Creating an admin group

Creating an admin user

Creating a dev group

Creating dev users

About Policies

Update

Share this:

Like this:

Share this:

Like this:

The scenario

The problem and the solution

Why a private cloud?

The new challenges

The new vision

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Share this:

Like this:

Enterprise Build and Test in the Cloud

Eclipse IAM, Maven integration for Eclipse

Share this:

Like this:

Share this:

Like this: