Tag Archives: serverless

Serverless Jenkins Pipelines with Google Cloud Run

Posted on by Carlos Sanchez
Reply
jenkins-google-cloud-run

Jenkinsfile-Runner-Google-Cloud-Run project is a Google Cloud Run (a container native, serverless platform) Docker image to run Jenkins pipelines. It will process a GitHub webhook, git clone the repository and execute the Jenkinsfile in that git repository. It allows high scalability and pay per use with zero cost if not used.

This image allows Jenkinsfile execution without needing a persistent Jenkins master running in the same way as Jenkins X Serverless, but using the Google Cloud Run platform instead of Kubernetes.

Google Cloud Run vs Project Fn vs AWS Lambda

I wrote three flavors of Jenkinsfile Runner

The image is similar to the other ones. The main difference between Lambda and Google Cloud Run is in the packaging, as Lambda layers are limited in size and are expanded in /opt while Google Cloud Run allows any custom Dockerfile where you can install whatever you want in a much easier way.

This image is extending the Jenkinsfile Runner image instead of doing a Maven build with it as a dependency as it simplifies classpath magement.

Limitations

Max build duration is 15 minutes but we can use a timeout value up tos 60 minutes by using gcloud beta.

Current implementation limitations:

  • checkout scm does not work, change it to sh 'git clone https://github.com/carlossg/jenkinsfile-runner-example.git'

Example

See the jenkinsfile-runner-example project for an example.

When the PRs are built Jenkins writes a comment back to the PR to show status, as defined in the Jenkinsfile, and totally customizable.

Check the PRs at carlossg/jenkinsfile-runner-example

Extending

You can add your plugins to plugins.txt. You could also add the Configuration as Code plugin for configuration, example at jenkins.yaml.

Other tools can be added to the Dockerfile.

Installation

GitHub webhooks execution will time out if the call takes too long, so we also create a nodejs Google function (index.js) that forwards the request to Google Cloud Run and returns the response to GitHub while the build runs.

Building

Build the package

mvn verify 
docker build -t jenkinsfile-runner-google-cloud-run .

Publishing

Both the function and the Google Cloud Run need to be deployed.

Set GITHUB_TOKEN_JENKINSFILE_RUNNER to a token that allows posting PR comments. A more secure way would be to use Google Cloud Secret Manager.

export GITHUB_TOKEN_JENKINSFILE_RUNNER=... 
PROJECT_ID=$(gcloud config get-value project 2> /dev/null) 
make deploy

Note the function url and use it to create a GitHub webhook of type json.

Execution

To test the Google Cloud Run execution

URL=$(gcloud run services describe jenkinsfile-runner \ 
  --platform managed \ 
  --region us-east1 \ 
  --format 'value(status.address.url)') 

curl -v -H "Content-Type: application/json" ${URL}/handle \
  -d @src/test/resources/github.json

Logging

gcloud logging read \
  "resource.type=cloud_run_revision AND resource.labels.service_name=jenkinsfile-runner" \ 
  --format "value(textPayload)" --limit 100

or

gcloud alpha logging tail \
  "resource.type=cloud_run_revision AND resource.labels.service_name=jenkinsfile-runner" \ 
  --format "value(textPayload)"

GitHub events

Add a GitHub json webhook to your git repo pointing to the Google Cloud Function url than you can get with

gcloud functions describe jenkinsfile-runner-function \
  --format 'value(httpsTrigger.url)'

Testing

The image can be run locally

docker run -ti --rm -p 8080:8080 \
  -e GITHUB_TOKEN=${GITHUB_TOKEN_JENKINSFILE_RUNNER} \
  jenkinsfile-runner-google-cloud-run
curl -v -H "Content-Type: application/json" \
  -X POST http://localhost:8080/handle \
  -d @src/test/resources/github.json

More information in the Jenkinsfile-Runner-Google-Cloud-Run GitHub page.

Serverless Jenkins Pipelines with Fn Project

Posted on by Carlos Sanchez
1

jenkins-lambdaThe Jenkinsfile-Runner-Fn project is a Fn Project (a container native, cloud agnostic serverless platform) function to run Jenkins pipelines. It will process a GitHub webhook, git clone the repository and execute the Jenkinsfile in that git repository. It allows scalability and pay per use with zero cost if not used.

This function allows Jenkinsfile execution without needing a persistent Jenkins master running in the same way as Jenkins X Serverless, but using the Fn Project platform (and supported providers like Oracle Functions) instead of Kubernetes.

Fn Project vs AWS Lambda

The function is very similar to the one in jenkinsfile-runner-lambda with just a small change in the signature. The main difference between Lambda and Fn is in the packaging, as Lambda layers are limited in size and are expanded in /optwhile Fn allows a custom Dockerfile where you can install whatever you want in a much easier way, just need to include the function code and entrypoint from fnproject/fn-java-fdk.

Oracle Functions

Oracle Functions is a cloud service providing Project Fn function execution (currently in limited availability). jenkinsfile-runner-fn function runs in Oracle Functions, with the caveat that it needs a syslog server running somewhere to get the logs (see below).

Limitations

Current implementation limitations:

  • checkout scm does not work, change it to sh 'git clone https://github.com/carlossg/jenkinsfile-runner-fn-example.git'
  • Jenkinsfile must use /tmp for any tool that needs writing files, see the example

Example

See the jenkinsfile-runner-fn-example project for an example that is tested and works.

Extending

You can add your plugins to plugins.txt. You could also add the Configuration as Code plugin for configuration.

Other tools can be added to the Dockerfile.

Installation

Install Fn

Building

Build the function

mvn clean package

Publishing

Create and deploy the function locally

fn create app jenkinsfile-runner
fn --verbose deploy --app jenkinsfile-runner --local

Execution

Invoke the function

cat src/test/resources/github.json | fn invoke jenkinsfile-runner jenkinsfile-runner

Logging

Get the logs for the last execution

fn get logs jenkinsfile-runner jenkinsfile-runner \
$(fn ls calls jenkinsfile-runner jenkinsfile-runner | grep 'ID:' | head -n 1 | sed -e 's/ID: //')

Syslog

Alternatively, start a syslog server to see the logs

docker run -d --rm -it -p 5140:514 --name syslog-ng balabit/syslog-ng:latest
docker exec -ti syslog-ng tail -f /var/log/messages-kv.log

Update the function to send logs to the syslog server

fn update app jenkinsfile-runner --syslog-url tcp://logs-01.loggly.com:514

GitHub events

Add a GitHub json webhook to your git repo pointing to the function url.

More information in the Jenkinsfile-Runner-Fn GitHub page.