Showing posts with label Gradle. Show all posts
Showing posts with label Gradle. Show all posts

Wednesday, July 6, 2022

Google Cloud Function Gradle Plugin

 It is easy to develop a Google Cloud Function using Java with Gradle as the build tool. It is however not so simple to test it locally.

The current recommended approach to testing especially with gradle is very complicated. It requires pulling in Invoker libraries and adding a custom task to run the invoker function.

I have now authored a gradle plugin which makes local testing way more easier!


Problem

The way the Invoker is added in for a Cloud Function Gradle project looks like this today:

This has a lot of opaque details, for eg, what does the configurations of invoker even mean, what is the magical task that is being registered?

Fix

Now contrast it with the approach with the plugin:


All the boiler plate is now gone, configuration around the function class, which port to start it up on much more simplified. Adding this new plugin contributes a task that can be invoked the following way:

./gradlew cloudFunctionRun
It would start up an endpoint using which the function can be tested locally.

Conclusion

It may be far easier to see fully working samples incorporating this plugin. These samples are available here —


Thursday, July 19, 2018

Jib - Building docker image for a Spring Boot App

I was pleasantly surprised by how easy it was to create a docker image for a sample Spring Boot application using Jib.

Let me first contrast Jib with an approach that I was using before.

I was creating docker images using bmuschko's excellent gradle-docker plugin. Given access to a docker daemon and a gradle dsl based description of the Dockerfile or a straight Dockerfile, it would create the docker image using a gradle task. In my case, the task to create the docker image looks something like this:

task createDockerImage(type: DockerBuildImage) {
    inputDir = file('.')
    dockerFile = project.file('docker/Dockerfile')
    tags = ['sample-micrometer-app:' + project.version]
}

createDockerImage.dependsOn build

and my Dockerfile itself derived off "java:8" base image:

FROM java:8
...

gradle-docker-plugin made it simple to create a docker image right from gradle with the catch that the plugin needs access to a docker daemon to create the image. Also since the base "java:8" image is large the final docker image turns out to be around 705MB on my machine. Again no fault of the gradle-docker plugin but based on my choice of base image.


Now with Jib, all I have to do is to add the plugin:

plugins {
    id 'com.google.cloud.tools.jib' version '0.9.6'
}

Configure it to give the image a name:

jib {
    to {
        image = "sample-micrometer-app:0.0.1-SNAPSHOT"
    }
}

And that is it. With a local docker daemon available, I can create my docker image using the following task:


./gradlew jibDockerBuild

Jib automatically selects a very lightweight base image - my new image is just about 150 MB in size.

If I had access to a docker registry available then the local docker daemon is not required, it can directly create and publish the image to a docker registry!

Jib gradle plugin provides an interesting task - "jibExportDockerContext" to export out the docker file, this way if needed a docker build can be run using this Dockerfile, for my purposes I wanted to see the contents of this file and it looks something like this:

FROM gcr.io/distroless/java

COPY libs /app/libs/
COPY resources /app/resources/
COPY classes /app/classes/

ENTRYPOINT ["java","-cp","/app/libs/*:/app/resources/:/app/classes/","sample.meter.SampleServiceAppKt"]


All in all, a very smooth experience and Jib does live up to its goals. My sample project with jib integrated with a gradle build is available here.


Monday, August 21, 2017

Gradle Kotlin DSL

Gradle build scripts can now be written using a dsl with Kotlin Language. All the concepts that work with traditional gradle build translate to a very intuitive dsl in Kotlin and have two additional features - it is typesafe and the script has excellent IDE support using Intellij IDEA.

My experience with the Gralde Kotlin DSL is fairly limited - all of one build script which is the subject of this article.

If you want to simply see how a sample script looks, I have a sample github repo with just that here - https://github.com/bijukunjummen/cf-show-env


Just to compare:

1. Consider the way different plugins are applied with gradle:

plugins {
 id "com.github.pivotalservices.cf-app" version "1.0.9"
}

apply plugin: 'kotlin"
apply plugin: 'java'
apply plugin: 'org.springframework.boot'
apply from: 'gradle/gatling.gradle'


An equivalent kotlin dsl is the following:

plugins {
    id("com.github.pivotalservices.cf-app").version("1.0.9")
}

apply {
    plugin("kotlin")
    plugin("java")
    plugin("org.springframework.boot")    
    from("gradle/gatling.gradle")
}



2. Adding project dependencies:

dependencies {
    compile('org.springframework.boot:spring-boot-starter-actuator')
    compile('org.springframework.boot:spring-boot-devtools')
    compile('org.springframework.boot:spring-boot-starter-thymeleaf')
    compile('org.springframework.boot:spring-boot-starter-web')
    compile('com.google.guava:guava:19.0')
    compile("org.webjars:bootstrap:3.3.7")
    compile("org.webjars:jquery:3.1.1")
    compile("io.prometheus:simpleclient:${prometheus_client_version}")
    compile("io.prometheus:simpleclient_spring_boot:${prometheus_client_version}")
    compile('nz.net.ultraq.thymeleaf:thymeleaf-layout-dialect')
    testCompile('org.springframework.boot:spring-boot-starter-test')
}

an equivalent code using kotlin DSL:

dependencies {
    val prometheus_client_version = "0.0.21"

    compile("org.springframework.boot:spring-boot-starter-actuator")
    compile("org.springframework.boot:spring-boot-devtools")
    compile("org.springframework.boot:spring-boot-starter-thymeleaf")
    compile("org.springframework.boot:spring-boot-starter-web")
    compile("com.google.guava:guava:19.0")
    compile("org.webjars:bootstrap:3.3.7")
    compile("org.webjars:jquery:3.1.1")
    compile("io.prometheus:simpleclient:${prometheus_client_version}")
    compile("io.prometheus:simpleclient_spring_boot:${prometheus_client_version}")
    compile("nz.net.ultraq.thymeleaf:thymeleaf-layout-dialect")
    testCompile("org.springframework.boot:spring-boot-starter-test")
}

3. Configuring Plugins - I have a plugin which helps deploy applications to Cloud Foundry, and works off a configuration which looks like this, when expressed using normal gradle build:

cfConfig {
    //CF Details
    ccHost = "api.local.pcfdev.io"
    ccUser = "admin"
    ccPassword = "admin"
    org = "pcfdev-org"
    space = "pcfdev-space"

    //App Details
    name = "cf-show-env"
    hostName = "cf-show-env"
    filePath = "build/libs/cf-show-env-0.1.3-SNAPSHOT.jar"
    path = ""
    domain = "local.pcfdev.io"
    instances = 2
    memory = 1024
    timeout = 180

    //Env and services
    buildpack = "https://github.com/cloudfoundry/java-buildpack.git"


    environment = ["JAVA_OPTS": "-Djava.security.egd=file:/dev/./urandom", "SPRING_PROFILES_ACTIVE": "cloud"]

    cfService {
        name = "p-mysql"
        plan = "512mb"
        instanceName = "test-db"
    }
    
 
    
    cfUserProvidedService {
        instanceName = "mydb1"
        credentials = ["jdbcUri": "someuri1"]
    }
}

this can now be configured in a typesafe way with full auto-completion support in IntelliJ the following way using Kotlin DSL:

configure< CfPluginExtension> {
    //CF Details
    ccHost = "api.local.pcfdev.io"
    ccUser = "admin"
    ccPassword = "admin"
    org = "pcfdev-org"
    space = "pcfdev-space"

    //App Details
    name = "cf-show-env"
    hostName = "cf-show-env"
    filePath = "build/libs/cf-show-env-1.0.0-M1.jar"
    path = ""
    domain = "local.pcfdev.io"
    instances = 2
    memory = 1024
    timeout = 180

    //Env and services
    buildpack = "https://github.com/cloudfoundry/java-buildpack.git"

    environment = mapOf(
            "JAVA_OPTS" to "-Djava.security.egd=file:/dev/./urandom", 
            "SPRING_PROFILES_ACTIVE" to "cloud"
    )

    cfService(closureOf<CfService> {
        name = "p-mysql"
        plan = "512mb"
        instanceName = "test-db"
    })
    
    cfUserProvidedService(closureOf<CfUserProvidedService> { 
        instanceName = "myups"
        credentials = mapOf(
                "user" to "someuser",
                "uri" to "someuri"
        )
    })

}

4. And finally a straight task:
task "hello-world" {
    doLast {
        println("Hello World")
    }
}

task showAppUrls(dependsOn: "cf-get-app-detail") << {
    print "${project.cfConfig.applicationDetail}"
}

looks more or less the same in Kotlin DSL:

task("hello-world") {
    doLast {
        println("Hello World")
    }
}


task("showAppUrls").dependsOn("cf-get-app-detail").doLast {
       println(cfConfig);
}


I am excited about using Kotlin DSL to configure my gradle builds, there are a few quirks to keep in mind though - the Intellij support tends to be a little flaky, it took a few tries for the IDEA to start helping with the auto-completions, also I needed to google quite a bit and look at some of the sample projects in gradle kotlin dsl, all in all though this has an awesome potential.

Monday, August 14, 2017

Concourse caching for Java Maven and Gradle builds

Concourse CI 3.3.x has introduced the ability to cache paths between task runs. This feature helps speed up tasks which cache content in specific folders - here I will demonstrate how this feature can be used for speeding up maven and gradle based java builds.

The code and the pipeline that I am using for this post is available at my github repo here - https://github.com/bijukunjummen/ci-concourse-caching-sample

Let me start with the gradle build, if I were to build the project using a gradle wrapper using the following command:

./gradlew clean build

then gradle would download the dependent libraries into a ".gradle" folder in the users home folder by default. This location of this folder can be changed using a "GRADLE_USER_HOME" environment variable, which is what I will be using in a concourse task to control the location of a cached path.

A concourse task which builds my project looks like this:

---
platform: linux
image_resource:
  type: docker-image
  source:
    repository: openjdk
    tag: 8-jdk
inputs:
  - name: repo
outputs:
  - name: out
run:
  path: /bin/bash
  args:
    - repo/ci/tasks/build.sh

caches:
  - path: .gradle/
  - path: .m2/

params:
  PROJECT_TYPE: 

See the caches parameter is specified as ".gradle" above. So all I have to do now is to ensure that Gradle uses this location as its home folder, which I would do in my build script:

export ROOT_FOLDER=$( pwd )
export GRADLE_USER_HOME="${ROOT_FOLDER}/.gradle"


The process to cache maven resources for a maven build is along the same lines, maven caches the dependent jars in a location that can be specified in a variety of ways, the one I have used is to specify this location via a dynamically generated settings.xml file the following way:

M2_HOME=${HOME}/.m2
mkdir -p ${M2_HOME}

M2_LOCAL_REPO="${ROOT_FOLDER}/.m2"

mkdir -p "${M2_LOCAL_REPO}/repository"

cat > ${M2_HOME}/settings.xml <<EOF

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                          https://maven.apache.org/xsd/settings-1.0.0.xsd">
      <localRepository>${M2_LOCAL_REPO}/repository</localRepository>
</settings>

EOF

which is quite a bit of bash scripting, all it is doing is generating a settings.xml with a localRepository tag set to ".m2/repository" folder which is relative to the temporary folder created by concourse for the build and thus can be cached.

With these changes in place, the behavior is that the downloads happen for the first run of the task but then get cached for subsequent runs. In my local concourse set-up a gradle build taking about 2 mins for a first time build takes about 20 seconds for a subsequent build !

You can try out this feature in my demo project here - https://github.com/bijukunjummen/ci-concourse-caching-sample


UPDATE: 06/08/2018

A far simpler approach to caching maven and gradle dependencies, than what I have detailed previously, is possible and this is based on an approach followed by Spring Cloud Pipelines project. My github sample already contains this change.

Again like before, first indicate in the task the folders(maven and gradle) that need to be cached across different runs of the task, remember that the caching is scoped to a task and a worker:

---
platform: linux
image_resource:
  type: docker-image
  source:
    repository: openjdk
    tag: 8-jdk
inputs:
  - name: repo
outputs:
  - name: out
run:
  path: /bin/bash
  args:
    - repo/ci/tasks/build.sh

caches:
  - path: gradle
  - path: maven

params:
  PROJECT_TYPE: 

Gradle expects the cached content to be available at "${HOME}/.gradle" folder and Maven at "${HOME}/.m2" folder, the trick then is to simply create a symbolic link of the cached folders to these folders, the following way in a bash script:

export ROOT_FOLDER=$( pwd )
export REPO=repo

M2_HOME="${HOME}/.m2"
M2_CACHE="${ROOT_FOLDER}/maven"
GRADLE_HOME="${HOME}/.gradle"
GRADLE_CACHE="${ROOT_FOLDER}/gradle"

echo "Generating symbolic links for caches"

[[ -d "${M2_CACHE}" && ! -d "${M2_HOME}" ]] && ln -s "${M2_CACHE}" "${M2_HOME}"
[[ -d "${GRADLE_CACHE}" && ! -d "${GRADLE_HOME}" ]] && ln -s "${GRADLE_CACHE}" "${GRADLE_HOME}"

And that should be it, since Maven and Gradle see their default local repositories the caching of dependencies should just work without any additional changes needed!, this is far simpler than the approach that I have previously described.

Thursday, May 4, 2017

Integrating Gatling into a Gradle build - Understanding SourceSets and Configuration

I recently worked on a project where we had to integrate the excellent load testing tool Gatling into a Gradle based build. There are gradle plugins available which make this easy, two of them being this and this, however for most of the needs a simple execution of the command line tool itself suffices, so this post will go into some details of how gatling can be hooked up into a gradle build and in the process understand some good gradle concepts.


SourceSets and Configuration


To execute the gatling cli I need to do two things, I need a location for the source code and related content of the Gatling simulations, and I need a way to get the gatling libraries. This is where two concepts of Gradle(SourceSets and Configuration) come into play.

Let us start with the first one - SourceSets.

SourceSets


SourceSets are simply a logical grouping of related files and are best demonstrated with an example. If I were to add a "java" plugin to a gradle build:

apply plugin: 'java'


sourceSets property would now show up with two values "main" and "test" and if I wanted to find details of the these sourceSets, a gradle task can be used for printing the details:

task sourceSetDetails {
    doLast {
        sourceSets {
            main {
                println java.properties
                println resources.properties
            }
        
            test {
                println java.properties
                println resources.properties
            }
        }
    }
}

Coming back to gatling, I can essentially create a new sourceSet to hold the gatling simulations:

sourceSets {
    simulations
}

This would now expect the gatling simulations to reside in "src/simulations/java" and the resources related to it in "src/simulations/resources" folders, which is okay, but ideally I would want to keep it totally separate from the project sources. I would want my folder structure to be with load simulations in "simulations/load" and resources in "simulations/resources" folder. This can be tweaked by first applying the "scala" plugin, which brings in scala compilation support to the project and then modifying the "simulations" source set along these lines:

apply plugin: 'scala'

sourceSets {
    simulations {
        scala {
            srcDirs = ['simulations/load']
        }
        resources {
            srcDirs = ['simulations/resources']
        }
    }
}

With these set of changes, I can now put my simulations in the right place, but the dependency of gatling and scala has not been pulled in, this is where the "configuration" feature of gradle comes in.

Configuration


Gradle Configuration is a way of grouping related dependencies together. If I were to print the existing set of configurations using a task:

task showConfigurations  {
    doLast {
        configurations.all { conf -> println(conf) }
    }
}

these show up:

configuration ':archives'
configuration ':compile'
configuration ':compileClasspath'
configuration ':compileOnly'
configuration ':default'
configuration ':runtime'
configuration ':simulationsCompile'
configuration ':simulationsCompileClasspath'
configuration ':simulationsCompileOnly'
configuration ':simulationsRuntime'
configuration ':testCompile'
configuration ':testCompileClasspath'
configuration ':testCompileOnly'
configuration ':testRuntime'
configuration ':zinc'

"compile" and "testCompile" should be familiar one's, that is where a normal source dependency and a test dependency is typically declared like this:

dependencies {
    compile 'org.slf4j:slf4j-api:1.7.21'
    testCompile 'junit:junit:4.12'   
}

However, it also looks like there is now configuration for "simulations" sourceset also available - "simulationsCompile" and "simulationsRuntime" etc, so with this I can declare the dependencies required for my gatling simulations using these configurations, however my intention is to declare a custom configuration just to go over the concept a little more, so let us explicitly declare one:

configurations {
    gatling
}

and use this configuration for declaring the dependencies of gatling:
dependencies {
    gatling 'org.scala-lang:scala-library:2.11.8'
    gatling 'io.gatling.highcharts:gatling-charts-highcharts:2.2.5'
}

Almost there, now how do we tell the sources in simulations source set to use dependency from gatling configuration..by tweaking the sourceSet a little:

sourceSets {
    simulations {
        scala {
            srcDirs = ['simulations/load']
        }
        resources {
            srcDirs = ['simulations/resources']
        }

        compileClasspath += configurations.gatling
    }
}


Running a Gatling Scenario

With the source sets and the configuration defined, all we need to do is to write a task to run a gatling simulation, which can be along these lines:

task gatlingRun(type: JavaExec) {
    description = 'Run gatling tests'
    new File("${buildDir}/reports/gatling").mkdirs()

    classpath = sourceSets.simulations.runtimeClasspath + configurations.gatling

    main = "io.gatling.app.Gatling"
    args = ['-s', 'simulations.SimpleSimulation',
            '-sf', 'simulations/resources',
            '-df', 'simulations/resources',
            '-rf', "${buildDir}/reports/gatling"
    ]
}

See how the compiled sources of the simulations and the dependencies from the gatling configuration is being set as the classpath for the "JavaExec" task


A good way to review this would be to look at a complete working sample that I have here in my github repo - https://github.com/bijukunjummen/cf-show-env

Sunday, January 15, 2017

Gradle Plugins DSL and Spring-Boot Plugin

Gradle Plugins DSL is a new gradle feature which provides a very succinct way of adding a plugin to a Gradle based project. A good way to show the utility of this new mechanism is in how it simplifies a sample Spring Boot based gradle build file.

If I were to generate a sample gradle based Spring boot project from the excellent http://start.spring.io site, a snippet of the gradle file which adds in the Spring Boot gradle plugin looks like this:

buildscript {
 ext {
  springBootVersion = '1.4.3.RELEASE'
 }
 repositories {
  mavenCentral()
 }
 dependencies {
  classpath("org.springframework.boot:spring-boot-gradle-plugin:${springBootVersion}")
 }
}

apply plugin: 'org.springframework.boot'

The new Gradle Plugins DSL simplifies this boilerplate drastically. An equivalent declaration using the new Plugins DSL is the following:

plugins {
  id "org.springframework.boot" version "1.4.3.RELEASE"
}

This IMHO reads far better, though it does require some level of mental parsing. The best way to understand this new syntax though may to know that this works in concert with the Gradle plugins portal, a centralized repository of plugins, to resolve the plugin related dependencies. The page for the Spring Boot plugin is here - https://plugins.gradle.org/plugin/org.springframework.boot.

Saturday, August 6, 2016

No downtime deployment using "Yet another" Cloud Foundry Gradle plugin

I have been trying my hand at writing a gradle plugin for deploying applications to Cloud Foundry and wrote about this plugin in my previous post. I have now enhanced this plugin with support for no-downtime deploys into Cloud Foundry using two approaches - an Autopilot style deployment and a more commonly used Blue-Green style deployment.

To jump into the meat of the plugin, once it is configured cleanly all you have to do is the following:

For an autopilot style

./gradlew cf-push-autopilot

and for a Blue-Green deployment:

./gradlew cf-push-blue-green

and the plugin tasks would take care of the rest.

What is being solved


If you use Cloud Foundry CLI to push an application to Cloud Foundry, then existing instances of the application is stopped, replaced and started up. This introduces a downtime for the application until the new instance of the application is up. Just to demonstrate this behavior, the following graph represents a steady traffic to a website while an application is pushed to Cloud Foundry - the 30 second blip is when the new app is being started up.


Autopilot and Blue-Green style deployments


Autopilot and Blue-Green styles of deployment fix the issue by carefully orchestrating the deployment of an application such that the external facing route always points to a working version of the application.

The plugin now natively performs all the steps needed for these two styles of no-downtime deployments.

Here is how the same graph looks with an Autopilot style type deployment using the plugin, note that there is a slightly higher response time around the time the new application switches in. Once primed though the response times smooth out:



and with a Blue-Green style deployment using this plugin


References:


1. The details about how to install and configure the plugin is available here - https://github.com/pivotalservices/ya-cf-app-gradle-plugin

2. A sample application configured with the plugin is here - https://github.com/bijukunjummen/cf-show-env

3. The load test using gatling is available here