all and sundry: August 2017

Monday, August 21, 2017

Gradle Kotlin DSL

Gradle build scripts can now be written using a dsl with Kotlin Language. All the concepts that work with traditional gradle build translate to a very intuitive dsl in Kotlin and have two additional features - it is typesafe and the script has excellent IDE support using Intellij IDEA.

My experience with the Gralde Kotlin DSL is fairly limited - all of one build script which is the subject of this article.

If you want to simply see how a sample script looks, I have a sample github repo with just that here - https://github.com/bijukunjummen/cf-show-env

Just to compare:

1. Consider the way different plugins are applied with gradle:

plugins {
 id "com.github.pivotalservices.cf-app" version "1.0.9"
}
 
apply plugin: 'kotlin"
apply plugin: 'java'
apply plugin: 'org.springframework.boot'
apply from: 'gradle/gatling.gradle'

An equivalent kotlin dsl is the following:

plugins {
    id("com.github.pivotalservices.cf-app").version("1.0.9")
}
 
apply {
    plugin("kotlin")
    plugin("java")
    plugin("org.springframework.boot")    
    from("gradle/gatling.gradle")
}

2. Adding project dependencies:

dependencies {
    compile('org.springframework.boot:spring-boot-starter-actuator')
    compile('org.springframework.boot:spring-boot-devtools')
    compile('org.springframework.boot:spring-boot-starter-thymeleaf')
    compile('org.springframework.boot:spring-boot-starter-web')
    compile('com.google.guava:guava:19.0')
    compile("org.webjars:bootstrap:3.3.7")
    compile("org.webjars:jquery:3.1.1")
    compile("io.prometheus:simpleclient:${prometheus_client_version}")
    compile("io.prometheus:simpleclient_spring_boot:${prometheus_client_version}")
    compile('nz.net.ultraq.thymeleaf:thymeleaf-layout-dialect')
    testCompile('org.springframework.boot:spring-boot-starter-test')
}

an equivalent code using kotlin DSL:

dependencies {
    val prometheus_client_version = "0.0.21"
 
    compile("org.springframework.boot:spring-boot-starter-actuator")
    compile("org.springframework.boot:spring-boot-devtools")
    compile("org.springframework.boot:spring-boot-starter-thymeleaf")
    compile("org.springframework.boot:spring-boot-starter-web")
    compile("com.google.guava:guava:19.0")
    compile("org.webjars:bootstrap:3.3.7")
    compile("org.webjars:jquery:3.1.1")
    compile("io.prometheus:simpleclient:${prometheus_client_version}")
    compile("io.prometheus:simpleclient_spring_boot:${prometheus_client_version}")
    compile("nz.net.ultraq.thymeleaf:thymeleaf-layout-dialect")
    testCompile("org.springframework.boot:spring-boot-starter-test")
}

3. Configuring Plugins - I have a plugin which helps deploy applications to Cloud Foundry, and works off a configuration which looks like this, when expressed using normal gradle build:

cfConfig {
    //CF Details
    ccHost = "api.local.pcfdev.io"
    ccUser = "admin"
    ccPassword = "admin"
    org = "pcfdev-org"
    space = "pcfdev-space"
 
    //App Details
    name = "cf-show-env"
    hostName = "cf-show-env"
    filePath = "build/libs/cf-show-env-0.1.3-SNAPSHOT.jar"
    path = ""
    domain = "local.pcfdev.io"
    instances = 2
    memory = 1024
    timeout = 180
 
    //Env and services
    buildpack = "https://github.com/cloudfoundry/java-buildpack.git"
 
 
    environment = ["JAVA_OPTS": "-Djava.security.egd=file:/dev/./urandom", "SPRING_PROFILES_ACTIVE": "cloud"]
 
    cfService {
        name = "p-mysql"
        plan = "512mb"
        instanceName = "test-db"
    }
     
  
     
    cfUserProvidedService {
        instanceName = "mydb1"
        credentials = ["jdbcUri": "someuri1"]
    }
}

this can now be configured in a typesafe way with full auto-completion support in IntelliJ the following way using Kotlin DSL:

configure< CfPluginExtension> {
    //CF Details
    ccHost = "api.local.pcfdev.io"
    ccUser = "admin"
    ccPassword = "admin"
    org = "pcfdev-org"
    space = "pcfdev-space"
 
    //App Details
    name = "cf-show-env"
    hostName = "cf-show-env"
    filePath = "build/libs/cf-show-env-1.0.0-M1.jar"
    path = ""
    domain = "local.pcfdev.io"
    instances = 2
    memory = 1024
    timeout = 180
 
    //Env and services
    buildpack = "https://github.com/cloudfoundry/java-buildpack.git"
 
    environment = mapOf(
            "JAVA_OPTS" to "-Djava.security.egd=file:/dev/./urandom", 
            "SPRING_PROFILES_ACTIVE" to "cloud"
    )
 
    cfService(closureOf<CfService> {
        name = "p-mysql"
        plan = "512mb"
        instanceName = "test-db"
    })
     
    cfUserProvidedService(closureOf<CfUserProvidedService> { 
        instanceName = "myups"
        credentials = mapOf(
                "user" to "someuser",
                "uri" to "someuri"
        )
    })
 
}

4. And finally a straight task:

task "hello-world" {
    doLast {
        println("Hello World")
    }
}
 
task showAppUrls(dependsOn: "cf-get-app-detail") << {
    print "${project.cfConfig.applicationDetail}"
}

looks more or less the same in Kotlin DSL:

task("hello-world") {
    doLast {
        println("Hello World")
    }
}
 
 
task("showAppUrls").dependsOn("cf-get-app-detail").doLast {
       println(cfConfig);
}

I am excited about using Kotlin DSL to configure my gradle builds, there are a few quirks to keep in mind though - the Intellij support tends to be a little flaky, it took a few tries for the IDEA to start helping with the auto-completions, also I needed to google quite a bit and look at some of the sample projects in gradle kotlin dsl, all in all though this has an awesome potential.

Monday, August 14, 2017

Concourse caching for Java Maven and Gradle builds

Concourse CI 3.3.x has introduced the ability to cache paths between task runs. This feature helps speed up tasks which cache content in specific folders - here I will demonstrate how this feature can be used for speeding up maven and gradle based java builds.

The code and the pipeline that I am using for this post is available at my github repo here - https://github.com/bijukunjummen/ci-concourse-caching-sample

Let me start with the gradle build, if I were to build the project using a gradle wrapper using the following command:

1	`./gradlew clean build`

then gradle would download the dependent libraries into a ".gradle" folder in the users home folder by default. This location of this folder can be changed using a "GRADLE_USER_HOME" environment variable, which is what I will be using in a concourse task to control the location of a cached path.

A concourse task which builds my project looks like this:

---
platform: linux
image_resource:
  type: docker-image
  source:
    repository: openjdk
    tag: 8-jdk
inputs:
  - name: repo
outputs:
  - name: out
run:
  path: /bin/bash
  args:
    - repo/ci/tasks/build.sh
 
caches:
  - path: .gradle/
  - path: .m2/
 
params:
  PROJECT_TYPE: 

See the caches parameter is specified as ".gradle" above. So all I have to do now is to ensure that Gradle uses this location as its home folder, which I would do in my build script:

1 2	`export ROOT_FOLDER=$( pwd )` `export GRADLE_USER_HOME="${ROOT_FOLDER}/.gradle"`

The process to cache maven resources for a maven build is along the same lines, maven caches the dependent jars in a location that can be specified in a variety of ways, the one I have used is to specify this location via a dynamically generated settings.xml file the following way:

M2_HOME=${HOME}/.m2
mkdir -p ${M2_HOME}
 
M2_LOCAL_REPO="${ROOT_FOLDER}/.m2"
 
mkdir -p "${M2_LOCAL_REPO}/repository"
 
cat > ${M2_HOME}/settings.xml <<EOF
 
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                          https://maven.apache.org/xsd/settings-1.0.0.xsd">
      <localRepository>${M2_LOCAL_REPO}/repository</localRepository>
</settings>
 
EOF

which is quite a bit of bash scripting, all it is doing is generating a settings.xml with a localRepository tag set to ".m2/repository" folder which is relative to the temporary folder created by concourse for the build and thus can be cached.

With these changes in place, the behavior is that the downloads happen for the first run of the task but then get cached for subsequent runs. In my local concourse set-up a gradle build taking about 2 mins for a first time build takes about 20 seconds for a subsequent build !

You can try out this feature in my demo project here - https://github.com/bijukunjummen/ci-concourse-caching-sample

UPDATE: 06/08/2018

A far simpler approach to caching maven and gradle dependencies, than what I have detailed previously, is possible and this is based on an approach followed by Spring Cloud Pipelines project. My github sample already contains this change.

Again like before, first indicate in the task the folders(maven and gradle) that need to be cached across different runs of the task, remember that the caching is scoped to a task and a worker:

---
platform: linux
image_resource:
  type: docker-image
  source:
    repository: openjdk
    tag: 8-jdk
inputs:
  - name: repo
outputs:
  - name: out
run:
  path: /bin/bash
  args:
    - repo/ci/tasks/build.sh
 
caches:
  - path: gradle
  - path: maven
 
params:
  PROJECT_TYPE: 

Gradle expects the cached content to be available at "${HOME}/.gradle" folder and Maven at "${HOME}/.m2" folder, the trick then is to simply create a symbolic link of the cached folders to these folders, the following way in a bash script:

export ROOT_FOLDER=$( pwd )
export REPO=repo
 
M2_HOME="${HOME}/.m2"
M2_CACHE="${ROOT_FOLDER}/maven"
GRADLE_HOME="${HOME}/.gradle"
GRADLE_CACHE="${ROOT_FOLDER}/gradle"
 
echo "Generating symbolic links for caches"
 
[[ -d "${M2_CACHE}" && ! -d "${M2_HOME}" ]] && ln -s "${M2_CACHE}" "${M2_HOME}"
[[ -d "${GRADLE_CACHE}" && ! -d "${GRADLE_HOME}" ]] && ln -s "${GRADLE_CACHE}" "${GRADLE_HOME}"

And that should be it, since Maven and Gradle see their default local repositories the caching of dependencies should just work without any additional changes needed!, this is far simpler than the approach that I have previously described.