Concourse caching for Java Maven and Gradle builds

Concourse CI 3.3.x has introduced the ability to cache paths between task runs. This feature helps speed up tasks which cache content in specific folders - here I will demonstrate how this feature can be used for speeding up maven and gradle based java builds.

The code and the pipeline that I am using for this post is available at my github repo here - https://github.com/bijukunjummen/ci-concourse-caching-sample

Let me start with the gradle build, if I were to build the project using a gradle wrapper using the following command:

./gradlew clean build

then gradle would download the dependent libraries into a ".gradle" folder in the users home folder by default. This location of this folder can be changed using a "GRADLE_USER_HOME" environment variable, which is what I will be using in a concourse task to control the location of a cached path.

A concourse task which builds my project looks like this:

---
platform: linux
image_resource:
  type: docker-image
  source:
    repository: openjdk
    tag: 8-jdk
inputs:
  - name: repo
outputs:
  - name: out
run:
  path: /bin/bash
  args:
    - repo/ci/tasks/build.sh

caches:
  - path: .gradle/
  - path: .m2/

params:
  PROJECT_TYPE: 

See the caches parameter is specified as ".gradle" above. So all I have to do now is to ensure that Gradle uses this location as its home folder, which I would do in my build script:

export ROOT_FOLDER=$( pwd )
export GRADLE_USER_HOME="${ROOT_FOLDER}/.gradle"

The process to cache maven resources for a maven build is along the same lines, maven caches the dependent jars in a location that can be specified in a variety of ways, the one I have used is to specify this location via a dynamically generated settings.xml file the following way:

M2_HOME=${HOME}/.m2
mkdir -p ${M2_HOME}

M2_LOCAL_REPO="${ROOT_FOLDER}/.m2"

mkdir -p "${M2_LOCAL_REPO}/repository"

cat > ${M2_HOME}/settings.xml <<EOF

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                          https://maven.apache.org/xsd/settings-1.0.0.xsd">
      <localRepository>${M2_LOCAL_REPO}/repository</localRepository>
</settings>

EOF

which is quite a bit of bash scripting, all it is doing is generating a settings.xml with a localRepository tag set to ".m2/repository" folder which is relative to the temporary folder created by concourse for the build and thus can be cached.

With these changes in place, the behavior is that the downloads happen for the first run of the task but then get cached for subsequent runs. In my local concourse set-up a gradle build taking about 2 mins for a first time build takes about 20 seconds for a subsequent build !

You can try out this feature in my demo project here - https://github.com/bijukunjummen/ci-concourse-caching-sample


UPDATE: 06/08/2018

A far simpler approach to caching maven and gradle dependencies, than what I have detailed previously, is possible and this is based on an approach followed by Spring Cloud Pipelines project. My github sample already contains this change.

Again like before, first indicate in the task the folders(maven and gradle) that need to be cached across different runs of the task, remember that the caching is scoped to a task and a worker:

---
platform: linux
image_resource:
  type: docker-image
  source:
    repository: openjdk
    tag: 8-jdk
inputs:
  - name: repo
outputs:
  - name: out
run:
  path: /bin/bash
  args:
    - repo/ci/tasks/build.sh

caches:
  - path: gradle
  - path: maven

params:
  PROJECT_TYPE: 

Gradle expects the cached content to be available at "${HOME}/.gradle" folder and Maven at "${HOME}/.m2" folder, the trick then is to simply create a symbolic link of the cached folders to these folders, the following way in a bash script:

export ROOT_FOLDER=$( pwd )
export REPO=repo

M2_HOME="${HOME}/.m2"
M2_CACHE="${ROOT_FOLDER}/maven"
GRADLE_HOME="${HOME}/.gradle"
GRADLE_CACHE="${ROOT_FOLDER}/gradle"

echo "Generating symbolic links for caches"

[[ -d "${M2_CACHE}" && ! -d "${M2_HOME}" ]] && ln -s "${M2_CACHE}" "${M2_HOME}"
[[ -d "${GRADLE_CACHE}" && ! -d "${GRADLE_HOME}" ]] && ln -s "${GRADLE_CACHE}" "${GRADLE_HOME}"

And that should be it, since Maven and Gradle see their default local repositories the caching of dependencies should just work without any additional changes needed!, this is far simpler than the approach that I have previously described.