Concourse caching for Java Maven and Gradle builds

Concourse CI 3.3.x has introduced the ability to cache paths between task runs. This feature helps speed up tasks which cache content in specific folders - here I will demonstrate how this feature can be used for speeding up maven and gradle based java builds.

The code and the pipeline that I am using for this post is available at my github repo here - https://github.com/bijukunjummen/ci-concourse-caching-sample

Let me start with the gradle build, if I were to build the project using a gradle wrapper using the following command:

./gradlew clean build

then gradle would download the dependent libraries into a ".gradle" folder in the users home folder by default. This location of this folder can be changed using a "GRADLE_USER_HOME" environment variable, which is what I will be using in a concourse task to control the location of a cached path.

A concourse task which builds my project looks like this:

---
platform: linux
image_resource:
  type: docker-image
  source:
    repository: openjdk
    tag: 8-jdk
inputs:
  - name: repo
outputs:
  - name: out
run:
  path: /bin/bash
  args:
    - repo/ci/tasks/build.sh

caches:
  - path: .gradle/
  - path: .m2/

params:
  PROJECT_TYPE: 

See the caches parameter is specified as ".gradle" above. So all I have to do now is to ensure that Gradle uses this location as its home folder, which I would do in my build script:

export ROOT_FOLDER=$( pwd )
export GRADLE_USER_HOME="${ROOT_FOLDER}/.gradle"

The process to cache maven resources for a maven build is along the same lines, maven caches the dependent jars in a location that can be specified in a variety of ways, the one I have used is to specify this location via a dynamically generated settings.xml file the following way:

M2_HOME=${HOME}/.m2
mkdir -p ${M2_HOME}

M2_LOCAL_REPO="${ROOT_FOLDER}/.m2"

mkdir -p "${M2_LOCAL_REPO}/repository"

cat > ${M2_HOME}/settings.xml <<EOF

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                          https://maven.apache.org/xsd/settings-1.0.0.xsd">
      <localRepository>${M2_LOCAL_REPO}/repository</localRepository>
</settings>

EOF

which is quite a bit of bash scripting, all it is doing is generating a settings.xml with a localRepository tag set to ".m2/repository" folder which is relative to the temporary folder created by concourse for the build and thus can be cached.

With these changes in place, the behavior is that the downloads happen for the first run of the task but then get cached for subsequent runs. In my local concourse set-up a gradle build taking about 2 mins for a first time build takes about 20 seconds for a subsequent build !

You can try out this feature in my demo project here - https://github.com/bijukunjummen/ci-concourse-caching-sample


UPDATE: 06/08/2018

A far simpler approach to caching maven and gradle dependencies, than what I have detailed previously, is possible and this is based on an approach followed by Spring Cloud Pipelines project. My github sample already contains this change.

Again like before, first indicate in the task the folders(maven and gradle) that need to be cached across different runs of the task, remember that the caching is scoped to a task and a worker:

---
platform: linux
image_resource:
  type: docker-image
  source:
    repository: openjdk
    tag: 8-jdk
inputs:
  - name: repo
outputs:
  - name: out
run:
  path: /bin/bash
  args:
    - repo/ci/tasks/build.sh

caches:
  - path: gradle
  - path: maven

params:
  PROJECT_TYPE: 

Gradle expects the cached content to be available at "${HOME}/.gradle" folder and Maven at "${HOME}/.m2" folder, the trick then is to simply create a symbolic link of the cached folders to these folders, the following way in a bash script:

export ROOT_FOLDER=$( pwd )
export REPO=repo

M2_HOME="${HOME}/.m2"
M2_CACHE="${ROOT_FOLDER}/maven"
GRADLE_HOME="${HOME}/.gradle"
GRADLE_CACHE="${ROOT_FOLDER}/gradle"

echo "Generating symbolic links for caches"

[[ -d "${M2_CACHE}" && ! -d "${M2_HOME}" ]] && ln -s "${M2_CACHE}" "${M2_HOME}"
[[ -d "${GRADLE_CACHE}" && ! -d "${GRADLE_HOME}" ]] && ln -s "${GRADLE_CACHE}" "${GRADLE_HOME}"

And that should be it, since Maven and Gradle see their default local repositories the caching of dependencies should just work without any additional changes needed!, this is far simpler than the approach that I have previously described.

Reasons to consider spring-boot for your next Spring based application!

Spring-boot  provides a quick way to create a Spring based application. There are some very compelling reasons to consider spring-boot for your next project:

Reason 1 : Simpler Dependency management using spring-boot starter projects.

Consider the effort required to start up a CRUD web application using Spring-boot, assuming that the CRUD is implemented using a h2 database with Spring-Data providing the data access abstraction. The maven pom required for such a project is the following:

<?xml version="1.0" encoding="UTF-8"?>
<project ...

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>0.5.0.BUILD-SNAPSHOT</version>
    </parent>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-jpa</artifactId>
        </dependency>
  <dependency>
   <groupId>org.thymeleaf</groupId>
   <artifactId>thymeleaf-spring3</artifactId>
  </dependency>
  <dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-jetty</artifactId>
  </dependency>  
  <dependency>
   <groupId>org.hsqldb</groupId>
   <artifactId>hsqldb</artifactId>
   <scope>runtime</scope>
  </dependency>  
  <dependency>
   <groupId>junit</groupId>
   <artifactId>junit</artifactId>
   <scope>test</scope>
  </dependency>
  <dependency>
   <groupId>org.hamcrest</groupId>
   <artifactId>hamcrest-library</artifactId>
   <scope>test</scope>
  </dependency>      
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>           
        </plugins>
    </build>

</project>

I have removed a few things for brevity.

Now, compare this to a traditional pom.xml where all the dependencies are typically spelled out:

.....

 <dependencies>
  <dependency>
   <groupId>junit</groupId>
   <artifactId>junit</artifactId>
   <version>4.11</version>
   <scope>test</scope>
  </dependency>
  <dependency>
   <groupId>org.hamcrest</groupId>
   <artifactId>hamcrest-core</artifactId>
   <version>1.2.1</version>
  </dependency>
  <dependency>
   <groupId>org.hamcrest</groupId>
   <artifactId>hamcrest-library</artifactId>
   <version>1.2.1</version>
  </dependency>

  <dependency>
   <groupId>log4j</groupId>
   <artifactId>log4j</artifactId>
   <version>1.2.16</version>
  </dependency>
  <dependency>
   <groupId>org.slf4j</groupId>
   <artifactId>slf4j-api</artifactId>
   <version>${slf4j.version}</version>
  </dependency>
  <dependency>
   <groupId>org.slf4j</groupId>
   <artifactId>jcl-over-slf4j</artifactId>
   <version>${slf4j.version}</version>
  </dependency>
  <dependency>
   <groupId>org.slf4j</groupId>
   <artifactId>slf4j-log4j12</artifactId>
   <version>${slf4j.version}</version>
  </dependency>
  <dependency>
   <groupId>org.aspectj</groupId>
   <artifactId>aspectjrt</artifactId>
   <version>${aspectj.version}</version>
  </dependency>
  <dependency>
   <groupId>org.aspectj</groupId>
   <artifactId>aspectjweaver</artifactId>
   <version>${aspectj.version}</version>
  </dependency>
        <!-- dependency>
            <groupId>javax.servlet</groupId>
            <artifactId>servlet-api</artifactId>
            <version>2.5</version>
            <scope>provided</scope>
        </dependency -->
        <dependency>
            <groupId>javax.servlet</groupId>
            <artifactId>javax.servlet-api</artifactId>
            <version>3.0.1</version>
            <scope>provided</scope>
        </dependency>        
  <dependency>
   <groupId>net.sf.flexjson</groupId>
   <artifactId>flexjson</artifactId>
   <version>2.0</version>
  </dependency>
  <!-- Spring dependencies -->
  <dependency>
   <groupId>org.springframework</groupId>
   <artifactId>spring-core</artifactId>
   <version>${spring.version}</version>
   <exclusions>
    <exclusion>
     <groupId>commons-logging</groupId>
     <artifactId>commons-logging</artifactId>
    </exclusion>
   </exclusions>
  </dependency>
  <dependency>
   <groupId>org.springframework</groupId>
   <artifactId>spring-test</artifactId>
   <version>${spring.version}</version>
   <scope>test</scope>
   <exclusions>
    <exclusion>
     <groupId>commons-logging</groupId>
     <artifactId>commons-logging</artifactId>
    </exclusion>
   </exclusions>
  </dependency>
  <dependency>
   <groupId>org.springframework</groupId>
   <artifactId>spring-context</artifactId>
   <version>${spring.version}</version>
  </dependency>
  <dependency>
   <groupId>org.springframework</groupId>
   <artifactId>spring-aop</artifactId>
   <version>${spring.version}</version>
  </dependency>
  <dependency>
   <groupId>org.springframework</groupId>
   <artifactId>spring-aspects</artifactId>
   <version>${spring.version}</version>
  </dependency>
  <dependency>
   <groupId>org.springframework</groupId>
   <artifactId>spring-tx</artifactId>
   <version>${spring.version}</version>
  </dependency>
  <dependency>
   <groupId>com.h2database</groupId>
   <artifactId>h2</artifactId>
   <version>${h2.version}</version>
  </dependency>
  <dependency>
   <groupId>org.hibernate</groupId>
   <artifactId>hibernate-core</artifactId>
   <version>${hibernate.version}</version>
  </dependency>
  <dependency>
   <groupId>org.hibernate</groupId>
   <artifactId>hibernate-entitymanager</artifactId>
   <version>${hibernate.version}</version>
   <exclusions>
    <exclusion>
     <groupId>cglib</groupId>
     <artifactId>cglib</artifactId>
    </exclusion>
    <exclusion>
     <groupId>dom4j</groupId>
     <artifactId>dom4j</artifactId>
    </exclusion>
   </exclusions>
  </dependency>
  <dependency>
   <groupId>org.hibernate.javax.persistence</groupId>
   <artifactId>hibernate-jpa-2.0-api</artifactId>
   <version>1.0.0.Final</version>
  </dependency>
  <dependency>
   <groupId>org.hibernate</groupId>
   <artifactId>hibernate-validator</artifactId>
   <version>4.3.0.Final</version>
   <exclusions>
    <exclusion>
     <groupId>javax.xml.bind</groupId>
     <artifactId>jaxb-api</artifactId>
    </exclusion>
    <exclusion>
     <groupId>com.sun.xml.bind</groupId>
     <artifactId>jaxb-impl</artifactId>
    </exclusion>
   </exclusions>
  </dependency>
  <dependency>
   <groupId>javax.validation</groupId>
   <artifactId>validation-api</artifactId>
   <version>1.0.0.GA</version>
  </dependency>

  <dependency>
   <groupId>javax.transaction</groupId>
   <artifactId>jta</artifactId>
   <version>1.1</version>
  </dependency>
  <dependency>
   <groupId>org.springframework</groupId>
   <artifactId>spring-jdbc</artifactId>
   <version>${spring.version}</version>
  </dependency>
  <dependency>
   <groupId>org.springframework</groupId>
   <artifactId>spring-orm</artifactId>
   <version>${spring.version}</version>
  </dependency>
  <dependency>
   <groupId>commons-pool</groupId>
   <artifactId>commons-pool</artifactId>
   <version>1.5.4</version>
   <exclusions>
    <exclusion>
     <groupId>commons-logging</groupId>
     <artifactId>commons-logging</artifactId>
    </exclusion>
   </exclusions>
  </dependency>
  <dependency>
   <groupId>commons-io</groupId>
   <artifactId>commons-io</artifactId>
   <version>2.4</version>
  </dependency>  
  <dependency>
   <groupId>commons-dbcp</groupId>
   <artifactId>commons-dbcp</artifactId>
   <version>1.3</version>
   <exclusions>
    <exclusion>
     <groupId>commons-logging</groupId>
     <artifactId>commons-logging</artifactId>
    </exclusion>
    <exclusion>
     <groupId>commons-pool</groupId>
     <artifactId>commons-pool</artifactId>
    </exclusion>
    <exclusion>
     <groupId>xerces</groupId>
     <artifactId>xerces</artifactId>
    </exclusion>
    <exclusion>
     <groupId>xerces</groupId>
     <artifactId>xercesImpl</artifactId>
    </exclusion>
    <exclusion>
     <groupId>xml-apis</groupId>
     <artifactId>xml-apis</artifactId>
    </exclusion>
   </exclusions>
  </dependency>
  <!-- Jackson JSON Processor -->
  <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-core</artifactId>
      <version>2.2.1</version>
  </dependency>
  <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-databind</artifactId>
      <version>2.2.1</version>
  </dependency>
  <dependency>
      <groupId>com.fasterxml.jackson.module</groupId>
      <artifactId>jackson-module-jaxb-annotations</artifactId>
      <version>2.2.1</version>
  </dependency>
  <dependency>
   <groupId>org.apache.tiles</groupId>
   <artifactId>tiles-core</artifactId>
   <version>2.2.1</version>
   <exclusions>
    <exclusion>
     <groupId>commons-logging</groupId>
     <artifactId>commons-logging</artifactId>
    </exclusion>
   </exclusions>
  </dependency>
  <dependency>
   <groupId>org.apache.tiles</groupId>
   <artifactId>tiles-jsp</artifactId>
   <version>2.2.1</version>
  </dependency>
  <dependency>
   <groupId>org.springframework</groupId>
   <artifactId>spring-web</artifactId>
   <version>${spring.version}</version>
   <exclusions>
    <exclusion>
     <groupId>commons-logging</groupId>
     <artifactId>commons-logging</artifactId>
    </exclusion>
   </exclusions>
  </dependency>
  <dependency>
   <groupId>org.springframework</groupId>
   <artifactId>spring-webmvc</artifactId>
   <version>${spring.version}</version>
   <exclusions>
    <exclusion>
     <groupId>commons-logging</groupId>
     <artifactId>commons-logging</artifactId>
    </exclusion>
   </exclusions>
  </dependency>
  <dependency>
   <groupId>commons-digester</groupId>
   <artifactId>commons-digester</artifactId>
   <version>2.0</version>
   <exclusions>
    <exclusion>
     <groupId>commons-logging</groupId>
     <artifactId>commons-logging</artifactId>
    </exclusion>
   </exclusions>
  </dependency>
  <dependency>
   <groupId>commons-fileupload</groupId>
   <artifactId>commons-fileupload</artifactId>
   <version>1.2.1</version>
   <exclusions>
    <exclusion>
     <groupId>commons-logging</groupId>
     <artifactId>commons-logging</artifactId>
    </exclusion>
   </exclusions>
  </dependency>
  <dependency>
   <groupId>org.apache.commons</groupId>
   <artifactId>commons-lang3</artifactId>
   <version>3.1</version>
  </dependency>
  <dependency>
   <groupId>javax.servlet</groupId>
   <artifactId>jstl</artifactId>
   <version>1.2</version>
  </dependency>
  <dependency>
   <groupId>javax.el</groupId>
   <artifactId>el-api</artifactId>
   <version>1.0</version>
   <scope>provided</scope>
  </dependency>
  <dependency>
   <groupId>joda-time</groupId>
   <artifactId>joda-time</artifactId>
   <version>1.6</version>
  </dependency>
  <dependency>
   <groupId>javax.servlet.jsp</groupId>
   <artifactId>jsp-api</artifactId>
   <version>2.1</version>
   <scope>provided</scope>
  </dependency>
  <dependency>
   <groupId>org.springframework.security</groupId>
   <artifactId>spring-security-core</artifactId>
   <version>${spring-security.version}</version>
   <exclusions>
    <exclusion>
     <groupId>commons-logging</groupId>
     <artifactId>commons-logging</artifactId>
    </exclusion>
   </exclusions>
  </dependency>
  <dependency>
   <groupId>org.springframework.security</groupId>
   <artifactId>spring-security-config</artifactId>
   <version>${spring-security.version}</version>
   <exclusions>
    <exclusion>
     <groupId>commons-logging</groupId>
     <artifactId>commons-logging</artifactId>
    </exclusion>
   </exclusions>
  </dependency>
  <dependency>
   <groupId>org.springframework.security</groupId>
   <artifactId>spring-security-web</artifactId>
   <version>${spring-security.version}</version>
  </dependency>
  <dependency>
   <groupId>org.springframework.security</groupId>
   <artifactId>spring-security-taglibs</artifactId>
   <version>${spring-security.version}</version>
  </dependency>
  <dependency>
      <groupId>com.googlecode.flyway</groupId>
      <artifactId>flyway-core</artifactId>
      <version>1.7</version>
  </dependency>
  <dependency>
   <groupId>org.springframework.data</groupId>
   <artifactId>spring-data-jpa</artifactId>
   <version>${spring-data-jpa.version}</version>
  </dependency>   
 
    
 </dependencies>
...

This is indeed a massive improvement in the way Spring specific dependencies are managed.

Reason 2: Quick standalone mode with auto configuration
All that needs to be done to start up a working web application is to run - "mvn spring:boot" and Spring boot does the rest. It provides a way to quickly run the application with an embedded tomcat or jetty container and auto-configures the application based on the libraries available in the classpath without needing any explicit configuration to be provided by the user - for eg. if it finds hsqldb libraries in the classpath it automatically configures a datasource with hsqldb as the embedded database, if it finds thymeleaf jars in the classpath it automatically configures thymeleaf as the templating engine, if it finds hibernate libraries in the classpath then it automatically uses hibernate as the JPA provider. This allows for a progressive enhancement in the development of a working application - Spring boot provides a way for a developer to quickly bootstrap and prototype the application and then slowly replace some of the auto-configured components with more robust alternatives - say a Postgres DB instead of hsqldb etc.

Reason 3: Opinionated Dependencies
Spring boot provides an opinionated and well tested set of dependencies that work well with the Spring ecosystem of projects. for eg. logback with slf4j as the logging dependency, using Servlet 3.0+ apis, Jackson 2.2 for json handling, thymeleaf for web page templating.


Conclusion:
The Spring-boot project is still under heavy development but even in its current state it holds a lot of potential in greatly simplifying application development using Spring umbrella of projects.


Here is a small reference application using Spring-boot - https://github.com/bijukunjummen/spring-boot-mvc-test.git

Debugging with Maven

For Non-Forked Process

The way to debug any project using maven is to use "mvnDebug" command line tool instead of "mvn". So, if I want to debug a tomcat based project, I would do:

mvnDebug tomcat:run

which would enable the debugging on the JVM and would wait for the debugger to connect to the vm:

By default, the JDWP port that mvnDebug listens is on 8000. 

The next step is to connect to this vm through the debugger in Eclipse - 

Go to Run->Debug Configurations->Remote Java Application and create a new "launch configuration" along these lines:

Click on debug and eclipse should connect to the maven vm and the tomcat should start up at this point. 

For Forked Process

The above approach will however not work for forked processes like JUnit tests - tests by default are forked by maven. 

There are two workarounds for debugging JUnit tests:

To prevent forking of Junit tests, this can be done using a forkMode parameter this way:

mvnDebug test -Dtest=GtdProjectDaoIntegrationTest -DforkMode=never

The second workaround is to use the "maven.surefire.debug" property:

mvn -Dmaven.surefire.debug test -Dtest=GtdProjectDaoIntegrationTest

This would, by default, start up the debugger at port 5005. A variation of this is to explicitly specify the port where the debugger is to be started and with additional JDWP options:

mvn -Dmaven.surefire.debug="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 -Xnoagent -Djava.compiler=NONE" test

References
http://maven.apache.org/plugins/maven-surefire-plugin/examples/debugging.html