Kotlintest and property based testing

I was very happy to see that Kotlintest, a port of the excellent scalatest in Kotlin, supports property based testing.

I was introduced to property based testing through the excellent "Functional programming in Scala" book.

The idea behind property based testing is simple - the behavior of a program is described as a property and the testing framework generates random data to validate the property. This is best illustrated with an example using the excellent scalacheck library:

import org.scalacheck.Prop.forAll
import org.scalacheck.Properties

object ListSpecification extends Properties("List") {
  property("reversing a list twice should return the list") = forAll { (a: List[Int]) =>
    a.reverse.reverse == a
  }
}

scalacheck would generate a random list(of integer) of varying sizes and would validate that this property holds for the lists. A similar specification expressed through Kotlintest looks like this:

import io.kotlintest.properties.forAll
import io.kotlintest.specs.StringSpec


class ListSpecification : StringSpec({
    "reversing a list twice should return the list" {
        forAll{ list: List<Int> ->
            list.reversed().reversed().toList() == list
        }
    }
})

If the generators have to be a little more constrained, say if we wanted to test this behavior on lists of integer in the range 1 to 1000 then an explicit generator can be passed in the following way, again starting with scalacheck:

import org.scalacheck.Prop.forAll
import org.scalacheck.{Gen, Properties}

object ListSpecification extends Properties("List") {
  val intList = Gen.listOf(Gen.choose(1, 1000))
  property("reversing a list twice should return the list") = forAll(intList) { (a: List[Int]) =>
    a.reverse.reverse == a
  }
}

and an equivalent kotlintest code:

import io.kotlintest.properties.Gen
import io.kotlintest.properties.forAll
import io.kotlintest.specs.StringSpec

class BehaviorOfListSpecs : StringSpec({
    "reversing a list twice should return the list" {
        val intList = Gen.list(Gen.choose(1, 1000))

        forAll(intList) { list ->
            list.reversed().reversed().toList() == list
        }
    }
})

Given this let me now jump onto another example from the scalacheck site, this time to illustrate a failure:

import org.scalacheck.Prop.forAll
import org.scalacheck.Properties

object StringSpecification extends Properties("String") {

  property("startsWith") = forAll { (a: String, b: String) =>
    (a + b).startsWith(a)
  }

  property("concatenate") = forAll { (a: String, b: String) =>
    (a + b).length > a.length && (a + b).length > b.length
  }

  property("substring") = forAll { (a: String, b: String, c: String) =>
    (a + b + c).substring(a.length, a.length + b.length) == b
  }
}

the second property described above is wrong - if two strings are concatenated together they are ALWAYS larger than each of the parts, this is not true if one of the strings is blank. If I were to run this test using scalacheck it correctly catches this wrongly specified behavior:

+ String.startsWith: OK, passed 100 tests.
! String.concatenate: Falsified after 0 passed tests.
> ARG_0: ""
> ARG_1: ""
+ String.substring: OK, passed 100 tests.
Found 1 failing properties.

An equivalent kotlintest is the following:

import io.kotlintest.properties.forAll
import io.kotlintest.specs.StringSpec

class StringSpecification : StringSpec({
    "startsWith" {
        forAll { a: String, b: String ->
            (a + b).startsWith(a)
        }
    }

    "concatenate" {
        forAll { a: String, b: String ->
            (a + b).length > a.length && (a + b).length > b.length
        }
    }

    "substring" {
        forAll { a: String, b: String, c: String ->
            (a + b + c).substring(a.length, a.length + b.length) == b
        }
    }
})

on running, it correctly catches the issue with concatenate and produces the following result:

java.lang.AssertionError: Property failed for

Y{_DZ<vGnzLQHf9|3$i|UE,;!%8^SRF;JX%EH+<5d:p`Y7dxAd;I+J5LB/:O)

 at io.kotlintest.properties.PropertyTestingKt.forAll(PropertyTesting.kt:27)

However there is an issue here, scalacheck found a simpler failure case, it does this by a process called "Test Case minimization" where in case of a failure it tries to find the smallest test case that can fail, something that the Kotlintest can learn from.


There are other features where Kotlintest lags with respect to scalacheck, a big one being able to combine generators:

case class Person(name: String, age: Int)

val genPerson = for {
  name <- Gen.alphaStr
  age <- Gen.choose(1, 50)
} yield Person(name, age)

genPerson.sample

However all in all, I have found the DSL of Kotlintest and its support for property based testing to be a good start so far and look forward to how this library evolves over time.

If you want to play with these samples a little more, it is available in my github repo here - https://github.com/bijukunjummen/kotlintest-scalacheck-sample

Cloud Foundry Application manifest using Kotlin DSL

I had a blast working with and getting my head around the excellent support for creating DSL's in Kotlin Language.
Kotlin DSL is now being used for creating gradle build files, for defining routes in Spring Webflux, for creating html templates using kotlinx.html library.

Here I am going to demonstrate creating a kotlin based DSL to represent a Cloud Foundry Application Manifest content.

A sample manifest looks like this when represented as a yaml file:

applications:
 - name: myapp
   memory: 512M
   instances: 1
   path: target/someapp.jar
   routes:
     - somehost.com
     - antother.com/path
   envs:
    ENV_NAME1: VALUE1
    ENV_NAME2: VALUE2

And here is the kind of DSL I am aiming for:

cf {
    name = "myapp"
    memory = 512(M)
    instances = 1
    path = "target/someapp.jar"
    routes {
        +"somehost.com"
        +"another.com/path"
    }
    envs {
        env["ENV_NAME1"] = "VALUE1"
        env["ENV_NAME2"] = "VALUE2"
    }
}

Getting the basic structure

Let me start with a simpler structure that looks like this:

cf {
    name = "myapp"
    instances = 1
    path = "target/someapp.jar"
}

and want this kind of a DSL to map to a structure which looks like this:

data class CfManifest(
        var name: String = "",
        var instances: Int? = 0,
        var path: String? = null
)

It would translate to a Kotlin function which takes a Lambda expression:

fun cf(init: CfManifest.() -> Unit) {
 ...
}

The parameter which looks like this:

() -> Unit

is fairly self-explanatory, a lambda expression which does not take any parameters and does not return anything.

The part that took a while to seep into my mind is this modified lambda expression, referred to as a Lambda expression with receiver:

CfManifest.() -> Unit

It does two things the way I have understood it:

1. It defines in the scope of the wrapped function an extension function for the receiver type - in my case the CfManifest class
2. this within the lambda expression now refers to the receiver function.

Given this, the cf function translates to :

fun cf(init: CfManifest.() -> Unit): CfManifest {
    val manifest = CfManifest()
    manifest.init()
    return manifest
}

which can be succinctly expressed as:

fun cf(init: CfManifest.() -> Unit) = CfManifest().apply(init)

so now when I call:

cf {
    name = "myapp"
    instances = 1
    path = "target/someapp.jar"
}

It translates to:

CFManifest().apply {
  this.name = "myapp"
  this.instances = 1
  this.path = "target/someapp.jar"
}

More DSL

Expanding on the basic structure:

cf {
    name = "myapp"
    memory = 512(M)
    instances = 1
    path = "target/someapp.jar"
    routes {
        +"somehost.com"
        +"another.com/path"
    }
    envs {
        env["ENV_NAME1"] = "VALUE1"
        env["ENV_NAME2"] = "VALUE2"
    }
}

The routes and the envs in turn become methods on the CfManifest class and look like this:

data class CfManifest(
        var name: String = "",
        var path: String? = null,
        var memory: MEM? = null,
        ...
        var routes: ROUTES? = null,
        var envs: ENVS = ENVS()
) {

    fun envs(block: ENVS.() -> Unit) {
        this.envs = ENVS().apply(block)
    }

    ...

    fun routes(block: ROUTES.() -> Unit) {
        this.routes = ROUTES().apply(block)
    }
}

data class ENVS(
        var env: MutableMap<String, String> = mutableMapOf()
)

data class ROUTES(
        private val routes: MutableList<String> = mutableListOf()
) {
    operator fun String.unaryPlus() {
        routes.add(this)
    }
}

See how the routes method takes in a Lambda expression with a receiver type of ROUTES, this allows me to define an expression like this:

cf {
    ...
    routes {
        +"somehost.com"
        +"another.com/path"
    }
    ...
}

Another trick here is way a route is being added is using :

+"somehost.com"

which is enabled using a Kotlin convention which translates specific method names to operators, here the unaryPlus method. The cool thing for me is that this operator is visible only in the scope of ROUTES instance!


Another feature of the DSL making use of Kotlin features is the way a memory is specified, there are two parts to it - a number and the modifier, 2G, 500M etc.
This is being specified in a slightly modified way via the DSL as 2(G) and 500(M).

The way it is implemented is using another Kotlin convention where if a class has an invoke method then instances can call it the following way:

class ClassWithInvoke() {
    operator fun invoke(n: Int): String = "" + n
}
val c = ClassWithInvoke()
c(10)

So implementing invoke method as an extension function on Int in the scope of the CFManifest class allows this kind of a DSL:

data class CfManifest(
        var name: String = "",
        ...
) {
    ...
    operator fun Int.invoke(m: MemModifier): MEM = MEM(this, m)
}

This is pure experimentation on my part, I am both new to Kotlin as well as Kotlin DSL's so very likely there are a lot of things that can be improved in this implementation, any feedback and suggestions are welcome. You can play with this sample code at my github repo here