Tracing a reactive flow - Using Spring Cloud Sleuth with Boot 2

Spring Cloud Sleuth which adds Spring instrumentation support on top of OpenZipkin Brave makes distributed tracing trivially simple for Spring Boot applications. This is a quick write up on what it takes to add support for distributed tracing using this excellent library.

Consider two applications - a client application which uses an upstream service application, both using Spring WebFlux, the reactive web stack for Spring:

My objective is to ensure that flows from user to the client application to the service application can be traced and latencies cleanly recorded for requests.


The final topology that Spring Cloud Sleuth enables is the following:

The sampled trace information from the client and the service app is exported to Zipkin via a queuing mechanism like RabbitMQ.


So what are the changes required to the client and the service app - like I said it is trivially simple! The following libraries need to be pulled in - in my case via gradle:

compile("org.springframework.cloud:spring-cloud-starter-sleuth")
 compile("org.springframework.cloud:spring-cloud-starter-zipkin")
 compile("org.springframework.amqp:spring-rabbit")

The versions are not specified as they are expected to be pulled in via Spring Cloud BOM and thanks to Spring Gradle Dependency Management plugin:

ext {
    springCloudVersion = 'Finchley.RELEASE'
}

apply plugin: 'io.spring.dependency-management'

dependencyManagement {
    imports {
        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${springCloudVersion}"
    }
}

And that is actually it, any logs from the application should now start recording the trace and the spans, see how he traceid carried forward in the following logs spanning two different services:

2018-06-22 04:06:28.579  INFO [sample-client-app,c3d507df405b8aaf,c3d507df405b8aaf,true] 9 --- [server-epoll-13] sample.load.PassThroughHandler           : handling message: Message(id=null, payload=Test, delay=1000)
2018-06-22 04:06:28.586  INFO [sample-service-app,c3d507df405b8aaf,829fde759da15e63,true] 8 --- [server-epoll-11] sample.load.MessageHandler               : Handling message: Message(id=5e7ba240-f97d-405a-9633-5540bbfe0df1, payload=Test, delay=1000)

Further the Zipkin UI records the exported information and can visually show a sample trace the following way:

This sample is available in my github repository here - https://github.com/bijukunjummen/sleuth-webflux-sample and can be started up easily using docker-compose with all the dependencies wired in.

Zuul 2 - Sample filter

Zuul 2 has finally been open sourced. I first heard of Zuul 2 during the Spring One 2016 talk by Mikey Cohen which is available here, it is good to finally be able to play with it.

To quickly touch on the purpose of a Gateway like Zuul 2 - Gateways provide an entry point to an ecosystem of microservices. Since all the customer requests are routed through the Gateway, it can control aspects of routing, request and response flowing through it:

• Routing based on different criteria - uri patterns, headers etc.
• Monitors service health
• Loadbalancing and throttling requests to origin servers
• Security
• Canary testing

My objective in this post is simple - to write a Zuul2 filter that can remove a path prefix and send a request to a downstream service and back.

Zuul2 filters are the mechanism by which Zuul is customized. Say if a client sends a request to /passthrough/someapi call, then I want the Zuul 2 layer to forward the request to a downstream service using /someapi uri. Zuul2 filters are typically packaged up as groovy files and are dynamically loaded(and potentially refreshed) and applied. My sample here will be a little different though, my filters are coded in Java and I had to bypass the loading mechanism built into Zuul.

It may be easier simply to follow the code, which is available in my github repository here - https://github.com/bijukunjummen/boot2-load-demo/tree/master/applications/zuul2-sample, it is packaged in with a set of samples which provide a similar functionality. The code is based on the Zuul 2 samples available here.



This is how my filter looks:

import com.netflix.zuul.context.SessionContext;
import com.netflix.zuul.filters.http.HttpInboundSyncFilter;
import com.netflix.zuul.message.http.HttpRequestMessage;

import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;

public class StripPrefixFilter extends HttpInboundSyncFilter {
    private final List<String> prefixPatterns;

    public StripPrefixFilter(List<String> prefixPatterns) {
        this.prefixPatterns = prefixPatterns;
    }

    @Override
    public HttpRequestMessage apply(HttpRequestMessage input) {
        SessionContext context = input.getContext();
        String path = input.getPath();
        String[] parts = path.split("/");
        if (parts.length > 0) {
            String targetPath = Arrays.stream(parts)
                    .skip(1).collect(Collectors.joining("/"));
            context.set("overrideURI", targetPath);
        }
        return input;
    }

    @Override
    public int filterOrder() {
        return 501;
    }

    @Override
    public boolean shouldFilter(HttpRequestMessage msg) {
        for (String target: prefixPatterns) {
            if (msg.getPath().matches(target)) {
                return true;
            }
        }
        return false;
    }
}

It extends "HttpInboundSyncFilter", these are filters which handle the request inbound to origin servers. As you can imagine there is a "HttpOutboundSyncFilter" which intercept calls outbound from the origin servers. There is a "HttpInboundFilter" and "HttpOutboundFilter" counterpart to these "sync" filters, they return RxJava Observable type.

There is a magic string "overrideUri" in my filter implementation. If you are curious about how I found that to be the override uri, it is by scanning through the Zuul2 codebase. There is likely a lot of filters used internally at Netflix which haven't been released for general consumption yet.

With this filter in place, I have bypassed the dynamic groovy scripts loading feature of Zuul2 by explicitly registering my custom filter using this component:

import com.netflix.zuul.filters.FilterRegistry;
import com.netflix.zuul.filters.ZuulFilter;

import javax.annotation.PostConstruct;
import javax.inject.Inject;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;

public class FiltersRegisteringService {

    private final List<ZuulFilter> filters;
    private final FilterRegistry filterRegistry;

    @Inject
    public FiltersRegisteringService(FilterRegistry filterRegistry, Set<ZuulFilter> filters) {
        this.filters = new ArrayList<>(filters);
        this.filterRegistry = filterRegistry;
    }

    public List<ZuulFilter> getFilters() {
        return filters;
    }

    @PostConstruct
    public void initialize() {
        for (ZuulFilter filter: filters) {
            this.filterRegistry.put(filter.filterName(), filter);
        }
    }
}

I had to make a few more minor tweaks to get this entire set-up with my custom filter bootstrapped, these can be followed in the github repo


Once the Zuul2 sample with this custom filter is started up, the behavior is that any request to /passthrough/messages is routed to a downstream system after the prefix "/passthrough" is stipped out. The instructions to start-up the Zuul 2 app is part of the README of the repo.

This concludes a quick intro to writing a custom Zuul2 filter, I hope this gives just enough of a feel to evaluate Zuul 2.