Service to Service call patterns - GKE with Anthos Service Mesh on a single cluster

 This is second in a series of posts exploring service to service call patterns in some of the application runtimes on Google Cloud. The first in the series explored service to service call patterns in GKE. 

This post will expand on it by adding in a Service Mesh, specifically Anthos Service Mesh, and explore how the service to service patterns change in the presence of a mesh. The service to service call with be across services in a single cluster. The next post will explore services deployed to multiple GKE clusters.

Set-Up

The steps to set-up a GKE cluster and install Anthos service mesh on top of it is described in this document - https://cloud.google.com/service-mesh/docs/unified-install/install, in brief these are the commands that I had to run in my GCP Project to get a cluster running:

	export PROJECT=$(gcloud config get-value project)
	export ZONE="us-west1-a"
	# Create a GKE Standard cluster named "solo"
	gcloud container clusters create solo \
	--project=${PROJECT} \
	--zone=${ZONE} \
	--enable-ip-alias \
	--machine-type=e2-standard-4 \
	--num-nodes=2 \
	--workload-pool=${PROJECT}.svc.id.goog
	# Download the asmcli utility
	curl https://storage.googleapis.com/csm-artifacts/asm/asmcli_1.11 > asmcli
	# Install service mesh on the newly created "solo" cluster
	./asmcli install \
	--project_id ${PROJECT} \
	--cluster_name solo \
	--cluster_location us-west1-a \
	--fleet_id ${PROJECT} \
	--enable_all \
	--ca mesh_ca
	# Determine the ASM revision
	ASM_REVISION=$(kubectl get deploy -n istio-system -l app=istiod -o jsonpath={.items[*].metadata.labels.'istio\.io\/rev'}'{"\n"}')
	# Install an ingress gateway that can work with the service mesh
	kubectl create namespace gw-namespace
	kubectl label namespace gw-namespace \
	istio.io/rev=${ASM_REVISION} --overwrite
	git clone https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages.git
	kubectl apply -n gw-namespace \
	-f anthos-service-mesh-packages/samples/gateways/istio-ingressgateway
	# Create a namespace to host applications with mesh proxy injected in
	kubectl create namespace istio-apps
	kubectl label namespace istio-apps istio-injection- istio.io/rev=${ASM_REVISION} --overwrite

view raw create_gke_cluster_inject_anthos_mesh.sh hosted with ❤ by GitHub

If the installation of cluster and the mesh has run through cleanly, a good way to verify the installation is to see if the cluster gets registered as a Anthos managed cluster in the Google Cloud Console.

The services that I will be installing is fairly simple and looks like this:

Using a UI, the caller can make the producer behave in certain ways:

• Introduce response time delays
• Respond with certain status codes

This will help check how the mesh environment will behave in the face of these behaviors.

The codebase for the "caller" and "producer" are in this repository - https://github.com/bijukunjummen/sample-service-to-service, there are kubernetes manifests available in the repository to bring up these services.

Behavior 1 - Mutual TLS

The first behavior that I want to see is for the the caller and the producer to verify each others identities by presenting and validating their certificates.

This can be done by adding in a istio DestinationRule for the producer, along these lines:

	apiVersion: networking.istio.io/v1alpha3
	kind: DestinationRule
	metadata:
	name: sample-producer-dl
	namespace: istio-apps
	spec:
	host: sample-producer.istio-apps.svc.cluster.local
	trafficPolicy:
	tls:
	mode: ISTIO_MUTUAL
	---
	apiVersion: networking.istio.io/v1alpha3
	kind: DestinationRule
	metadata:
	name: sample-caller-dl
	namespace: istio-apps
	spec:
	host: sample-caller.istio-apps.svc.cluster.local
	trafficPolicy:
	tls:
	mode: ISTIO_MUTUAL

view raw mtls-service-to-service.yaml hosted with ❤ by GitHub

This also adds in the DestinationRule for the caller, this is because the caller gets the call from the browser via an Ingress Gateway and even this call needs to be authenticated using mtls

Alright now that the set-up in place, the following is what gets captured as the request flows from the Browser to the Ingress Gateway to the Caller to the Producer.

The sign that the mTLS works is seeing the "x-forwarded-client-cert" header, this is in both the Callers headers coming in from Ingress-gateway, and in the "Producers" headers coming in from the Caller.

Behavior 2 - Timeout

The second behavior that I want to explore is the timeouts. A request timeout can be set for the call from the Caller to Producer by creating a Virtual Service for the Producer with the value set, along these lines:

	apiVersion: networking.istio.io/v1alpha3
	kind: VirtualService
	metadata:
	name: sample-producer-route
	namespace: istio-apps
	spec:
	hosts:
	- "sample-producer.istio-apps.svc.cluster.local"
	http:
	- timeout: 5s
	route:
	- destination:
	host: sample-producer
	port:
	number: 8080

view raw istio-timeout.yaml hosted with ❤ by GitHub

With this configuration in place a request from the caller with a delay of 6 seconds, causes the Mesh to timeout and present an error that looks like this:

The mesh responds with a http status code of 504 with a message of "Upstream timed out". 

Behavior 3 - Circuit Breaker

Circuit breaker is implemented using a Destination Rule resource

	apiVersion: networking.istio.io/v1alpha3
	kind: DestinationRule
	metadata:
	name: sample-producer-dl
	namespace: istio-apps
	spec:
	host: sample-producer.istio-apps.svc.cluster.local
	trafficPolicy:
	tls:
	mode: ISTIO_MUTUAL
	connectionPool:
	outlierDetection:
	consecutive5xxErrors: 3
	interval: 15s
	baseEjectionTime: 15s

view raw istio-circuit-breaker.yml hosted with ❤ by GitHub

Here I have configuration which breaks the circuit if 3 continuous 5XX responses are received from the Producer in a 15 second interval, and then does not make a request for another 15 seconds

With this configuration in place a request with broken circuit looks like this:

The mesh responds with a http status code of 503 and a message of "no healthy upstream"

Conclusion

The neat thing is that in all scenarios so far, the way the Caller calls the Producer remains exactly the same, it is the mesh which injects in the appropriate security controls through mTLS and the resilience of calling service through timeouts and circuit breaker.