Cloud Deploy with Cloud Run

Google Cloud Deploy is a service to continuously deploy to Google Cloud Application runtimes. It has supported Google Kubernetes Engine(GKE) so far, and now is starting to support Cloud Run. This post is about a quick trial of this new and exciting support in Cloud Deploy. 

It may be simpler to explore the entire sample which is available in my github repo here - https://github.com/bijukunjummen/clouddeploy-cloudrun-sample 

End to end Flow

The sample attempts to do the following:

A Cloud Build based build first builds an image. This image is handed over to Cloud Deploy which deploys to Cloud Run. A "dev" and "prod" target is simulated by the Cloud Run applications having names prefixed with the environment name.

Building an image

There are way too many ways to build a container image, my personal favorite is  the excellent Google jib tool which requires a simple plugin to be in place to create AND publish a container image. Once an image is created, the next task is to get the tagged image name for use with say a Kubernetes deployment manifest. 

Skaffold does a great job of orchestrating these two steps, creating an image and rendering the application runtime manifests with the image locations. Since the deployment is to a Cloud Run environment, the manifest looks something like this:

Now, manifest for each target environment may look a little different, so for eg in my case the application name targeted towards dev environment has a "dev-" prefix and for prod environment has a "prod-" prefix. This is where another tool called Kustomize fits in. Kustomize is fairly intuitive, it expresses the variations for each environment as a patch file, so for eg, in my case where I want to prefix the name of the application in the dev environment with a "dev-", the Kustomize configuration looks something like this:

So now, we have 3 tools:

1. For building an image - Google Jib
2. Generating the manifests based on environment - Kustomize
3. Rending the image name in the manifests - Skaffold

Skaffold does a great job of wiring all the tools together, and looks something like this for my example:

Deploying the Image

In the Google Cloud Environment, Cloud Build is used for calling Skaffold and building the image, I have a cloudbuild.yaml file available with my sample, which shows how skaffold is invoked and the image built.

Let's come to the topic of the post, about deploying this image to Cloud Run using Cloud Deploy. Cloud Deploy uses a configuration file to describe where the image needs to be deployed, which is Cloud Run in this instance and how the deployment needs to be promoted across environments. The environments are referred to as "targets" and look like this in my configuration:

They point to the project and region for the Cloud Run service.

Next is the configuration to describe how the pipeline will take the application through the targets:

This simply shows that application will be first deployed to the "dev" target and then promoted to the "prod" target after approval.

The "profiles" in the each of the stages show the profile that will be activated in skaffold, which simply determines which overlay of kustomize will be used to create the manifest.

That covers the entire Cloud Deploy configuration. The next step once the configuration file is ready is to create the deployment pipeline, which is done using a command which looks like this:

gcloud deploy apply --file=clouddeploy.yaml --region=us-west1

and registers the pipeline with Cloud Deploy service.

So just to quickly recap, I now have the image built by Cloud Build, the manifests generated using skaffold, kustomize, and a pipeline registered with Cloud Deploy, the next step is to trigger the pipeline for the image and the artifacts, which is done through another command, which is hooked up to Cloud Build:

gcloud deploy releases create release-$SHORT_SHA --delivery-pipeline clouddeploy-cloudrun-sample --region us-west1 --build-artifacts artifacts.json

This would trigger the deploy to the different Cloud Run targets - "dev" in my case to start with:

Once deployed, I have a shiny Cloud Run app all ready to accept requests!

This can now be promoted to my "prod" target with a manual approval process:

Conclusion

Cloud Deploy's support for Cloud Run works great, it takes a familiar tooling with Skaffold typically meant for Kubernetes manifests and uses it cleverly for Cloud Run deployment flows. I look forward to more capabilities in Cloud Deploy with support for Blue/Green, Canary deployment models.

The entire working code is available in my github repo here - https://github.com/bijukunjummen/clouddeploy-cloudrun-sample 

Skaffold for Local Java App Development

Skaffold is a tool which handles the workflow of building, pushing and deploying container images and has the added benefit of facilitating an excellent local dev loop. 

In this post I will be exploring using Skaffold for local development of a Java based application

Installing Skaffold

Installing Skaffold locally is straightforward, and explained well here. It works great with minikube as a local kubernetes development environment. 

Skaffold Configuration

My sample application is available in a github repository here - https://github.com/bijukunjummen/hello-skaffold-gke

Skaffold requires at a minimum, a configuration expressed in a skaffold.yml file, with details of 

• How to build an image
• Where to push the image 
• How to deploy the image - Kubernetes artifacts which should be hydrated with the details of the published image and used for deployment.

In my project, the skaffold.yml file looks like this:

apiVersion: skaffold/v2beta16
kind: Config
metadata:
  name: hello-skaffold-gke
build:
  artifacts:
  - image: hello-skaffold-gke
    jib: {}
deploy:
  kubectl:
    manifests:
    - kubernetes/hello-deployment.yaml
    - kubernetes/hello-service.yaml

This tells Skaffold:

• that the container image should be built using the excellent jib tool
• The location of the kubernetes deployment artifacts, in my case a deployment and a service describing the application

The Kubernetes manifests need not hardcode the container image tag, instead  they can use a placeholder which gets hydrated by Skaffold:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-skaffold-gke-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-skaffold-gke
  template:
    metadata:
      labels:
        app: hello-skaffold-gke
    spec:
      containers:
        - name: hello-skaffold-gke
          image: hello-skaffold-gke
          ports:
            - containerPort: 8080

The image section gets populated with real tagged image name by Skaffold. 

Now that we have a Skaffold descriptor in terms of skaffold.yml file and Kubernetes manifests, let's see some uses of Skaffold.

Building a local Image

A local image is built using the "skaffold build" command, trying it on my local environment:

skaffold build --file-output artifacts.json

results in an image published to the local docker registry, along with a artifact.json file with a content pointing to the created image

{
  "builds": [
    {
      "imageName": "hello-skaffold-gke",
      "tag": "hello-skaffold-gke:a44382e0cd08ba65be1847b5a5aad099071d8e6f351abd88abedee1fa9a52041"
    }
  ]
}

If I wanted to tag the image with the coordinates to the Artifact Registry, I can specify an additional flag "default-repo", the following way:

skaffold build --file-output artifacts.json --default-repo=us-west1-docker.pkg.dev/myproject/sample-repo

resulting in a artifacts.json file with content that looks like this:

{
  "builds": [
    {
      "imageName": "hello-skaffold-gke",
      "tag": "us-west1-docker.pkg.dev/myproject/sample-repo/hello-skaffold-gke:a44382e0c008bf65be1847b5a5aad099071d8e6f351abd88abedee1fa9a52041"
    }
  ]
}

The kubernetes manifests can now be hydrated using a command which looks like this:

skaffold render -a artifacts.json --digest-source=local

which hydrates the manifests, and the output looks like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-skaffold-gke-deployment
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-skaffold-gke
  template:
    metadata:
      labels:
        app: hello-skaffold-gke
    spec:
      containers:
      - image: us-west1-docker.pkg.dev/myproject/sample-repo/hello-skaffold-gke:a44382e0c008bf65be1847b5a5aad099071d8e6f351abd88abedee1fa9a52041
        name: hello-skaffold-gke
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: hello-skaffold-gke-service
  namespace: default
spec:
  ports:
  - name: hello-skaffold-gke
    port: 8080
  selector:
    app: hello-skaffold-gke
  type: LoadBalancer

The right image name now gets plugged into the Kubernetes manifests and can be used for deploying to any Kubernetes environment.

Deploying

Local Development loop with Skaffold

The additional benefit of having a Skaffold configuration file is in the excellent local development loop provided by Skaffold. All that needs to be done to get into the development loop is to run the following command:

skaffold dev --port-forward

which builds an image, renders the kubernetes artifacts pointing to the image and deploying the Kubernetes artifacts to the relevant local Kubernetes environment, minikube in my case:

➜  hello-skaffold-gke git:(main) ✗ skaffold dev --port-forward
Listing files to watch...
 - hello-skaffold-gke
Generating tags...
 - hello-skaffold-gke -> hello-skaffold-gke:5aa5435-dirty
Checking cache...
 - hello-skaffold-gke: Found Locally
Tags used in deployment:
 - hello-skaffold-gke -> hello-skaffold-gke:a44382e0c008bf65be1847b5a5aad099071d8e6f351abd88abedee1fa9a52041
Starting deploy...
 - deployment.apps/hello-skaffold-gke-deployment created
 - service/hello-skaffold-gke-service created
Waiting for deployments to stabilize...
 - deployment/hello-skaffold-gke-deployment is ready.
Deployments stabilized in 2.175 seconds
Port forwarding service/hello-skaffold-gke-service in namespace default, remote port 8080 -> http://127.0.0.1:8080
Press Ctrl+C to exit
Watching for changes...

The dev loops kicks in if any of the file is changed in the project, the image gets rebuilt and deployed again and is surprisingly quick with a tool like jib for creating images.

Debugging with Skaffold

Debugging also works great with skaffold, it starts the appropriate debugging agent for the language being used, so for java, if I were to run the following command:

skaffold debug --port-forward

and attach a debugger in Intellij using a "Remote process" pointing to the debug port

It would pause execution when a code with breakpoint is invoked!

Debugging Kubernetes artifacts

Since real Kubernetes artifacts are being used in the dev loop, we get to test the artifacts and see if there is any typos in them. So for eg, if I were to make a mistake and refer to "port" as "por", it would show up in the dev loop with an error the following way:

WARN[0003] deployer cleanup:kubectl create: running [kubectl --context minikube create --dry-run=client -oyaml -f /Users/biju/learn/hello-skaffold-gke/kubernetes/hello-deployment.yaml -f /Users/biju/learn/hello-skaffold-gke/kubernetes/hello-service.yaml]
 - stdout: "apiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: hello-skaffold-gke-deployment\n  namespace: default\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: hello-skaffold-gke\n  template:\n    metadata:\n      labels:\n        app: hello-skaffold-gke\n    spec:\n      containers:\n      - image: hello-skaffold-gke\n        name: hello-skaffold-gke\n        ports:\n        - containerPort: 8080\n"
 - stderr: "error: error validating \"/Users/biju/learn/hello-skaffold-gke/kubernetes/hello-service.yaml\": error validating data: [ValidationError(Service.spec.ports[0]): unknown field \"por\" in io.k8s.api.core.v1.ServicePort, ValidationError(Service.spec.ports[0]): missing required field \"port\" in io.k8s.api.core.v1.ServicePort]; if you choose to ignore these errors, turn validation off with --validate=false\n"
 - cause: exit status 1  subtask=-1 task=DevLoop
kubectl create: running [kubectl --context minikube create --dry-run=client -oyaml -f /Users/biju/learn/hello-skaffold-gke/kubernetes/hello-deployment.yaml -f /Users/biju/learn/hello-skaffold-gke/kubernetes/hello-service.yaml]
 - stdout: "apiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: hello-skaffold-gke-deployment\n  namespace: default\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: hello-skaffold-gke\n  template:\n    metadata:\n      labels:\n        app: hello-skaffold-gke\n    spec:\n      containers:\n      - image: hello-skaffold-gke\n        name: hello-skaffold-gke\n        ports:\n        - containerPort: 8080\n"
 - stderr: "error: error validating \"/Users/biju/learn/hello-skaffold-gke/kubernetes/hello-service.yaml\": error validating data: [ValidationError(Service.spec.ports[0]): unknown field \"por\" in io.k8s.api.core.v1.ServicePort, ValidationError(Service.spec.ports[0]): missing required field \"port\" in io.k8s.api.core.v1.ServicePort]; if you choose to ignore these errors, turn validation off with --validate=false\n"
 - cause: exit status 1

This is a great way to make sure that the Kubernetes manifests are tested in some way before deployment

Conclusion

Skaffold is an awesome tool to have in my toolbox, it facilitates building of container images, tagging them with sane names, hydrating the Kubernetes manifests using the images, deploying the manifests to a Kubernetes environment. In addition it provides a great development and debugging loop.