Skaffold for Cloud Run and Local Environments

In one of my previous posts, I had explored using Cloud Deploy to deploy to a Cloud Run environment. Cloud Deploy uses a Skaffold file to internally orchestrate the steps required to build an image, adding the coordinates of the image to the manifest files and deploying it to a runtime. This works out great, not so much for local development and testing though. The reason is a lack of local Cloud Run runtime.

A good alternative is to simply use a local distribution of Kubernetes — say a minikube or kind. This will allow Skaffold to be used to its full power — with an ability to provide a quick development loop, debug, etc. I have documented some of the features here. The catch however is that there will now need to be two different sets of details of the environments maintained along with their corresponding sets of manifests — ones targeting Cloud Run, targeting minikube.

Skaffold patching is a way to do this and this post will go into the high-level details of the approach.

Skaffold Profiles and Patches

My original Skaffold configuration looks like this, targeting a Cloud Run environment:

apiVersion: skaffold/v3alpha1
kind: Config
metadata:
  name: clouddeploy-cloudrun-skaffold
manifests:
  kustomize:
    paths:
      - manifests/base
build:
  artifacts:
    - image: clouddeploy-cloudrun-app-image
      jib: { }
profiles:
  - name: dev
    manifests:
      kustomize:
        paths:
          - manifests/overlays/dev
  - name: prod
    manifests:
      kustomize:
        paths:
          - manifests/overlays/prod
deploy:
  cloudrun:
    region: us-west1-a

The “deploy.cloudrun” part indicates that it is targeting a Cloud Run environment.

So now, I want a different behavior in “local” environment, the way to do this in skaffold is to create a Skaffold profile that specifies what is different about this environment:

apiVersion: skaffold/v3alpha1
kind: Config
metadata:
  name: clouddeploy-cloudrun-skaffold
manifests:
  kustomize:
    paths:
      - manifests/base
build:
  artifacts:
    - image: clouddeploy-cloudrun-app-image
      jib: { }
profiles:
  - name: local
    # Something different on local
  - name: dev
    manifests:
      kustomize:
        paths:
          - manifests/overlays/dev
  - name: prod
    manifests:
      kustomize:
        paths:
          - manifests/overlays/prod
deploy:
  cloudrun:
    region: us-west1-a

I have two things different on local,

the deploy environment will be a minikube-based Kubernetes environment

the manifests file will be for this Kubernetes environment.

For the first requirement:

apiVersion: skaffold/v3alpha1
kind: Config
metadata:
  name: clouddeploy-cloudrun-skaffold
manifests:
  kustomize:
    paths:
      - manifests/base
build:
  artifacts:
    - image: clouddeploy-cloudrun-app-image
      jib: { }
profiles:
  - name: local
    patches:
      - op: remove
        path: /deploy/cloudrun
    deploy:
      kubectl: { }
  - name: dev
    manifests:
      kustomize:
        paths:
          - manifests/overlays/dev
  - name: prod
    manifests:
      kustomize:
        paths:
          - manifests/overlays/prod
deploy:
  cloudrun:
    region: us-west1-a

To specify the deploy environment where patches come, here the patch indicates that I want to remove Cloudrun as a deployment environment and add in Kubernetes.

And for the second requirement of generating a Kubernetes manifest, a rawYaml tag is introduced:

apiVersion: skaffold/v3alpha1
kind: Config
metadata:
  name: clouddeploy-cloudrun-skaffold
manifests:
  kustomize:
    paths:
      - manifests/base
build:
  artifacts:
    - image: clouddeploy-cloudrun-app-image
      jib: { }
profiles:
  - name: local
    manifests:
      kustomize: { }
      rawYaml:
        - kube/app.yaml
    patches:
      - op: remove
        path: /deploy/cloudrun
    deploy:
      kubectl: { }
  - name: dev
    manifests:
      kustomize:
        paths:
          - manifests/overlays/dev
  - name: prod
    manifests:
      kustomize:
        paths:
          - manifests/overlays/prod
deploy:
  cloudrun:
    region: us-west1-a

In this way a combination of Skaffold profiles and patches are used for tweaking the local deployment for Minikube.

Activating Profiles

When testing on local the “local” profile can be activated this way with Skaffold — with a -p flag:

skaffold dev -p local

One of the most useful command that I got to use is the “diagnose” command in skaffold which clearly showed what skaffold configuration is active for specific profiles:

skaffold diagnose -p local

which generated this resolved configuration for me:

apiVersion: skaffold/v3
kind: Config
metadata:
  name: clouddeploy-cloudrun-skaffold
build:
  artifacts:
  - image: clouddeploy-cloudrun-app-image
    context: .
    jib: {}
  tagPolicy:
    gitCommit: {}
  local:
    concurrency: 1
manifests:
  rawYaml:
  - /Users/biju/learn/clouddeploy-cloudrun-sample/kube/app.yaml
  kustomize: {}
deploy:
  kubectl: {}
  logs:
    prefix: container

Conclusion

There will likely be better support for Cloud Run on a local environment, for now, a minikube based Kubernetes is a good stand-in. Skaffold with profiles and patches can target this environment on a local box. This allows Skaffold features like quick development loop, debugging, etc to be activated while an application is in the process of being developed.