Bigtable Pagination in Java

 Consider a set of rows stored in Bigtable table called “people”:

My objective is to be able to paginate a few records at a time, say with each page containing 4 records:

Page 1:

Page 2:

Page 3:

High-Level Approach

A high level approach to doing this is to introduce two parameters:

• Offset — the point from which to retrieve the records.
• Limit — the number of records to retrieve per page

Limit in all cases is 4 in my example. Offset provides some way to indicate where to retrieve the next set of records from. Bigtable orders the record lexicographically using the key of each row, so one way to indicate offset is by using the key of the last record on a page. Given this, and using a marker offset of empty string for the first page, offset and record for each page looks like this:

Page 1 — offset: “”, limit: 4

Page 2 — offset: “person#id-004”, limit: 4

Page 3 — offset: “person#id-008”, limit: 4

The challenge now is in figuring out how to retrieve a set of records given a prefix, an offset, and a limit.

Retrieving records given a prefix, offset, limit

Bigtable java client provides a “readRows” api, that takes in a Query and returns a list of rows.

import com.google.cloud.bigtable.data.v2.BigtableDataClient
import com.google.cloud.bigtable.data.v2.models.Query
import com.google.cloud.bigtable.data.v2.models.Row

val rows: List<Row> = bigtableDataClient.readRows(query).toList()

Now, Query has a variant that takes in a prefix and returns rows matching the prefix:

import com.google.cloud.bigtable.data.v2.BigtableDataClient
import com.google.cloud.bigtable.data.v2.models.Query
import com.google.cloud.bigtable.data.v2.models.Row

val query: Query = Query.create("people").limit(limit).prefix(keyPrefix)
val rows: List<Row> = bigtableDataClient.readRows(query).toList()        

This works for the first page, however, for subsequent pages, the offset needs to be accounted for.

A way to get this to work is to use a Query that takes in a range:

import com.google.cloud.bigtable.data.v2.BigtableDataClient
import com.google.cloud.bigtable.data.v2.models.Query
import com.google.cloud.bigtable.data.v2.models.Row
import com.google.cloud.bigtable.data.v2.models.Range

val range: Range.ByteStringRange = 
    Range.ByteStringRange
        .unbounded()
        .startOpen(offset)
        .endOpen(end)

val query: Query = Query.create("people")
                    .limit(limit)
                    .range(range)

The problem with this is to figure out what the end of the range should be. This is where a neat utility that the Bigtable Java library provides comes in. This utility given a prefix of “abc”, calculates the end of the range to be “abd”

import com.google.cloud.bigtable.data.v2.models.Range

val range = Range.ByteStringRange.prefix("abc")

Putting this all together, a query that fetches paginated rows at an offset looks like this:

val query: Query =
    Query.create("people")
        .limit(limit)
        .range(Range.ByteStringRange
            .prefix(keyPrefix)
            .startOpen(offset))

val rows: List<Row> = bigtableDataClient.readRows(query).toList()

When returning the result, the final key needs to be returned so that it can be used as the offset for the next page, this can be done in Kotlin by having the following type:

data class Page<T>(val data: List<T>, val nextOffset: String)

Conclusion

I have a full example available here — this pulls in the right library dependencies and has all the mechanics of pagination wrapped into a working sample.

Cloud Run Health Checks — Spring Boot App

 Cloud Run services now can configure startup and liveness probes for a running container.

The startup probe is for determining when a container has cleanly started up and is ready to take traffic. A Liveness probe kicks off once a container has started up, to ensure that the container remains functional — Cloud Run would restart a container if the liveness probe fails.

Implementing Health Check Probes

A Cloud Run service can be described using a manifest file and a sample manifest looks like this:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  annotations:
    run.googleapis.com/ingress: all
  name: health-cloudrun-sample
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/maxScale: '5'
        autoscaling.knative.dev/minScale: '1'
    spec:
      containers:
        image: us-west1-docker.pkg.dev/sample-proj/sample-repo/health-app-image:latest

        startupProbe:
          httpGet:
            httpHeaders:
            - name: HOST
              value: localhost:8080
            path: /actuator/health/readiness
          initialDelaySeconds: 15
          timeoutSeconds: 1
          failureThreshold: 5
          periodSeconds: 10

        livenessProbe:
          httpGet:
            httpHeaders:
            - name: HOST
              value: localhost:8080
            path: /actuator/health/liveness
          timeoutSeconds: 1
          periodSeconds: 10
          failureThreshold: 5

        ports:
        - containerPort: 8080
          name: http1
        resources:
          limits:
            cpu: 1000m
            memory: 512Mi

This manifest can then be used for deployment to Cloud Run the following way:

gcloud run services replace sample-manifest.yaml --region=us-west1

Now, coming back to the manifest, the startup probe is defined this way:

startupProbe:
  httpGet:
    httpHeaders:
    - name: HOST
      value: localhost:8080
    path: /actuator/health/readiness
  initialDelaySeconds: 15
  timeoutSeconds: 1
  failureThreshold: 5
  periodSeconds: 10

It is set to make an http request to a /actuator/health/readiness path. There is an explicit HOST header also provided, this is temporary though as Cloud Run health checks currently have a bug where this header is missing from the health check requests.

The rest of the properties indicate the following:

• initialDelaySeconds — delay for performing the first probe
• timeoutSeconds — timeout for the health check request
• failureThreshold — number of tries before the container is marked as not ready
• periodSeconds — the delay between probes

Once the startup probe succeeds, Cloud Run would mark the container as being available to handle the traffic.

A livenessProbe follows a similar pattern:

livenessProbe:
  httpGet:
    httpHeaders:
    - name: HOST
      value: localhost:8080
    path: /actuator/health/liveness
  timeoutSeconds: 1
  periodSeconds: 10
  failureThreshold: 5

From a Spring Boot application perspective, all that needs to be done is to enable the Health check endpoints as described here

Conclusion

Start-Up probe ensures that a container receives traffic only when ready and a Liveness probe ensures that the container remains healthy during its operation, else gets restarted by the infrastructure. These health probes are a welcome addition to the already excellent feature set of Cloud Run.

Skaffold for Cloud Run and Local Environments

In one of my previous posts, I had explored using Cloud Deploy to deploy to a Cloud Run environment. Cloud Deploy uses a Skaffold file to internally orchestrate the steps required to build an image, adding the coordinates of the image to the manifest files and deploying it to a runtime. This works out great, not so much for local development and testing though. The reason is a lack of local Cloud Run runtime.

A good alternative is to simply use a local distribution of Kubernetes — say a minikube or kind. This will allow Skaffold to be used to its full power — with an ability to provide a quick development loop, debug, etc. I have documented some of the features here. The catch however is that there will now need to be two different sets of details of the environments maintained along with their corresponding sets of manifests — ones targeting Cloud Run, targeting minikube.

Skaffold patching is a way to do this and this post will go into the high-level details of the approach.

Skaffold Profiles and Patches

My original Skaffold configuration looks like this, targeting a Cloud Run environment:

apiVersion: skaffold/v3alpha1
kind: Config
metadata:
  name: clouddeploy-cloudrun-skaffold
manifests:
  kustomize:
    paths:
      - manifests/base
build:
  artifacts:
    - image: clouddeploy-cloudrun-app-image
      jib: { }
profiles:
  - name: dev
    manifests:
      kustomize:
        paths:
          - manifests/overlays/dev
  - name: prod
    manifests:
      kustomize:
        paths:
          - manifests/overlays/prod
deploy:
  cloudrun:
    region: us-west1-a

The “deploy.cloudrun” part indicates that it is targeting a Cloud Run environment.

So now, I want a different behavior in “local” environment, the way to do this in skaffold is to create a Skaffold profile that specifies what is different about this environment:

apiVersion: skaffold/v3alpha1
kind: Config
metadata:
  name: clouddeploy-cloudrun-skaffold
manifests:
  kustomize:
    paths:
      - manifests/base
build:
  artifacts:
    - image: clouddeploy-cloudrun-app-image
      jib: { }
profiles:
  - name: local
    # Something different on local
  - name: dev
    manifests:
      kustomize:
        paths:
          - manifests/overlays/dev
  - name: prod
    manifests:
      kustomize:
        paths:
          - manifests/overlays/prod
deploy:
  cloudrun:
    region: us-west1-a

I have two things different on local,

the deploy environment will be a minikube-based Kubernetes environment

the manifests file will be for this Kubernetes environment.

For the first requirement:

apiVersion: skaffold/v3alpha1
kind: Config
metadata:
  name: clouddeploy-cloudrun-skaffold
manifests:
  kustomize:
    paths:
      - manifests/base
build:
  artifacts:
    - image: clouddeploy-cloudrun-app-image
      jib: { }
profiles:
  - name: local
    patches:
      - op: remove
        path: /deploy/cloudrun
    deploy:
      kubectl: { }
  - name: dev
    manifests:
      kustomize:
        paths:
          - manifests/overlays/dev
  - name: prod
    manifests:
      kustomize:
        paths:
          - manifests/overlays/prod
deploy:
  cloudrun:
    region: us-west1-a

To specify the deploy environment where patches come, here the patch indicates that I want to remove Cloudrun as a deployment environment and add in Kubernetes.

And for the second requirement of generating a Kubernetes manifest, a rawYaml tag is introduced:

apiVersion: skaffold/v3alpha1
kind: Config
metadata:
  name: clouddeploy-cloudrun-skaffold
manifests:
  kustomize:
    paths:
      - manifests/base
build:
  artifacts:
    - image: clouddeploy-cloudrun-app-image
      jib: { }
profiles:
  - name: local
    manifests:
      kustomize: { }
      rawYaml:
        - kube/app.yaml
    patches:
      - op: remove
        path: /deploy/cloudrun
    deploy:
      kubectl: { }
  - name: dev
    manifests:
      kustomize:
        paths:
          - manifests/overlays/dev
  - name: prod
    manifests:
      kustomize:
        paths:
          - manifests/overlays/prod
deploy:
  cloudrun:
    region: us-west1-a

In this way a combination of Skaffold profiles and patches are used for tweaking the local deployment for Minikube.

Activating Profiles

When testing on local the “local” profile can be activated this way with Skaffold — with a -p flag:

skaffold dev -p local

One of the most useful command that I got to use is the “diagnose” command in skaffold which clearly showed what skaffold configuration is active for specific profiles:

skaffold diagnose -p local

which generated this resolved configuration for me:

apiVersion: skaffold/v3
kind: Config
metadata:
  name: clouddeploy-cloudrun-skaffold
build:
  artifacts:
  - image: clouddeploy-cloudrun-app-image
    context: .
    jib: {}
  tagPolicy:
    gitCommit: {}
  local:
    concurrency: 1
manifests:
  rawYaml:
  - /Users/biju/learn/clouddeploy-cloudrun-sample/kube/app.yaml
  kustomize: {}
deploy:
  kubectl: {}
  logs:
    prefix: container

Conclusion

There will likely be better support for Cloud Run on a local environment, for now, a minikube based Kubernetes is a good stand-in. Skaffold with profiles and patches can target this environment on a local box. This allows Skaffold features like quick development loop, debugging, etc to be activated while an application is in the process of being developed.