Consider a set of rows stored in Bigtable table called “people”:
My objective is to be able to paginate a few records at a time, say with each page containing 4 records:
Page 1:
Page 2:
High-Level Approach
A high level approach to doing this is to introduce two parameters:
- Offset — the point from which to retrieve the records.
- Limit — the number of records to retrieve per page
Limit in all cases is 4 in my example. Offset provides some way to indicate where to retrieve the next set of records from. Bigtable orders the record lexicographically using the key of each row, so one way to indicate offset is by using the key of the last record on a page. Given this, and using a marker offset of empty string for the first page, offset and record for each page looks like this:
Page 1 — offset: “”, limit: 4
Page 2 — offset: “person#id-004”, limit: 4
The challenge now is in figuring out how to retrieve a set of records given a prefix, an offset, and a limit.
Retrieving records given a prefix, offset, limit
Bigtable java client provides a “readRows” api, that takes in a Query and returns a list of rows.
import com.google.cloud.bigtable.data.v2.BigtableDataClient import com.google.cloud.bigtable.data.v2.models.Query import com.google.cloud.bigtable.data.v2.models.Row val rows: List<Row> = bigtableDataClient.readRows(query).toList()
Now, Query has a variant that takes in a prefix and returns rows matching the prefix:
import com.google.cloud.bigtable.data.v2.BigtableDataClient
import com.google.cloud.bigtable.data.v2.models.Query
import com.google.cloud.bigtable.data.v2.models.Row
val query: Query = Query.create("people").limit(limit).prefix(keyPrefix)
val rows: List<Row> = bigtableDataClient.readRows(query).toList()
This works for the first page, however, for subsequent pages, the offset needs to be accounted for.
A way to get this to work is to use a Query that takes in a range:
import com.google.cloud.bigtable.data.v2.BigtableDataClient
import com.google.cloud.bigtable.data.v2.models.Query
import com.google.cloud.bigtable.data.v2.models.Row
import com.google.cloud.bigtable.data.v2.models.Range
val range: Range.ByteStringRange =
Range.ByteStringRange
.unbounded()
.startOpen(offset)
.endOpen(end)
val query: Query = Query.create("people")
.limit(limit)
.range(range)
The problem with this is to figure out what the end of the range should be. This is where a neat utility that the Bigtable Java library provides comes in. This utility given a prefix of “abc”, calculates the end of the range to be “abd”
import com.google.cloud.bigtable.data.v2.models.Range
val range = Range.ByteStringRange.prefix("abc")Putting this all together, a query that fetches paginated rows at an offset looks like this:
val query: Query =
Query.create("people")
.limit(limit)
.range(Range.ByteStringRange
.prefix(keyPrefix)
.startOpen(offset))
val rows: List<Row> = bigtableDataClient.readRows(query).toList()
When returning the result, the final key needs to be returned so that it can be used as the offset for the next page, this can be done in Kotlin by having the following type:
data class Page<T>(val data: List<T>, val nextOffset: String)
Conclusion
I have a full example available here — this pulls in the right library dependencies and has all the mechanics of pagination wrapped into a working sample.
