AWS DynamoDB is described as a NoSQL key-value and a document database. In my work I mostly use the key-value behavior of the database but rarely use the document database features, however the document database part is growing on me and this post highlights some ways of using the document database feature of DynamoDB along with introducing a small utility library built on top of AWS SDK 2.X for Java that simplifies using document database features of AWS DynamoDB
The treatment of the document database features will be very high level in this post, I will plan a follow up which goes into more details later
DynamoDB as a document database
So what does it mean for AWS DynamoDB to be treated as a document database. Consider a json representation of an entity, say something representing a Hotel:
{
"id": "1",
"name": "test",
"address": "test address",
"state": "OR",
"properties": {
"amenities":{
"rooms": 100,
"gym": 2,
"swimmingPool": true
}
},
"zip": "zip"
}
This json has some top level attributes like "id", a name, an address etc. But it also has a free form "properties" holding some additional "nested" attributes of this hotel.
A document database can store this document representing the hotel in its entirety OR can treat individual fields say the "properties" field of the hotel as a document.
A naive way to do this will be to simply serialize the entire content into a json string and store it in, say for eg, for the properties field transform into a string representation of the json and store in the database, this works, but there are a few issues with it.
- None of the attributes of the field like properties can be queried for, say if I wanted to know whether the hotel has a swimming pool, there is no way just to get this information of of the stored content.
- The attributes cannot be filtered on - so say if wanted hotels with atleast 2 gyms, this is not something that can be filtered down to.
A document database would allow for the the entire document to be saved, individual attributes, both top level and nested ones, to be queried/filtered on.
So for eg, in the example of "hotel" document the top level attributes are "id", "name", "address", "state", "zip" and the nested attributes are "properties.amenities.rooms", "properties.amenities.gym", "properties.amenities.swimmingPool" and so on.
AWS SDK 2 for DynamoDB and Document database support
If you are writing a Java based application to interact with a AWS DynamoDB database, then you would have likely used the new
AWS SDK 2 library to make the API calls. However one issue with the library is that it natively does not support a json based document model. Let me go into a little more detail here.
From the AWS SDK 2 for AWS DynamoDB's perspective every attribute that is saved is an instance of something called an
AttributeValue.
A row of data, say for a hotel, is a simple map of "attribute" names to Attribute values, and a sample code looks something like this:
val putItemRequest = PutItemRequest.builder()
.tableName(TABLE_NAME)
.item(
mapOf(
ID to AttributeValue.builder().s(hotel.id).build(),
NAME to AttributeValue.builder().s(hotel.name).build(),
ZIP to AttributeValue.builder().s(hotel.zip).build(),
STATE to AttributeValue.builder().s(hotel.state).build(),
ADDRESS to AttributeValue.builder().s(hotel.address).build(),
PROPERTIES to objectMapper.writeValueAsString(hotel.properties),
VERSION to AttributeValue.builder().n(hotel.version.toString()).build()
)
)
.build()
dynamoClient.putItem(putItemRequest)
Here a map of each attribute to an AttributeValue is being created with an appropriate "type" of content, "s" indicates a string, "n" a number in the above sample.
There are other AttributeValue types like "m" representing a map and "l" representing a list.
The neat thing is that "m" and "l" types can have nested AttributeValues, which maps to a structured json document, however there is no simple way to convert a json to this kind of an Attribute Value and back.
So for eg. if I were to handle the raw "properties" of a hotel which understands the nested attributes, an approach could be this:
val putItemRequest = PutItemRequest.builder()
.tableName(TABLE_NAME)
.item(
mapOf(
ID to AttributeValue.builder().s(hotel.id).build(),
NAME to AttributeValue.builder().s(hotel.name).build(),
ZIP to AttributeValue.builder().s(hotel.zip).build(),
STATE to AttributeValue.builder().s(hotel.state).build(),
ADDRESS to AttributeValue.builder().s(hotel.address).build(),
PROPERTIES to AttributeValue.builder()
.m(
mapOf(
"amenities" to AttributeValue.builder()
.m(
mapOf(
"rooms" to AttributeValue.builder().n("200").build(),
"gym" to AttributeValue.builder().n("2").build(),
"swimmingPool" to AttributeValue.builder().bool(true).build()
)
)
.build()
)
)
.build(),
VERSION to AttributeValue.builder().n(hotel.version.toString()).build()
)
)
.build()
See how the nested attributes are being expanded out recursively.
Introducing the Json to AttributeValue utility library
This is exactly where the utility library that I have developed comes in.
Given a json structure as a
Jackson JsonNode it converts the Json into an appropriately nested AttributeValue type and when retrieving back from DynamoDB, can convert the resulting nested AttributeValue type back to a json.
The structure would look exactly similar to the handcrafted sample shown before. So using the utility saving the "properties" would look like this:
val putItemRequest = PutItemRequest.builder()
.tableName(TABLE_NAME)
.item(
mapOf(
ID to AttributeValue.builder().s(hotel.id).build(),
NAME to AttributeValue.builder().s(hotel.name).build(),
ZIP to AttributeValue.builder().s(hotel.zip).build(),
STATE to AttributeValue.builder().s(hotel.state).build(),
ADDRESS to AttributeValue.builder().s(hotel.address).build(),
PROPERTIES to JsonAttributeValueUtil.toAttributeValue(hotel.properties),
VERSION to AttributeValue.builder().n(hotel.version.toString()).build()
)
)
.build()
dynamoClient.putItem(putItemRequest)
and when querying back from DynamoDB, the resulting nested AttributeValue converted back to a json this way(Kotlin code in case you are baffled by the "?let"):
properties = map[PROPERTIES]?.let { attributeValue ->
JsonAttributeValueUtil.fromAttributeValue(
attributeValue
)
} ?: JsonNodeFactory.instance.objectNode()
The neat thing is even the top level attributes can be generated given a json representing the entire Hotel type. So say a json representing a Hotel is provided:
val hotel = """
{
"id": "1",
"name": "test",
"address": "test address",
"state": "OR",
"properties": {
"amenities":{
"rooms": 100,
"gym": 2,
"swimmingPool": true
}
},
"zip": "zip"
}
""".trimIndent()
val attributeValue = JsonAttributeValueUtil.toAttributeValue(hotel, objectMapper)
dynamoDbClient.putItem(
PutItemRequest.builder()
.tableName(DynamoHotelRepo.TABLE_NAME)
.item(attributeValue.m())
.build()
)
Using the Library
The utility library is available
here - https://github.com/bijukunjummen/aws-sdk2-dynamo-json-helper and provides details of how to get the binaries in place and use it with code.
Conclusion
AWS SDK 2 is an excellent and highly performant client, providing non-blocking support for client calls. I like how it provides a synchronous API and an asynchronous API and remains highly opionionated in consistenly providing a low level client API for calling the different AWS services. This utlility library provides a nice bridge for AWS SDK 2 to remain low level but be able to manage a json based document persistence and back.
All the samples in this post are available in my github repository
here - https://github.com/bijukunjummen/dynamodb-document-sample
