CloudEvent Basics

CloudEvent is a way of describing events in a common way. This specification is starting to be adopted across different event producers across Cloud Providers, which over time will provide these benefits:

• Consistency: The format of an event looks the same irrespective of the source producing the event, systems which transmit the event and systems consuming the event. 
• Tooling: Since there is a consistency in format, tooling and libraries can depend on this common format

Cloud Event Sample

One of the ways I have got my head around CloudEvent is to look at samples. Here is a sample Cloud Event published by a Google Cloud Pub/Sub topic, this is in a json format(there are other formats to represent a CloudEvent, for eg, avro or protobuf):

{
  "data": {
    "subscription": "projects/test-project/subscriptions/my-subscription",
    "message": {
      "attributes": {
        "attr1": "attr1-value"
      },
      "data": "dGVzdCBtZXNzYWdlIDM=",
      "messageId": "message-id",
      "publishTime": "2021-02-05T04:06:14.109Z",
      "orderingKey": "ordering-key"
    }
  },
  "datacontenttype": "application/json",
  "id": "3103425958877813",
  "source": "//pubsub.googleapis.com/projects/test-project/topics/my-topic",
  "specversion": "1.0",
  "time": "2021-02-05T04:06:14.109Z",
  "type": "google.cloud.pubsub.topic.v1.messagePublished"
}

Some of the elements in this event are:

1. “id” which uniquely identifies the event
2. “source” which identifies the system generating the event
3. “specversion” identifies the CloudEvent specificiation that this event complies with
4. “type” defining the type of event produced by the source system
5. “datacontenttype” which describes the content type of the data
6. “data”, which is the actual event payload, the structure of this specifically can change based on the “type” of event.

The “id”, “source”, “specversion” and “type” fields are mandatory

Cloud Event Extensions

In certain cases there will be additional attributes that may be needed to be understood across systems which produce and consume messages. A good example is distributed tracing where tracing attributes may need to be present in event data, to support these cases, events can have extension attributes. An example is the following:

{
  "data": {
    "subscription": "projects/test-project/subscriptions/my-subscription",
    "message": {
      "attributes": {
        "attr1": "attr1-value"
      },
      "data": "dGVzdCBtZXNzYWdlIDM=",
      "messageId": "message-id",
      "publishTime": "2021-02-05T04:06:14.109Z",
      "orderingKey": "ordering-key"
    }
  },
  "datacontenttype": "application/json",
  "id": "3103425958877813",
  "source": "//pubsub.googleapis.com/projects/test-project/topics/my-topic",
  "specversion": "1.0",
  "time": "2021-02-05T04:06:14.109Z",
  "type": "google.cloud.pubsub.topic.v1.messagePublished",
  "traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01",
  "tracestate": "rojo=00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01,congo=lZWRzIHRoNhcm5hbCBwbGVhc3VyZS4"
}

where “traceparent” and “tracestate” capture the distribution tracing related attributes. Some of the other extension types are documented here.

Data Attribute

The event payload is contained in the “data” attribute (or can be base 64 encoded into a “data_base64” attribute). The structure of the data attribute is entirely depends on the event type. There is a level of specification that can be specified by the event type using an additional attribute called “dataschema”.

Consider another sample for a log entry data related event in Google Cloud:

{
  "data": {
    "insertId": "1234567",
    "logName": "projects/test-project/logs/cloudaudit.googleapis.com%2Fdata_access",
    "protoPayload": {
      "authenticationInfo": {
        "principalEmail": "robot@test-project.iam.gserviceaccount.com"
      },
      "methodName": "jobservice.jobcompleted",
      "requestMetadata": {
        "callerIp": "2620:15c:0:200:1a75:e914:115b:e970",
        "callerSuppliedUserAgent": "google-cloud-sdk357.0.0 (gzip),gzip(gfe)",
        "destinationAttributes": {
          
        },
        "requestAttributes": {
          
        }
      },
      "resourceName": "projects/test-project/jobs/sample-job",
      "serviceData": {
        "jobCompletedEvent": {
          "eventName": "query_job_completed",
          "job": {
            "jobConfiguration": {
              "query": {
                "createDisposition": "CREATE_IF_NEEDED",
                "defaultDataset": {
                  
                },
                "destinationTable": {
                  "datasetId": "sample-dataset",
                  "projectId": "test-project",
                  "tableId": "sample-table"
                },
                "query": "sample-query",
                "queryPriority": "QUERY_INTERACTIVE",
                "statementType": "SELECT",
                "writeDisposition": "WRITE_TRUNCATE"
              }
            }
          }
        }
      },
      "serviceName": "bigquery.googleapis.com",
      "status": {
        
      }
    },
    "receiveTimestamp": "2021-11-25T21:56:00.653866570Z",
    "resource": {
      "labels": {
        "project_id": "test-project"
      },
      "type": "bigquery_resource"
    },
    "severity": "INFO",
    "timestamp": "2021-11-25T21:56:00.276607Z"
  },
  "datacontenttype": "application/json; charset=utf-8",
  "dataschema": "https://googleapis.github.io/google-cloudevents/jsonschema/google/events/cloud/audit/v1/LogEntryData.json",
  "id": "projects/test-project/logs/cloudaudit.googleapis.com%2Fdata_access1234567123456789",
  "methodName": "jobservice.jobcompleted",
  "recordedTime": "2021-11-25T21:56:00.276607Z",
  "resourceName": "projects/test-project/jobs/sample-job",
  "serviceName": "bigquery.googleapis.com",
  "source": "//cloudaudit.googleapis.com/projects/test-project/logs/data_access",
  "specversion": "1.0",
  "subject": "bigquery.googleapis.com/projects/test-project/jobs/sample-job",
  "time": "2021-11-25T21:56:00.653866570Z",
  "type": "google.cloud.audit.log.v1.written"
}

The “data” field is fairly complicated here, however see how there is a reference to a “dataschema” pointing to this document — https://googleapis.github.io/google-cloudevents/jsonschema/google/events/cloud/audit/v1/LogEntryData.json

which describes the elements in the “data”, using json schema specification

Conclusion

CloudEvents attempts to solve the issue of different event sources using different ways to represent an event, by providing a common specification.

This blog post provides a quick overview of the specification, in a future post I will go over how this is useful for writing eventing systems on Google Cloud.