Producer Writeable metrics

How to get the same metrics as the official KPL

Sample code based on Prom Client

const promClient = require('prom-client')

User Record Received

Count of how many logical user records were received by the KPL core for putting.

User Records Pending

Periodic sample of how many user records are currently pending. A record is pending if it is either currently buffered and waiting for to be sent, or sent and in-flight to the backend.

User Records Put

Count of how many logical user records were put successfully.

userRecordsPutCounter = new promClient.Counter({
  name: 'kinesis_user_records_put_total',
  help: 'Count of how many logical user records were put successfully',
  labelNames: ['topic', 'shard'],
})

kinesisWriter.on('kinesis.putRecords', (response) => {
  response.Records.forEach((x) => {
    if (x.ShardId) {
      userRecordsPutCounter.labels(TOPIC, x.ShardId).inc();
    }
  });
});

User Records Data Put

Bytes in the logical user records successfully put.

Kinesis Records Put

Count of how many Amazon Kinesis records were put successfully. Recall that each Amazon Kinesis record can contain multiple user records.

exports.kinesisRecordsPutCounter = new promClient.Counter({
  name: 'kinesis_records_put_total',
  help: 'Count of how many Amazon Kinesis records were put successfully',
  labelNames: ['topic'],
})

kinesisWriter.on('kinesis.putRecords', (response) => {
  kinesisRecordsPutCounter.labels(TOPIC).inc()
})

Kinesis Records Data Put

Bytes in the Amazon Kinesis records.

Errors by Code

Count of each type of error code. This introduces an additional dimension of ErrorCode, on top of the normal ones like StreamName and ShardId.

All Errors

This is triggered by the same errors as above, but does not distinguish between types. This is useful as a general monitor of the error rate without having to manually sum the counts from all the different types of errors.

Retries per Record

Number of retries performed per user record. Zero is emitted for records that succeed in one try.

Buffering Time

The time between a user record arriving at the KPL and leaving for the backend. This information is transmitted back to the user on a per-record basis, but is also available as an aggregated statistic.

Request Time

The time it takes to perform PutRecordsRequests.

User Records per Kinesis Record

The number of logical user records aggregated into a single Amazon Kinesis record.

Amazon Kinesis Records per PutRecordsRequest

The number of Amazon Kinesis records aggregated into a single PutRecordsRequest.

User Records per PutRecordsRequest

The total number of user records contained within a PutRecordsRequest. This is roughly equivalent to the product of the previous two metrics.