Throttling exceptions being triggered by AWS JS SDK

[ecout]

When enough events need to be processed a throttling exception is thrown. These are retryable, does the lambda retry those?

2021-10-19T15:24:15.278-07:00

Copy
2021-10-19T22:24:15.278Z	d2ca43df-6984-45e9-ad7d-571b56160f00	INFO	ThrottlingException: Rate exceeded
    at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)
    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:55:8)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:688:14)
    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:690:12) {
  code: 'ThrottlingException',
  time: 2021-10-19T22:24:15.275Z,
  requestId: '7d3c79c0-193b-463b-a237-30493848771d',
  statusCode: 400,
  retryable: true
}

Reference:
aws/aws-sdk-js#412

[rayjanoka]

hmm you can slow it down by setting --lambda_threads (or --lambda-threads with the latest commit) to less than 3. I'm pretty sure the JS SDK will retry 3 times. I'm not sure if Lambda will retry in the synchronous mode like we have it.

It would probably be good to check for errors in the ruby script and retry a few times there.

You can use the audit script to get the inventory and then run retro again and see if anything gets picked up on a 2nd run.