checkpointing for streams #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

abhineshada opened this issue Apr 29, 2019 · 2 comments

abhineshada commented Apr 29, 2019

Common approach to handle checkpointing while using GO sdks?
How to handle missing/duplicated data?

Owner

urakozz commented Apr 29, 2019 •

edited

Loading

What the hell is checkpointing? I didn't use DynamoDB for some years, is is new term there?

How to handle missing/duplicated data?

However you like, this library only wraps complexity of iterating over overengineered dynamo streams

Author

abhineshada commented Apr 29, 2019 •

edited

Loading

By checkpoint, I mean tracking the status of shards/data processed.

Refer: https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-ddb.html
https://www.infoq.com/articles/demystifying-dynamodb-streams

KCL adapter for dynamoDB streams has checkpoint feature (only available in Java): https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.KCLAdapter.html

The library is really well written. I had few doubts regarding how data accuracy can be maintained using go SDK.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment