Skip to content

exposing num threads as option #2349

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 30, 2025
Merged

Conversation

joy91227
Copy link
Contributor

@joy91227 joy91227 commented May 4, 2025

Key parallelism is limited by NUM_THREADS which is hardcoded to 100 [1]. This could cause commit too large exceptions [2].

Exposing this option for users to tweak is a easy workaround before implementing a long-term solution that ensures that it never generates outputs that are too large.

[1]
https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v2/datastream-to-sql/src/main/java/com/google/cloud/teleport/v2/transforms/CreateDml.java#L44

[2] https://cloud.google.com/dataflow/docs/guides/common-errors#key-commit-too-large-exception.

@joy91227 joy91227 requested a review from a team as a code owner May 4, 2025 20:57
Copy link

codecov bot commented May 29, 2025

Codecov Report

Attention: Patch coverage is 0% with 8 lines in your changes missing coverage. Please review.

Project coverage is 49.71%. Comparing base (04a7052) to head (9c6d307).
Report is 18 commits behind head on main.

Files with missing lines Patch % Lines
...e/cloud/teleport/v2/templates/DataStreamToSQL.java 0.00% 4 Missing ⚠️
...google/cloud/teleport/v2/transforms/CreateDml.java 0.00% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##               main    #2349   +/-   ##
=========================================
  Coverage     49.71%   49.71%           
- Complexity     4714     4715    +1     
=========================================
  Files           928      928           
  Lines         56300    56305    +5     
  Branches       6083     6083           
=========================================
+ Hits          27987    27990    +3     
- Misses        26308    26310    +2     
  Partials       2005     2005           
Components Coverage Δ
spanner-templates 70.38% <ø> (ø)
spanner-import-export 68.50% <ø> (ø)
spanner-live-forward-migration 78.36% <ø> (ø)
spanner-live-reverse-replication 80.38% <ø> (ø)
spanner-bulk-migration 87.74% <ø> (ø)
Files with missing lines Coverage Δ
...e/cloud/teleport/v2/templates/DataStreamToSQL.java 4.47% <0.00%> (-0.21%) ⬇️
...google/cloud/teleport/v2/transforms/CreateDml.java 0.00% <0.00%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@liferoad liferoad merged commit 98878a9 into GoogleCloudPlatform:main May 30, 2025
14 of 15 checks passed
@joy91227 joy91227 deleted the datastream branch May 30, 2025 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants