Skip to content

Commit 49598d0

Browse files
committed
[Bugfix] [Connector-V2] Fix SourceCommonOptions Parallelism Not Working with Spark Engine (apache#9302)
This bug was caused by EnvCommonOptions overriding SourceCommonOptions when setting the parallelism in the sparkRuntimeEnvironment.
1 parent 3783816 commit 49598d0

File tree

1 file changed

+1
-1
lines changed
  • seatunnel-core/seatunnel-spark-starter/seatunnel-spark-starter-common/src/main/java/org/apache/seatunnel/core/starter/spark/execution

1 file changed

+1
-1
lines changed

seatunnel-core/seatunnel-spark-starter/seatunnel-spark-starter-common/src/main/java/org/apache/seatunnel/core/starter/spark/execution/SourceExecuteProcessor.java

+1-1
Original file line numberDiff line numberDiff line change
@@ -92,12 +92,12 @@ public List<DatasetTableInfo> execute(List<DatasetTableInfo> upstreamDataStreams
9292
EnvCommonOptions.PARALLELISM.key(),
9393
EnvCommonOptions.PARALLELISM.defaultValue());
9494
}
95+
envOption.put(EnvCommonOptions.PARALLELISM.key(), String.valueOf(parallelism));
9596
Dataset<Row> dataset =
9697
sparkRuntimeEnvironment
9798
.getSparkSession()
9899
.read()
99100
.format(SeaTunnelSource.class.getSimpleName())
100-
.option(EnvCommonOptions.PARALLELISM.key(), parallelism)
101101
.option(
102102
Constants.SOURCE_SERIALIZATION,
103103
SerializationUtils.objectToString(source))

0 commit comments

Comments
 (0)