Skip to content

Pipe: serialize sink transfers by region#17946

Merged
jt2594838 merged 1 commit into
masterfrom
pipe-serialize-sink-by-region
Jun 16, 2026
Merged

Pipe: serialize sink transfers by region#17946
jt2594838 merged 1 commit into
masterfrom
pipe-serialize-sink-by-region

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Jun 16, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR adds an optional Pipe sink config to serialize transfers within the same data region while keeping different data regions independently parallel.

  • Add connector.serialize-by-region / sink.serialize-by-region, defaulting to true.
  • When enabled, data region sink subtasks are grouped by region and use one sink subtask per region, so the same region is sent serially while different regions can still run in parallel.
  • When sink/connector is not explicitly declared and serialize-by-region is enabled, default the IoTDB thrift sink to iotdb-thrift-sync-connector.
  • Preserve explicit sink selection: iotdb-thrift-connector and iotdb-thrift-async-connector still use the async data-region sink.
  • Preserve old connector.parallel.tasks / sink.parallel.tasks behavior when serialize-by-region=false.
  • Remove the async sink in-flight handler waiting changes from this PR.

Tests

  • git diff --check
  • git diff --cached --check
  • git diff origin/master -- iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/sink/protocol/thrift/async/IoTDBDataRegionAsyncSink.java
  • mvn -DskipTests spotless:apply -pl iotdb-core/node-commons,iotdb-core/datanode
  • mvn -DskipTests spotless:apply -pl integration-test -P with-integration-tests
  • Attempted mvn -pl iotdb-core/datanode -Dtest=PipeDataNodePluginAgentTest test -Ddevelocity.off=true, but it is blocked before running tests by existing datanode compile errors in generated/queryengine/subscription sources, including missing parser contexts, missing IFill/IFillFilter, missing ComparatorChain, missing PERCENTILE, missing getModeMapSizeThreshold, TopicMeta owner methods, and a DataNodeTableOperatorGenerator#buildAggregator signature mismatch.

@codecov

codecov Bot commented Jun 16, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 29.62963% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 40.93%. Comparing base (c39061a) to head (dc0e1a5).
⚠️ Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
...gent/task/subtask/sink/PipeSinkSubtaskManager.java 0.00% 19 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #17946      +/-   ##
============================================
- Coverage     41.05%   40.93%   -0.12%     
  Complexity      318      318              
============================================
  Files          5258     5258              
  Lines        365541   365550       +9     
  Branches      47256    47259       +3     
============================================
- Hits         150081   149654     -427     
- Misses       215460   215896     +436     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Caideyipi Caideyipi force-pushed the pipe-serialize-sink-by-region branch from 3789288 to 766d3e3 Compare June 16, 2026 02:51
@Caideyipi Caideyipi force-pushed the pipe-serialize-sink-by-region branch from 766d3e3 to dc0e1a5 Compare June 16, 2026 02:58
@sonarqubecloud

Copy link
Copy Markdown

@jt2594838 jt2594838 merged commit 5e6f1c2 into master Jun 16, 2026
40 of 43 checks passed
@jt2594838 jt2594838 deleted the pipe-serialize-sink-by-region branch June 16, 2026 08:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants