[feature](iceberg) support insert into iceberg table with sort-order - #60540
Merged
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 31149 ms |
ClickBench: Total hot run time: 28.15 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds support for writing Iceberg tables with sort-order, allowing data to be sorted during writes and adding lower/upper bounds statistics to enable query-time data file pruning.
Changes:
- Added column statistics collection (column sizes, value counts, null counts, lower/upper bounds) for both Parquet and ORC formats
- Introduced VIcebergSortWriter that sorts data in-memory before writing to files, with spilling support when memory limits are exceeded
- Extended Iceberg table sink to support sort-info from table properties and use the sort writer when sort order is configured
- Added SpillIcebergTableSinkOperator to support spillable sorting operations in the pipeline execution model
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| gensrc/thrift/DataSinks.thrift | Added TIcebergColumnStats structure and integrated it into TIcebergCommitData; added sort_info to TIcebergTableSink |
| fe/fe-core/src/main/java/org/apache/doris/planner/IcebergTableSink.java | Modified to convert Iceberg table sort order to SortInfo and pass output expressions for sorting |
| fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java | Added logic to set output expressions on IcebergTableSink for sort writer |
| fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/helper/IcebergWriterHelper.java | Enhanced to build Metrics from column statistics and pass to DataFile builder |
| fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergTransaction.java | Updated to use new convertToWriterResult signature with Table parameter |
| be/src/vec/sink/writer/iceberg/vpartition_writer_base.h | New base interface for partition writers to support polymorphism |
| be/src/vec/sink/writer/iceberg/viceberg_table_writer.h | Modified to use IPartitionWriterBase interface and track current writer |
| be/src/vec/sink/writer/iceberg/viceberg_table_writer.cpp | Updated to create VIcebergSortWriter when sort_info is present |
| be/src/vec/sink/writer/iceberg/viceberg_sort_writer.h | New writer that sorts data and writes to files with spilling support |
| be/src/vec/sink/writer/iceberg/viceberg_partition_writer.h | Modified to implement IPartitionWriterBase interface |
| be/src/vec/sink/writer/iceberg/viceberg_partition_writer.cpp | Added statistics collection after closing file writers |
| be/src/vec/runtime/vparquet_transformer.h | Added collect_file_statistics_after_close method |
| be/src/vec/runtime/vparquet_transformer.cpp | Implemented statistics collection by reading Parquet file metadata |
| be/src/vec/runtime/vorc_transformer.h | Added collect_file_statistics_after_close method and helper methods |
| be/src/vec/runtime/vorc_transformer.cpp | Implemented statistics collection by reading ORC file metadata |
| be/src/vec/exec/format/table/parquet_utils.h | Added merge_stats function declaration |
| be/src/vec/exec/format/table/parquet_utils.cpp | Implemented merge_stats to merge Parquet statistics across row groups |
| be/src/vec/common/sort/sorter.h | Exposed merge_sort_state and _do_sort methods for external use |
| be/src/pipeline/pipeline_fragment_context.cpp | Added logic to create SpillIcebergTableSinkOperator when sort_info is present |
| be/src/pipeline/exec/spill_iceberg_table_sink_operator.h | New operator for spillable Iceberg table sink operations |
| be/src/pipeline/exec/spill_iceberg_table_sink_operator.cpp | Implemented spillable sink operator with memory revocation support |
| be/src/pipeline/exec/operator.cpp | Registered new SpillIcebergTableSinkLocalState operator |
| be/src/pipeline/exec/iceberg_table_sink_operator.cpp | Updated init_properties call with new signature |
| regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_stats2.groovy | Added comprehensive test for statistics collection in Parquet and ORC |
| regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_stats2.out | Test expected output |
| docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/iceberg/run26.sql | Setup script for test tables with sort order |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
morningman
reviewed
Feb 10, 2026
zhangstar333
requested review from
CalvinKirs and
dataroaring
as code owners
February 10, 2026 14:21
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 30584 ms |
ClickBench: Total hot run time: 28.88 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
1 similar comment
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
github-actions
Bot
removed
the
approved
Contributor
FE UT Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 28969 ms |
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
morningman
approved these changes
Feb 14, 2026
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
run check_coverage |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
morningman
approved these changes
Feb 20, 2026
morningman
added a commit
to morningman/doris
that referenced
this pull request
Mar 3, 2026
Based on PR apache#60540 review suggestions: 6.1: Encapsulate _current_writer in VIcebergTableWriter - Move _current_writer from public to private - Add const getter method current_writer() - Update SpillIcebergTableSinkLocalState to use getter 6.2: Split VIcebergSortWriter header-only into .h + .cpp - Move all method implementations to viceberg_sort_writer.cpp - Replace heavy includes (spill_stream.h, spill_stream_manager.h) with forward declarations in header to reduce compilation dependencies - Add comprehensive documentation comments to all methods and members 6.4: Fix error handling in VIcebergSortWriter::close() - Track internal_status for intermediate operations - Pass actual error status to underlying partition writer's close() instead of the original caller status (which could be OK) 6.5: Add explicit do_sort() in _do_spill() - Call _sorter->do_sort() before prepare_for_read() for clarity - Add detailed comments explaining the relationship with FullSorter::prepare_for_read() internal sorting 6.6: Fix typo 'SpillReadDerializeBlockTime' -> 'SpillReadDeserializeBlockTime' - Fix counter name string in spill_iceberg_table_sink_operator.cpp - Fix counter name string in operator.h (PipelineXSpillLocalState) - Fix counter lookup string in spill_reader.h - Fix member variable name _spill_read_derialize_block_timer
16 tasks
morningman
added a commit
that referenced
this pull request
Mar 6, 2026
### What problem does this PR solve? Based on PR #60540 review suggestions: 1: Encapsulate _current_writer in VIcebergTableWriter - Move _current_writer from public to private - Add const getter method current_writer() - Update SpillIcebergTableSinkLocalState to use getter 2: Split VIcebergSortWriter header-only into .h + .cpp - Move all method implementations to viceberg_sort_writer.cpp - Replace heavy includes (spill_stream.h, spill_stream_manager.h) with forward declarations in header to reduce compilation dependencies - Add comprehensive documentation comments to all methods and members 4: Fix error handling in VIcebergSortWriter::close() - Track internal_status for intermediate operations - Pass actual error status to underlying partition writer's close() instead of the original caller status (which could be OK) 5: Add explicit do_sort() in _do_spill() - Call _sorter->do_sort() before prepare_for_read() for clarity - Add detailed comments explaining the relationship with FullSorter::prepare_for_read() internal sorting 6: Fix typo 'SpillReadDerializeBlockTime' -> 'SpillReadDeserializeBlockTime' - Fix counter name string in spill_iceberg_table_sink_operator.cpp - Fix counter name string in operator.h (PipelineXSpillLocalState) - Fix counter lookup string in spill_reader.h - Fix member variable name _spill_read_derialize_block_timer
zhangstar333
added a commit
to zhangstar333/incubator-doris
that referenced
this pull request
Mar 17, 2026
…pache#60540) Problem Summary: support write iceberg table with sort-order, the write data have been local sorted, and have add lower/upper_bounds metadata. so the iceberg plan could use it to prune datafile. **Notes**: this is only a local sort, not global sort. so if you are more parallel about iceberg writer, you many see overlapping of lower/upper_bounds between files. if you need a global sort, maybe could add order by cluster in the insert SQL. you could create table, and then alter table eg: ``` CREATE TABLE test_table2 ( id INT, name STRING, score DOUBLE, create_time datetime ) ORDER BY ( id ASC NULLS FIRST, score DESC NULLS LAST) PROPERTIES ( 'write-format'='ORC' ); ```
morningman
added a commit
to morningman/doris
that referenced
this pull request
Mar 18, 2026
…e#60978) ### What problem does this PR solve? Based on PR apache#60540 review suggestions: 1: Encapsulate _current_writer in VIcebergTableWriter - Move _current_writer from public to private - Add const getter method current_writer() - Update SpillIcebergTableSinkLocalState to use getter 2: Split VIcebergSortWriter header-only into .h + .cpp - Move all method implementations to viceberg_sort_writer.cpp - Replace heavy includes (spill_stream.h, spill_stream_manager.h) with forward declarations in header to reduce compilation dependencies - Add comprehensive documentation comments to all methods and members 4: Fix error handling in VIcebergSortWriter::close() - Track internal_status for intermediate operations - Pass actual error status to underlying partition writer's close() instead of the original caller status (which could be OK) 5: Add explicit do_sort() in _do_spill() - Call _sorter->do_sort() before prepare_for_read() for clarity - Add detailed comments explaining the relationship with FullSorter::prepare_for_read() internal sorting 6: Fix typo 'SpillReadDerializeBlockTime' -> 'SpillReadDeserializeBlockTime' - Fix counter name string in spill_iceberg_table_sink_operator.cpp - Fix counter name string in operator.h (PipelineXSpillLocalState) - Fix counter lookup string in spill_reader.h - Fix member variable name _spill_read_derialize_block_timer
zhangstar333
added a commit
to zhangstar333/incubator-doris
that referenced
this pull request
Apr 10, 2026
…pache#60540) Problem Summary: support write iceberg table with sort-order, the write data have been local sorted, and have add lower/upper_bounds metadata. so the iceberg plan could use it to prune datafile. **Notes**: this is only a local sort, not global sort. so if you are more parallel about iceberg writer, you many see overlapping of lower/upper_bounds between files. if you need a global sort, maybe could add order by cluster in the insert SQL. you could create table, and then alter table eg: ``` CREATE TABLE test_table2 ( id INT, name STRING, score DOUBLE, create_time datetime ) ORDER BY ( id ASC NULLS FIRST, score DESC NULLS LAST) PROPERTIES ( 'write-format'='ORC' ); ```
16 tasks
76 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Problem Summary:
support write iceberg table with sort-order, the write data have been local sorted, and have add lower/upper_bounds metadata. so the iceberg plan could use it to prune datafile.
Notes: this is only a local sort, not global sort. so if you are more parallel about iceberg writer, you many see overlapping of lower/upper_bounds between files.
if you need a global sort, maybe could add order by cluster in the insert SQL.
you could create table, and then alter table eg:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)