refactor: Mutil Column Aggregate Function State Serialization Interface #18398

forsaken628 · 2025-07-21T10:18:45Z

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Serializing the Aggregate Function State into multiple columns instead of a single binary column helps to reduce the size of the serialization and reduce io.

break change: aggregate index must rebuild handly

Tests

Unit Test
Logic Test
Benchmark Test
No Test - Explain why

Type of change

Bug Fix (non-breaking change which fixes an issue)
New Feature (non-breaking change which adds functionality)
Breaking Change (fix or feature that could cause existing functionality not to work as expected)
Documentation Update
Refactoring
Performance Improvement
Other (please describe):

This change is

tests/sqllogictests/suites/ee/02_ee_aggregating_index/02_0001_agg_index_projected_scan.test

src/query/storages/fuse/src/operations/analyze.rs

src/query/functions/src/aggregates/adaptors/aggregate_null_adaptor.rs

sundy-li · 2025-07-24T03:43:02Z

Do you have any performance test results for the agg state serialized size and query latency?

forsaken628 · 2025-07-24T04:13:04Z

Do you have any performance test results for the agg state serialized size and query latency?

Waiting to add. But optimization for each function is the next step, and the interface is refactored here first.

github-actions · 2025-07-24T05:09:12Z

Docker Image for PR

tag: pr-18398-ccff018-1753333673

note: this image tag is only available for internal use.

github-actions · 2025-07-25T03:53:57Z

Docker Image for PR

tag: pr-18398-00301bc-1753415560

note: this image tag is only available for internal use.

github-actions · 2025-07-25T05:24:32Z

ClickBench Report

github-actions · 2025-07-25T16:56:53Z

Docker Image for PR

tag: pr-18398-0467ff9-1753462537

note: this image tag is only available for internal use.

github-actions · 2025-07-25T18:25:33Z

ClickBench Report

github-actions · 2025-07-25T23:37:20Z

ClickBench Report

github-actions · 2025-07-26T10:03:48Z

Docker Image for PR

tag: pr-18398-d586479-1753524149

note: this image tag is only available for internal use.

github-actions · 2025-07-26T11:32:12Z

ClickBench Report

forsaken628 · 2025-07-26T14:52:58Z

benchmark:

sql

explain analyze select sum(number+1) from numbers(100000000) group by number IGNORE_RESULT;

1.2.778
size: medium
duration: 2.2s

    └── AggregateFinal
        ├── output columns: [sum(number) (#1), count(number) (#2), numbers.number (#0)]
        ├── group by: [number]
        ├── aggregate functions: [sum(number), count()]
        ├── estimated rows: 100000000.00
        ├── cpu time: 32.391624724s
        ├── wait time: 3.605208609s
        ├── exchange bytes: 818.73 MiB
        ├── output rows: 95.7 million
        ├── output bytes: 2.15 GiB

pr
size: medium
duration: 1.8s

    └── AggregateFinal
        ├── output columns: [sum(number) (#1), count(number) (#2), numbers.number (#0)]
        ├── group by: [number]
        ├── aggregate functions: [sum(number), count()]
        ├── estimated rows: 100000000.00
        ├── cpu time: 26.94341643s
        ├── wait time: 3.161924207s
        ├── exchange bytes: 415.09 MiB
        ├── output rows: 89.46 million
        ├── output bytes: 2.01 GiB

BohuTANG · 2025-07-27T02:43:18Z

👍
Let me benchmark this PR next

BohuTANG · 2025-07-27T07:07:43Z

checksb test passed!

BohuTANG · 2025-07-27T07:32:14Z

Databend Query Performance Comparison

Query: select sum(number+1),count(number) from numbers(10000000000) group by number IGNORE_RESULT;
Setup: 2 nodes (32C256G each), 10B numbers (74.51 GiB)

Results Summary

Metric	This PR	Main	Difference
Execution Time	2m 50s	3m 3s	🟢 13s faster
Spilled Data	46.54 GiB	86.79 GiB	🟢 40.25 GiB less

Key Differences

Component	This PR	Main	Difference
AggregateFinal - CPU Time	4470.8s	5206.9s	🟢 736.1s less
AggregateFinal - Wait Time	1245.5s	2082.9s	🟢 837.4s less
AggregateFinal - Exchange Bytes	20.09 GiB	37.38 GiB	🟢 17.29 GiB less
Spill Write Time	1729.7s	3105.9s	🟢 1376.2s less
Spill Read Time	1223.4s	1853.7s	🟢 630.3s less
Bytes Spilled (Write)	46.54 GiB	86.79 GiB	🟢 40.25 GiB less
Bytes Spilled (Read)	46.54 GiB	86.79 GiB	🟢 40.25 GiB less
EvalScalar - CPU Time	86.2s	77.7s	🔴 8.5s more

Execution Plans

This PR (2m 50s)

Exchange
├── output columns: [count(number) (#2), sum(number) + 1 * count(number) (#3)]
├── exchange type: Merge
└── EvalScalar
    ├── output columns: [count(number) (#2), sum(number) + 1 * count(number) (#3)]
    ├── expressions: [sum(number) (#1) + CAST(1 * count(number) (#2) AS UInt64 NULL)]
    ├── estimated rows: 10000000000.00
    ├── cpu time: 86.236814334s
    ├── output rows: 10 billion
    ├── output bytes: 150.18 GiB
    └── AggregateFinal
        ├── output columns: [sum(number) (#1), count(number) (#2), numbers.number (#0)]
        ├── group by: [number]
        ├── aggregate functions: [sum(number), count()]
        ├── estimated rows: 10000000000.00
        ├── cpu time: 4470.835086584s
        ├── wait time: 1245.548067478s
        ├── exchange bytes: 20.09 GiB
        ├── output rows: 10 billion
        ├── output bytes: 224.68 GiB
        ├── numbers remote spilled by write: 128
        ├── bytes remote spilled by write: 46.54 GiB
        ├── remote spilled time by write: 1729.697s
        ├── numbers remote spilled by read: 16384
        ├── bytes remote spilled by read: 46.54 GiB
        ├── remote spilled time by read: 1223.388s
        └── Exchange
            ├── output columns: [sum(number) (#1), count(number) (#2), numbers.number (#0)]
            ├── exchange type: Hash(0)
            └── AggregatePartial
                ├── group by: [number]
                ├── aggregate functions: [sum(number), count()]
                ├── estimated rows: 10000000000.00
                ├── cpu time: 1197.109558377s
                └── TableScan
                    ├── table: default.system.numbers
                    ├── output columns: [number (#0)]
                    ├── read rows: 10000000000
                    ├── read size: 74.51 GiB
                    ├── partitions total: 152588
                    ├── partitions scanned: 152588
                    ├── push downs: [filters: [], limit: NONE]
                    ├── estimated rows: 10000000000.00
                    ├── cpu time: 26.056168242s
                    ├── output rows: 10 billion
                    ├── output bytes: 74.51 GiB
                    └── bytes scanned: 74.51 GiB

Main (3m 3s)

Exchange
├── output columns: [count(number) (#2), sum(number) + 1 * count(number) (#3)]
├── exchange type: Merge
└── EvalScalar
    ├── output columns: [count(number) (#2), sum(number) + 1 * count(number) (#3)]
    ├── expressions: [sum(number) (#1) + CAST(1 * count(number) (#2) AS UInt64 NULL)]
    ├── estimated rows: 10000000000.00
    ├── cpu time: 77.694080059s
    ├── output rows: 10 billion
    ├── output bytes: 150.18 GiB
    └── AggregateFinal
        ├── output columns: [sum(number) (#1), count(number) (#2), numbers.number (#0)]
        ├── group by: [number]
        ├── aggregate functions: [sum(number), count()]
        ├── estimated rows: 10000000000.00
        ├── cpu time: 5206.939129139s
        ├── wait time: 2082.870426479s
        ├── exchange bytes: 37.38 GiB
        ├── output rows: 10 billion
        ├── output bytes: 224.68 GiB
        ├── numbers remote spilled by write: 128
        ├── bytes remote spilled by write: 86.79 GiB
        ├── remote spilled time by write: 3105.909s
        ├── numbers remote spilled by read: 16384
        ├── bytes remote spilled by read: 86.79 GiB
        ├── remote spilled time by read: 1853.687s
        └── Exchange
            ├── output columns: [sum(number) (#1), count(number) (#2), numbers.number (#0)]
            ├── exchange type: Hash(0)
            └── AggregatePartial
                ├── group by: [number]
                ├── aggregate functions: [sum(number), count()]
                ├── estimated rows: 10000000000.00
                ├── cpu time: 1172.800108227s
                └── TableScan
                    ├── table: default.system.numbers
                    ├── output columns: [number (#0)]
                    ├── read rows: 10000000000
                    ├── read size: 74.51 GiB
                    ├── partitions total: 152588
                    ├── partitions scanned: 152588
                    ├── push downs: [filters: [], limit: NONE]
                    ├── estimated rows: 10000000000.00
                    ├── cpu time: 25.544488872s
                    ├── output rows: 10 billion
                    ├── output bytes: 74.51 GiB
                    └── bytes scanned: 74.51 GiB

github-actions bot added the pr-refactor this PR changes the code base without new features or bugfix label Jul 21, 2025

trait

028584e

forsaken628 force-pushed the aggr-ser branch from 1ac6fd1 to 028584e Compare July 21, 2025 10:27

fix

d64d2ec

forsaken628 force-pushed the aggr-ser branch from 09770c7 to d64d2ec Compare July 22, 2025 07:29

forsaken628 added 5 commits July 22, 2025 17:35

fix

f303632

fix

64d74db

null adaptor

de26741

serialize_type

dfec718

ee

2d76884

forsaken628 force-pushed the aggr-ser branch from 071ddb0 to 2d76884 Compare July 23, 2025 08:48

fix

5b16525

forsaken628 commented Jul 23, 2025

View reviewed changes

tests/sqllogictests/suites/ee/02_ee_aggregating_index/02_0001_agg_index_projected_scan.test Outdated Show resolved Hide resolved

src/query/storages/fuse/src/operations/analyze.rs Show resolved Hide resolved

forsaken628 changed the title ~~refactor: New Aggregate Function State Serialization Interface~~ refactor: Mutil Column Aggregate Function State Serialization Interface Jul 23, 2025

forsaken628 marked this pull request as ready for review July 23, 2025 12:42

forsaken628 requested review from sundy-li and b41sh July 23, 2025 12:42

forsaken628 added 2 commits July 23, 2025 20:52

if

828da4a

fix

72f3a29

sundy-li reviewed Jul 24, 2025

View reviewed changes

src/query/functions/src/aggregates/adaptors/aggregate_null_adaptor.rs Outdated Show resolved Hide resolved

sundy-li reviewed Jul 24, 2025

View reviewed changes

src/query/functions/src/aggregates/adaptors/aggregate_null_adaptor.rs Outdated Show resolved Hide resolved

sundy-li requested a review from zhang2014 July 24, 2025 03:43

fix

f16744f

forsaken628 force-pushed the aggr-ser branch from 7267f58 to f16744f Compare July 24, 2025 03:54

forsaken628 added the ci-cloud Build docker image for cloud test label Jul 24, 2025

zhang2014 approved these changes Jul 24, 2025

View reviewed changes

zhang2014 added the ci-benchmark Benchmark: run all test label Jul 24, 2025

forsaken628 added the ci-benchmark Benchmark: run all test label Jul 25, 2025

forsaken628 added 2 commits July 25, 2025 15:27

test

3942864

sum

36fcb59

forsaken628 force-pushed the aggr-ser branch 2 times, most recently from b3ea7a9 to 0078deb Compare July 25, 2025 11:56

batch_serialize

45a3f7b

forsaken628 force-pushed the aggr-ser branch from 0078deb to 45a3f7b Compare July 25, 2025 13:54

forsaken628 added ci-benchmark Benchmark: run all test and removed ci-benchmark Benchmark: run all test labels Jul 25, 2025

forsaken628 added 4 commits July 26, 2025 08:13

count

a8743ee

clean up

db81bfe

Merge remote-tracking branch 'up/main' into aggr-ser

abc6ae8

fix

3b18b07

forsaken628 added ci-benchmark Benchmark: run all test and removed ci-benchmark Benchmark: run all test labels Jul 26, 2025

Merge remote-tracking branch 'up/main' into aggr-ser

9780b92

sundy-li approved these changes Jul 27, 2025

View reviewed changes

forsaken628 merged commit cb63b57 into databendlabs:main Jul 27, 2025
86 checks passed

forsaken628 deleted the aggr-ser branch July 27, 2025 13:01

refactor: Mutil Column Aggregate Function State Serialization Interface #18398

refactor: Mutil Column Aggregate Function State Serialization Interface #18398

Uh oh!

Conversation

forsaken628 commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tests

Type of change

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sundy-li commented Jul 24, 2025

Uh oh!

forsaken628 commented Jul 24, 2025

Uh oh!

github-actions bot commented Jul 24, 2025

Docker Image for PR

Uh oh!

github-actions bot commented Jul 25, 2025

Docker Image for PR

Uh oh!

github-actions bot commented Jul 25, 2025

ClickBench Report

Uh oh!

github-actions bot commented Jul 25, 2025

Docker Image for PR

Uh oh!

github-actions bot commented Jul 25, 2025

ClickBench Report

Uh oh!

github-actions bot commented Jul 25, 2025

ClickBench Report

Uh oh!

github-actions bot commented Jul 26, 2025

Docker Image for PR

Uh oh!

github-actions bot commented Jul 26, 2025

ClickBench Report

Uh oh!

forsaken628 commented Jul 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BohuTANG commented Jul 27, 2025

Uh oh!

BohuTANG commented Jul 27, 2025

Uh oh!

BohuTANG commented Jul 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Databend Query Performance Comparison

Results Summary

Key Differences

Execution Plans

This PR (2m 50s)

Main (3m 3s)

Uh oh!

Uh oh!

Uh oh!

forsaken628 commented Jul 21, 2025 •

edited

Loading

forsaken628 commented Jul 26, 2025 •

edited

Loading

BohuTANG commented Jul 27, 2025 •

edited

Loading