Add implementation for exponential histogram merging and percentiles #131220

JonasKunz · 2025-07-14T14:35:50Z

Still a WIP.

Have you signed the contributor license agreement?
Have you followed the contributor guidelines?
If submitting code, have you built your formula locally prior to submission with gradle check?
If submitting code, is your pull request against main? Unless there is a good reason otherwise, we prefer pull requests against main and will backport as needed.
If submitting code, have you checked that your submission is for an OS and architecture that we support?
If you are submitting this code for a class then read our policy for that.

libs/exponential-histogram/README.md

kkrik-es · 2025-07-15T13:05:35Z

libs/exponential-histogram/README.md

+
+This offers significant benefits for distributions with fewer distinct values:
+If we have at least as many buckets as we have distinct values to store in the histogram, we can almost exactly represent this distribution.
+This can be achieved by simply maintaining the scale at the maximum supported value (so the buckets become the smallest).


These two lines are tricky to grasp. Is there some external reference explaining the differences between dense and sparse storage, and how the number of buckets and scale are associated?

I don't think there are any sources explaining this clearly. The DDSketch and UDDSketch papers use sparse representations in the papers, but practically all implementations are dense due to the O(1) insertion time I assume.

I added a concrete example in 5e1ca08, hope this makes it clearer?

...ponential-histogram/src/main/java/org/elasticsearch/exponentialhistogram/DownscaleStats.java

kkrik-es · 2025-07-15T15:00:11Z

...ogram/src/main/java/org/elasticsearch/exponentialhistogram/ExponentialHistogramQuantile.java

+        long negCount = getTotalCount(histo.negativeBuckets());
+        long posCount = getTotalCount(histo.positiveBuckets());
+
+        long totalCount = zeroCount + negCount + posCount;


It's likely that we'll get many quantile calls for the same histogram, so we might as well store this number in the histogram.

Yeah this was part of the TODOs, but should be simple enough to do it now.

If we expect many calls for the same histogram, it could make sense to build an array representing the prefix-sum of the counts. This allows for percentile computation in O(log n) instead of O(n) at the cost of a little memory. But this is definitely something for later if we see the need for it.

...ogram/src/main/java/org/elasticsearch/exponentialhistogram/ExponentialHistogramQuantile.java

...al-histogram/src/test/java/org/elasticsearch/exponentialhistogram/QuantileAccuracyTests.java

.../src/main/java/org/elasticsearch/exponentialhistogram/FixedCapacityExponentialHistogram.java

JonasKunz · 2025-07-18T13:19:58Z

Applied the changes of the first reviews and a bit of cleanup / refactoring. I also am hopeful that the tests do not introduce flakes, as I ran the randomized ones locally continously for quite a while.

elasticsearchmachine · 2025-07-18T14:37:59Z

Pinging @elastic/es-storage-engine (Team:StorageEngine)

JonasKunz added 18 commits July 8, 2025 11:14

Initial experiments commit

43dd073

Clean up generator

da159a9

Clean up merger

fa4efe0

More tests, a bit of cleanup

fba967f

Stash benchmark changes

cef3b11

Merge remote-tracking branch 'elastic/main' into exponentional-histos

aac9f6d

spotless, checkstyle

6a2b62f

more build fixes

eb955cd

Fix license headers

2eb5fdd

spotless round 2

fd7064e

Fix tests, implement benchmarks

66b5e2c

Reduce max scale to preserve numeric accuracy

2f293d0

Check for sane scale and indices

91193bc

Fix and clean percentile computation

92efdcf

Add some tests based on TDigestTest

e6924e9

Clean up, bug fixes and javadoc

cab3fdf

Remove dead code

454a9cc

A bit more javadoc

486a8bd

elasticsearchmachine added v9.2.0 external-contributor Pull request authored by a developer outside the Elasticsearch team labels Jul 14, 2025

JonasKunz added 3 commits July 15, 2025 11:41

AI-assisted javadoc and spotless

7c6655b

Readme bullet points

a980c0e

Add readme

a46914a

JonasKunz force-pushed the exponentional-histos branch from 7711093 to a46914a Compare July 15, 2025 12:40