CHIA-2136 have generate-chain fill compressed blocks #19932

jack60612 · 2025-08-10T07:19:30Z

Purpose:

Changes generate-chain to generate and fill compressed blocks.

Current Behavior:

no compressed blocks

New Behavior:

compressed blocks using in memory coinstore, and also std mempool

Testing Notes:

arvidn

did you explore adding another option to BlockTools, next to include_transactions that fills the block to the max, using the BlockBuilder directly? I suspect that would end up being simpler without affecting existing tests

chia/_tests/util/full_sync.py

arvidn · 2025-08-11T10:20:35Z

chia/simulator/block_tools.py

@@ -403,7 +403,10 @@ def setup_new_gen(
            assert rng is not None
            bundle, additions = make_spend_bundle(available_coins, wallet, rng)
            removals = bundle.removals()
-            program = simple_solution_generator(bundle).program
+            if curr.height >= self.constants.HARD_FORK_HEIGHT:


this is a reasonable change. It doesn't affect the number of transactions per block (which is good, that should stay the same). It does, however, change the chain, so the test_cache would have to be updated. Doing so expands the scope of this task quite a lot, so I think it probably best to avoid it.

tools/generate_chain.py

jack60612 · 2025-08-11T16:19:49Z

did you explore adding another option to BlockTools, next to include_transactions that fills the block to the max, using the BlockBuilder directly? I suspect that would end up being simpler without affecting existing tests

I'll look at this, I keep getting blocked and having to go down consensus rabbit holes.

correct spends and additions amounts

arvidn · 2025-08-14T06:56:53Z

chia/simulator/block_tools.py

+            assert rng is not None
+            assert curr.height >= self.constants.HARD_FORK_HEIGHT  # we need new compression for BlockBuilder
+            # function constants
+            adjusted_max_cost: uint64 = uint64(self.constants.MAX_BLOCK_COST_CLVM * block_fillrate / 100)


I suspect you could make this much simpler by not allowing the fill rate to be configured. We really just need full blocks (for the benchmark).

arvidn · 2025-08-14T06:58:19Z

chia/simulator/block_tools.py

+            adjusted_max_cost: uint64 = uint64(self.constants.MAX_BLOCK_COST_CLVM * block_fillrate / 100)
+            static_cost: uint64 = uint64(4839648)  # cond + exec cost per sb
+            static_comp_cost: uint64 = uint64(7684000)  # byte cost per sb after compression
+            max_batch_size: int = int(1400 * 0.10 * block_fillrate / 100)  # 10% of the way done with the block


The fewer iterations you need, the better performance. Did you experience some issue that was solved by having a max batch size?

arvidn · 2025-08-14T07:03:22Z

chia/simulator/block_tools.py

+            # function constants
+            adjusted_max_cost: uint64 = uint64(self.constants.MAX_BLOCK_COST_CLVM * block_fillrate / 100)
+            static_cost: uint64 = uint64(4839648)  # cond + exec cost per sb
+            static_comp_cost: uint64 = uint64(7684000)  # byte cost per sb after compression


this constant looks really suspicious. BlockBuilder will tell what the current block cost is (as long as the cost passed to add_spend_bundles() is correct).

if we want to reduce iterations we have to guesstimate the compressed block cost, so this kinda helps us skip iterations

arvidn · 2025-08-14T07:06:56Z

chia/simulator/block_tools.py

+                val_after_next_sb = (len(batch_bundles) + 1) * static_comp_cost + total_cost
+                if val_after_next_sb > adjusted_max_cost or len(batch_bundles) == max_batch_size:


Suggested change

val_after_next_sb = (len(batch_bundles) + 1) * static_comp_cost + total_cost

if val_after_next_sb > adjusted_max_cost or len(batch_bundles) == max_batch_size:

if len(batch_bundles) * static_cost + total_cost > adjusted_max_cost:

arvidn · 2025-08-14T07:07:38Z

chia/simulator/block_tools.py

+            batch_removals: list[Coin] = []
+            batch_additions: list[Coin] = []
+            while len(avail_coins) > 0:
+                val_after_next_sb = (len(batch_bundles) + 1) * static_comp_cost + total_cost


if static_comp_cost is really the true execution + condition cost for one transaction, why do you add 1 to len(batch_bundles)?

to account for the next one that's about to be added, but ill revisit

arvidn · 2025-08-14T07:09:57Z

chia/simulator/block_tools.py

@@ -712,9 +798,14 @@ def get_consecutive_blocks(
        include_transactions: bool = False,
        skip_overflow: bool = False,
        min_signage_point: int = -1,
+        block_fillrate: Optional[int] = None,


an alternative would be to make include_transactions be an integer, 0=no transactions, 1=5 transactions per block, 2=full blocks

sounds good

arvidn · 2025-08-14T07:11:15Z

chia/simulator/block_tools.py

-                            include_transactions=include_transactions,
-                            block_refs=block_refs,
-                        )
+                        if new_gen_cache is None:


maybe this cache should be moved into setup_new_gen(), since it's duplicated now

how would that persist across executions?

oh wait the class, i forgot it was part of that.

jack60612 added the Changed Required label for PR that categorizes merge commit message as "Changed" for changelog label Aug 10, 2025

arvidn reviewed Aug 11, 2025

View reviewed changes

jack60612 force-pushed the jn.bl-test branch from 40fd666 to 795b29e Compare August 11, 2025 17:25

jack60612 changed the title ~~have generate-chain fill compressed blocks~~ CHIA-2136 have generate-chain fill compressed blocks Aug 11, 2025

github-actions bot added the merge_conflict Branch has conflicts that prevent merge to main label Aug 13, 2025

jack60612 added 4 commits August 13, 2025 14:55

arvids changes

973e095

fix ci

8ee0147

update generate_chain to support new changes

f11a50f

correct spends and additions amounts

finalize block_tools changes

9e17324

jack60612 force-pushed the jn.bl-test branch from eec4de6 to 9e17324 Compare August 13, 2025 18:58

github-actions bot removed the merge_conflict Branch has conflicts that prevent merge to main label Aug 13, 2025

jack60612 requested a review from arvidn August 13, 2025 18:59

jack60612 added 3 commits August 13, 2025 15:02

revert extra changes

61b599d

fix regression

ffb9418

another reg

668c8ff

arvidn reviewed Aug 14, 2025

View reviewed changes

change logic

02e14ac

		val_after_next_sb = (len(batch_bundles) + 1) * static_comp_cost + total_cost
		if val_after_next_sb > adjusted_max_cost or len(batch_bundles) == max_batch_size:

	val_after_next_sb = (len(batch_bundles) + 1) * static_comp_cost + total_cost
	if val_after_next_sb > adjusted_max_cost or len(batch_bundles) == max_batch_size:
	if len(batch_bundles) * static_cost + total_cost > adjusted_max_cost:

CHIA-2136 have generate-chain fill compressed blocks #19932

Are you sure you want to change the base?

CHIA-2136 have generate-chain fill compressed blocks #19932

Uh oh!

Conversation

jack60612 commented Aug 10, 2025

Purpose:

Current Behavior:

New Behavior:

Testing Notes:

Uh oh!

arvidn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jack60612 commented Aug 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!