Add zebrad migrate-from-zcashd command

[str4d]

Motivation

If a user already has a zcashd node, they should be able to use its local block data to bootstrap a Zebra node, instead of requiring a fresh download.

Solution

This PR adds a command that parses the blkNNNNN.dat files from a zcashd datadir, and replays them to the Zebra state as if it just received them from the network.

Tests

Currently WIP, tested with my local zcashd node and it doesn't work due to Zebra not implementing large parts of the historic Zcash protocol spec.

Specifications & References

Follow-up Work

Needs advice on how to alter Zebra to enable this. We either need:

• some way to alter Zebra so we can leverage its most-work chain-following logic while skipping all validation for old blocks in a way that also ensures it never persists orphan blocks (because zcashd saves all blocks as-received, even if they are later orphaned) or invalid blocks to state (which can exist in the block files on disk, if they were individually valid but contextually invalid).
• a LevelDB dependency (which would enable Zebra to read the zcashd block index and directly find the main chain blocks, ignoring orphans.

PR Checklist

• The PR name is suitable for the release notes.
• The solution is tested.
• The documentation is up to date.

[arya2]

some way to alter Zebra so we can leverage its most-work chain-following logic while skipping all validation for old blocks in a way that also ensures it never persists orphan blocks (because zcashd saves all blocks as-received, even if they are later orphaned) or invalid blocks to state (which can exist in the block files on disk, if they were individually valid but contextually invalid).

I would suggest using the block verifier router, Zebra will skip validation for checkpointed blocks.

[arya2]

Suggested change

	//! * TODO: We currently perform full validaton because we don't read from the
	//! `zcashd` block index.

[arya2]

Suggested change

	io::{AsyncReadExt, BufReader},
	io::{AsyncReadExt, BufReader},
	sync::oneshot,

[arya2]

Suggested change

	use tower::{Service, ServiceExt};
	use tower::{buffer::Buffer, util::BoxService, Service, ServiceBuilder, ServiceExt};

[arya2]

Suggested change

	};
	chain_tip::ChainTip,
	};
	use zebra_node_services::mempool;

[arya2]

Suggested change

	/// The maximum number of unprocessed messages to buffer for
	/// the state service when migrating from zcashd.
	const STATE_BUFFER_BOUND: usize = 100;

[arya2]

Suggested change

	self.migrate(app_config.network.network.clone(), app_config.state.clone())
	.await
	.map_err(|e| eyre!(e))
	self.migrate(
	app_config.network.network.clone(),
	app_config.state.clone(),
	app_config.consensus.clone(),
	)
	.await
	.map_err(|e| eyre!(e))

[arya2]

Suggested change

	target_config: zebra_state::Config,
	target_state_config: zebra_state::Config,
	target_consensus_config: zebra_consensus::Config,

[arya2]

Suggested change

	info!(?target_config, "initializing target state service");
	info!(
	?target_state_config,
	?target_consensus_config,
	"initializing target state service"
	);

[arya2]

I'm skeptical of there being any noticeable benefit from using PrepareForBulkLoad() until we switch to shardtrees, but it would be nice to have.

Note: The transaction verifier works with a dummy mempool setup channel receiver.

Suggested change

	let (
	mut target_state,
	_target_read_only_state_service,
	_target_latest_chain_tip,
	_target_chain_tip_change,
	) = zebra_state::spawn_init(target_config.clone(), &network, Height::MAX, 0).await?;
	let (
	target_state_service,
	_target_read_state,
	target_latest_chain_tip,
	_target_chain_tip_change,
	) = zebra_state::spawn_init(target_state_config.clone(), &network, Height::MAX, 0).await?;
	let target_state = ServiceBuilder::new()
	.buffer(STATE_BUFFER_BOUND)
	.service(target_state_service);
	let (
	mut block_verifier_router,
	_tx_verifier,
	_consensus_task_handles,
	_max_checkpoint_height,
	) = zebra_consensus::router::init(
	target_consensus_config,
	&network,
	target_state,
	oneshot::channel::<
	Buffer<BoxService<mempool::Request, mempool::Response, BoxError>, mempool::Request>,
	>()
	.1,
	)
	.await;

[arya2]

The first two checks here:

            if source_block_hash != target_block_commit_hash
                || source_block_hash != target_block_data_hash
                || source_block != target_block

are now happening here in the block verifier and here in the checkpoint verifier.

Is the third check necessary?

Suggested change

	let target_block_commit_hash = target_state
	.ready()
	.await?
	.call(if height == Height::MIN {
	// We can always trust the genesis block from a `zcashd` datadir to be
	// the only block with height 0 due to how `zcashd` sideloads it into
	// new datadirs.
	zebra_state::Request::CommitCheckpointVerifiedBlock(source_block.clone().into())
	} else {
	// We can't use `CommitCheckpointVerifiedBlock` here because `zcashd`
	// block files contain the blocks as-received from the network, and
	// can include orphaned blocks that aren't within the checkpoint.
	// TODO: The only consensus logic we need Zebra to do for historic
	// blocks is to find the most-work chain; every other consensus rule
	// can be presumed-valid for blocks that end up in the main chain
	// (and certainly for blocks that end up in the checkpoint).
	zebra_state::Request::CommitSemanticallyVerifiedBlock(
	source_block.clone().into(),
	)
	})
	.await?;
	let target_block_commit_hash = match target_block_commit_hash {
	zebra_state::Response::Committed(target_block_commit_hash) => {
	trace!(?target_block_commit_hash, "wrote Zebra block");
	target_block_commit_hash
	}
	response => Err(format!(
	"unexpected response to CommitSemanticallyVerifiedBlock request, height: {}\n \
	response: {response:?}",
	height.0,
	))?,
	};
	// Read written block from target
	let target_block = target_state
	.ready()
	.await?
	.call(zebra_state::Request::Block(height.into()))
	.await?;
	let target_block = match target_block {
	zebra_state::Response::Block(Some(target_block)) => {
	trace!(?height, %target_block, "read Zebra block");
	target_block
	}
	zebra_state::Response::Block(None) => Err(format!(
	"unexpected missing Zebra block, height: {}",
	height.0,
	))?,
	response => Err(format!(
	"unexpected response to Block request, height: {},\n \
	response: {response:?}",
	height.0,
	))?,
	};
	let target_block_data_hash = target_block.hash();
	// Check for data errors
	//
	// These checks make sure that Zebra doesn't corrupt the block data
	// when serializing it.
	// Zebra currently serializes `Block` structs into bytes while writing,
	// then deserializes bytes into new `Block` structs when reading.
	// So these checks are sufficient to detect block data corruption.
	//
	// If Zebra starts reusing cached `Block` structs after writing them,
	// we'll also need to check `Block` structs created from the actual database bytes.
	if source_block_hash != target_block_commit_hash
	|| source_block_hash != target_block_data_hash
	|| source_block != target_block
	{
	Err(format!(
	"unexpected mismatch between zcashd and Zebra blocks,\n \
	max copy height: {max_copy_height:?},\n \
	zcashd hash: {source_block_hash:?},\n \
	Zebra commit hash: {target_block_commit_hash:?},\n \
	Zebra data hash: {target_block_data_hash:?},\n \
	zcashd block: {source_block:?},\n \
	Zebra block: {target_block:?}",
	))?;
	}
	block_verifier_router
	.ready()
	.await?
	.call(zebra_consensus::Request::Commit(source_block))
	.await?;

[arya2]

Suggested change

	let initial_target_tip = match initial_target_tip {
	zebra_state::Response::Tip(target_tip) => target_tip,
	response => Err(format!("unexpected response to Tip request: {response:?}",))?,
	};
	let min_target_height = initial_target_tip
	.map(|target_tip| target_tip.0 .0 + 1)
	.unwrap_or(0);
	let initial_target_tip = target_latest_chain_tip.best_tip_height();
	let min_target_height = initial_target_tip
	.map(|Height(target_tip)| target_tip + 1)
	.unwrap_or(0);

[oxarbitrage]

Note: During the Arborist call, there was a suggestion to move this code to the zebra-utils crate as a new binary instead of a new zebrad command.

[arya2]

Note: During the Arborist call, there was a suggestion to move this code to the zebra-utils crate as a new binary instead of a new zebrad command.

It would be nice to avoid adding a dependency to zebrad, especially one like levelsdb, but if that can be avoided, having a command in zebrad seems like a better UX while needing fewer modifications.