foster/bitcoin-core
0
0
Fork 0
mirror of https://github.com/bitcoin/bitcoin.git synced 2025-03-05 14:06:27 -05:00
Code Issues Activity

Merge bitcoin/bitcoin#27596: assumeutxo (2)

Browse source 
edbed31066 chainparams: add signet assumeutxo param at height 160_000 (Sjors Provoost)
b8cafe3871 chainparams: add testnet assumeutxo param at height 2_500_000 (Sjors Provoost)
99839bbfa7 doc: add note about confusing HaveTxsDownloaded name (James O'Beirne)
7ee46a755f contrib: add script to demo/test assumeutxo (James O'Beirne)
42cae39356 test: add feature_assumeutxo functional test (James O'Beirne)
0f64bac603 rpc: add getchainstates (James O'Beirne)
bb05857794 refuse to activate a UTXO snapshot if mempool not empty (James O'Beirne)
ce585a9a15 rpc: add loadtxoutset (James O'Beirne)
62ac519e71 validation: do not activate snapshot if behind active chain (James O'Beirne)
9511fb3616 validation: assumeutxo: swap m_mempool on snapshot activation (James O'Beirne)
7fcd21544a blockstorage: segment normal/assumedvalid blockfiles (James O'Beirne)
4c3b8ca35c validation: populate nChainTx value for assumedvalid chainstates (James O'Beirne)
49ef778158 test: adjust chainstate tests to use recognized snapshot base (James O'Beirne)
1019c39982 validation: pruning for multiple chainstates (James O'Beirne)
373cf91531 validation: indexing changes for assumeutxo (James O'Beirne)
1fffdd76a1 net_processing: validationinterface: ignore some events for bg chain (James O'Beirne)
fbe0a7d7ca wallet: validationinterface: only handle active chain notifications (James O'Beirne)
f073917a9e validationinterface: only send zmq notifications for active (James O'Beirne)
4d8f4dcb45 validation: pass ChainstateRole for validationinterface calls (James O'Beirne)
1e59acdf17 validation: only call UpdatedBlockTip for active chainstate (James O'Beirne)
c6af23c517 validation: add ChainstateRole (James O'Beirne)
9f2318c76c validation: MaybeRebalanceCaches when chain leaves IBD (James O'Beirne)
434495a8c1 chainparams: add blockhash to AssumeutxoData (James O'Beirne)
c711ca186f assumeutxo: remove snapshot during -reindex{-chainstate} (James O'Beirne)
c93ef43e4f bugfix: correct is_snapshot_cs in VerifyDB (James O'Beirne)
b73d3bbd23 net_processing: Request assumeutxo background chain blocks (Suhas Daftuar)

Pull request description:

  - Background and FAQ: https://github.com/jamesob/assumeutxo-docs/tree/2019-04-proposal/proposal
  - Prior progress/project: https://github.com/bitcoin/bitcoin/projects/11
  - Replaces https://github.com/bitcoin/bitcoin/pull/15606, which was closed due to Github slowness. Original description and commentary can be found there.

  ---

  This changeset finishes the first phase of the assumeutxo project. It makes UTXO snapshots loadable via RPC (`loadtxoutset`) and adds `assumeutxo` parameters to chainparams. It contains all the remaining changes necessary to both use an assumedvalid snapshot chainstate and do a full validation sync in the background.

  This may look like a lot to review, but note that
  - ~200 lines are a (non-essential) demo shell script
  - Many lines are functional test, documentation, and relatively dilute RPC code.

  So it shouldn't be as burdensome to review as the linecount might suggest.

  - **P2P**: minor changes are made to `init.cpp` and `net_processing.cpp` to make simultaneous IBD across multiple chainstates work.
  - **Pruning**: implement correct pruning behavior when using a background chainstate
  - **Blockfile separation**: to prevent "fragmentation" in blockfile storage, have background chainstates use separate blockfiles from active snapshot chainstates to avoid interleaving heights and impairing pruning.
  - **Indexing**: some `CValidationInterface` events are given with an additional parameter, ChainstateRole, and all indexers ignore events from ChainstateRole::ASSUMEDVALID so that indexation only happens sequentially.
  - Have `-reindex` properly wipe snapshot chainstates.
  - **RPC**: introduce RPC commands `loadtxoutset` and (hidden) `getchainstates`.
  - **Release docs & first assumeutxo commitment**: add notes and a particular assumeutxo hash value for first AU-enabled release.
    - This will complete the project and allow use of UTXO snapshots for faster node bootstrap.

  The next phase, if it were to be pursued, would be coming up with a way to distribute the UTXO snapshots over the P2P network.

  ---

  ### UTXO snapshots

  Create your own with `./contrib/devtools/utxo_snapshot.sh`, e.g.
  ```shell
  ./contrib/devtools/utxo_snapshot.sh 788000 utxo.dat ./src/bitcoin-cli -datadir=$(pwd)/testdata`)
  ```
  or use the pre-generated ones listed below.

  - Testnet: **2'500'000** (Sjors):
    - torrent: `magnet:?xt=urn:btih:511e09f4bf853aefab00de5c070b1e031f0ecbe9&dn=utxo-testnet-2500000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969`
    - sha256: `79db4b025448cc0ac388d8589a28eab02de53055d181e34eb47391717aa16388`
  - Signet: **160'000** (Sjors):
    - torrent: `magnet:?xt=urn:btih:9da986cb27b3980ea7fd06b21e199b148d486880&dn=utxo-signet-160000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969`
    - sha256: `eeeca845385ba91e84ef58c09d38f98f246a24feadaad57fe1e5874f3f92ef8c`
  - Mainnet: **800'000** (Sjors):
    - Note: this needs the following commit cherry-picked in: 24deb2022b
    - torrent: `magnet:?xt=urn:btih:50ee955bef37f5ec3e5b0df4cf0288af3d715a2e&dn=utxo-800000.dat&tr=udp%3A%2F%2Ftracker.bitcoin.sprovoost.nl%3A6969`

  ### Testing

  #### For fun (~5min)

  If you want to do a quick test, you can run `./contrib/devtools/test_utxo_snapshots.sh` and follow the instructions. This is mostly obviated by the functional tests, though.

  #### For real (longer)

  If you'd like to experience a real usage of assumeutxo, you can do that too.
  I've cut a new snapshot at height 788'000 (http://img.jameso.be/utxo-788000.dat - but you can do it yourself with `./contrib/devtools/utxo_snapshot.sh` if you want). Download that, and then create a datadir for testing:
  ```sh
  $ cd ~/src/bitcoin  # or whatever

  # get the snapshot
  $ curl http://img.jameso.be/utxo-788000.dat > utxo-788000.dat

  # you'll want to do this if you like copy/pasting
  $ export AU_DATADIR=/home/${USER}/au-test # or wherever

  $ mkdir ${AU_DATADIR}
  $ vim ${AU_DATADIR}/bitcoin.conf

  dbcache=8000  # or, you know, something high
  blockfilterindex=1
  coinstatsindex=1
  prune=3000
  logthreadnames=1
  ```
  Obtain this branch, build it, and then start bitcoind:
  ```sh
  $ git remote add jamesob https://github.com/jamesob/bitcoin
  $ git fetch jamesob assumeutxo
  $ git checkout jamesob/assumeutxo

  $ ./configure $conf_args && make  # (whatever you like to do here)

  # start 'er up and watch the logs
  $ ./src/bitcoind -datadir=${AU_DATADIR}
  ```
  Then, in some other window, load the snapshot
  ```sh
  $ ./src/bitcoin-cli -datadir=${AU_DATADIR} loadtxoutset $(pwd)/utxo-788000.dat
  ```

  You'll see some log messages about headers retrieval and waiting to see the snapshot in the headers chain. Once you get the full headers chain, you'll spend a decent amount of time (~10min) loading the snapshot, checking it, and flushing it to disk. After all that happens, you should be syncing to tip in pretty short order, and you'll see the occasional `[background validation]` log message go by.

  In yet another window, you can check out chainstate status with
  ```sh
  $ ./src/bitcoin-cli -datadir=${AU_DATADIR} getchainstates
  ```
  as well as usual favorites like `getblockchaininfo`.

ACKs for top commit:
  achow101:
    ACK edbed31066

Tree-SHA512: 6086fb9a38dc7df85fedc76b30084dd8154617a2a91e89a84fb41326d34ef8e7d7ea593107afba01369093bf8cc91770621d98f0ea42a5b3b99db868d2f14dc2
This commit is contained in:
Andrew Chow 2023-10-02 17:00:28 -04:00
commit e7b0004b37
No known key found for this signature in database
GPG key ID: 17565732E08E5E41
43 changed files with 1599 additions and 291 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

200
contrib/devtools/test_utxo_snapshots.sh Executable file
View file

@ -0,0 +1,200 @@
#!/usr/bin/env bash
# Demonstrate the creation and usage of UTXO snapshots.
#
# A server node starts up, IBDs up to a certain height, then generates a UTXO
# snapshot at that point.
#
# The server then downloads more blocks (to create a diff from the snapshot).
#
# We bring a client up, load the UTXO snapshot, and we show the client sync to
# the "network tip" and then start a background validation of the snapshot it
# loaded. We see the background validation chainstate removed after validation
# completes.
#
export LC_ALL=C
set -e
BASE_HEIGHT=${1:-30000}
INCREMENTAL_HEIGHT=20000
FINAL_HEIGHT=$(($BASE_HEIGHT + $INCREMENTAL_HEIGHT))
SERVER_DATADIR="$(pwd)/utxodemo-data-server-$BASE_HEIGHT"
CLIENT_DATADIR="$(pwd)/utxodemo-data-client-$BASE_HEIGHT"
UTXO_DAT_FILE="$(pwd)/utxo.$BASE_HEIGHT.dat"
# Chosen to try to not interfere with any running bitcoind processes.
SERVER_PORT=8633
SERVER_RPC_PORT=8632
CLIENT_PORT=8733
CLIENT_RPC_PORT=8732
SERVER_PORTS="-port=${SERVER_PORT} -rpcport=${SERVER_RPC_PORT}"
CLIENT_PORTS="-port=${CLIENT_PORT} -rpcport=${CLIENT_RPC_PORT}"
# Ensure the client exercises all indexes to test that snapshot use works
# properly with indexes.
ALL_INDEXES="-txindex -coinstatsindex -blockfilterindex=1"
if ! command -v jq >/dev/null ; then
echo "This script requires jq to parse JSON RPC output. Please install it."
echo "(e.g. sudo apt install jq)"
exit 1
fi
DUMP_OUTPUT="dumptxoutset-output-$BASE_HEIGHT.json"
finish() {
echo
echo "Killing server and client PIDs ($SERVER_PID, $CLIENT_PID) and cleaning up datadirs"
echo
rm -f "$UTXO_DAT_FILE" "$DUMP_OUTPUT"
rm -rf "$SERVER_DATADIR" "$CLIENT_DATADIR"
kill -9 "$SERVER_PID" "$CLIENT_PID"
}
trap finish EXIT
# Need to specify these to trick client into accepting server as a peer
# it can IBD from, otherwise the default values prevent IBD from the server node.
EARLY_IBD_FLAGS="-maxtipage=9223372036854775207 -minimumchainwork=0x00"
server_rpc() {
./src/bitcoin-cli -rpcport=$SERVER_RPC_PORT -datadir="$SERVER_DATADIR" "$@"
}
client_rpc() {
./src/bitcoin-cli -rpcport=$CLIENT_RPC_PORT -datadir="$CLIENT_DATADIR" "$@"
}
server_sleep_til_boot() {
while ! server_rpc ping >/dev/null 2>&1; do sleep 0.1; done
}
client_sleep_til_boot() {
while ! client_rpc ping >/dev/null 2>&1; do sleep 0.1; done
}
mkdir -p "$SERVER_DATADIR" "$CLIENT_DATADIR"
echo "Hi, welcome to the assumeutxo demo/test"
echo
echo "We're going to"
echo
echo " - start up a 'server' node, sync it via mainnet IBD to height ${BASE_HEIGHT}"
echo " - create a UTXO snapshot at that height"
echo " - IBD ${INCREMENTAL_HEIGHT} more blocks on top of that"
echo
echo "then we'll demonstrate assumeutxo by "
echo
echo " - starting another node (the 'client') and loading the snapshot in"
echo " * first you'll have to modify the code slightly (chainparams) and recompile"
echo " * don't worry, we'll make it easy"
echo " - observing the client sync ${INCREMENTAL_HEIGHT} blocks on top of the snapshot from the server"
echo " - observing the client validate the snapshot chain via background IBD"
echo
read -p "Press [enter] to continue" _
echo
echo "-- Starting the demo. You might want to run the two following commands in"
echo " separate terminal windows:"
echo
echo " watch -n0.1 tail -n 30 $SERVER_DATADIR/debug.log"
echo " watch -n0.1 tail -n 30 $CLIENT_DATADIR/debug.log"
echo
read -p "Press [enter] to continue" _
echo
echo "-- IBDing the blocks (height=$BASE_HEIGHT) required to the server node..."
./src/bitcoind -logthreadnames=1 $SERVER_PORTS \
-datadir="$SERVER_DATADIR" $EARLY_IBD_FLAGS -stopatheight="$BASE_HEIGHT" >/dev/null
echo
echo "-- Creating snapshot at ~ height $BASE_HEIGHT ($UTXO_DAT_FILE)..."
sleep 2
./src/bitcoind -logthreadnames=1 $SERVER_PORTS \
-datadir="$SERVER_DATADIR" $EARLY_IBD_FLAGS -connect=0 -listen=0 >/dev/null &
SERVER_PID="$!"
server_sleep_til_boot
server_rpc dumptxoutset "$UTXO_DAT_FILE" > "$DUMP_OUTPUT"
cat "$DUMP_OUTPUT"
kill -9 "$SERVER_PID"
RPC_BASE_HEIGHT=$(jq -r .base_height < "$DUMP_OUTPUT")
RPC_AU=$(jq -r .txoutset_hash < "$DUMP_OUTPUT")
RPC_NCHAINTX=$(jq -r .nchaintx < "$DUMP_OUTPUT")
RPC_BLOCKHASH=$(jq -r .base_hash < "$DUMP_OUTPUT")
# Wait for server to shutdown...
while server_rpc ping >/dev/null 2>&1; do sleep 0.1; done
echo
echo "-- Now: add the following to CMainParams::m_assumeutxo_data"
echo " in src/kernel/chainparams.cpp, and recompile:"
echo
echo " {${RPC_BASE_HEIGHT}, AssumeutxoHash{uint256S(\"0x${RPC_AU}\")}, ${RPC_NCHAINTX}, uint256S(\"0x${RPC_BLOCKHASH}\")},"
echo
echo
echo "-- IBDing more blocks to the server node (height=$FINAL_HEIGHT) so there is a diff between snapshot and tip..."
./src/bitcoind $SERVER_PORTS -logthreadnames=1 -datadir="$SERVER_DATADIR" \
$EARLY_IBD_FLAGS -stopatheight="$FINAL_HEIGHT" >/dev/null
echo
echo "-- Starting the server node to provide blocks to the client node..."
./src/bitcoind $SERVER_PORTS -logthreadnames=1 -debug=net -datadir="$SERVER_DATADIR" \
$EARLY_IBD_FLAGS -connect=0 -listen=1 >/dev/null &
SERVER_PID="$!"
server_sleep_til_boot
echo
echo "-- Okay, what you're about to see is the client starting up and activating the snapshot."
echo " I'm going to display the top 14 log lines from the client on top of an RPC called"
echo " getchainstates, which is like getblockchaininfo but for both the snapshot and "
echo " background validation chainstates."
echo
echo " You're going to first see the snapshot chainstate sync to the server's tip, then"
echo " the background IBD chain kicks in to validate up to the base of the snapshot."
echo
echo " Once validation of the snapshot is done, you should see log lines indicating"
echo " that we've deleted the background validation chainstate."
echo
echo " Once everything completes, exit the watch command with CTRL+C."
echo
read -p "When you're ready for all this, hit [enter]" _
echo
echo "-- Starting the client node to get headers from the server, then load the snapshot..."
./src/bitcoind $CLIENT_PORTS $ALL_INDEXES -logthreadnames=1 -datadir="$CLIENT_DATADIR" \
-connect=0 -addnode=127.0.0.1:$SERVER_PORT -debug=net $EARLY_IBD_FLAGS >/dev/null &
CLIENT_PID="$!"
client_sleep_til_boot
echo
echo "-- Initial state of the client:"
client_rpc getchainstates
echo
echo "-- Loading UTXO snapshot into client..."
client_rpc loadtxoutset "$UTXO_DAT_FILE"
watch -n 0.3 "( tail -n 14 $CLIENT_DATADIR/debug.log ; echo ; ./src/bitcoin-cli -rpcport=$CLIENT_RPC_PORT -datadir=$CLIENT_DATADIR getchainstates) | cat"
echo
echo "-- Okay, now I'm going to restart the client to make sure that the snapshot chain reloads "
echo " as the main chain properly..."
echo
echo " Press CTRL+C after you're satisfied to exit the demo"
echo
read -p "Press [enter] to continue"
while kill -0 "$CLIENT_PID"; do
sleep 1
done
./src/bitcoind $CLIENT_PORTS $ALL_INDEXES -logthreadnames=1 -datadir="$CLIENT_DATADIR" -connect=0 \
-addnode=127.0.0.1:$SERVER_PORT "$EARLY_IBD_FLAGS" >/dev/null &
CLIENT_PID="$!"
client_sleep_til_boot
watch -n 0.3 "( tail -n 14 $CLIENT_DATADIR/debug.log ; echo ; ./src/bitcoin-cli -rpcport=$CLIENT_RPC_PORT -datadir=$CLIENT_DATADIR getchainstates) | cat"
echo
echo "-- Done!"

2
doc/design/assumeutxo.md
View file

@ -3,7 +3,7 @@
Assumeutxo is a feature that allows fast bootstrapping of a validating bitcoind
instance with a very similar security model to assumevalid.
The RPC commands `dumptxoutset` and `loadtxoutset` (yet to be merged) are used to
The RPC commands `dumptxoutset` and `loadtxoutset` are used to
respectively generate and load UTXO snapshots. The utility script
`./contrib/devtools/utxo_snapshot.sh` may be of use.

28
doc/release-notes-27596.md Normal file
View file

@ -0,0 +1,28 @@
Pruning
-------
When using assumeutxo with `-prune`, the prune budget may be exceeded if it is set
lower than 1100MB (i.e. `MIN_DISK_SPACE_FOR_BLOCK_FILES * 2`). Prune budget is normally
split evenly across each chainstate, unless the resulting prune budget per chainstate
is beneath `MIN_DISK_SPACE_FOR_BLOCK_FILES` in which case that value will be used.
RPC
---
`loadtxoutset` has been added, which allows loading a UTXO snapshot of the format
generated by `dumptxoutset`. Once this snapshot is loaded, its contents will be
deserialized into a second chainstate data structure, which is then used to sync to
the network's tip under a security model very much like `assumevalid`.
Meanwhile, the original chainstate will complete the initial block download process in
the background, eventually validating up to the block that the snapshot is based upon.
The result is a usable bitcoind instance that is current with the network tip in a
matter of minutes rather than hours. UTXO snapshot are typically obtained via
third-party sources (HTTP, torrent, etc.) which is reasonable since their contents
are always checked by hash.
You can find more information on this process in the `assumeutxo` design
document (<https://github.com/bitcoin/bitcoin/blob/master/doc/design/assumeutxo.md>).
`getchainstates` has been added to aid in monitoring the assumeutxo sync process.

4
doc/zmq.md
View file

@ -113,11 +113,11 @@ Where the 8-byte uints correspond to the mempool sequence number.
| hashtx | <32-byte transaction hash in Little Endian> | <uint32 sequence number in Little Endian>
`rawblock`: Notifies when the chain tip is updated. Messages are ZMQ multipart messages with three parts. The first part is the topic (`rawblock`), the second part is the serialized block, and the last part is a sequence number (representing the message count to detect lost messages).
`rawblock`: Notifies when the chain tip is updated. When assumeutxo is in use, this notification will not be issued for historical blocks connected to the background validation chainstate. Messages are ZMQ multipart messages with three parts. The first part is the topic (`rawblock`), the second part is the serialized block, and the last part is a sequence number (representing the message count to detect lost messages).
| rawblock | <serialized block> | <uint32 sequence number in Little Endian>
`hashblock`: Notifies when the chain tip is updated. Messages are ZMQ multipart messages with three parts. The first part is the topic (`hashblock`), the second part is the 32-byte block hash, and the last part is a sequence number (representing the message count to detect lost messages).
`hashblock`: Notifies when the chain tip is updated. When assumeutxo is in use, this notification will not be issued for historical blocks connected to the background validation chainstate. Messages are ZMQ multipart messages with three parts. The first part is the topic (`hashblock`), the second part is the 32-byte block hash, and the last part is a sequence number (representing the message count to detect lost messages).
| hashblock | <32-byte block hash in Little Endian> | <uint32 sequence number in Little Endian>

2
src/bench/wallet_create_tx.cpp
View file

@ -70,7 +70,7 @@ void generateFakeBlock(const CChainParams& params,
// notify wallet
const auto& pindex = WITH_LOCK(::cs_main, return context.chainman->ActiveChain().Tip());
wallet.blockConnected(kernel::MakeBlockInfo(pindex, &block));
wallet.blockConnected(ChainstateRole::NORMAL, kernel::MakeBlockInfo(pindex, &block));
}
struct PreSelectInputs {

6
src/chain.h
View file

@ -276,6 +276,12 @@ public:
*
* Does not imply the transactions are consensus-valid (ConnectTip might fail)
* Does not imply the transactions are still stored on disk. (IsBlockPruned might return true)
*
* Note that this will be true for the snapshot base block, if one is loaded (and
* all subsequent assumed-valid blocks) since its nChainTx value will have been set
* manually based on the related AssumeutxoData entry.
*
* TODO: potentially change the name of this based on the fact above.
*/
bool HaveTxsDownloaded() const { return nChainTx != 0; }

36
src/index/base.cpp
View file

@ -79,9 +79,15 @@ BaseIndex::~BaseIndex()
bool BaseIndex::Init()
{
AssertLockNotHeld(cs_main);
// May need reset if index is being restarted.
m_interrupt.reset();
// m_chainstate member gives indexing code access to node internals. It is
// removed in followup https://github.com/bitcoin/bitcoin/pull/24230
m_chainstate = &m_chain->context()->chainman->ActiveChainstate();
m_chainstate = WITH_LOCK(::cs_main,
return &m_chain->context()->chainman->GetChainstateForIndexing());
// Register to validation interface before setting the 'm_synced' flag, so that
// callbacks are not missed once m_synced is true.
RegisterValidationInterface(this);
@ -92,7 +98,8 @@ bool BaseIndex::Init()
}
LOCK(cs_main);
CChain& active_chain = m_chainstate->m_chain;
CChain& index_chain = m_chainstate->m_chain;
if (locator.IsNull()) {
SetBestBlockIndex(nullptr);
} else {
@ -114,7 +121,7 @@ bool BaseIndex::Init()
// Note: this will latch to true immediately if the user starts up with an empty
// datadir and an index enabled. If this is the case, indexation will happen solely
// via `BlockConnected` signals until, possibly, the next restart.
m_synced = start_block == active_chain.Tip();
m_synced = start_block == index_chain.Tip();
m_init = true;
return true;
}
@ -143,6 +150,8 @@ void BaseIndex::ThreadSync()
std::chrono::steady_clock::time_point last_locator_write_time{0s};
while (true) {
if (m_interrupt) {
LogPrintf("%s: m_interrupt set; exiting ThreadSync\n", GetName());
SetBestBlockIndex(pindex);
// No need to handle errors in Commit. If it fails, the error will be already be
// logged. The best way to recover is to continue, as index cannot be corrupted by
@ -250,8 +259,19 @@ bool BaseIndex::Rewind(const CBlockIndex* current_tip, const CBlockIndex* new_ti
return true;
}
void BaseIndex::BlockConnected(const std::shared_ptr<const CBlock>& block, const CBlockIndex* pindex)
void BaseIndex::BlockConnected(ChainstateRole role, const std::shared_ptr<const CBlock>& block, const CBlockIndex* pindex)
{
// Ignore events from the assumed-valid chain; we will process its blocks
// (sequentially) after it is fully verified by the background chainstate. This
// is to avoid any out-of-order indexing.
//
// TODO at some point we could parameterize whether a particular index can be
// built out of order, but for now just do the conservative simple thing.
if (role == ChainstateRole::ASSUMEDVALID) {
return;
}
// Ignore BlockConnected signals until we have fully indexed the chain.
if (!m_synced) {
return;
}
@ -296,8 +316,14 @@ void BaseIndex::BlockConnected(const std::shared_ptr<const CBlock>& block, const
}
}
void BaseIndex::ChainStateFlushed(const CBlockLocator& locator)
void BaseIndex::ChainStateFlushed(ChainstateRole role, const CBlockLocator& locator)
{
// Ignore events from the assumed-valid chain; we will process its blocks
// (sequentially) after it is fully verified by the background chainstate.
if (role == ChainstateRole::ASSUMEDVALID) {
return;
}
if (!m_synced) {
return;
}

16
src/index/base.h
View file

@ -15,6 +15,7 @@
class CBlock;
class CBlockIndex;
class Chainstate;
class ChainstateManager;
namespace interfaces {
class Chain;
} // namespace interfaces
@ -30,6 +31,11 @@ struct IndexSummary {
* Base class for indices of blockchain data. This implements
* CValidationInterface and ensures blocks are indexed sequentially according
* to their position in the active chain.
*
* In the presence of multiple chainstates (i.e. if a UTXO snapshot is loaded),
* only the background "IBD" chainstate will be indexed to avoid building the
* index out of order. When the background chainstate completes validation, the
* index will be reinitialized and indexing will continue.
*/
class BaseIndex : public CValidationInterface
{
@ -102,9 +108,9 @@ protected:
Chainstate* m_chainstate{nullptr};
const std::string m_name;
void BlockConnected(const std::shared_ptr<const CBlock>& block, const CBlockIndex* pindex) override;
void BlockConnected(ChainstateRole role, const std::shared_ptr<const CBlock>& block, const CBlockIndex* pindex) override;
void ChainStateFlushed(const CBlockLocator& locator) override;
void ChainStateFlushed(ChainstateRole role, const CBlockLocator& locator) override;
/// Initialize internal state from the database and block index.
[[nodiscard]] virtual bool CustomInit(const std::optional<interfaces::BlockKey>& block) { return true; }
@ -122,9 +128,6 @@ protected:
virtual DB& GetDB() const = 0;
/// Get the name of the index for display in logs.
const std::string& GetName() const LIFETIMEBOUND { return m_name; }
/// Update the internal best block index as well as the prune lock.
void SetBestBlockIndex(const CBlockIndex* block);
@ -133,6 +136,9 @@ public:
/// Destructor interrupts sync thread if running and blocks until it exits.
virtual ~BaseIndex();
/// Get the name of the index for display in logs.
const std::string& GetName() const LIFETIMEBOUND { return m_name; }
/// Blocks the current thread until the index is caught up to the current
/// state of the block chain. This only blocks if the index has gotten in
/// sync once and only needs to process blocks in the ValidationInterface

30
src/init.cpp
View file

@ -1478,6 +1478,25 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
node.chainman = std::make_unique<ChainstateManager>(node.kernel->interrupt, chainman_opts, blockman_opts);
ChainstateManager& chainman = *node.chainman;
// This is defined and set here instead of inline in validation.h to avoid a hard
// dependency between validation and index/base, since the latter is not in
// libbitcoinkernel.
chainman.restart_indexes = [&node]() {
LogPrintf("[snapshot] restarting indexes\n");
// Drain the validation interface queue to ensure that the old indexes
// don't have any pending work.
SyncWithValidationInterfaceQueue();
for (auto* index : node.indexes) {
index->Interrupt();
index->Stop();
if (!(index->Init() && index->StartBackgroundSync())) {
LogPrintf("[snapshot] WARNING failed to restart index %s on snapshot chain\n", index->GetName());
}
}
};
node::ChainstateLoadOptions options;
options.mempool = Assert(node.mempool.get());
options.reindex = node::fReindex;
@ -1906,18 +1925,19 @@ bool StartIndexBackgroundSync(NodeContext& node)
// indexes_start_block='nullptr' means "start from height 0".
std::optional<const CBlockIndex*> indexes_start_block;
std::string older_index_name;
ChainstateManager& chainman = *Assert(node.chainman);
const Chainstate& chainstate = WITH_LOCK(::cs_main, return chainman.GetChainstateForIndexing());
const CChain& index_chain = chainstate.m_chain;
for (auto index : node.indexes) {
const IndexSummary& summary = index->GetSummary();
if (summary.synced) continue;
// Get the last common block between the index best block and the active chain
LOCK(::cs_main);
const CChain& active_chain = chainman.ActiveChain();
const CBlockIndex* pindex = chainman.m_blockman.LookupBlockIndex(summary.best_block_hash);
if (!active_chain.Contains(pindex)) {
pindex = active_chain.FindFork(pindex);
if (!index_chain.Contains(pindex)) {
pindex = index_chain.FindFork(pindex);
}
if (!indexes_start_block || !pindex || pindex->nHeight < indexes_start_block.value()->nHeight) {
@ -1932,7 +1952,7 @@ bool StartIndexBackgroundSync(NodeContext& node)
LOCK(::cs_main);
const CBlockIndex* start_block = *indexes_start_block;
if (!start_block) start_block = chainman.ActiveChain().Genesis();
if (!chainman.m_blockman.CheckBlockDataAvailability(*chainman.ActiveChain().Tip(), *Assert(start_block))) {
if (!chainman.m_blockman.CheckBlockDataAvailability(*index_chain.Tip(), *Assert(start_block))) {
return InitError(strprintf(Untranslated("%s best block of the index goes beyond pruned data. Please disable the index or reindex (which will download the whole blockchain again)"), older_index_name));
}
}

5
src/interfaces/chain.h
View file

@ -27,6 +27,7 @@ class Coin;
class uint256;
enum class MemPoolRemovalReason;
enum class RBFTransactionState;
enum class ChainstateRole;
struct bilingual_str;
struct CBlockLocator;
struct FeeCalculation;
@ -310,10 +311,10 @@ public:
virtual ~Notifications() {}
virtual void transactionAddedToMempool(const CTransactionRef& tx) {}
virtual void transactionRemovedFromMempool(const CTransactionRef& tx, MemPoolRemovalReason reason) {}
virtual void blockConnected(const BlockInfo& block) {}
virtual void blockConnected(ChainstateRole role, const BlockInfo& block) {}
virtual void blockDisconnected(const BlockInfo& block) {}
virtual void updatedBlockTip() {}
virtual void chainStateFlushed(const CBlockLocator& locator) {}
virtual void chainStateFlushed(ChainstateRole role, const CBlockLocator& locator) {}
};
//! Register handler for notifications.

11
src/kernel/chain.cpp
View file

@ -4,6 +4,7 @@
#include <chain.h>
#include <interfaces/chain.h>
#include <kernel/chain.h>
#include <sync.h>
#include <uint256.h>
@ -25,3 +26,13 @@ interfaces::BlockInfo MakeBlockInfo(const CBlockIndex* index, const CBlock* data
return info;
}
} // namespace kernel
std::ostream& operator<<(std::ostream& os, const ChainstateRole& role) {
switch(role) {
case ChainstateRole::NORMAL: os << "normal"; break;
case ChainstateRole::ASSUMEDVALID: os << "assumedvalid"; break;
case ChainstateRole::BACKGROUND: os << "background"; break;
default: os.setstate(std::ios_base::failbit);
}
return os;
}

20
src/kernel/chain.h
View file

@ -5,6 +5,8 @@
#ifndef BITCOIN_KERNEL_CHAIN_H
#define BITCOIN_KERNEL_CHAIN_H
#include<iostream>
class CBlock;
class CBlockIndex;
namespace interfaces {
@ -14,6 +16,24 @@ struct BlockInfo;
namespace kernel {
//! Return data from block index.
interfaces::BlockInfo MakeBlockInfo(const CBlockIndex* block_index, const CBlock* data = nullptr);
} // namespace kernel
//! This enum describes the various roles a specific Chainstate instance can take.
//! Other parts of the system sometimes need to vary in behavior depending on the
//! existence of a background validation chainstate, e.g. when building indexes.
enum class ChainstateRole {
// Single chainstate in use, "normal" IBD mode.
NORMAL,
// Doing IBD-style validation in the background. Implies use of an assumed-valid
// chainstate.
BACKGROUND,
// Active assumed-valid chainstate. Implies use of a background IBD chainstate.
ASSUMEDVALID,
};
std::ostream& operator<<(std::ostream& os, const ChainstateRole& role);
#endif // BITCOIN_KERNEL_CHAIN_H

35
src/kernel/chainparams.cpp
View file

@ -172,7 +172,7 @@ public:
}
};
m_assumeutxo_data = MapAssumeutxo{
m_assumeutxo_data = {
// TODO to be specified in a future patch.
};
@ -266,8 +266,13 @@ public:
}
};
m_assumeutxo_data = MapAssumeutxo{
// TODO to be specified in a future patch.
m_assumeutxo_data = {
{
.height = 2'500'000,
.hash_serialized = AssumeutxoHash{uint256S("0x2a8fdefef3bf75fa00540ccaaaba4b5281bea94229327bdb0f7416ef1e7a645c")},
.nChainTx = 66484552,
.blockhash = uint256S("0x0000000000000093bcb68c03a9a168ae252572d348a2eaeba2cdf9231d73206f")
}
};
chainTxData = ChainTxData{
@ -370,6 +375,15 @@ public:
vFixedSeeds.clear();
m_assumeutxo_data = {
{
.height = 160'000,
.hash_serialized = AssumeutxoHash{uint256S("0x5225141cb62dee63ab3be95f9b03d60801f264010b1816d4bd00618b2736e7be")},
.nChainTx = 2289496,
.blockhash = uint256S("0x0000003ca3c99aff040f2563c2ad8f8ec88bd0fd6b8f0895cfaf1ef90353a62c")
}
};
base58Prefixes[PUBKEY_ADDRESS] = std::vector<unsigned char>(1,111);
base58Prefixes[SCRIPT_ADDRESS] = std::vector<unsigned char>(1,196);
base58Prefixes[SECRET_KEY] = std::vector<unsigned char>(1,239);
@ -477,14 +491,19 @@ public:
}
};
m_assumeutxo_data = MapAssumeutxo{
m_assumeutxo_data = {
{
110,
{AssumeutxoHash{uint256S("0x1ebbf5850204c0bdb15bf030f47c7fe91d45c44c712697e4509ba67adb01c618")}, 110},
.height = 110,
.hash_serialized = AssumeutxoHash{uint256S("0x1ebbf5850204c0bdb15bf030f47c7fe91d45c44c712697e4509ba67adb01c618")},
.nChainTx = 110,
.blockhash = uint256S("0x696e92821f65549c7ee134edceeeeaaa4105647a3c4fd9f298c0aec0ab50425c")
},
{
200,
{AssumeutxoHash{uint256S("0x51c8d11d8b5c1de51543c579736e786aa2736206d1e11e627568029ce092cf62")}, 200},
// For use by test/functional/feature_assumeutxo.py
.height = 299,
.hash_serialized = AssumeutxoHash{uint256S("0xef45ccdca5898b6c2145e4581d2b88c56564dd389e4bd75a1aaf6961d3edd3c0")},
.nChainTx = 300,
.blockhash = uint256S("0x7e0517ef3ea6ecbed9117858e42eedc8eb39e8698a38dcbd1b3962a283233f4c")
},
};

26
src/kernel/chainparams.h
View file

@ -12,6 +12,7 @@
#include <uint256.h>
#include <util/chaintype.h>
#include <util/hash_type.h>
#include <util/vector.h>
#include <cstdint>
#include <iterator>
@ -44,17 +45,21 @@ struct AssumeutxoHash : public BaseHash<uint256> {
* as valid.
*/
struct AssumeutxoData {
int height;
//! The expected hash of the deserialized UTXO set.
const AssumeutxoHash hash_serialized;
AssumeutxoHash hash_serialized;
//! Used to populate the nChainTx value, which is used during BlockManager::LoadBlockIndex().
//!
//! We need to hardcode the value here because this is computed cumulatively using block data,
//! which we do not necessarily have at the time of snapshot load.
const unsigned int nChainTx;
};
unsigned int nChainTx;
using MapAssumeutxo = std::map<int, const AssumeutxoData>;
//! The hash of the base block for this snapshot. Used to refer to assumeutxo data
//! prior to having a loaded blockindex.
uint256 blockhash;
};
/**
* Holds various statistics on transactions within a chain. Used to estimate
@ -114,9 +119,14 @@ public:
const std::vector<uint8_t>& FixedSeeds() const { return vFixedSeeds; }
const CCheckpointData& Checkpoints() const { return checkpointData; }
//! Get allowed assumeutxo configuration.
//! @see ChainstateManager
const MapAssumeutxo& Assumeutxo() const { return m_assumeutxo_data; }
std::optional<AssumeutxoData> AssumeutxoForHeight(int height) const
{
return FindFirst(m_assumeutxo_data, [&](const auto& d) { return d.height == height; });
}
std::optional<AssumeutxoData> AssumeutxoForBlockhash(const uint256& blockhash) const
{
return FindFirst(m_assumeutxo_data, [&](const auto& d) { return d.blockhash == blockhash; });
}
const ChainTxData& TxData() const { return chainTxData; }
@ -169,7 +179,7 @@ protected:
bool fDefaultConsistencyChecks;
bool m_is_mockable_chain;
CCheckpointData checkpointData;
MapAssumeutxo m_assumeutxo_data;
std::vector<AssumeutxoData> m_assumeutxo_data;
ChainTxData chainTxData;
};

127
src/net_processing.cpp
View file

@ -18,6 +18,7 @@
#include <index/blockfilterindex.h>
#include <kernel/mempool_entry.h>
#include <logging.h>
#include <kernel/chain.h>
#include <merkleblock.h>
#include <netbase.h>
#include <netmessagemaker.h>
@ -483,7 +484,7 @@ public:
CTxMemPool& pool, Options opts);
/** Overridden from CValidationInterface. */
void BlockConnected(const std::shared_ptr<const CBlock>& pblock, const CBlockIndex* pindexConnected) override
void BlockConnected(ChainstateRole role, const std::shared_ptr<const CBlock>& pblock, const CBlockIndex* pindexConnected) override
EXCLUSIVE_LOCKS_REQUIRED(!m_recent_confirmed_transactions_mutex);
void BlockDisconnected(const std::shared_ptr<const CBlock> &block, const CBlockIndex* pindex) override
EXCLUSIVE_LOCKS_REQUIRED(!m_recent_confirmed_transactions_mutex);
@ -892,6 +893,38 @@ private:
*/
void FindNextBlocksToDownload(const Peer& peer, unsigned int count, std::vector<const CBlockIndex*>& vBlocks, NodeId& nodeStaller) EXCLUSIVE_LOCKS_REQUIRED(cs_main);
/** Request blocks for the background chainstate, if one is in use. */
void TryDownloadingHistoricalBlocks(const Peer& peer, unsigned int count, std::vector<const CBlockIndex*>& vBlocks, const CBlockIndex* from_tip, const CBlockIndex* target_block) EXCLUSIVE_LOCKS_REQUIRED(cs_main);
/**
* \brief Find next blocks to download from a peer after a starting block.
*
* \param vBlocks Vector of blocks to download which will be appended to.
* \param peer Peer which blocks will be downloaded from.
* \param state Pointer to the state of the peer.
* \param pindexWalk Pointer to the starting block to add to vBlocks.
* \param count Maximum number of blocks to allow in vBlocks. No more
* blocks will be added if it reaches this size.
* \param nWindowEnd Maximum height of blocks to allow in vBlocks. No
* blocks will be added above this height.
* \param activeChain Optional pointer to a chain to compare against. If
* provided, any next blocks which are already contained
* in this chain will not be appended to vBlocks, but
* instead will be used to update the
* state->pindexLastCommonBlock pointer.
* \param nodeStaller Optional pointer to a NodeId variable that will receive
* the ID of another peer that might be causing this peer
* to stall. This is set to the ID of the peer which
* first requested the first in-flight block in the
* download window. It is only set if vBlocks is empty at
* the end of this function call and if increasing
* nWindowEnd by 1 would cause it to be non-empty (which
* indicates the download might be stalled because every
* block in the window is in flight and no other peer is
* trying to download the next block).
*/
void FindNextBlocks(std::vector<const CBlockIndex*>& vBlocks, const Peer& peer, CNodeState *state, const CBlockIndex *pindexWalk, unsigned int count, int nWindowEnd, const CChain* activeChain=nullptr, NodeId* nodeStaller=nullptr) EXCLUSIVE_LOCKS_REQUIRED(cs_main);
/* Multimap used to preserve insertion order */
typedef std::multimap<uint256, std::pair<NodeId, std::list<QueuedBlock>::iterator>> BlockDownloadMap;
BlockDownloadMap mapBlocksInFlight GUARDED_BY(cs_main);
@ -1312,6 +1345,7 @@ void PeerManagerImpl::UpdateBlockAvailability(NodeId nodeid, const uint256 &hash
}
}
// Logic for calculating which blocks to download from a given peer, given our current tip.
void PeerManagerImpl::FindNextBlocksToDownload(const Peer& peer, unsigned int count, std::vector<const CBlockIndex*>& vBlocks, NodeId& nodeStaller)
{
if (count == 0)
@ -1341,12 +1375,47 @@ void PeerManagerImpl::FindNextBlocksToDownload(const Peer& peer, unsigned int co
if (state->pindexLastCommonBlock == state->pindexBestKnownBlock)
return;
std::vector<const CBlockIndex*> vToFetch;
const CBlockIndex *pindexWalk = state->pindexLastCommonBlock;
// Never fetch further than the best block we know the peer has, or more than BLOCK_DOWNLOAD_WINDOW + 1 beyond the last
// linked block we have in common with this peer. The +1 is so we can detect stalling, namely if we would be able to
// download that next block if the window were 1 larger.
int nWindowEnd = state->pindexLastCommonBlock->nHeight + BLOCK_DOWNLOAD_WINDOW;
FindNextBlocks(vBlocks, peer, state, pindexWalk, count, nWindowEnd, &m_chainman.ActiveChain(), &nodeStaller);
}
void PeerManagerImpl::TryDownloadingHistoricalBlocks(const Peer& peer, unsigned int count, std::vector<const CBlockIndex*>& vBlocks, const CBlockIndex *from_tip, const CBlockIndex* target_block)
{
Assert(from_tip);
Assert(target_block);
if (vBlocks.size() >= count) {
return;
}
vBlocks.reserve(count);
CNodeState *state = Assert(State(peer.m_id));
if (state->pindexBestKnownBlock == nullptr || state->pindexBestKnownBlock->GetAncestor(target_block->nHeight) != target_block) {
// This peer can't provide us the complete series of blocks leading up to the
// assumeutxo snapshot base.
//
// Presumably this peer's chain has less work than our ActiveChain()'s tip, or else we
// will eventually crash when we try to reorg to it. Let other logic
// deal with whether we disconnect this peer.
//
// TODO at some point in the future, we might choose to request what blocks
// this peer does have from the historical chain, despite it not having a
// complete history beneath the snapshot base.
return;
}
FindNextBlocks(vBlocks, peer, state, from_tip, count, std::min<int>(from_tip->nHeight + BLOCK_DOWNLOAD_WINDOW, target_block->nHeight));
}
void PeerManagerImpl::FindNextBlocks(std::vector<const CBlockIndex*>& vBlocks, const Peer& peer, CNodeState *state, const CBlockIndex *pindexWalk, unsigned int count, int nWindowEnd, const CChain* activeChain, NodeId* nodeStaller)
{
std::vector<const CBlockIndex*> vToFetch;
int nMaxHeight = std::min<int>(state->pindexBestKnownBlock->nHeight, nWindowEnd + 1);
NodeId waitingfor = -1;
while (pindexWalk->nHeight < nMaxHeight) {
@ -1374,8 +1443,8 @@ void PeerManagerImpl::FindNextBlocksToDownload(const Peer& peer, unsigned int co
// We wouldn't download this block or its descendants from this peer.
return;
}
if (pindex->nStatus & BLOCK_HAVE_DATA || m_chainman.ActiveChain().Contains(pindex)) {
if (pindex->HaveTxsDownloaded())
if (pindex->nStatus & BLOCK_HAVE_DATA || (activeChain && activeChain->Contains(pindex))) {
if (activeChain && pindex->HaveTxsDownloaded())
state->pindexLastCommonBlock = pindex;
} else if (!IsBlockRequested(pindex->GetBlockHash())) {
// The block is not already downloaded, and not yet in flight.
@ -1383,7 +1452,7 @@ void PeerManagerImpl::FindNextBlocksToDownload(const Peer& peer, unsigned int co
// We reached the end of the window.
if (vBlocks.size() == 0 && waitingfor != peer.m_id) {
// We aren't able to fetch anything, but we would be if the download window was one larger.
nodeStaller = waitingfor;
if (nodeStaller) *nodeStaller = waitingfor;
}
return;
}
@ -1843,11 +1912,30 @@ void PeerManagerImpl::StartScheduledTasks(CScheduler& scheduler)
* announcements for them. Also save the time of the last tip update and
* possibly reduce dynamic block stalling timeout.
*/
void PeerManagerImpl::BlockConnected(const std::shared_ptr<const CBlock>& pblock, const CBlockIndex* pindex)
void PeerManagerImpl::BlockConnected(
ChainstateRole role,
const std::shared_ptr<const CBlock>& pblock,
const CBlockIndex* pindex)
{
m_orphanage.EraseForBlock(*pblock);
// Update this for all chainstate roles so that we don't mistakenly see peers
// helping us do background IBD as having a stale tip.
m_last_tip_update = GetTime<std::chrono::seconds>();
// In case the dynamic timeout was doubled once or more, reduce it slowly back to its default value
auto stalling_timeout = m_block_stalling_timeout.load();
Assume(stalling_timeout >= BLOCK_STALLING_TIMEOUT_DEFAULT);
if (stalling_timeout != BLOCK_STALLING_TIMEOUT_DEFAULT) {
const auto new_timeout = std::max(std::chrono::duration_cast<std::chrono::seconds>(stalling_timeout * 0.85), BLOCK_STALLING_TIMEOUT_DEFAULT);
if (m_block_stalling_timeout.compare_exchange_strong(stalling_timeout, new_timeout)) {
LogPrint(BCLog::NET, "Decreased stalling timeout to %d seconds\n", count_seconds(new_timeout));
}
}
if (role == ChainstateRole::BACKGROUND) {
return;
}
m_orphanage.EraseForBlock(*pblock);
{
LOCK(m_recent_confirmed_transactions_mutex);
for (const auto& ptx : pblock->vtx) {
@ -1864,16 +1952,6 @@ void PeerManagerImpl::BlockConnected(const std::shared_ptr<const CBlock>& pblock
m_txrequest.ForgetTxHash(ptx->GetWitnessHash());
}
}
// In case the dynamic timeout was doubled once or more, reduce it slowly back to its default value
auto stalling_timeout = m_block_stalling_timeout.load();
Assume(stalling_timeout >= BLOCK_STALLING_TIMEOUT_DEFAULT);
if (stalling_timeout != BLOCK_STALLING_TIMEOUT_DEFAULT) {
const auto new_timeout = std::max(std::chrono::duration_cast<std::chrono::seconds>(stalling_timeout * 0.85), BLOCK_STALLING_TIMEOUT_DEFAULT);
if (m_block_stalling_timeout.compare_exchange_strong(stalling_timeout, new_timeout)) {
LogPrint(BCLog::NET, "Decreased stalling timeout to %d seconds\n", count_seconds(new_timeout));
}
}
}
void PeerManagerImpl::BlockDisconnected(const std::shared_ptr<const CBlock> &block, const CBlockIndex* pindex)
@ -5847,7 +5925,20 @@ bool PeerManagerImpl::SendMessages(CNode* pto)
if (CanServeBlocks(*peer) && ((sync_blocks_and_headers_from_peer && !IsLimitedPeer(*peer)) || !m_chainman.IsInitialBlockDownload()) && state.vBlocksInFlight.size() < MAX_BLOCKS_IN_TRANSIT_PER_PEER) {
std::vector<const CBlockIndex*> vToDownload;
NodeId staller = -1;
FindNextBlocksToDownload(*peer, MAX_BLOCKS_IN_TRANSIT_PER_PEER - state.vBlocksInFlight.size(), vToDownload, staller);
auto get_inflight_budget = [&state]() {
return std::max(0, MAX_BLOCKS_IN_TRANSIT_PER_PEER - static_cast<int>(state.vBlocksInFlight.size()));
};
// If a snapshot chainstate is in use, we want to find its next blocks
// before the background chainstate to prioritize getting to network tip.
FindNextBlocksToDownload(*peer, get_inflight_budget(), vToDownload, staller);
if (m_chainman.BackgroundSyncInProgress() && !IsLimitedPeer(*peer)) {
TryDownloadingHistoricalBlocks(
*peer,
get_inflight_budget(),
vToDownload, m_chainman.GetBackgroundSyncTip(),
Assert(m_chainman.GetSnapshotBaseBlock()));
}
for (const CBlockIndex *pindex : vToDownload) {
uint32_t nFetchFlags = GetFetchFlags(*peer);
vGetData.push_back(CInv(MSG_BLOCK | nFetchFlags, pindex->GetBlockHash()));

218
src/node/blockstorage.cpp
View file

@ -10,6 +10,7 @@
#include <dbwrapper.h>
#include <flatfile.h>
#include <hash.h>
#include <kernel/chain.h>
#include <kernel/chainparams.h>
#include <kernel/messagestartchars.h>
#include <logging.h>
@ -257,40 +258,56 @@ void BlockManager::PruneOneBlockFile(const int fileNumber)
m_dirty_fileinfo.insert(fileNumber);
}
void BlockManager::FindFilesToPruneManual(std::set<int>& setFilesToPrune, int nManualPruneHeight, int chain_tip_height)
void BlockManager::FindFilesToPruneManual(
std::set<int>& setFilesToPrune,
int nManualPruneHeight,
const Chainstate& chain,
ChainstateManager& chainman)
{
assert(IsPruneMode() && nManualPruneHeight > 0);
LOCK2(cs_main, cs_LastBlockFile);
if (chain_tip_height < 0) {
if (chain.m_chain.Height() < 0) {
return;
}
// last block to prune is the lesser of (user-specified height, MIN_BLOCKS_TO_KEEP from the tip)
unsigned int nLastBlockWeCanPrune = std::min((unsigned)nManualPruneHeight, chain_tip_height - MIN_BLOCKS_TO_KEEP);
const auto [min_block_to_prune, last_block_can_prune] = chainman.GetPruneRange(chain, nManualPruneHeight);
int count = 0;
for (int fileNumber = 0; fileNumber < m_last_blockfile; fileNumber++) {
if (m_blockfile_info[fileNumber].nSize == 0 || m_blockfile_info[fileNumber].nHeightLast > nLastBlockWeCanPrune) {
for (int fileNumber = 0; fileNumber < this->MaxBlockfileNum(); fileNumber++) {
const auto& fileinfo = m_blockfile_info[fileNumber];
if (fileinfo.nSize == 0 || fileinfo.nHeightLast > (unsigned)last_block_can_prune || fileinfo.nHeightFirst < (unsigned)min_block_to_prune) {
continue;
}
PruneOneBlockFile(fileNumber);
setFilesToPrune.insert(fileNumber);
count++;
}
LogPrintf("Prune (Manual): prune_height=%d removed %d blk/rev pairs\n", nLastBlockWeCanPrune, count);
LogPrintf("[%s] Prune (Manual): prune_height=%d removed %d blk/rev pairs\n",
chain.GetRole(), last_block_can_prune, count);
}
void BlockManager::FindFilesToPrune(std::set<int>& setFilesToPrune, uint64_t nPruneAfterHeight, int chain_tip_height, int prune_height, bool is_ibd)
void BlockManager::FindFilesToPrune(
std::set<int>& setFilesToPrune,
int last_prune,
const Chainstate& chain,
ChainstateManager& chainman)
{
LOCK2(cs_main, cs_LastBlockFile);
if (chain_tip_height < 0 || GetPruneTarget() == 0) {
// Distribute our -prune budget over all chainstates.
const auto target = std::max(
MIN_DISK_SPACE_FOR_BLOCK_FILES, GetPruneTarget() / chainman.GetAll().size());
if (chain.m_chain.Height() < 0 || target == 0) {
return;
}
if ((uint64_t)chain_tip_height <= nPruneAfterHeight) {
if (static_cast<uint64_t>(chain.m_chain.Height()) <= chainman.GetParams().PruneAfterHeight()) {
return;
}
unsigned int nLastBlockWeCanPrune{(unsigned)std::min(prune_height, chain_tip_height - static_cast<int>(MIN_BLOCKS_TO_KEEP))};
const auto [min_block_to_prune, last_block_can_prune] = chainman.GetPruneRange(chain, last_prune);
uint64_t nCurrentUsage = CalculateCurrentUsage();
// We don't check to prune until after we've allocated new space for files
// So we should leave a buffer under our target to account for another allocation
@ -299,29 +316,31 @@ void BlockManager::FindFilesToPrune(std::set<int>& setFilesToPrune, uint64_t nPr
uint64_t nBytesToPrune;
int count = 0;
if (nCurrentUsage + nBuffer >= GetPruneTarget()) {
if (nCurrentUsage + nBuffer >= target) {
// On a prune event, the chainstate DB is flushed.
// To avoid excessive prune events negating the benefit of high dbcache
// values, we should not prune too rapidly.
// So when pruning in IBD, increase the buffer a bit to avoid a re-prune too soon.
if (is_ibd) {
if (chainman.IsInitialBlockDownload()) {
// Since this is only relevant during IBD, we use a fixed 10%
nBuffer += GetPruneTarget() / 10;
nBuffer += target / 10;
}
for (int fileNumber = 0; fileNumber < m_last_blockfile; fileNumber++) {
nBytesToPrune = m_blockfile_info[fileNumber].nSize + m_blockfile_info[fileNumber].nUndoSize;
for (int fileNumber = 0; fileNumber < this->MaxBlockfileNum(); fileNumber++) {
const auto& fileinfo = m_blockfile_info[fileNumber];
nBytesToPrune = fileinfo.nSize + fileinfo.nUndoSize;
if (m_blockfile_info[fileNumber].nSize == 0) {
if (fileinfo.nSize == 0) {
continue;
}
if (nCurrentUsage + nBuffer < GetPruneTarget()) { // are we below our target?
if (nCurrentUsage + nBuffer < target) { // are we below our target?
break;
}
// don't prune files that could have a block within MIN_BLOCKS_TO_KEEP of the main chain's tip but keep scanning
if (m_blockfile_info[fileNumber].nHeightLast > nLastBlockWeCanPrune) {
// don't prune files that could have a block that's not within the allowable
// prune range for the chain being pruned.
if (fileinfo.nHeightLast > (unsigned)last_block_can_prune || fileinfo.nHeightFirst < (unsigned)min_block_to_prune) {
continue;
}
@ -333,10 +352,10 @@ void BlockManager::FindFilesToPrune(std::set<int>& setFilesToPrune, uint64_t nPr
}
}
LogPrint(BCLog::PRUNE, "target=%dMiB actual=%dMiB diff=%dMiB max_prune_height=%d removed %d blk/rev pairs\n",
GetPruneTarget() / 1024 / 1024, nCurrentUsage / 1024 / 1024,
(int64_t(GetPruneTarget()) - int64_t(nCurrentUsage)) / 1024 / 1024,
nLastBlockWeCanPrune, count);
LogPrint(BCLog::PRUNE, "[%s] target=%dMiB actual=%dMiB diff=%dMiB min_height=%d max_prune_height=%d removed %d blk/rev pairs\n",
chain.GetRole(), target / 1024 / 1024, nCurrentUsage / 1024 / 1024,
(int64_t(target) - int64_t(nCurrentUsage)) / 1024 / 1024,
min_block_to_prune, last_block_can_prune, count);
}
void BlockManager::UpdatePruneLock(const std::string& name, const PruneLockInfo& lock_info) {
@ -360,13 +379,32 @@ CBlockIndex* BlockManager::InsertBlockIndex(const uint256& hash)
return pindex;
}
bool BlockManager::LoadBlockIndex()
bool BlockManager::LoadBlockIndex(const std::optional<uint256>& snapshot_blockhash)
{
if (!m_block_tree_db->LoadBlockIndexGuts(
GetConsensus(), [this](const uint256& hash) EXCLUSIVE_LOCKS_REQUIRED(cs_main) { return this->InsertBlockIndex(hash); }, m_interrupt)) {
return false;
}
if (snapshot_blockhash) {
const AssumeutxoData au_data = *Assert(GetParams().AssumeutxoForBlockhash(*snapshot_blockhash));
m_snapshot_height = au_data.height;
CBlockIndex* base{LookupBlockIndex(*snapshot_blockhash)};
// Since nChainTx (responsible for estimated progress) isn't persisted
// to disk, we must bootstrap the value for assumedvalid chainstates
// from the hardcoded assumeutxo chainparams.
base->nChainTx = au_data.nChainTx;
LogPrintf("[snapshot] set nChainTx=%d for %s\n", au_data.nChainTx, snapshot_blockhash->ToString());
} else {
// If this isn't called with a snapshot blockhash, make sure the cached snapshot height
// is null. This is relevant during snapshot completion, when the blockman may be loaded
// with a height that then needs to be cleared after the snapshot is fully validated.
m_snapshot_height.reset();
}
Assert(m_snapshot_height.has_value() == snapshot_blockhash.has_value());
// Calculate nChainWork
std::vector<CBlockIndex*> vSortedByHeight{GetAllBlockIndices()};
std::sort(vSortedByHeight.begin(), vSortedByHeight.end(),
@ -383,7 +421,11 @@ bool BlockManager::LoadBlockIndex()
// Pruned nodes may have deleted the block.
if (pindex->nTx > 0) {
if (pindex->pprev) {
if (pindex->pprev->nChainTx > 0) {
if (m_snapshot_height && pindex->nHeight == *m_snapshot_height &&
pindex->GetBlockHash() == *snapshot_blockhash) {
// Should have been set above; don't disturb it with code below.
Assert(pindex->nChainTx > 0);
} else if (pindex->pprev->nChainTx > 0) {
pindex->nChainTx = pindex->pprev->nChainTx + pindex->nTx;
} else {
pindex->nChainTx = 0;
@ -420,27 +462,29 @@ bool BlockManager::WriteBlockIndexDB()
vBlocks.push_back(*it);
m_dirty_blockindex.erase(it++);
}
if (!m_block_tree_db->WriteBatchSync(vFiles, m_last_blockfile, vBlocks)) {
int max_blockfile = WITH_LOCK(cs_LastBlockFile, return this->MaxBlockfileNum());
if (!m_block_tree_db->WriteBatchSync(vFiles, max_blockfile, vBlocks)) {
return false;
}
return true;
}
bool BlockManager::LoadBlockIndexDB()
bool BlockManager::LoadBlockIndexDB(const std::optional<uint256>& snapshot_blockhash)
{
if (!LoadBlockIndex()) {
if (!LoadBlockIndex(snapshot_blockhash)) {
return false;
}
int max_blockfile_num{0};
// Load block file info
m_block_tree_db->ReadLastBlockFile(m_last_blockfile);
m_blockfile_info.resize(m_last_blockfile + 1);
LogPrintf("%s: last block file = %i\n", __func__, m_last_blockfile);
for (int nFile = 0; nFile <= m_last_blockfile; nFile++) {
m_block_tree_db->ReadLastBlockFile(max_blockfile_num);
m_blockfile_info.resize(max_blockfile_num + 1);
LogPrintf("%s: last block file = %i\n", __func__, max_blockfile_num);
for (int nFile = 0; nFile <= max_blockfile_num; nFile++) {
m_block_tree_db->ReadBlockFileInfo(nFile, m_blockfile_info[nFile]);
}
LogPrintf("%s: last block file info: %s\n", __func__, m_blockfile_info[m_last_blockfile].ToString());
for (int nFile = m_last_blockfile + 1; true; nFile++) {
LogPrintf("%s: last block file info: %s\n", __func__, m_blockfile_info[max_blockfile_num].ToString());
for (int nFile = max_blockfile_num + 1; true; nFile++) {
CBlockFileInfo info;
if (m_block_tree_db->ReadBlockFileInfo(nFile, info)) {
m_blockfile_info.push_back(info);
@ -464,6 +508,15 @@ bool BlockManager::LoadBlockIndexDB()
}
}
{
// Initialize the blockfile cursors.
LOCK(cs_LastBlockFile);
for (size_t i = 0; i < m_blockfile_info.size(); ++i) {
const auto last_height_in_file = m_blockfile_info[i].nHeightLast;
m_blockfile_cursors[BlockfileTypeForHeight(last_height_in_file)] = {static_cast<int>(i), 0};
}
}
// Check whether we have ever pruned block & undo files
m_block_tree_db->ReadFlag("prunedblockfiles", m_have_pruned);
if (m_have_pruned) {
@ -481,12 +534,13 @@ bool BlockManager::LoadBlockIndexDB()
void BlockManager::ScanAndUnlinkAlreadyPrunedFiles()
{
AssertLockHeld(::cs_main);
int max_blockfile = WITH_LOCK(cs_LastBlockFile, return this->MaxBlockfileNum());
if (!m_have_pruned) {
return;
}
std::set<int> block_files_to_prune;
for (int file_number = 0; file_number < m_last_blockfile; file_number++) {
for (int file_number = 0; file_number < max_blockfile; file_number++) {
if (m_blockfile_info[file_number].nSize == 0) {
block_files_to_prune.insert(file_number);
}
@ -661,7 +715,7 @@ bool BlockManager::FlushUndoFile(int block_file, bool finalize)
return true;
}
bool BlockManager::FlushBlockFile(bool fFinalize, bool finalize_undo)
bool BlockManager::FlushBlockFile(int blockfile_num, bool fFinalize, bool finalize_undo)
{
bool success = true;
LOCK(cs_LastBlockFile);
@ -673,9 +727,9 @@ bool BlockManager::FlushBlockFile(bool fFinalize, bool finalize_undo)
// have populated `m_blockfile_info` via LoadBlockIndexDB().
return true;
}
assert(static_cast<int>(m_blockfile_info.size()) > m_last_blockfile);
assert(static_cast<int>(m_blockfile_info.size()) > blockfile_num);
FlatFilePos block_pos_old(m_last_blockfile, m_blockfile_info[m_last_blockfile].nSize);
FlatFilePos block_pos_old(blockfile_num, m_blockfile_info[blockfile_num].nSize);
if (!BlockFileSeq().Flush(block_pos_old, fFinalize)) {
m_opts.notifications.flushError("Flushing block file to disk failed. This is likely the result of an I/O error.");
success = false;
@ -683,13 +737,33 @@ bool BlockManager::FlushBlockFile(bool fFinalize, bool finalize_undo)
// we do not always flush the undo file, as the chain tip may be lagging behind the incoming blocks,
// e.g. during IBD or a sync after a node going offline
if (!fFinalize || finalize_undo) {
if (!FlushUndoFile(m_last_blockfile, finalize_undo)) {
if (!FlushUndoFile(blockfile_num, finalize_undo)) {
success = false;
}
}
return success;
}
BlockfileType BlockManager::BlockfileTypeForHeight(int height)
{
if (!m_snapshot_height) {
return BlockfileType::NORMAL;
}
return (height >= *m_snapshot_height) ? BlockfileType::ASSUMED : BlockfileType::NORMAL;
}
bool BlockManager::FlushChainstateBlockFile(int tip_height)
{
LOCK(cs_LastBlockFile);
auto& cursor = m_blockfile_cursors[BlockfileTypeForHeight(tip_height)];
if (cursor) {
// The cursor may not exist after a snapshot has been loaded but before any
// blocks have been downloaded.
return FlushBlockFile(cursor->file_num, /*fFinalize=*/false, /*finalize_undo=*/false);
}
return false;
}
uint64_t BlockManager::CalculateCurrentUsage()
{
LOCK(cs_LastBlockFile);
@ -744,8 +818,19 @@ bool BlockManager::FindBlockPos(FlatFilePos& pos, unsigned int nAddSize, unsigne
{
LOCK(cs_LastBlockFile);
unsigned int nFile = fKnown ? pos.nFile : m_last_blockfile;
if (m_blockfile_info.size() <= nFile) {
const BlockfileType chain_type = BlockfileTypeForHeight(nHeight);
if (!m_blockfile_cursors[chain_type]) {
// If a snapshot is loaded during runtime, we may not have initialized this cursor yet.
assert(chain_type == BlockfileType::ASSUMED);
const auto new_cursor = BlockfileCursor{this->MaxBlockfileNum() + 1};
m_blockfile_cursors[chain_type] = new_cursor;
LogPrint(BCLog::BLOCKSTORAGE, "[%s] initializing blockfile cursor to %s\n", chain_type, new_cursor);
}
const int last_blockfile = m_blockfile_cursors[chain_type]->file_num;
int nFile = fKnown ? pos.nFile : last_blockfile;
if (static_cast<int>(m_blockfile_info.size()) <= nFile) {
m_blockfile_info.resize(nFile + 1);
}
@ -762,13 +847,20 @@ bool BlockManager::FindBlockPos(FlatFilePos& pos, unsigned int nAddSize, unsigne
}
}
assert(nAddSize < max_blockfile_size);
while (m_blockfile_info[nFile].nSize + nAddSize >= max_blockfile_size) {
// when the undo file is keeping up with the block file, we want to flush it explicitly
// when it is lagging behind (more blocks arrive than are being connected), we let the
// undo block write case handle it
finalize_undo = (m_blockfile_info[nFile].nHeightLast == m_undo_height_in_last_blockfile);
nFile++;
if (m_blockfile_info.size() <= nFile) {
finalize_undo = (static_cast<int>(m_blockfile_info[nFile].nHeightLast) ==
Assert(m_blockfile_cursors[chain_type])->undo_height);
// Try the next unclaimed blockfile number
nFile = this->MaxBlockfileNum() + 1;
// Set to increment MaxBlockfileNum() for next iteration
m_blockfile_cursors[chain_type] = BlockfileCursor{nFile};
if (static_cast<int>(m_blockfile_info.size()) <= nFile) {
m_blockfile_info.resize(nFile + 1);
}
}
@ -776,9 +868,10 @@ bool BlockManager::FindBlockPos(FlatFilePos& pos, unsigned int nAddSize, unsigne
pos.nPos = m_blockfile_info[nFile].nSize;
}
if ((int)nFile != m_last_blockfile) {
if (nFile != last_blockfile) {
if (!fKnown) {
LogPrint(BCLog::BLOCKSTORAGE, "Leaving block file %i: %s\n", m_last_blockfile, m_blockfile_info[m_last_blockfile].ToString());
LogPrint(BCLog::BLOCKSTORAGE, "Leaving block file %i: %s (onto %i) (height %i)\n",
last_blockfile, m_blockfile_info[last_blockfile].ToString(), nFile, nHeight);
}
// Do not propagate the return code. The flush concerns a previous block
@ -788,13 +881,13 @@ bool BlockManager::FindBlockPos(FlatFilePos& pos, unsigned int nAddSize, unsigne
// data may be inconsistent after a crash if the flush is called during
// a reindex. A flush error might also leave some of the data files
// untrimmed.
if (!FlushBlockFile(!fKnown, finalize_undo)) {
if (!FlushBlockFile(last_blockfile, !fKnown, finalize_undo)) {
LogPrintLevel(BCLog::BLOCKSTORAGE, BCLog::Level::Warning,
"Failed to flush previous block file %05i (finalize=%i, finalize_undo=%i) before opening new block file %05i\n",
m_last_blockfile, !fKnown, finalize_undo, nFile);
last_blockfile, !fKnown, finalize_undo, nFile);
}
m_last_blockfile = nFile;
m_undo_height_in_last_blockfile = 0; // No undo data yet in the new file, so reset our undo-height tracking.
// No undo data yet in the new file, so reset our undo-height tracking.
m_blockfile_cursors[chain_type] = BlockfileCursor{nFile};
}
m_blockfile_info[nFile].AddBlock(nHeight, nTime);
@ -868,6 +961,9 @@ bool BlockManager::WriteBlockToDisk(const CBlock& block, FlatFilePos& pos) const
bool BlockManager::WriteUndoDataForBlock(const CBlockUndo& blockundo, BlockValidationState& state, CBlockIndex& block)
{
AssertLockHeld(::cs_main);
const BlockfileType type = BlockfileTypeForHeight(block.nHeight);
auto& cursor = *Assert(WITH_LOCK(cs_LastBlockFile, return m_blockfile_cursors[type]));
// Write undo information to disk
if (block.GetUndoPos().IsNull()) {
FlatFilePos _pos;
@ -882,7 +978,7 @@ bool BlockManager::WriteUndoDataForBlock(const CBlockUndo& blockundo, BlockValid
// in the block file info as below; note that this does not catch the case where the undo writes are keeping up
// with the block writes (usually when a synced up node is getting newly mined blocks) -- this case is caught in
// the FindBlockPos function
if (_pos.nFile < m_last_blockfile && static_cast<uint32_t>(block.nHeight) == m_blockfile_info[_pos.nFile].nHeightLast) {
if (_pos.nFile < cursor.file_num && static_cast<uint32_t>(block.nHeight) == m_blockfile_info[_pos.nFile].nHeightLast) {
// Do not propagate the return code, a failed flush here should not
// be an indication for a failed write. If it were propagated here,
// the caller would assume the undo data not to be written, when in
@ -891,8 +987,8 @@ bool BlockManager::WriteUndoDataForBlock(const CBlockUndo& blockundo, BlockValid
if (!FlushUndoFile(_pos.nFile, true)) {
LogPrintLevel(BCLog::BLOCKSTORAGE, BCLog::Level::Warning, "Failed to flush undo file %05i\n", _pos.nFile);
}
} else if (_pos.nFile == m_last_blockfile && static_cast<uint32_t>(block.nHeight) > m_undo_height_in_last_blockfile) {
m_undo_height_in_last_blockfile = block.nHeight;
} else if (_pos.nFile == cursor.file_num && block.nHeight > cursor.undo_height) {
cursor.undo_height = block.nHeight;
}
// update nUndoPos in block index
block.nUndoPos = _pos.nPos;
@ -1091,4 +1187,18 @@ void ImportBlocks(ChainstateManager& chainman, std::vector<fs::path> vImportFile
}
} // End scope of ImportingNow
}
std::ostream& operator<<(std::ostream& os, const BlockfileType& type) {
switch(type) {
case BlockfileType::NORMAL: os << "normal"; break;
case BlockfileType::ASSUMED: os << "assumed"; break;
default: os.setstate(std::ios_base::failbit);
}
return os;
}
std::ostream& operator<<(std::ostream& os, const BlockfileCursor& cursor) {
os << strprintf("BlockfileCursor(file_num=%d, undo_height=%d)", cursor.file_num, cursor.undo_height);
return os;
}
} // namespace node

101
src/node/blockstorage.h
View file

@ -9,6 +9,7 @@
#include <chain.h>
#include <dbwrapper.h>
#include <kernel/blockmanager_opts.h>
#include <kernel/chain.h>
#include <kernel/chainparams.h>
#include <kernel/cs_main.h>
#include <kernel/messagestartchars.h>
@ -97,6 +98,35 @@ struct PruneLockInfo {
int height_first{std::numeric_limits<int>::max()}; //! Height of earliest block that should be kept and not pruned
};
enum BlockfileType {
// Values used as array indexes - do not change carelessly.
NORMAL = 0,
ASSUMED = 1,
NUM_TYPES = 2,
};
std::ostream& operator<<(std::ostream& os, const BlockfileType& type);
struct BlockfileCursor {
// The latest blockfile number.
int file_num{0};
// Track the height of the highest block in file_num whose undo
// data has been written. Block data is written to block files in download
// order, but is written to undo files in validation order, which is
// usually in order by height. To avoid wasting disk space, undo files will
// be trimmed whenever the corresponding block file is finalized and
// the height of the highest block written to the block file equals the
// height of the highest block written to the undo file. This is a
// heuristic and can sometimes preemptively trim undo files that will write
// more data later, and sometimes fail to trim undo files that can't have
// more data written later.
int undo_height{0};
};
std::ostream& operator<<(std::ostream& os, const BlockfileCursor& cursor);
/**
* Maintains a tree of blocks (stored in `m_block_index`) which is consulted
* to determine where the most-work tip is.
@ -117,16 +147,17 @@ private:
* per index entry (nStatus, nChainWork, nTimeMax, etc.) as well as peripheral
* collections like m_dirty_blockindex.
*/
bool LoadBlockIndex()
bool LoadBlockIndex(const std::optional<uint256>& snapshot_blockhash)
EXCLUSIVE_LOCKS_REQUIRED(cs_main);
/** Return false if block file or undo file flushing fails. */
[[nodiscard]] bool FlushBlockFile(bool fFinalize = false, bool finalize_undo = false);
[[nodiscard]] bool FlushBlockFile(int blockfile_num, bool fFinalize, bool finalize_undo);
/** Return false if undo file flushing fails. */
[[nodiscard]] bool FlushUndoFile(int block_file, bool finalize = false);
[[nodiscard]] bool FindBlockPos(FlatFilePos& pos, unsigned int nAddSize, unsigned int nHeight, uint64_t nTime, bool fKnown);
[[nodiscard]] bool FlushChainstateBlockFile(int tip_height);
bool FindUndoPos(BlockValidationState& state, int nFile, FlatFilePos& pos, unsigned int nAddSize);
FlatFileSeq BlockFileSeq() const;
@ -138,7 +169,11 @@ private:
bool UndoWriteToDisk(const CBlockUndo& blockundo, FlatFilePos& pos, const uint256& hashBlock) const;
/* Calculate the block/rev files to delete based on height specified by user with RPC command pruneblockchain */
void FindFilesToPruneManual(std::set<int>& setFilesToPrune, int nManualPruneHeight, int chain_tip_height);
void FindFilesToPruneManual(
std::set<int>& setFilesToPrune,
int nManualPruneHeight,
const Chainstate& chain,
ChainstateManager& chainman);
/**
* Prune block and undo files (blk???.dat and rev???.dat) so that the disk space used is less than a user-defined target.
@ -154,24 +189,39 @@ private:
* A db flag records the fact that at least some block files have been pruned.
*
* @param[out] setFilesToPrune The set of file indices that can be unlinked will be returned
* @param last_prune The last height we're able to prune, according to the prune locks
*/
void FindFilesToPrune(std::set<int>& setFilesToPrune, uint64_t nPruneAfterHeight, int chain_tip_height, int prune_height, bool is_ibd);
void FindFilesToPrune(
std::set<int>& setFilesToPrune,
int last_prune,
const Chainstate& chain,
ChainstateManager& chainman);
RecursiveMutex cs_LastBlockFile;
std::vector<CBlockFileInfo> m_blockfile_info;
int m_last_blockfile = 0;
// Track the height of the highest block in m_last_blockfile whose undo
// data has been written. Block data is written to block files in download
// order, but is written to undo files in validation order, which is
// usually in order by height. To avoid wasting disk space, undo files will
// be trimmed whenever the corresponding block file is finalized and
// the height of the highest block written to the block file equals the
// height of the highest block written to the undo file. This is a
// heuristic and can sometimes preemptively trim undo files that will write
// more data later, and sometimes fail to trim undo files that can't have
// more data written later.
unsigned int m_undo_height_in_last_blockfile = 0;
//! Since assumedvalid chainstates may be syncing a range of the chain that is very
//! far away from the normal/background validation process, we should segment blockfiles
//! for assumed chainstates. Otherwise, we might have wildly different height ranges
//! mixed into the same block files, which would impair our ability to prune
//! effectively.
//!
//! This data structure maintains separate blockfile number cursors for each
//! BlockfileType. The ASSUMED state is initialized, when necessary, in FindBlockPos().
//!
//! The first element is the NORMAL cursor, second is ASSUMED.
std::array<std::optional<BlockfileCursor>, BlockfileType::NUM_TYPES>
m_blockfile_cursors GUARDED_BY(cs_LastBlockFile) = {
BlockfileCursor{},
std::nullopt,
};
int MaxBlockfileNum() const EXCLUSIVE_LOCKS_REQUIRED(cs_LastBlockFile)
{
static const BlockfileCursor empty_cursor;
const auto& normal = m_blockfile_cursors[BlockfileType::NORMAL].value_or(empty_cursor);
const auto& assumed = m_blockfile_cursors[BlockfileType::ASSUMED].value_or(empty_cursor);
return std::max(normal.file_num, assumed.file_num);
}
/** Global flag to indicate we should check to see if there are
* block/undo files that should be deleted. Set on startup
@ -195,6 +245,8 @@ private:
*/
std::unordered_map<std::string, PruneLockInfo> m_prune_locks GUARDED_BY(::cs_main);
BlockfileType BlockfileTypeForHeight(int height);
const kernel::BlockManagerOpts m_opts;
public:
@ -210,6 +262,20 @@ public:
BlockMap m_block_index GUARDED_BY(cs_main);
/**
* The height of the base block of an assumeutxo snapshot, if one is in use.
*
* This controls how blockfiles are segmented by chainstate type to avoid
* comingling different height regions of the chain when an assumedvalid chainstate
* is in use. If heights are drastically different in the same blockfile, pruning
* suffers.
*
* This is set during ActivateSnapshot() or upon LoadBlockIndex() if a snapshot
* had been previously loaded. After the snapshot is validated, this is unset to
* restore normal LoadBlockIndex behavior.
*/
std::optional<int> m_snapshot_height;
std::vector<CBlockIndex*> GetAllBlockIndices() EXCLUSIVE_LOCKS_REQUIRED(::cs_main);
/**
@ -221,7 +287,8 @@ public:
std::unique_ptr<BlockTreeDB> m_block_tree_db GUARDED_BY(::cs_main);
bool WriteBlockIndexDB() EXCLUSIVE_LOCKS_REQUIRED(::cs_main);
bool LoadBlockIndexDB() EXCLUSIVE_LOCKS_REQUIRED(::cs_main);
bool LoadBlockIndexDB(const std::optional<uint256>& snapshot_blockhash)
EXCLUSIVE_LOCKS_REQUIRED(::cs_main);
/**
* Remove any pruned block & undo files that are still on disk.

9
src/node/chainstate.cpp
View file

@ -185,7 +185,14 @@ ChainstateLoadResult LoadChainstate(ChainstateManager& chainman, const CacheSize
chainman.InitializeChainstate(options.mempool);
// Load a chain created from a UTXO snapshot, if any exist.
chainman.DetectSnapshotChainstate(options.mempool);
bool has_snapshot = chainman.DetectSnapshotChainstate(options.mempool);
if (has_snapshot && (options.reindex || options.reindex_chainstate)) {
LogPrintf("[snapshot] deleting snapshot chainstate due to reindexing\n");
if (!chainman.DeleteSnapshotChainstate()) {
return {ChainstateLoadStatus::FAILURE_FATAL, Untranslated("Couldn't remove snapshot chainstate.")};
}
}
auto [init_status, init_error] = CompleteChainstateInitialization(chainman, cache_sizes, options);
if (init_status != ChainstateLoadStatus::SUCCESS) {

8
src/node/interfaces.cpp
View file

@ -434,9 +434,9 @@ public:
{
m_notifications->transactionRemovedFromMempool(tx, reason);
}
void BlockConnected(const std::shared_ptr<const CBlock>& block, const CBlockIndex* index) override
void BlockConnected(ChainstateRole role, const std::shared_ptr<const CBlock>& block, const CBlockIndex* index) override
{
m_notifications->blockConnected(kernel::MakeBlockInfo(index, block.get()));
m_notifications->blockConnected(role, kernel::MakeBlockInfo(index, block.get()));
}
void BlockDisconnected(const std::shared_ptr<const CBlock>& block, const CBlockIndex* index) override
{
@ -446,7 +446,9 @@ public:
{
m_notifications->updatedBlockTip();
}
void ChainStateFlushed(const CBlockLocator& locator) override { m_notifications->chainStateFlushed(locator); }
void ChainStateFlushed(ChainstateRole role, const CBlockLocator& locator) override {
m_notifications->chainStateFlushed(role, locator);
}
std::shared_ptr<Chain::Notifications> m_notifications;
};

177
src/rpc/blockchain.cpp
View file

@ -8,6 +8,7 @@
#include <blockfilter.h>
#include <chain.h>
#include <chainparams.h>
#include <clientversion.h>
#include <coins.h>
#include <common/args.h>
#include <consensus/amount.h>
@ -2699,6 +2700,178 @@ UniValue CreateUTXOSnapshot(
return result;
}
static RPCHelpMan loadtxoutset()
{
return RPCHelpMan{
"loadtxoutset",
"Load the serialized UTXO set from disk.\n"
"Once this snapshot is loaded, its contents will be "
"deserialized into a second chainstate data structure, which is then used to sync to "
"the network's tip under a security model very much like `assumevalid`. "
"Meanwhile, the original chainstate will complete the initial block download process in "
"the background, eventually validating up to the block that the snapshot is based upon.\n\n"
"The result is a usable bitcoind instance that is current with the network tip in a "
"matter of minutes rather than hours. UTXO snapshot are typically obtained from "
"third-party sources (HTTP, torrent, etc.) which is reasonable since their "
"contents are always checked by hash.\n\n"
"You can find more information on this process in the `assumeutxo` design "
"document (<https://github.com/bitcoin/bitcoin/blob/master/doc/design/assumeutxo.md>).",
{
{"path",
RPCArg::Type::STR,
RPCArg::Optional::NO,
"path to the snapshot file. If relative, will be prefixed by datadir."},
},
RPCResult{
RPCResult::Type::OBJ, "", "",
{
{RPCResult::Type::NUM, "coins_loaded", "the number of coins loaded from the snapshot"},
{RPCResult::Type::STR_HEX, "tip_hash", "the hash of the base of the snapshot"},
{RPCResult::Type::NUM, "base_height", "the height of the base of the snapshot"},
{RPCResult::Type::STR, "path", "the absolute path that the snapshot was loaded from"},
}
},
RPCExamples{
HelpExampleCli("loadtxoutset", "utxo.dat")
},
[&](const RPCHelpMan& self, const JSONRPCRequest& request) -> UniValue
{
NodeContext& node = EnsureAnyNodeContext(request.context);
fs::path path{AbsPathForConfigVal(EnsureArgsman(node), fs::u8path(request.params[0].get_str()))};
FILE* file{fsbridge::fopen(path, "rb")};
AutoFile afile{file};
if (afile.IsNull()) {
throw JSONRPCError(
RPC_INVALID_PARAMETER,
"Couldn't open file " + path.u8string() + " for reading.");
}
SnapshotMetadata metadata;
afile >> metadata;
uint256 base_blockhash = metadata.m_base_blockhash;
int max_secs_to_wait_for_headers = 60 * 10;
CBlockIndex* snapshot_start_block = nullptr;
LogPrintf("[snapshot] waiting to see blockheader %s in headers chain before snapshot activation\n",
base_blockhash.ToString());
ChainstateManager& chainman = *node.chainman;
while (max_secs_to_wait_for_headers > 0) {
snapshot_start_block = WITH_LOCK(::cs_main,
return chainman.m_blockman.LookupBlockIndex(base_blockhash));
max_secs_to_wait_for_headers -= 1;
if (!IsRPCRunning()) {
throw JSONRPCError(RPC_CLIENT_NOT_CONNECTED, "Shutting down");
}
if (!snapshot_start_block) {
std::this_thread::sleep_for(std::chrono::seconds(1));
} else {
break;
}
}
if (!snapshot_start_block) {
LogPrintf("[snapshot] timed out waiting for snapshot start blockheader %s\n",
base_blockhash.ToString());
throw JSONRPCError(
RPC_INTERNAL_ERROR,
"Timed out waiting for base block header to appear in headers chain");
}
if (!chainman.ActivateSnapshot(afile, metadata, false)) {
throw JSONRPCError(RPC_INTERNAL_ERROR, "Unable to load UTXO snapshot " + fs::PathToString(path));
}
CBlockIndex* new_tip{WITH_LOCK(::cs_main, return chainman.ActiveTip())};
UniValue result(UniValue::VOBJ);
result.pushKV("coins_loaded", metadata.m_coins_count);
result.pushKV("tip_hash", new_tip->GetBlockHash().ToString());
result.pushKV("base_height", new_tip->nHeight);
result.pushKV("path", fs::PathToString(path));
return result;
},
};
}
const std::vector<RPCResult> RPCHelpForChainstate{
{RPCResult::Type::NUM, "blocks", "number of blocks in this chainstate"},
{RPCResult::Type::STR_HEX, "bestblockhash", "blockhash of the tip"},
{RPCResult::Type::NUM, "difficulty", "difficulty of the tip"},
{RPCResult::Type::NUM, "verificationprogress", "progress towards the network tip"},
{RPCResult::Type::STR_HEX, "snapshot_blockhash", /*optional=*/true, "the base block of the snapshot this chainstate is based on, if any"},
{RPCResult::Type::NUM, "coins_db_cache_bytes", "size of the coinsdb cache"},
{RPCResult::Type::NUM, "coins_tip_cache_bytes", "size of the coinstip cache"},
};
static RPCHelpMan getchainstates()
{
return RPCHelpMan{
"getchainstates",
"\nReturn information about chainstates.\n",
{},
RPCResult{
RPCResult::Type::OBJ, "", "", {
{RPCResult::Type::NUM, "headers", "the number of headers seen so far"},
{RPCResult::Type::OBJ, "normal", /*optional=*/true, "fully validated chainstate containing blocks this node has validated starting from the genesis block", RPCHelpForChainstate},
{RPCResult::Type::OBJ, "snapshot", /*optional=*/true, "only present if an assumeutxo snapshot is loaded. Partially validated chainstate containing blocks this node has validated starting from the snapshot. After the snapshot is validated (when the 'normal' chainstate advances far enough to validate it), this chainstate will replace and become the 'normal' chainstate.", RPCHelpForChainstate},
}
},
RPCExamples{
HelpExampleCli("getchainstates", "")
+ HelpExampleRpc("getchainstates", "")
},
[&](const RPCHelpMan& self, const JSONRPCRequest& request) -> UniValue
{
LOCK(cs_main);
UniValue obj(UniValue::VOBJ);
NodeContext& node = EnsureAnyNodeContext(request.context);
ChainstateManager& chainman = *node.chainman;
auto make_chain_data = [&](const Chainstate& cs) EXCLUSIVE_LOCKS_REQUIRED(::cs_main) {
AssertLockHeld(::cs_main);
UniValue data(UniValue::VOBJ);
if (!cs.m_chain.Tip()) {
return data;
}
const CChain& chain = cs.m_chain;
const CBlockIndex* tip = chain.Tip();
data.pushKV("blocks", (int)chain.Height());
data.pushKV("bestblockhash", tip->GetBlockHash().GetHex());
data.pushKV("difficulty", (double)GetDifficulty(tip));
data.pushKV("verificationprogress", GuessVerificationProgress(Params().TxData(), tip));
data.pushKV("coins_db_cache_bytes", cs.m_coinsdb_cache_size_bytes);
data.pushKV("coins_tip_cache_bytes", cs.m_coinstip_cache_size_bytes);
if (cs.m_from_snapshot_blockhash) {
data.pushKV("snapshot_blockhash", cs.m_from_snapshot_blockhash->ToString());
}
return data;
};
if (chainman.GetAll().size() > 1) {
for (Chainstate* chainstate : chainman.GetAll()) {
obj.pushKV(
chainstate->m_from_snapshot_blockhash ? "snapshot" : "normal",
make_chain_data(*chainstate));
}
} else {
obj.pushKV("normal", make_chain_data(chainman.ActiveChainstate()));
}
obj.pushKV("headers", chainman.m_best_header ? chainman.m_best_header->nHeight : -1);
return obj;
}
};
}
void RegisterBlockchainRPCCommands(CRPCTable& t)
{
static const CRPCCommand commands[]{
@ -2722,13 +2895,15 @@ void RegisterBlockchainRPCCommands(CRPCTable& t)
{"blockchain", &scantxoutset},
{"blockchain", &scanblocks},
{"blockchain", &getblockfilter},
{"blockchain", &dumptxoutset},
{"blockchain", &loadtxoutset},
{"blockchain", &getchainstates},
{"hidden", &invalidateblock},
{"hidden", &reconsiderblock},
{"hidden", &waitfornewblock},
{"hidden", &waitforblock},
{"hidden", &waitforblockheight},
{"hidden", &syncwithvalidationinterfacequeue},
{"hidden", &dumptxoutset},
};
for (const auto& c : commands) {
t.appendCommand(c.name, &c);

2
src/test/coinstatsindex_tests.cpp
View file

@ -105,7 +105,7 @@ BOOST_FIXTURE_TEST_CASE(coinstatsindex_unclean_shutdown, TestChain100Setup)
// Send block connected notification, then stop the index without
// sending a chainstate flushed notification. Prior to #24138, this
// would cause the index to be corrupted and fail to reload.
ValidationInterfaceTest::BlockConnected(index, new_block, new_block_index);
ValidationInterfaceTest::BlockConnected(ChainstateRole::NORMAL, index, new_block, new_block_index);
index.Stop();
}

2
src/test/fuzz/rpc.cpp
View file

@ -80,6 +80,7 @@ const std::vector<std::string> RPC_COMMANDS_NOT_SAFE_FOR_FUZZING{
"gettxoutproof", // avoid prohibitively slow execution
"importmempool", // avoid reading from disk
"importwallet", // avoid reading from disk
"loadtxoutset", // avoid reading from disk
"loadwallet", // avoid reading from disk
"savemempool", // disabled as a precautionary measure: may take a file path argument in the future
"setban", // avoid DNS lookups
@ -122,6 +123,7 @@ const std::vector<std::string> RPC_COMMANDS_SAFE_FOR_FUZZING{
"getblockstats",
"getblocktemplate",
"getchaintips",
"getchainstates",
"getchaintxstats",
"getconnectioncount",
"getdeploymentinfo",

18
src/test/util/chainstate.h
View file

@ -109,7 +109,23 @@ CreateAndActivateUTXOSnapshot(
0 == WITH_LOCK(node.chainman->GetMutex(), return node.chainman->ActiveHeight()));
}
return node.chainman->ActivateSnapshot(auto_infile, metadata, in_memory_chainstate);
auto& new_active = node.chainman->ActiveChainstate();
auto* tip = new_active.m_chain.Tip();
// Disconnect a block so that the snapshot chainstate will be ahead, otherwise
// it will refuse to activate.
//
// TODO this is a unittest-specific hack, and we should probably rethink how to
// better generate/activate snapshots in unittests.
if (tip->pprev) {
new_active.m_chain.SetTip(*(tip->pprev));
}
bool res = node.chainman->ActivateSnapshot(auto_infile, metadata, in_memory_chainstate);
// Restore the old tip.
new_active.m_chain.SetTip(*tip);
return res;
}

8
src/test/util/validation.cpp
View file

@ -22,7 +22,11 @@ void TestChainstateManager::JumpOutOfIbd()
Assert(!IsInitialBlockDownload());
}
void ValidationInterfaceTest::BlockConnected(CValidationInterface& obj, const std::shared_ptr<const CBlock>& block, const CBlockIndex* pindex)
void ValidationInterfaceTest::BlockConnected(
ChainstateRole role,
CValidationInterface& obj,
const std::shared_ptr<const CBlock>& block,
const CBlockIndex* pindex)
{
obj.BlockConnected(block, pindex);
obj.BlockConnected(role, block, pindex);
}

6
src/test/util/validation.h
View file

@ -19,7 +19,11 @@ struct TestChainstateManager : public ChainstateManager {
class ValidationInterfaceTest
{
public:
static void BlockConnected(CValidationInterface& obj, const std::shared_ptr<const CBlock>& block, const CBlockIndex* pindex);
static void BlockConnected(
ChainstateRole role,
CValidationInterface& obj,
const std::shared_ptr<const CBlock>& block,
const CBlockIndex* pindex);
};
#endif // BITCOIN_TEST_UTIL_VALIDATION_H

2
src/test/validation_block_tests.cpp
View file

@ -43,7 +43,7 @@ struct TestSubscriber final : public CValidationInterface {
BOOST_CHECK_EQUAL(m_expected_tip, pindexNew->GetBlockHash());
}
void BlockConnected(const std::shared_ptr<const CBlock>& block, const CBlockIndex* pindex) override
void BlockConnected(ChainstateRole role, const std::shared_ptr<const CBlock>& block, const CBlockIndex* pindex) override
{
BOOST_CHECK_EQUAL(m_expected_tip, block->hashPrevBlock);
BOOST_CHECK_EQUAL(m_expected_tip, pindex->pprev->GetBlockHash());

118
src/test/validation_chainstatemanager_tests.cpp
View file

@ -30,30 +30,22 @@ using node::BlockManager;
using node::KernelNotifications;
using node::SnapshotMetadata;
BOOST_FIXTURE_TEST_SUITE(validation_chainstatemanager_tests, ChainTestingSetup)
BOOST_FIXTURE_TEST_SUITE(validation_chainstatemanager_tests, TestingSetup)
//! Basic tests for ChainstateManager.
//!
//! First create a legacy (IBD) chainstate, then create a snapshot chainstate.
BOOST_AUTO_TEST_CASE(chainstatemanager)
BOOST_FIXTURE_TEST_CASE(chainstatemanager, TestChain100Setup)
{
ChainstateManager& manager = *m_node.chainman;
CTxMemPool& mempool = *m_node.mempool;
std::vector<Chainstate*> chainstates;
BOOST_CHECK(!manager.SnapshotBlockhash().has_value());
// Create a legacy (IBD) chainstate.
//
Chainstate& c1 = WITH_LOCK(::cs_main, return manager.InitializeChainstate(&mempool));
Chainstate& c1 = manager.ActiveChainstate();
chainstates.push_back(&c1);
c1.InitCoinsDB(
/*cache_size_bytes=*/1 << 23, /*in_memory=*/true, /*should_wipe=*/false);
WITH_LOCK(::cs_main, c1.InitCoinsCache(1 << 23));
c1.LoadGenesisBlock();
BlockValidationState val_state;
BOOST_CHECK(c1.ActivateBestChain(val_state, nullptr));
BOOST_CHECK(!manager.IsSnapshotActive());
BOOST_CHECK(WITH_LOCK(::cs_main, return !manager.IsSnapshotValidated()));
@ -63,8 +55,9 @@ BOOST_AUTO_TEST_CASE(chainstatemanager)
auto& active_chain = WITH_LOCK(manager.GetMutex(), return manager.ActiveChain());
BOOST_CHECK_EQUAL(&active_chain, &c1.m_chain);
BOOST_CHECK_EQUAL(WITH_LOCK(manager.GetMutex(), return manager.ActiveHeight()), 0);
// Get to a valid assumeutxo tip (per chainparams);
mineBlocks(10);
BOOST_CHECK_EQUAL(WITH_LOCK(manager.GetMutex(), return manager.ActiveHeight()), 110);
auto active_tip = WITH_LOCK(manager.GetMutex(), return manager.ActiveTip());
auto exp_tip = c1.m_chain.Tip();
BOOST_CHECK_EQUAL(active_tip, exp_tip);
@ -74,19 +67,21 @@ BOOST_AUTO_TEST_CASE(chainstatemanager)
// Create a snapshot-based chainstate.
//
const uint256 snapshot_blockhash = active_tip->GetBlockHash();
Chainstate& c2 = WITH_LOCK(::cs_main, return manager.ActivateExistingSnapshot(
&mempool, snapshot_blockhash));
Chainstate& c2 = WITH_LOCK(::cs_main, return manager.ActivateExistingSnapshot(snapshot_blockhash));
chainstates.push_back(&c2);
BOOST_CHECK_EQUAL(manager.SnapshotBlockhash().value(), snapshot_blockhash);
c2.InitCoinsDB(
/*cache_size_bytes=*/1 << 23, /*in_memory=*/true, /*should_wipe=*/false);
WITH_LOCK(::cs_main, c2.InitCoinsCache(1 << 23));
c2.m_chain.SetTip(*active_tip);
{
LOCK(::cs_main);
c2.InitCoinsCache(1 << 23);
c2.CoinsTip().SetBestBlock(active_tip->GetBlockHash());
c2.setBlockIndexCandidates.insert(manager.m_blockman.LookupBlockIndex(active_tip->GetBlockHash()));
c2.LoadChainTip();
}
BlockValidationState _;
BOOST_CHECK(c2.ActivateBestChain(_, nullptr));
BOOST_CHECK_EQUAL(manager.SnapshotBlockhash().value(), snapshot_blockhash);
BOOST_CHECK(manager.IsSnapshotActive());
BOOST_CHECK(WITH_LOCK(::cs_main, return !manager.IsSnapshotValidated()));
BOOST_CHECK_EQUAL(&c2, &manager.ActiveChainstate());
@ -97,13 +92,15 @@ BOOST_AUTO_TEST_CASE(chainstatemanager)
auto& active_chain2 = WITH_LOCK(manager.GetMutex(), return manager.ActiveChain());
BOOST_CHECK_EQUAL(&active_chain2, &c2.m_chain);
BOOST_CHECK_EQUAL(WITH_LOCK(manager.GetMutex(), return manager.ActiveHeight()), 0);
BOOST_CHECK_EQUAL(WITH_LOCK(manager.GetMutex(), return manager.ActiveHeight()), 110);
mineBlocks(1);
BOOST_CHECK_EQUAL(WITH_LOCK(manager.GetMutex(), return manager.ActiveHeight()), 111);
BOOST_CHECK_EQUAL(WITH_LOCK(manager.GetMutex(), return c1.m_chain.Height()), 110);
auto active_tip2 = WITH_LOCK(manager.GetMutex(), return manager.ActiveTip());
auto exp_tip2 = c2.m_chain.Tip();
BOOST_CHECK_EQUAL(active_tip2, exp_tip2);
BOOST_CHECK_EQUAL(exp_tip, exp_tip2);
BOOST_CHECK_EQUAL(active_tip, active_tip2->pprev);
BOOST_CHECK_EQUAL(active_tip, c1.m_chain.Tip());
BOOST_CHECK_EQUAL(active_tip2, c2.m_chain.Tip());
// Let scheduler events finish running to avoid accessing memory that is going to be unloaded
SyncWithValidationInterfaceQueue();
@ -113,7 +110,6 @@ BOOST_AUTO_TEST_CASE(chainstatemanager)
BOOST_FIXTURE_TEST_CASE(chainstatemanager_rebalance_caches, TestChain100Setup)
{
ChainstateManager& manager = *m_node.chainman;
CTxMemPool& mempool = *m_node.mempool;
size_t max_cache = 10000;
manager.m_total_coinsdb_cache = max_cache;
@ -125,9 +121,6 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_rebalance_caches, TestChain100Setup)
//
Chainstate& c1 = manager.ActiveChainstate();
chainstates.push_back(&c1);
c1.InitCoinsDB(
/*cache_size_bytes=*/1 << 23, /*in_memory=*/true, /*should_wipe=*/false);
{
LOCK(::cs_main);
c1.InitCoinsCache(1 << 23);
@ -140,7 +133,7 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_rebalance_caches, TestChain100Setup)
// Create a snapshot-based chainstate.
//
CBlockIndex* snapshot_base{WITH_LOCK(manager.GetMutex(), return manager.ActiveChain()[manager.ActiveChain().Height() / 2])};
Chainstate& c2 = WITH_LOCK(cs_main, return manager.ActivateExistingSnapshot(&mempool, *snapshot_base->phashBlock));
Chainstate& c2 = WITH_LOCK(cs_main, return manager.ActivateExistingSnapshot(*snapshot_base->phashBlock));
chainstates.push_back(&c2);
c2.InitCoinsDB(
/*cache_size_bytes=*/1 << 23, /*in_memory=*/true, /*should_wipe=*/false);
@ -289,10 +282,10 @@ struct SnapshotTestSetup : TestChain100Setup {
BOOST_CHECK(!chainman.ActiveChain().Genesis()->IsAssumedValid());
}
const AssumeutxoData& au_data = *ExpectedAssumeutxo(snapshot_height, ::Params());
const auto& au_data = ::Params().AssumeutxoForHeight(snapshot_height);
const CBlockIndex* tip = WITH_LOCK(chainman.GetMutex(), return chainman.ActiveTip());
BOOST_CHECK_EQUAL(tip->nChainTx, au_data.nChainTx);
BOOST_CHECK_EQUAL(tip->nChainTx, au_data->nChainTx);
// To be checked against later when we try loading a subsequent snapshot.
uint256 loaded_snapshot_blockhash{*chainman.SnapshotBlockhash()};
@ -426,18 +419,24 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_activate_snapshot, SnapshotTestSetup)
BOOST_FIXTURE_TEST_CASE(chainstatemanager_loadblockindex, TestChain100Setup)
{
ChainstateManager& chainman = *Assert(m_node.chainman);
CTxMemPool& mempool = *m_node.mempool;
Chainstate& cs1 = chainman.ActiveChainstate();
int num_indexes{0};
int num_assumed_valid{0};
// Blocks in range [assumed_valid_start_idx, last_assumed_valid_idx) will be
// marked as assumed-valid and not having data.
const int expected_assumed_valid{20};
const int last_assumed_valid_idx{40};
const int last_assumed_valid_idx{111};
const int assumed_valid_start_idx = last_assumed_valid_idx - expected_assumed_valid;
// Mine to height 120, past the hardcoded regtest assumeutxo snapshot at
// height 110
mineBlocks(20);
CBlockIndex* validated_tip{nullptr};
CBlockIndex* assumed_base{nullptr};
CBlockIndex* assumed_tip{WITH_LOCK(chainman.GetMutex(), return chainman.ActiveChain().Tip())};
BOOST_CHECK_EQUAL(assumed_tip->nHeight, 120);
auto reload_all_block_indexes = [&]() {
// For completeness, we also reset the block sequence counters to
@ -463,7 +462,7 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_loadblockindex, TestChain100Setup)
LOCK(::cs_main);
auto index = cs1.m_chain[i];
// Blocks with heights in range [20, 40) are marked ASSUMED_VALID
// Blocks with heights in range [91, 110] are marked ASSUMED_VALID
if (i < last_assumed_valid_idx && i >= assumed_valid_start_idx) {
index->nStatus = BlockStatus::BLOCK_VALID_TREE | BlockStatus::BLOCK_ASSUMED_VALID;
}
@ -489,7 +488,7 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_loadblockindex, TestChain100Setup)
// Note: cs2's tip is not set when ActivateExistingSnapshot is called.
Chainstate& cs2 = WITH_LOCK(::cs_main,
return chainman.ActivateExistingSnapshot(&mempool, *assumed_base->phashBlock));
return chainman.ActivateExistingSnapshot(*assumed_base->phashBlock));
// Set tip of the fully validated chain to be the validated tip
cs1.m_chain.SetTip(*validated_tip);
@ -497,10 +496,36 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_loadblockindex, TestChain100Setup)
// Set tip of the assume-valid-based chain to the assume-valid block
cs2.m_chain.SetTip(*assumed_base);
// Sanity check test variables.
BOOST_CHECK_EQUAL(num_indexes, 121); // 121 total blocks, including genesis
BOOST_CHECK_EQUAL(assumed_tip->nHeight, 120); // original chain has height 120
BOOST_CHECK_EQUAL(validated_tip->nHeight, 90); // current cs1 chain has height 90
BOOST_CHECK_EQUAL(assumed_base->nHeight, 110); // current cs2 chain has height 110
// Regenerate cs1.setBlockIndexCandidates and cs2.setBlockIndexCandidate and
// check contents below.
reload_all_block_indexes();
// The fully validated chain should have the current validated tip
// and the assumed valid base as candidates.
// The fully validated chain should only have the current validated tip and
// the assumed valid base as candidates, blocks 90 and 110. Specifically:
//
// - It does not have blocks 0-89 because they contain less work than the
// chain tip.
//
// - It has block 90 because it has data and equal work to the chain tip,
// (since it is the chain tip).
//
// - It does not have blocks 91-109 because they do not contain data.
//
// - It has block 110 even though it does not have data, because
// LoadBlockIndex has a special case to always add the snapshot block as a
// candidate. The special case is only actually intended to apply to the
// snapshot chainstate cs2, not the background chainstate cs1, but it is
// written broadly and applies to both.
//
// - It does not have any blocks after height 110 because cs1 is a background
// chainstate, and only blocks where are ancestors of the snapshot block
// are added as candidates for the background chainstate.
BOOST_CHECK_EQUAL(cs1.setBlockIndexCandidates.size(), 2);
BOOST_CHECK_EQUAL(cs1.setBlockIndexCandidates.count(validated_tip), 1);
BOOST_CHECK_EQUAL(cs1.setBlockIndexCandidates.count(assumed_base), 1);
@ -508,8 +533,25 @@ BOOST_FIXTURE_TEST_CASE(chainstatemanager_loadblockindex, TestChain100Setup)
// The assumed-valid tolerant chain has the assumed valid base as a
// candidate, but otherwise has none of the assumed-valid (which do not
// HAVE_DATA) blocks as candidates.
//
// Specifically:
// - All blocks below height 110 are not candidates, because cs2 chain tip
// has height 110 and they have less work than it does.
//
// - Block 110 is a candidate even though it does not have data, because it
// is the snapshot block, which is assumed valid.
//
// - Blocks 111-120 are added because they have data.
// Check that block 90 is absent
BOOST_CHECK_EQUAL(cs2.setBlockIndexCandidates.count(validated_tip), 0);
// Check that block 109 is absent
BOOST_CHECK_EQUAL(cs2.setBlockIndexCandidates.count(assumed_base->pprev), 0);
// Check that block 110 is present
BOOST_CHECK_EQUAL(cs2.setBlockIndexCandidates.count(assumed_base), 1);
// Check that block 120 is present
BOOST_CHECK_EQUAL(cs2.setBlockIndexCandidates.count(assumed_tip), 1);
// Check that 11 blocks total are present.
BOOST_CHECK_EQUAL(cs2.setBlockIndexCandidates.size(), num_indexes - last_assumed_valid_idx + 1);
}

10
src/test/validation_tests.cpp
View file

@ -132,17 +132,17 @@ BOOST_AUTO_TEST_CASE(test_assumeutxo)
std::vector<int> bad_heights{0, 100, 111, 115, 209, 211};
for (auto empty : bad_heights) {
const auto out = ExpectedAssumeutxo(empty, *params);
const auto out = params->AssumeutxoForHeight(empty);
BOOST_CHECK(!out);
}
const auto out110 = *ExpectedAssumeutxo(110, *params);
const auto out110 = *params->AssumeutxoForHeight(110);
BOOST_CHECK_EQUAL(out110.hash_serialized.ToString(), "1ebbf5850204c0bdb15bf030f47c7fe91d45c44c712697e4509ba67adb01c618");
BOOST_CHECK_EQUAL(out110.nChainTx, 110U);
const auto out210 = *ExpectedAssumeutxo(200, *params);
BOOST_CHECK_EQUAL(out210.hash_serialized.ToString(), "51c8d11d8b5c1de51543c579736e786aa2736206d1e11e627568029ce092cf62");
BOOST_CHECK_EQUAL(out210.nChainTx, 200U);
const auto out110_2 = *params->AssumeutxoForBlockhash(uint256S("0x696e92821f65549c7ee134edceeeeaaa4105647a3c4fd9f298c0aec0ab50425c"));
BOOST_CHECK_EQUAL(out110_2.hash_serialized.ToString(), "1ebbf5850204c0bdb15bf030f47c7fe91d45c44c712697e4509ba67adb01c618");
BOOST_CHECK_EQUAL(out110_2.nChainTx, 110U);
}
BOOST_AUTO_TEST_SUITE_END()

1
src/test/validationinterface_tests.cpp
View file

@ -8,6 +8,7 @@
#include <scheduler.h>
#include <test/util/setup_common.h>
#include <util/check.h>
#include <kernel/chain.h>
#include <validationinterface.h>
#include <atomic>

13
src/util/vector.h
View file

@ -5,7 +5,9 @@
#ifndef BITCOIN_UTIL_VECTOR_H
#define BITCOIN_UTIL_VECTOR_H
#include <functional>
#include <initializer_list>
#include <optional>
#include <type_traits>
#include <utility>
#include <vector>
@ -67,4 +69,15 @@ inline void ClearShrink(V& v) noexcept
V{}.swap(v);
}
template<typename V, typename L>
inline std::optional<V> FindFirst(const std::vector<V>& vec, const L fnc)
{
for (const auto& el : vec) {
if (fnc(el)) {
return el;
}
}
return std::nullopt;
}
#endif // BITCOIN_UTIL_VECTOR_H

233
src/validation.cpp
View file

@ -5,6 +5,7 @@
#include <validation.h>
#include <kernel/chain.h>
#include <kernel/coinstats.h>
#include <kernel/mempool_persist.h>
@ -68,6 +69,7 @@
#include <optional>
#include <string>
#include <utility>
#include <tuple>
using kernel::CCoinsStats;
using kernel::CoinStatsHashType;
@ -2551,11 +2553,14 @@ bool Chainstate::FlushStateToDisk(
if (nManualPruneHeight > 0) {
LOG_TIME_MILLIS_WITH_CATEGORY("find files to prune (manual)", BCLog::BENCH);
m_blockman.FindFilesToPruneManual(setFilesToPrune, std::min(last_prune, nManualPruneHeight), m_chain.Height());
m_blockman.FindFilesToPruneManual(
setFilesToPrune,
std::min(last_prune, nManualPruneHeight),
*this, m_chainman);
} else {
LOG_TIME_MILLIS_WITH_CATEGORY("find files to prune", BCLog::BENCH);
m_blockman.FindFilesToPrune(setFilesToPrune, m_chainman.GetParams().PruneAfterHeight(), m_chain.Height(), last_prune, m_chainman.IsInitialBlockDownload());
m_blockman.FindFilesToPrune(setFilesToPrune, last_prune, *this, m_chainman);
m_blockman.m_check_for_pruning = false;
}
if (!setFilesToPrune.empty()) {
@ -2596,7 +2601,7 @@ bool Chainstate::FlushStateToDisk(
// First make sure all block and undo data is flushed to disk.
// TODO: Handle return error, or add detailed comment why it is
// safe to not return an error upon failure.
if (!m_blockman.FlushBlockFile()) {
if (!m_blockman.FlushChainstateBlockFile(m_chain.Height())) {
LogPrintLevel(BCLog::VALIDATION, BCLog::Level::Warning, "%s: Failed to flush block file.\n", __func__);
}
}
@ -2645,7 +2650,7 @@ bool Chainstate::FlushStateToDisk(
}
if (full_flush_completed) {
// Update best block in wallet (so we can detect restored wallets).
GetMainSignals().ChainStateFlushed(m_chain.GetLocator());
GetMainSignals().ChainStateFlushed(this->GetRole(), m_chain.GetLocator());
}
} catch (const std::runtime_error& e) {
return FatalError(m_chainman.GetNotifications(), state, std::string("System error while flushing: ") + e.what());
@ -3192,6 +3197,7 @@ bool Chainstate::ActivateBestChain(BlockValidationState& state, std::shared_ptr<
CBlockIndex *pindexMostWork = nullptr;
CBlockIndex *pindexNewTip = nullptr;
bool exited_ibd{false};
do {
// Block until the validation queue drains. This should largely
// never happen in normal operation, however may happen during
@ -3205,6 +3211,7 @@ bool Chainstate::ActivateBestChain(BlockValidationState& state, std::shared_ptr<
LOCK(cs_main);
// Lock transaction pool for at least as long as it takes for connectTrace to be consumed
LOCK(MempoolMutex());
const bool was_in_ibd = m_chainman.IsInitialBlockDownload();
CBlockIndex* starting_tip = m_chain.Tip();
bool blocks_connected = false;
do {
@ -3237,7 +3244,7 @@ bool Chainstate::ActivateBestChain(BlockValidationState& state, std::shared_ptr<
for (const PerBlockConnectTrace& trace : connectTrace.GetBlocksConnected()) {
assert(trace.pblock && trace.pindex);
GetMainSignals().BlockConnected(trace.pblock, trace.pindex);
GetMainSignals().BlockConnected(this->GetRole(), trace.pblock, trace.pindex);
}
// This will have been toggled in
@ -3252,16 +3259,21 @@ bool Chainstate::ActivateBestChain(BlockValidationState& state, std::shared_ptr<
if (!blocks_connected) return true;
const CBlockIndex* pindexFork = m_chain.FindFork(starting_tip);
bool fInitialDownload = m_chainman.IsInitialBlockDownload();
bool still_in_ibd = m_chainman.IsInitialBlockDownload();
if (was_in_ibd && !still_in_ibd) {
// Active chainstate has exited IBD.
exited_ibd = true;
}
// Notify external listeners about the new tip.
// Enqueue while holding cs_main to ensure that UpdatedBlockTip is called in the order in which blocks are connected
if (pindexFork != pindexNewTip) {
if (this == &m_chainman.ActiveChainstate() && pindexFork != pindexNewTip) {
// Notify ValidationInterface subscribers
GetMainSignals().UpdatedBlockTip(pindexNewTip, pindexFork, fInitialDownload);
GetMainSignals().UpdatedBlockTip(pindexNewTip, pindexFork, still_in_ibd);
// Always notify the UI if a new block tip was connected
if (kernel::IsInterrupted(m_chainman.GetNotifications().blockTip(GetSynchronizationState(fInitialDownload), *pindexNewTip))) {
if (kernel::IsInterrupted(m_chainman.GetNotifications().blockTip(GetSynchronizationState(still_in_ibd), *pindexNewTip))) {
// Just breaking and returning success for now. This could
// be changed to bubble up the kernel::Interrupted value to
// the caller so the caller could distinguish between
@ -3272,8 +3284,25 @@ bool Chainstate::ActivateBestChain(BlockValidationState& state, std::shared_ptr<
}
// When we reach this point, we switched to a new tip (stored in pindexNewTip).
if (exited_ibd) {
// If a background chainstate is in use, we may need to rebalance our
// allocation of caches once a chainstate exits initial block download.
LOCK(::cs_main);
m_chainman.MaybeRebalanceCaches();
}
if (WITH_LOCK(::cs_main, return m_disabled)) {
// Background chainstate has reached the snapshot base block, so exit.
// Restart indexes to resume indexing for all blocks unique to the snapshot
// chain. This resumes indexing "in order" from where the indexing on the
// background validation chain left off.
//
// This cannot be done while holding cs_main (within
// MaybeCompleteSnapshotValidation) or a cs_main deadlock will occur.
if (m_chainman.restart_indexes) {
m_chainman.restart_indexes();
}
break;
}
@ -3510,7 +3539,8 @@ void Chainstate::ResetBlockFailureFlags(CBlockIndex *pindex) {
void Chainstate::TryAddBlockIndexCandidate(CBlockIndex* pindex)
{
AssertLockHeld(cs_main);
// The block only is a candidate for the most-work-chain if it has more work than our current tip.
// The block only is a candidate for the most-work-chain if it has the same
// or more work than our current tip.
if (m_chain.Tip() != nullptr && setBlockIndexCandidates.value_comp()(pindex, m_chain.Tip())) {
return;
}
@ -4148,6 +4178,12 @@ bool ChainstateManager::ProcessNewBlock(const std::shared_ptr<const CBlock>& blo
return error("%s: ActivateBestChain failed (%s)", __func__, state.ToString());
}
Chainstate* bg_chain{WITH_LOCK(cs_main, return BackgroundSyncInProgress() ? m_ibd_chainstate.get() : nullptr)};
BlockValidationState bg_state;
if (bg_chain && !bg_chain->ActivateBestChain(bg_state, block)) {
return error("%s: [background] ActivateBestChain failed (%s)", __func__, bg_state.ToString());
}
return true;
}
@ -4275,7 +4311,7 @@ VerifyDBResult CVerifyDB::VerifyDB(
bool skipped_l3_checks{false};
LogPrintf("Verification progress: 0%%\n");
const bool is_snapshot_cs{!chainstate.m_from_snapshot_blockhash};
const bool is_snapshot_cs{chainstate.m_from_snapshot_blockhash};
for (pindex = chainstate.m_chain.Tip(); pindex && pindex->pprev; pindex = pindex->pprev) {
const int percentageDone = std::max(1, std::min(99, (int)(((double)(chainstate.m_chain.Height() - pindex->nHeight)) / (double)nCheckDepth * (nCheckLevel >= 4 ? 50 : 100))));
@ -4506,7 +4542,7 @@ bool ChainstateManager::LoadBlockIndex()
// Load block index from databases
bool needs_init = fReindex;
if (!fReindex) {
bool ret{m_blockman.LoadBlockIndexDB()};
bool ret{m_blockman.LoadBlockIndexDB(SnapshotBlockhash())};
if (!ret) return false;
m_blockman.ScanAndUnlinkAlreadyPrunedFiles();
@ -4802,6 +4838,10 @@ void ChainstateManager::CheckBlockIndex()
CBlockIndex* pindexFirstAssumeValid = nullptr; // Oldest ancestor of pindex which has BLOCK_ASSUMED_VALID
while (pindex != nullptr) {
nNodes++;
if (pindex->pprev && pindex->nTx > 0) {
// nChainTx should increase monotonically
assert(pindex->pprev->nChainTx <= pindex->nChainTx);
}
if (pindexFirstAssumeValid == nullptr && pindex->nStatus & BLOCK_ASSUMED_VALID) pindexFirstAssumeValid = pindex;
if (pindexFirstInvalid == nullptr && pindex->nStatus & BLOCK_FAILED_VALID) pindexFirstInvalid = pindex;
if (pindexFirstMissing == nullptr && !(pindex->nStatus & BLOCK_HAVE_DATA)) {
@ -5093,19 +5133,7 @@ Chainstate& ChainstateManager::InitializeChainstate(CTxMemPool* mempool)
return *m_active_chainstate;
}
const AssumeutxoData* ExpectedAssumeutxo(
const int height, const CChainParams& chainparams)
{
const MapAssumeutxo& valid_assumeutxos_map = chainparams.Assumeutxo();
const auto assumeutxo_found = valid_assumeutxos_map.find(height);
if (assumeutxo_found != valid_assumeutxos_map.end()) {
return &assumeutxo_found->second;
}
return nullptr;
}
static bool DeleteCoinsDBFromDisk(const fs::path db_path, bool is_snapshot)
[[nodiscard]] static bool DeleteCoinsDBFromDisk(const fs::path db_path, bool is_snapshot)
EXCLUSIVE_LOCKS_REQUIRED(::cs_main)
{
AssertLockHeld(::cs_main);
@ -5157,6 +5185,14 @@ bool ChainstateManager::ActivateSnapshot(
return false;
}
{
LOCK(::cs_main);
if (Assert(m_active_chainstate->GetMempool())->size() > 0) {
LogPrintf("[snapshot] can't activate a snapshot when mempool not empty\n");
return false;
}
}
int64_t current_coinsdb_cache_size{0};
int64_t current_coinstip_cache_size{0};
@ -5202,19 +5238,8 @@ bool ChainstateManager::ActivateSnapshot(
static_cast<size_t>(current_coinstip_cache_size * SNAPSHOT_CACHE_PERC));
}
bool snapshot_ok = this->PopulateAndValidateSnapshot(
*snapshot_chainstate, coins_file, metadata);
// If not in-memory, persist the base blockhash for use during subsequent
// initialization.
if (!in_memory) {
LOCK(::cs_main);
if (!node::WriteSnapshotBaseBlockhash(*snapshot_chainstate)) {
snapshot_ok = false;
}
}
if (!snapshot_ok) {
LOCK(::cs_main);
auto cleanup_bad_snapshot = [&](const char* reason) EXCLUSIVE_LOCKS_REQUIRED(::cs_main) {
LogPrintf("[snapshot] activation failed - %s\n", reason);
this->MaybeRebalanceCaches();
// PopulateAndValidateSnapshot can return (in error) before the leveldb datadir
@ -5231,23 +5256,48 @@ bool ChainstateManager::ActivateSnapshot(
}
}
return false;
};
if (!this->PopulateAndValidateSnapshot(*snapshot_chainstate, coins_file, metadata)) {
LOCK(::cs_main);
return cleanup_bad_snapshot("population failed");
}
LOCK(::cs_main); // cs_main required for rest of snapshot activation.
// Do a final check to ensure that the snapshot chainstate is actually a more
// work chain than the active chainstate; a user could have loaded a snapshot
// very late in the IBD process, and we wouldn't want to load a useless chainstate.
if (!CBlockIndexWorkComparator()(ActiveTip(), snapshot_chainstate->m_chain.Tip())) {
return cleanup_bad_snapshot("work does not exceed active chainstate");
}
// If not in-memory, persist the base blockhash for use during subsequent
// initialization.
if (!in_memory) {
if (!node::WriteSnapshotBaseBlockhash(*snapshot_chainstate)) {
return cleanup_bad_snapshot("could not write base blockhash");
}
}
{
LOCK(::cs_main);
assert(!m_snapshot_chainstate);
m_snapshot_chainstate.swap(snapshot_chainstate);
const bool chaintip_loaded = m_snapshot_chainstate->LoadChainTip();
assert(chaintip_loaded);
// Transfer possession of the mempool to the snapshot chainstate.
// Mempool is empty at this point because we're still in IBD.
Assert(m_active_chainstate->m_mempool->size() == 0);
Assert(!m_snapshot_chainstate->m_mempool);
m_snapshot_chainstate->m_mempool = m_active_chainstate->m_mempool;
m_active_chainstate->m_mempool = nullptr;
m_active_chainstate = m_snapshot_chainstate.get();
m_blockman.m_snapshot_height = this->GetSnapshotBaseHeight();
LogPrintf("[snapshot] successfully activated snapshot %s\n", base_blockhash.ToString());
LogPrintf("[snapshot] (%.2f MB)\n",
m_snapshot_chainstate->CoinsTip().DynamicMemoryUsage() / (1000 * 1000));
this->MaybeRebalanceCaches();
}
return true;
}
@ -5289,7 +5339,7 @@ bool ChainstateManager::PopulateAndValidateSnapshot(
CBlockIndex* snapshot_start_block = WITH_LOCK(::cs_main, return m_blockman.LookupBlockIndex(base_blockhash));
if (!snapshot_start_block) {
// Needed for ComputeUTXOStats and ExpectedAssumeutxo to determine the
// Needed for ComputeUTXOStats to determine the
// height and to avoid a crash when base_blockhash.IsNull()
LogPrintf("[snapshot] Did not find snapshot start blockheader %s\n",
base_blockhash.ToString());
@ -5297,7 +5347,7 @@ bool ChainstateManager::PopulateAndValidateSnapshot(
}
int base_height = snapshot_start_block->nHeight;
auto maybe_au_data = ExpectedAssumeutxo(base_height, GetParams());
const auto& maybe_au_data = GetParams().AssumeutxoForHeight(base_height);
if (!maybe_au_data) {
LogPrintf("[snapshot] assumeutxo height in snapshot metadata not recognized "
@ -5307,6 +5357,14 @@ bool ChainstateManager::PopulateAndValidateSnapshot(
const AssumeutxoData& au_data = *maybe_au_data;
// This work comparison is a duplicate check with the one performed later in
// ActivateSnapshot(), but is done so that we avoid doing the long work of staging
// a snapshot that isn't actually usable.
if (WITH_LOCK(::cs_main, return !CBlockIndexWorkComparator()(ActiveTip(), snapshot_start_block))) {
LogPrintf("[snapshot] activation failed - height does not exceed active chainstate\n");
return false;
}
COutPoint outpoint;
Coin coin;
const uint64_t coins_count = metadata.m_coins_count;
@ -5566,7 +5624,7 @@ SnapshotCompletionResult ChainstateManager::MaybeCompleteSnapshotValidation()
CCoinsViewDB& ibd_coins_db = m_ibd_chainstate->CoinsDB();
m_ibd_chainstate->ForceFlushStateToDisk();
auto maybe_au_data = ExpectedAssumeutxo(curr_height, m_options.chainparams);
const auto& maybe_au_data = m_options.chainparams.AssumeutxoForHeight(curr_height);
if (!maybe_au_data) {
LogPrintf("[snapshot] assumeutxo data not found for height "
"(%d) - refusing to validate snapshot\n", curr_height);
@ -5718,16 +5776,22 @@ bool ChainstateManager::DetectSnapshotChainstate(CTxMemPool* mempool)
LogPrintf("[snapshot] detected active snapshot chainstate (%s) - loading\n",
fs::PathToString(*path));
this->ActivateExistingSnapshot(mempool, *base_blockhash);
this->ActivateExistingSnapshot(*base_blockhash);
return true;
}
Chainstate& ChainstateManager::ActivateExistingSnapshot(CTxMemPool* mempool, uint256 base_blockhash)
Chainstate& ChainstateManager::ActivateExistingSnapshot(uint256 base_blockhash)
{
assert(!m_snapshot_chainstate);
m_snapshot_chainstate =
std::make_unique<Chainstate>(mempool, m_blockman, *this, base_blockhash);
std::make_unique<Chainstate>(nullptr, m_blockman, *this, base_blockhash);
LogPrintf("[snapshot] switching active chainstate to %s\n", m_snapshot_chainstate->ToString());
// Mempool is empty at this point because we're still in IBD.
Assert(m_active_chainstate->m_mempool->size() == 0);
Assert(!m_snapshot_chainstate->m_mempool);
m_snapshot_chainstate->m_mempool = m_active_chainstate->m_mempool;
m_active_chainstate->m_mempool = nullptr;
m_active_chainstate = m_snapshot_chainstate.get();
return *m_snapshot_chainstate;
}
@ -5744,15 +5808,20 @@ bool IsBIP30Unspendable(const CBlockIndex& block_index)
(block_index.nHeight==91812 && block_index.GetBlockHash() == uint256S("0x00000000000af0aed4792b1acee3d966af36cf5def14935db8de83d6f9306f2f"));
}
util::Result<void> Chainstate::InvalidateCoinsDBOnDisk()
static fs::path GetSnapshotCoinsDBPath(Chainstate& cs) EXCLUSIVE_LOCKS_REQUIRED(::cs_main)
{
AssertLockHeld(::cs_main);
// Should never be called on a non-snapshot chainstate.
assert(m_from_snapshot_blockhash);
auto storage_path_maybe = this->CoinsDB().StoragePath();
assert(cs.m_from_snapshot_blockhash);
auto storage_path_maybe = cs.CoinsDB().StoragePath();
// Should never be called with a non-existent storage path.
assert(storage_path_maybe);
fs::path snapshot_datadir = *storage_path_maybe;
return *storage_path_maybe;
}
util::Result<void> Chainstate::InvalidateCoinsDBOnDisk()
{
fs::path snapshot_datadir = GetSnapshotCoinsDBPath(*this);
// Coins views no longer usable.
m_coins_views.reset();
@ -5783,6 +5852,33 @@ util::Result<void> Chainstate::InvalidateCoinsDBOnDisk()
return {};
}
bool ChainstateManager::DeleteSnapshotChainstate()
{
AssertLockHeld(::cs_main);
Assert(m_snapshot_chainstate);
Assert(m_ibd_chainstate);
fs::path snapshot_datadir = GetSnapshotCoinsDBPath(*m_snapshot_chainstate);
if (!DeleteCoinsDBFromDisk(snapshot_datadir, /*is_snapshot=*/ true)) {
LogPrintf("Deletion of %s failed. Please remove it manually to continue reindexing.\n",
fs::PathToString(snapshot_datadir));
return false;
}
m_active_chainstate = m_ibd_chainstate.get();
m_snapshot_chainstate.reset();
return true;
}
ChainstateRole Chainstate::GetRole() const
{
if (m_chainman.GetAll().size() <= 1) {
return ChainstateRole::NORMAL;
}
return (this != &m_chainman.ActiveChainstate()) ?
ChainstateRole::BACKGROUND :
ChainstateRole::ASSUMEDVALID;
}
const CBlockIndex* ChainstateManager::GetSnapshotBaseBlock() const
{
return m_active_chainstate ? m_active_chainstate->SnapshotBase() : nullptr;
@ -5880,3 +5976,38 @@ bool ChainstateManager::ValidatedSnapshotCleanup()
}
return true;
}
Chainstate& ChainstateManager::GetChainstateForIndexing()
{
// We can't always return `m_ibd_chainstate` because after background validation
// has completed, `m_snapshot_chainstate == m_active_chainstate`, but it can be
// indexed.
return (this->GetAll().size() > 1) ? *m_ibd_chainstate : *m_active_chainstate;
}
std::pair<int, int> ChainstateManager::GetPruneRange(const Chainstate& chainstate, int last_height_can_prune)
{
if (chainstate.m_chain.Height() <= 0) {
return {0, 0};
}
int prune_start{0};
if (this->GetAll().size() > 1 && m_snapshot_chainstate.get() == &chainstate) {
// Leave the blocks in the background IBD chain alone if we're pruning
// the snapshot chain.
prune_start = *Assert(GetSnapshotBaseHeight()) + 1;
}
int max_prune = std::max<int>(
0, chainstate.m_chain.Height() - static_cast<int>(MIN_BLOCKS_TO_KEEP));
// last block to prune is the lesser of (caller-specified height, MIN_BLOCKS_TO_KEEP from the tip)
//
// While you might be tempted to prune the background chainstate more
// aggressively (i.e. fewer MIN_BLOCKS_TO_KEEP), this won't work with index
// building - specifically blockfilterindex requires undo data, and if
// we don't maintain this trailing window, we hit indexing failures.
int prune_end = std::min(last_height_can_prune, max_prune);
return {prune_start, prune_end};
}

70
src/validation.h
View file

@ -13,6 +13,7 @@
#include <arith_uint256.h>
#include <attributes.h>
#include <chain.h>
#include <kernel/chain.h>
#include <consensus/amount.h>
#include <deploymentstatus.h>
#include <kernel/chainparams.h>
@ -511,6 +512,12 @@ public:
ChainstateManager& chainman,
std::optional<uint256> from_snapshot_blockhash = std::nullopt);
//! Return the current role of the chainstate. See `ChainstateManager`
//! documentation for a description of the different types of chainstates.
//!
//! @sa ChainstateRole
ChainstateRole GetRole() const EXCLUSIVE_LOCKS_REQUIRED(::cs_main);
/**
* Initialize the CoinsViews UTXO set database management data structures. The in-memory
* cache is initialized separately.
@ -848,9 +855,6 @@ private:
//! Points to either the ibd or snapshot chainstate; indicates our
//! most-work chain.
//!
//! Once this pointer is set to a corresponding chainstate, it will not
//! be reset until init.cpp:Shutdown().
//!
//! This is especially important when, e.g., calling ActivateBestChain()
//! on all chainstates because we are not able to hold ::cs_main going into
//! that call.
@ -881,13 +885,6 @@ private:
/** Most recent headers presync progress update, for rate-limiting. */
std::chrono::time_point<std::chrono::steady_clock> m_last_presync_update GUARDED_BY(::cs_main) {};
//! Returns nullptr if no snapshot has been loaded.
const CBlockIndex* GetSnapshotBaseBlock() const EXCLUSIVE_LOCKS_REQUIRED(::cs_main);
//! Return the height of the base block of the snapshot in use, if one exists, else
//! nullopt.
std::optional<int> GetSnapshotBaseHeight() const EXCLUSIVE_LOCKS_REQUIRED(::cs_main);
std::array<ThresholdConditionCache, VERSIONBITS_NUM_BITS> m_warningcache GUARDED_BY(::cs_main);
//! Return true if a chainstate is considered usable.
@ -904,6 +901,10 @@ public:
explicit ChainstateManager(const util::SignalInterrupt& interrupt, Options options, node::BlockManager::Options blockman_options);
//! Function to restart active indexes; set dynamically to avoid a circular
//! dependency on `base/index.cpp`.
std::function<void()> restart_indexes = std::function<void()>();
const CChainParams& GetParams() const { return m_options.chainparams; }
const Consensus::Params& GetConsensus() const { return m_options.chainparams.GetConsensus(); }
bool ShouldCheckBlockIndex() const { return *Assert(m_options.check_block_index); }
@ -1034,12 +1035,25 @@ public:
//! Otherwise, revert to using the ibd chainstate and shutdown.
SnapshotCompletionResult MaybeCompleteSnapshotValidation() EXCLUSIVE_LOCKS_REQUIRED(::cs_main);
//! Returns nullptr if no snapshot has been loaded.
const CBlockIndex* GetSnapshotBaseBlock() const EXCLUSIVE_LOCKS_REQUIRED(::cs_main);
//! The most-work chain.
Chainstate& ActiveChainstate() const;
CChain& ActiveChain() const EXCLUSIVE_LOCKS_REQUIRED(GetMutex()) { return ActiveChainstate().m_chain; }
int ActiveHeight() const EXCLUSIVE_LOCKS_REQUIRED(GetMutex()) { return ActiveChain().Height(); }
CBlockIndex* ActiveTip() const EXCLUSIVE_LOCKS_REQUIRED(GetMutex()) { return ActiveChain().Tip(); }
//! The state of a background sync (for net processing)
bool BackgroundSyncInProgress() const EXCLUSIVE_LOCKS_REQUIRED(GetMutex()) {
return IsUsable(m_snapshot_chainstate.get()) && IsUsable(m_ibd_chainstate.get());
}
//! The tip of the background sync chain
const CBlockIndex* GetBackgroundSyncTip() const EXCLUSIVE_LOCKS_REQUIRED(GetMutex()) {
return BackgroundSyncInProgress() ? m_ibd_chainstate->m_chain.Tip() : nullptr;
}
node::BlockMap& BlockIndex() EXCLUSIVE_LOCKS_REQUIRED(::cs_main)
{
AssertLockHeld(::cs_main);
@ -1193,10 +1207,13 @@ public:
void ResetChainstates() EXCLUSIVE_LOCKS_REQUIRED(::cs_main);
//! Remove the snapshot-based chainstate and all on-disk artifacts.
//! Used when reindex{-chainstate} is called during snapshot use.
[[nodiscard]] bool DeleteSnapshotChainstate() EXCLUSIVE_LOCKS_REQUIRED(::cs_main);
//! Switch the active chainstate to one based on a UTXO snapshot that was loaded
//! previously.
Chainstate& ActivateExistingSnapshot(CTxMemPool* mempool, uint256 base_blockhash)
EXCLUSIVE_LOCKS_REQUIRED(::cs_main);
Chainstate& ActivateExistingSnapshot(uint256 base_blockhash) EXCLUSIVE_LOCKS_REQUIRED(::cs_main);
//! If we have validated a snapshot chain during this runtime, copy its
//! chainstate directory over to the main `chainstate` location, completing
@ -1209,6 +1226,26 @@ public:
//! @sa node/chainstate:LoadChainstate()
bool ValidatedSnapshotCleanup() EXCLUSIVE_LOCKS_REQUIRED(::cs_main);
//! @returns the chainstate that indexes should consult when ensuring that an
//! index is synced with a chain where we can expect block index entries to have
//! BLOCK_HAVE_DATA beneath the tip.
//!
//! In other words, give us the chainstate for which we can reasonably expect
//! that all blocks beneath the tip have been indexed. In practice this means
//! when using an assumed-valid chainstate based upon a snapshot, return only the
//! fully validated chain.
Chainstate& GetChainstateForIndexing() EXCLUSIVE_LOCKS_REQUIRED(::cs_main);
//! Return the [start, end] (inclusive) of block heights we can prune.
//!
//! start > end is possible, meaning no blocks can be pruned.
std::pair<int, int> GetPruneRange(
const Chainstate& chainstate, int last_height_can_prune) EXCLUSIVE_LOCKS_REQUIRED(::cs_main);
//! Return the height of the base block of the snapshot in use, if one exists, else
//! nullopt.
std::optional<int> GetSnapshotBaseHeight() const EXCLUSIVE_LOCKS_REQUIRED(::cs_main);
~ChainstateManager();
};
@ -1231,15 +1268,6 @@ bool DeploymentEnabled(const ChainstateManager& chainman, DEP dep)
return DeploymentEnabled(chainman.GetConsensus(), dep);
}
/**
* Return the expected assumeutxo value for a given height, if one exists.
*
* @param[in] height Get the assumeutxo value for this height.
*
* @returns empty if no assumeutxo configuration exists for the given height.
*/
const AssumeutxoData* ExpectedAssumeutxo(const int height, const CChainParams& params);
/** Identifies blocks that overwrote an existing coinbase output in the UTXO set (see BIP30) */
bool IsBIP30Repeat(const CBlockIndex& block_index);

13
src/validationinterface.cpp
View file

@ -8,6 +8,7 @@
#include <attributes.h>
#include <chain.h>
#include <consensus/validation.h>
#include <kernel/chain.h>
#include <logging.h>
#include <primitives/block.h>
#include <primitives/transaction.h>
@ -223,9 +224,9 @@ void CMainSignals::TransactionRemovedFromMempool(const CTransactionRef& tx, MemP
RemovalReasonToString(reason));
}
void CMainSignals::BlockConnected(const std::shared_ptr<const CBlock> &pblock, const CBlockIndex *pindex) {
auto event = [pblock, pindex, this] {
m_internals->Iterate([&](CValidationInterface& callbacks) { callbacks.BlockConnected(pblock, pindex); });
void CMainSignals::BlockConnected(ChainstateRole role, const std::shared_ptr<const CBlock> &pblock, const CBlockIndex *pindex) {
auto event = [role, pblock, pindex, this] {
m_internals->Iterate([&](CValidationInterface& callbacks) { callbacks.BlockConnected(role, pblock, pindex); });
};
ENQUEUE_AND_LOG_EVENT(event, "%s: block hash=%s block height=%d", __func__,
pblock->GetHash().ToString(),
@ -242,9 +243,9 @@ void CMainSignals::BlockDisconnected(const std::shared_ptr<const CBlock>& pblock
pindex->nHeight);
}
void CMainSignals::ChainStateFlushed(const CBlockLocator &locator) {
auto event = [locator, this] {
m_internals->Iterate([&](CValidationInterface& callbacks) { callbacks.ChainStateFlushed(locator); });
void CMainSignals::ChainStateFlushed(ChainstateRole role, const CBlockLocator &locator) {
auto event = [role, locator, this] {
m_internals->Iterate([&](CValidationInterface& callbacks) { callbacks.ChainStateFlushed(role, locator); });
};
ENQUEUE_AND_LOG_EVENT(event, "%s: block hash=%s", __func__,
locator.IsNull() ? "null" : locator.vHave.front().ToString());

19
src/validationinterface.h
View file

@ -7,6 +7,7 @@
#define BITCOIN_VALIDATIONINTERFACE_H
#include <kernel/cs_main.h>
#include <kernel/chain.h>
#include <primitives/transaction.h> // CTransaction(Ref)
#include <sync.h>
@ -87,7 +88,7 @@ protected:
* but may not be called on every intermediate tip. If the latter behavior is desired,
* subscribe to BlockConnected() instead.
*
* Called on a background thread.
* Called on a background thread. Only called for the active chainstate.
*/
virtual void UpdatedBlockTip(const CBlockIndex *pindexNew, const CBlockIndex *pindexFork, bool fInitialDownload) {}
/**
@ -136,11 +137,12 @@ protected:
*
* Called on a background thread.
*/
virtual void BlockConnected(const std::shared_ptr<const CBlock> &block, const CBlockIndex *pindex) {}
virtual void BlockConnected(ChainstateRole role, const std::shared_ptr<const CBlock> &block, const CBlockIndex *pindex) {}
/**
* Notifies listeners of a block being disconnected
*
* Called on a background thread.
* Called on a background thread. Only called for the active chainstate, since
* background chainstates should never disconnect blocks.
*/
virtual void BlockDisconnected(const std::shared_ptr<const CBlock> &block, const CBlockIndex* pindex) {}
/**
@ -159,17 +161,18 @@ protected:
*
* Called on a background thread.
*/
virtual void ChainStateFlushed(const CBlockLocator &locator) {}
virtual void ChainStateFlushed(ChainstateRole role, const CBlockLocator &locator) {}
/**
* Notifies listeners of a block validation result.
* If the provided BlockValidationState IsValid, the provided block
* is guaranteed to be the current best block at the time the
* callback was generated (not necessarily now)
* callback was generated (not necessarily now).
*/
virtual void BlockChecked(const CBlock&, const BlockValidationState&) {}
/**
* Notifies listeners that a block which builds directly on our current tip
* has been received and connected to the headers tree, though not validated yet */
* has been received and connected to the headers tree, though not validated yet.
*/
virtual void NewPoWValidBlock(const CBlockIndex *pindex, const std::shared_ptr<const CBlock>& block) {};
friend class CMainSignals;
friend class ValidationInterfaceTest;
@ -199,9 +202,9 @@ public:
void UpdatedBlockTip(const CBlockIndex *, const CBlockIndex *, bool fInitialDownload);
void TransactionAddedToMempool(const CTransactionRef&, uint64_t mempool_sequence);
void TransactionRemovedFromMempool(const CTransactionRef&, MemPoolRemovalReason, uint64_t mempool_sequence);
void BlockConnected(const std::shared_ptr<const CBlock> &, const CBlockIndex *pindex);
void BlockConnected(ChainstateRole, const std::shared_ptr<const CBlock> &, const CBlockIndex *pindex);
void BlockDisconnected(const std::shared_ptr<const CBlock> &, const CBlockIndex* pindex);
void ChainStateFlushed(const CBlockLocator &);
void ChainStateFlushed(ChainstateRole, const CBlockLocator &);
void BlockChecked(const CBlock&, const BlockValidationState&);
void NewPoWValidBlock(const CBlockIndex *, const std::shared_ptr<const CBlock>&);
};

5
src/wallet/test/fuzz/notifications.cpp
View file

@ -2,6 +2,7 @@
// Distributed under the MIT software license, see the accompanying
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
#include <kernel/chain.h>
#include <test/fuzz/FuzzedDataProvider.h>
#include <test/fuzz/fuzz.h>
#include <test/fuzz/util.h>
@ -145,8 +146,8 @@ FUZZ_TARGET(wallet_notifications, .init = initialize_setup)
// time to the maximum value. This ensures that the wallet's birth time is always
// earlier than this maximum time.
info.chain_time_max = std::numeric_limits<unsigned int>::max();
a.wallet->blockConnected(info);
b.wallet->blockConnected(info);
a.wallet->blockConnected(ChainstateRole::NORMAL, info);
b.wallet->blockConnected(ChainstateRole::NORMAL, info);
// Store the coins for the next block
Coins coins_new;
for (const auto& tx : block.vtx) {

14
src/wallet/wallet.cpp
View file

@ -22,6 +22,7 @@
#include <interfaces/chain.h>
#include <interfaces/handler.h>
#include <interfaces/wallet.h>
#include <kernel/chain.h>
#include <kernel/mempool_removal_reason.h>
#include <key.h>
#include <key_io.h>
@ -626,11 +627,11 @@ bool CWallet::ChangeWalletPassphrase(const SecureString& strOldWalletPassphrase,
return false;
}
void CWallet::chainStateFlushed(const CBlockLocator& loc)
void CWallet::chainStateFlushed(ChainstateRole role, const CBlockLocator& loc)
{
// Don't update the best block until the chain is attached so that in case of a shutdown,
// the rescan will be restarted at next startup.
if (m_attaching_chain) {
if (m_attaching_chain || role == ChainstateRole::BACKGROUND) {
return;
}
WalletBatch batch(GetDatabase());
@ -1465,8 +1466,11 @@ void CWallet::transactionRemovedFromMempool(const CTransactionRef& tx, MemPoolRe
}
}
void CWallet::blockConnected(const interfaces::BlockInfo& block)
void CWallet::blockConnected(ChainstateRole role, const interfaces::BlockInfo& block)
{
if (role == ChainstateRole::BACKGROUND) {
return;
}
assert(block.data);
LOCK(cs_wallet);
@ -2944,7 +2948,7 @@ std::shared_ptr<CWallet> CWallet::Create(WalletContext& context, const std::stri
}
if (chain) {
walletInstance->chainStateFlushed(chain->getTipLocator());
walletInstance->chainStateFlushed(ChainstateRole::NORMAL, chain->getTipLocator());
}
} else if (wallet_creation_flags & WALLET_FLAG_DISABLE_PRIVATE_KEYS) {
// Make it impossible to disable private keys after creation
@ -3230,7 +3234,7 @@ bool CWallet::AttachChain(const std::shared_ptr<CWallet>& walletInstance, interf
}
}
walletInstance->m_attaching_chain = false;
walletInstance->chainStateFlushed(chain.getTipLocator());
walletInstance->chainStateFlushed(ChainstateRole::NORMAL, chain.getTipLocator());
walletInstance->GetDatabase().IncrementUpdateCounter();
}
walletInstance->m_attaching_chain = false;

4
src/wallet/wallet.h
View file

@ -599,7 +599,7 @@ public:
CWalletTx* AddToWallet(CTransactionRef tx, const TxState& state, const UpdateWalletTxFn& update_wtx=nullptr, bool fFlushOnClose=true, bool rescanning_old_block = false);
bool LoadToWallet(const uint256& hash, const UpdateWalletTxFn& fill_wtx) EXCLUSIVE_LOCKS_REQUIRED(cs_wallet);
void transactionAddedToMempool(const CTransactionRef& tx) override;
void blockConnected(const interfaces::BlockInfo& block) override;
void blockConnected(ChainstateRole role, const interfaces::BlockInfo& block) override;
void blockDisconnected(const interfaces::BlockInfo& block) override;
void updatedBlockTip() override;
int64_t RescanFromTime(int64_t startTime, const WalletRescanReserver& reserver, bool update);
@ -777,7 +777,7 @@ public:
/** should probably be renamed to IsRelevantToMe */
bool IsFromMe(const CTransaction& tx) const;
CAmount GetDebit(const CTransaction& tx, const isminefilter& filter) const;
void chainStateFlushed(const CBlockLocator& loc) override;
void chainStateFlushed(ChainstateRole role, const CBlockLocator& loc) override;
DBErrors LoadWallet();
DBErrors ZapSelectTx(std::vector<uint256>& vHashIn, std::vector<uint256>& vHashOut) EXCLUSIVE_LOCKS_REQUIRED(cs_wallet);

6
src/zmq/zmqnotificationinterface.cpp
View file

@ -5,6 +5,7 @@
#include <zmq/zmqnotificationinterface.h>
#include <common/args.h>
#include <kernel/chain.h>
#include <logging.h>
#include <primitives/block.h>
#include <primitives/transaction.h>
@ -170,8 +171,11 @@ void CZMQNotificationInterface::TransactionRemovedFromMempool(const CTransaction
});
}
void CZMQNotificationInterface::BlockConnected(const std::shared_ptr<const CBlock>& pblock, const CBlockIndex* pindexConnected)
void CZMQNotificationInterface::BlockConnected(ChainstateRole role, const std::shared_ptr<const CBlock>& pblock, const CBlockIndex* pindexConnected)
{
if (role == ChainstateRole::BACKGROUND) {
return;
}
for (const CTransactionRef& ptx : pblock->vtx) {
const CTransaction& tx = *ptx;
TryForEachAndRemoveFailed(notifiers, [&tx](CZMQAbstractNotifier* notifier) {

2
src/zmq/zmqnotificationinterface.h
View file

@ -33,7 +33,7 @@ protected:
// CValidationInterface
void TransactionAddedToMempool(const CTransactionRef& tx, uint64_t mempool_sequence) override;
void TransactionRemovedFromMempool(const CTransactionRef& tx, MemPoolRemovalReason reason, uint64_t mempool_sequence) override;
void BlockConnected(const std::shared_ptr<const CBlock>& pblock, const CBlockIndex* pindexConnected) override;
void BlockConnected(ChainstateRole role, const std::shared_ptr<const CBlock>& pblock, const CBlockIndex* pindexConnected) override;
void BlockDisconnected(const std::shared_ptr<const CBlock>& pblock, const CBlockIndex* pindexDisconnected) override;
void UpdatedBlockTip(const CBlockIndex *pindexNew, const CBlockIndex *pindexFork, bool fInitialDownload) override;

246
test/functional/feature_assumeutxo.py Executable file
View file

@ -0,0 +1,246 @@
#!/usr/bin/env python3
# Copyright (c) 2021 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
"""Test for assumeutxo, a means of quickly bootstrapping a node using
a serialized version of the UTXO set at a certain height, which corresponds
to a hash that has been compiled into bitcoind.
The assumeutxo value generated and used here is committed to in
`CRegTestParams::m_assumeutxo_data` in `src/chainparams.cpp`.
## Possible test improvements
- TODO: test submitting a transaction and verifying it appears in mempool
- TODO: test what happens with -reindex and -reindex-chainstate before the
snapshot is validated, and make sure it's deleted successfully.
Interesting test cases could be loading an assumeutxo snapshot file with:
- TODO: An invalid hash
- TODO: Valid hash but invalid snapshot file (bad coin height or truncated file or
bad other serialization)
- TODO: Valid snapshot file, but referencing an unknown block
- TODO: Valid snapshot file, but referencing a snapshot block that turns out to be
invalid, or has an invalid parent
- TODO: Valid snapshot file and snapshot block, but the block is not on the
most-work chain
Interesting starting states could be loading a snapshot when the current chain tip is:
- TODO: An ancestor of snapshot block
- TODO: Not an ancestor of the snapshot block but has less work
- TODO: The snapshot block
- TODO: A descendant of the snapshot block
- TODO: Not an ancestor or a descendant of the snapshot block and has more work
"""
from test_framework.test_framework import BitcoinTestFramework
from test_framework.util import assert_equal, wait_until_helper
START_HEIGHT = 199
SNAPSHOT_BASE_HEIGHT = 299
FINAL_HEIGHT = 399
COMPLETE_IDX = {'synced': True, 'best_block_height': FINAL_HEIGHT}
class AssumeutxoTest(BitcoinTestFramework):
def set_test_params(self):
"""Use the pregenerated, deterministic chain up to height 199."""
self.num_nodes = 3
self.rpc_timeout = 120
self.extra_args = [
[],
["-fastprune", "-prune=1", "-blockfilterindex=1", "-coinstatsindex=1"],
["-txindex=1", "-blockfilterindex=1", "-coinstatsindex=1"],
]
def setup_network(self):
"""Start with the nodes disconnected so that one can generate a snapshot
including blocks the other hasn't yet seen."""
self.add_nodes(3)
self.start_nodes(extra_args=self.extra_args)
def run_test(self):
"""
Bring up two (disconnected) nodes, mine some new blocks on the first,
and generate a UTXO snapshot.
Load the snapshot into the second, ensure it syncs to tip and completes
background validation when connected to the first.
"""
n0 = self.nodes[0]
n1 = self.nodes[1]
n2 = self.nodes[2]
# Mock time for a deterministic chain
for n in self.nodes:
n.setmocktime(n.getblockheader(n.getbestblockhash())['time'])
self.sync_blocks()
def no_sync():
pass
# Generate a series of blocks that `n0` will have in the snapshot,
# but that n1 doesn't yet see. In order for the snapshot to activate,
# though, we have to ferry over the new headers to n1 so that it
# isn't waiting forever to see the header of the snapshot's base block
# while disconnected from n0.
for i in range(100):
self.generate(n0, nblocks=1, sync_fun=no_sync)
newblock = n0.getblock(n0.getbestblockhash(), 0)
# make n1 aware of the new header, but don't give it the block.
n1.submitheader(newblock)
n2.submitheader(newblock)
# Ensure everyone is seeing the same headers.
for n in self.nodes:
assert_equal(n.getblockchaininfo()["headers"], SNAPSHOT_BASE_HEIGHT)
self.log.info("-- Testing assumeutxo + some indexes + pruning")
assert_equal(n0.getblockcount(), SNAPSHOT_BASE_HEIGHT)
assert_equal(n1.getblockcount(), START_HEIGHT)
self.log.info(f"Creating a UTXO snapshot at height {SNAPSHOT_BASE_HEIGHT}")
dump_output = n0.dumptxoutset('utxos.dat')
assert_equal(
dump_output['txoutset_hash'],
'ef45ccdca5898b6c2145e4581d2b88c56564dd389e4bd75a1aaf6961d3edd3c0')
assert_equal(dump_output['nchaintx'], 300)
assert_equal(n0.getblockchaininfo()["blocks"], SNAPSHOT_BASE_HEIGHT)
# Mine more blocks on top of the snapshot that n1 hasn't yet seen. This
# will allow us to test n1's sync-to-tip on top of a snapshot.
self.generate(n0, nblocks=100, sync_fun=no_sync)
assert_equal(n0.getblockcount(), FINAL_HEIGHT)
assert_equal(n1.getblockcount(), START_HEIGHT)
assert_equal(n0.getblockchaininfo()["blocks"], FINAL_HEIGHT)
self.log.info(f"Loading snapshot into second node from {dump_output['path']}")
loaded = n1.loadtxoutset(dump_output['path'])
assert_equal(loaded['coins_loaded'], SNAPSHOT_BASE_HEIGHT)
assert_equal(loaded['base_height'], SNAPSHOT_BASE_HEIGHT)
monitor = n1.getchainstates()
assert_equal(monitor['normal']['blocks'], START_HEIGHT)
assert_equal(monitor['snapshot']['blocks'], SNAPSHOT_BASE_HEIGHT)
assert_equal(monitor['snapshot']['snapshot_blockhash'], dump_output['base_hash'])
assert_equal(n1.getblockchaininfo()["blocks"], SNAPSHOT_BASE_HEIGHT)
PAUSE_HEIGHT = FINAL_HEIGHT - 40
self.log.info("Restarting node to stop at height %d", PAUSE_HEIGHT)
self.restart_node(1, extra_args=[
f"-stopatheight={PAUSE_HEIGHT}", *self.extra_args[1]])
# Finally connect the nodes and let them sync.
self.connect_nodes(0, 1)
n1.wait_until_stopped(timeout=5)
self.log.info("Checking that blocks are segmented on disk")
assert self.has_blockfile(n1, "00000"), "normal blockfile missing"
assert self.has_blockfile(n1, "00001"), "assumed blockfile missing"
assert not self.has_blockfile(n1, "00002"), "too many blockfiles"
self.log.info("Restarted node before snapshot validation completed, reloading...")
self.restart_node(1, extra_args=self.extra_args[1])
self.connect_nodes(0, 1)
self.log.info(f"Ensuring snapshot chain syncs to tip. ({FINAL_HEIGHT})")
wait_until_helper(lambda: n1.getchainstates()['snapshot']['blocks'] == FINAL_HEIGHT)
self.sync_blocks(nodes=(n0, n1))
self.log.info("Ensuring background validation completes")
# N.B.: the `snapshot` key disappears once the background validation is complete.
wait_until_helper(lambda: not n1.getchainstates().get('snapshot'))
# Ensure indexes have synced.
completed_idx_state = {
'basic block filter index': COMPLETE_IDX,
'coinstatsindex': COMPLETE_IDX,
}
self.wait_until(lambda: n1.getindexinfo() == completed_idx_state)
for i in (0, 1):
n = self.nodes[i]
self.log.info(f"Restarting node {i} to ensure (Check|Load)BlockIndex passes")
self.restart_node(i, extra_args=self.extra_args[i])
assert_equal(n.getblockchaininfo()["blocks"], FINAL_HEIGHT)
assert_equal(n.getchainstates()['normal']['blocks'], FINAL_HEIGHT)
assert_equal(n.getchainstates().get('snapshot'), None)
if i != 0:
# Ensure indexes have synced for the assumeutxo node
self.wait_until(lambda: n.getindexinfo() == completed_idx_state)
# Node 2: all indexes + reindex
# -----------------------------
self.log.info("-- Testing all indexes + reindex")
assert_equal(n2.getblockcount(), START_HEIGHT)
self.log.info(f"Loading snapshot into third node from {dump_output['path']}")
loaded = n2.loadtxoutset(dump_output['path'])
assert_equal(loaded['coins_loaded'], SNAPSHOT_BASE_HEIGHT)
assert_equal(loaded['base_height'], SNAPSHOT_BASE_HEIGHT)
monitor = n2.getchainstates()
assert_equal(monitor['normal']['blocks'], START_HEIGHT)
assert_equal(monitor['snapshot']['blocks'], SNAPSHOT_BASE_HEIGHT)
assert_equal(monitor['snapshot']['snapshot_blockhash'], dump_output['base_hash'])
self.connect_nodes(0, 2)
wait_until_helper(lambda: n2.getchainstates()['snapshot']['blocks'] == FINAL_HEIGHT)
self.sync_blocks()
self.log.info("Ensuring background validation completes")
wait_until_helper(lambda: not n2.getchainstates().get('snapshot'))
completed_idx_state = {
'basic block filter index': COMPLETE_IDX,
'coinstatsindex': COMPLETE_IDX,
'txindex': COMPLETE_IDX,
}
self.wait_until(lambda: n2.getindexinfo() == completed_idx_state)
for i in (0, 2):
n = self.nodes[i]
self.log.info(f"Restarting node {i} to ensure (Check|Load)BlockIndex passes")
self.restart_node(i, extra_args=self.extra_args[i])
assert_equal(n.getblockchaininfo()["blocks"], FINAL_HEIGHT)
assert_equal(n.getchainstates()['normal']['blocks'], FINAL_HEIGHT)
assert_equal(n.getchainstates().get('snapshot'), None)
if i != 0:
# Ensure indexes have synced for the assumeutxo node
self.wait_until(lambda: n.getindexinfo() == completed_idx_state)
self.log.info("Test -reindex-chainstate of an assumeutxo-synced node")
self.restart_node(2, extra_args=[
'-reindex-chainstate=1', *self.extra_args[2]])
assert_equal(n2.getblockchaininfo()["blocks"], FINAL_HEIGHT)
wait_until_helper(lambda: n2.getblockcount() == FINAL_HEIGHT)
self.log.info("Test -reindex of an assumeutxo-synced node")
self.restart_node(2, extra_args=['-reindex=1', *self.extra_args[2]])
self.connect_nodes(0, 2)
wait_until_helper(lambda: n2.getblockcount() == FINAL_HEIGHT)
if __name__ == '__main__':
AssumeutxoTest().main()

4
test/functional/test_framework/test_framework.py
View file

@ -979,3 +979,7 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass):
def is_bdb_compiled(self):
"""Checks whether the wallet module was compiled with BDB support."""
return self.config["components"].getboolean("USE_BDB")
def has_blockfile(self, node, filenum: str):
blocksdir = os.path.join(node.datadir, self.chain, 'blocks', '')
return os.path.isfile(os.path.join(blocksdir, f"blk{filenum}.dat"))

1
test/functional/test_runner.py
View file

@ -324,6 +324,7 @@ BASE_SCRIPTS = [
'wallet_coinbase_category.py --descriptors',
'feature_filelock.py',
'feature_loadblock.py',
'feature_assumeutxo.py',
'p2p_dos_header_tree.py',
'p2p_add_connections.py',
'feature_bind_port_discover.py',

8
test/lint/lint-shell.py
View file

@ -67,9 +67,13 @@ def main():
'*.sh',
]
files = get_files(files_cmd)
# remove everything that doesn't match this regex
reg = re.compile(r'src/[leveldb,secp256k1,minisketch]')
files[:] = [file for file in files if not reg.match(file)]
def should_exclude(fname: str) -> bool:
return bool(reg.match(fname)) or 'test_utxo_snapshots.sh' in fname
# remove everything that doesn't match this regex
files[:] = [file for file in files if not should_exclude(file)]
# build the `shellcheck` command
shellcheck_cmd = [