0
0
Fork 0
mirror of https://github.com/bitcoin/bitcoin.git synced 2025-02-24 12:41:41 -05:00
Commit graph

98 commits

Author SHA1 Message Date
Andrew Chow
bb136aaf2c
Merge bitcoin/bitcoin#26533: prune: scan and unlink already pruned block files on startup
3141eab9c6 test: add functional test for ScanAndUnlinkAlreadyPrunedFiles (Andrew Toth)
e252909e56 test: add unit test for ScanAndUnlinkAlreadyPrunedFiles (Andrew Toth)
77557dda4a prune: scan and unlink already pruned block files on startup (Andrew Toth)

Pull request description:

  There are a few cases where we can mark a block and undo file as pruned in our block index, but not actually remove the files from disk.
  1. If we call `FindFilesToPrune` or `FindFilesToPruneManual` and crash before `UnlinkPrunedFiles`.
  2. If on Windows there is an open file handle to the file somewhere else when calling `fs::remove` in `UnlinkPrunedFiles` (https://en.cppreference.com/w/cpp/filesystem/remove, https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-deletefilew#remarks). This could be from another process, or if we are calling `ReadBlockFromDisk`/`ReadRawBlockFromDisk` without having a lock on `cs_main` (which has been allowed since ccd8ef65f9).

  This PR mitigates this by scanning all pruned block files on startup after `LoadBlockIndexDB` and unlinking them again.

ACKs for top commit:
  achow101:
    ACK 3141eab9c6
  pablomartin4btc:
    re-ACK with added functional test 3141eab9c6.
  furszy:
    Code review ACK 3141eab9
  theStack:
    Code-review ACK 3141eab9c6

Tree-SHA512: 6c73bc57838ad1b7e5d441af3c4d6bf4c61c4382e2b86485e57fbb74a61240710c0ceeceb8b4834e610ecfa3175c6955c81ea4b2285fee11ca6383f472979d8d
2023-02-28 09:54:10 -05:00
MarcoFalke
faf7b4f1fc
Add BlockManager::IsPruneMode() 2023-01-16 17:31:32 +01:00
MarcoFalke
fae71fe27e
Add BlockManager::GetPruneTarget() 2023-01-16 17:16:30 +01:00
MarcoFalke
fa0f0436d8
Add BlockManager::LoadingBlocks() 2023-01-16 16:38:11 +01:00
fanquake
282019cd3d
refactor: add kernel/cs_main.*
Co-authored-by: Anthony Towns <aj@erisian.com.au>
2023-01-05 09:05:14 +00:00
Hennadii Stepanov
306ccd4927
scripted-diff: Bump copyright headers
-BEGIN VERIFY SCRIPT-
./contrib/devtools/copyright_header.py update ./
-END VERIFY SCRIPT-

Commits of previous years:
- 2021: f47dda2c58
- 2020: fa0074e2d8
- 2019: aaaaad6ac9
2022-12-24 23:49:50 +00:00
Andrew Toth
77557dda4a prune: scan and unlink already pruned block files on startup 2022-12-20 12:25:36 -05:00
glozow
cc12b8947b
Merge bitcoin/bitcoin#24858: incorrect blk file size calculation during reindex results in recoverable blk file corruption
bcb0cacac2 reindex, log, test: fixes #21379 (mruddy)

Pull request description:

  Fixes #21379.

  The blocks/blk?????.dat files are mutated and become increasingly malformed, or corrupt, as a result of running the re-indexing process.
  The mutations occur after the re-indexing process has finished, as new blocks are appended, but are a result of a re-indexing process miscalculation that lingers in the block manager's `m_blockfile_info` `nSize` data until node restart.
  These additions to the blk files are non-fatal, but also not desirable.
  That is, this is a form of data corruption that the reading code is lenient enough to process (it skips the extra bytes), but it adds some scary looking log messages as it encounters them.

  The summary of the problem is that the re-index process double counts the size of the serialization header (magic message start bytes [4 bytes] + length [4 bytes] = 8 bytes) while calculating the blk data file size (both values already account for the serialization header's size, hence why it is over accounted).

  This bug manifests itself in a few different ways, after re-indexing, when a new block from a peer is processed:
  1. If the new block will not fit into the last blk file processed while re-indexing, while remaining under the 128MiB limit, then the blk file is flushed to disk and truncated to a size that is 8 greater than it should be. The truncation adds zero bytes (see `FlatFileSeq::Flush` and `TruncateFile`).
  1. If the last blk file processed while re-indexing has logical space for the new block under the 128 MiB limit:
      1. If the blk file was not already large enough to hold the new block, then the zeros are, in effect, added by `fseek` when the file is opened for writing. Eight zero bytes are added to the end of the last blk file just before the new block is written. This happens because the write offset is 8 too great due to the miscalculation. The result is 8 zero bytes between the end of the last block and the beginning of the next block's magic + length + block.
      1. If the blk file was already large enough to hold the new block, then the current existing file contents remain in the 8 byte gap between the end of the last block and the beginning of the next block's magic + length + block. Commonly, when this occcurs, it is due to the blk file containing blocks that are not connected to the block tree during reindex and are thus left behind by the reindex process and later overwritten when new blocks are added. The orphaned blocks can be valid blocks, but due to the nature of concurrent block download, the parent may not have been retrieved and written by the time the node was previously shutdown.

ACKs for top commit:
  LarryRuane:
    tested code-review ACK bcb0cacac2
  ryanofsky:
    Code review ACK bcb0cacac2. This is a disturbing bug with an easy fix which seems well-worth merging.
  mzumsande:
    ACK bcb0cacac2 (reviewed code and did some testing, I agree that it fixes the bug).
  w0xlt:
    tACK bcb0cacac2

Tree-SHA512: acc97927ea712916506772550451136b0f1e5404e92df24cc05e405bb09eb6fe7c3011af3dd34a7723c3db17fda657ae85fa314387e43833791e9169c0febe51
2022-10-12 14:13:54 -04:00
James O'Beirne
00eeb31c76 scripted-diff: rename CChainState -> Chainstate
-BEGIN VERIFY SCRIPT-
sed -i 's/CChainState/Chainstate/g' $(git grep -l CChainState ':(exclude)doc/release-notes*')
-END VERIFY SCRIPT-

Co-authored-by: MacroFake <falke.marco@gmail.com>
2022-09-09 11:47:27 -04:00
Carl Dong
06b88ffb8a LoadMempool: Pass in load_path, stop using gArgs
Also:
1. Have CChainState::LoadMempool and ::ThreadImport take in paths and
   pass it through untouched to LoadMempool.
2. Make LoadMempool exit early if the load_path is empty.
3. Adjust the call to ::ThreadImport in ::AppInitMain to correctly pass
   in an empty path if mempool persistence is disabled.
2022-07-15 12:26:20 -04:00
darosior
64f81a38b9
doc: Correct nPruneTarget misidentifying units of variable 2022-06-07 15:30:16 -05:00
Cory Fields
a4741bd8d4 kernel: pass params to BlockManager rather than using a global 2022-06-02 15:18:09 +00:00
mruddy
bcb0cacac2 reindex, log, test: fixes #21379
This fixes a blk file size calculation made during reindex that results in increased blk file malformity.
The fix is to avoid double counting the size of the serialization header during reindex.
This adds a unit test to reproduce the bug before the fix and to ensure that it does not recur.
These changes include a log message change also so as to not be as alarming. This is a common and recoverable
data corruption. These messages can now be filtered by the debug log reindex category.
2022-05-07 07:11:29 -04:00
Jon Atack
4cb9d21434 blockstorage: add LIFETIMEBOUND to GetFirstStoredBlock()::start_time
See PR 22278 for discussion.

Co-authored-by: MarcoFalke <falke.marco@gmail.com>
2022-05-03 22:20:31 +02:00
Jon Atack
86ce844d3b blockstorage, refactor: pass GetFirstStoredBlock() start_block by reference
instead of by pointer, so as to not accept a nullptr.
2022-04-28 20:42:08 +02:00
Jon Atack
ed12c0a49d blockstorage, refactor: make GetFirstStoredBlock() a member of BlockManager
instead of a global
2022-04-28 20:42:08 +02:00
Carl Dong
7ab07e0332 validation: Prune UnloadBlockIndex and callees
In previous commits in this patchset, we've made sure that every
Unload/UnloadBlockIndex member function resets its own members, and does
not reach out to globals.

This means that their corresponding classes' default destructors can now
replace them, and do an even more thorough job without the need to be
updated for every new member variable.

Therefore, we can remove them, and also remove UnloadBlockIndex since
that's not used anymore.

Unfortunately, chainstatemanager_loadblockindex relies on
CChainState::UnloadBlockIndex, so that needs to stay for now.
2022-04-27 11:13:38 -04:00
fanquake
bd616bc16a
Merge bitcoin/bitcoin#24917: Make BlockManager::LoadBlockIndex private
fa1970f075 Make BlockManager::LoadBlockIndex private (MarcoFalke)

Pull request description:

  * After commit fa27f03b49 `BlockManager::LoadBlockIndex` is only called by `BlockManager::LoadBlockIndexDB`. Thus, it can be made `private`.

  * After commit c600ee3816 `m_best_invalid` is no longer accessed by `BlockManager::LoadBlockIndex`. Thus, the unused `friend` can be removed.

ACKs for top commit:
  mruddy:
    ACK fa1970f075 I verified by double checking references, then applying the patch, and running `make check`. LGTM.

Tree-SHA512: 9b36b4c59bf7ad01171764ce61b1be9750fc92d105c4fe939b1a6a70027ab6300d5d2a2fc3e82f981e22c3987f2ca84e092d2e1f8463fa320af9f05048580c0a
2022-04-26 20:20:07 +01:00
Fabian Jahr
2561823531
blockstorage: Add prune locks to BlockManager
This change also introduces an aditional buffer of 10 blocks (PRUNE_LOCK_BUFFER) that will not be pruned before the best block.

Co-authored-by: Luke Dashjr <luke-jr+git@utopios.org>
2022-04-25 23:21:58 +02:00
Fabian Jahr
231fc7b035
refactor: Introduce GetFirstStoredBlock helper function 2022-04-25 23:18:01 +02:00
Carl Dong
f0a2fb3c5d scripted-diff: Rename pindexBestHeader, fHavePruned
...to m_best_header and m_have_pruned

-BEGIN VERIFY SCRIPT-
find_regex="\bpindexBestHeader\b" \
    && git grep -l -E "$find_regex" -- src \
        | xargs sed -i -E "s@$find_regex@m_best_header@g"
find_regex="\bfHavePruned\b" \
    && git grep -l -E "$find_regex" -- src \
        | xargs sed -i -E "s@$find_regex@m_have_pruned@g"
-END VERIFY SCRIPT-
2022-04-19 14:36:18 -04:00
Carl Dong
3308ecd3fc move-mostly: Make fHavePruned a BlockMan member
[META] In the next commit, we move the clearing of fHavePruned to
       BlockManager::Unload()
2022-04-19 14:34:56 -04:00
Carl Dong
0d567daf23 move-mostly: Make pindexBestHeader a ChainMan member
[META] In the next commit, we move the clearing of pindexBestHeader to
       ChainstateManager::Unload()
2022-04-19 14:34:55 -04:00
MarcoFalke
fa1970f075
Make BlockManager::LoadBlockIndex private 2022-04-19 11:32:49 +02:00
Carl Dong
f865cf8ded Add and use BlockManager::GetAllBlockIndices 2022-03-15 19:42:43 -04:00
Carl Dong
28ba0313ea Add and use CBlockIndexHeightOnlyComparator
...also use std::sort for clarity
2022-03-15 19:42:43 -04:00
Carl Dong
c600ee3816 Only load BlockMan in BlockMan member functions
This commit effectively splits the "load block index itself" logic from
"derive Chainstate variables from loaded block index" logic.

This means that BlockManager::LoadBlockIndex{,DB} will only load what's
relevant to the BlockManager.

I strongly recommend reviewing with the following git-diff flags:
  --color-moved=dimmed_zebra --color-moved-ws=allow-indentation-change
2022-03-15 19:42:41 -04:00
Anthony Towns
5be9ee3c54 refactor: more const annotations for uses of CBlockIndex* 2022-03-09 14:32:47 -05:00
Carl Dong
c2a1655799 style-only: Use using instead of typedef for BlockMap 2022-02-22 11:56:49 -05:00
Carl Dong
bec86ae326 blockstorage: Make m_block_index own CBlockIndex's
Instead of having CBlockIndex's live on the heap, which requires manual
memory management, have them be owned by m_block_index. This means that
they will live and die with BlockManager.

A change to BlockManager::LookupBlockIndex:
- Previously, it was a const member function returning a non-const CBlockIndex*
- Now, there's are const and non-const versions of
  BlockManager::LookupBlockIndex returning a CBlockIndex with the same
  const-ness as the member function:
    (e.g. const CBlockIndex* LookupBlockIndex(...) const)

See next commit for some weirdness that this eliminates.

The range based for-loops are modernize (using auto + destructuring) in
a future commit.
2022-02-22 11:52:19 -05:00
Hennadii Stepanov
5d59ae0ba8
Remove/inline ReadRawBlockFromDisk(block_data, pindex, message_start) 2022-01-25 20:43:37 +01:00
Jon Atack
eaeeb88768
Require IsBlockPruned() to hold mutex cs_main
Co-authored-by: Vasil Dimov <vd@FreeBSD.org>
2022-01-25 20:43:34 +01:00
Jon Atack
2e557ced28
Require WriteUndoDataForBlock() to hold mutex cs_main
Mutex cs_main is already held by the caller of WriteUndoDataForBlock().
This change is needed to require CBlockIndex::GetUndoPos() to hold
cs_main and CBlockIndex::nStatus to be guarded by cs_main in the
following commits without adding 2 unnecessary cs_main locks to
WriteUndoDataForBlock().
2022-01-25 20:43:19 +01:00
Russell Yanofsky
90fc8b089d Add src/node/* code to node:: namespace 2022-01-06 22:14:16 -05:00
MarcoFalke
fa68a6c2fc
scripted-diff: Rename touched member variables
-BEGIN VERIFY SCRIPT-

 ren() { sed -i "s/\<$1\>/$2/g" $( git grep -l "$1" ./src/ ) ; }

 ren vinfoBlockFile     m_blockfile_info
 ren nLastBlockFile     m_last_blockfile
 ren fCheckForPruning   m_check_for_pruning
 ren setDirtyBlockIndex m_dirty_blockindex
 ren setDirtyFileInfo   m_dirty_fileinfo

-END VERIFY SCRIPT-
2022-01-05 16:19:11 +01:00
MarcoFalke
facd3df21f
Make blockstorage globals private members of BlockManager 2022-01-05 16:18:50 +01:00
MarcoFalke
faa8c2d7d7
doc: Clarify nPruneAfterHeight for signet 2022-01-05 16:17:22 +01:00
MarcoFalke
fa467f3913
move-only: Create WriteBlockIndexDB helper
Can be reviewed with --color-moved=dimmed-zebra --color-moved-ws=ignore-all-space
2022-01-05 15:08:06 +01:00
MarcoFalke
fa88cfd3f9
Move functions to BlockManager
Needed for a later commit
2022-01-05 15:07:28 +01:00
MarcoFalke
fa7efc915b
Fixup style of moved code
Can be reviewed with --word-diff-regex=. -U0 --ignore-all-space
2022-01-02 17:05:22 +01:00
MarcoFalke
fade2a44f4
Move BlockManager to node/blockstorage
Can be reviewed with --color-moved=dimmed-zebra
2022-01-02 17:05:14 +01:00
MarcoFalke
fa2d21fec8
add missing atomic include 2021-06-22 11:06:30 +02:00
MarcoFalke
fa09a9eac8
style: Add { } to multi-line if
Can be reviewed with --word-diff-regex=. --ignore-all-space
2021-04-27 10:36:41 +02:00
MarcoFalke
fadafab833
move-only: Move functions to blockstorage 2021-04-27 10:36:23 +02:00
MarcoFalke
fa7e64d586
move-only: Move constants to blockstorage 2021-04-27 10:32:54 +02:00
MarcoFalke
fa81c30c6f
refactor: Move pruning/reindex/importing globals to blockstorage
Can be reviewed with --color-moved=dimmed-zebra
2021-04-27 10:32:24 +02:00
MarcoFalke
fa0c7d9ad2
move-only: Move *Disk functions to blockstorage
Can be reviewed with the git options
--color-moved=dimmed-zebra --color-moved-ws=ignore-all-space
2021-04-05 20:26:14 +02:00
MarcoFalke
fa413f07a1
move-only: Move ThreadImport to blockstorage
Can be reviewed with the git options
--color-moved=dimmed-zebra --color-moved-ws=ignore-all-space
2021-04-04 18:07:24 +02:00