1ea7e45a1f test: raise explicit error if any of the needed release binaries is missing (Sebastian Falbesoner)
Pull request description:
If the `releases` directory exists, but still only a subset of the necessary previous release binaries are available, the test fails by throwing an exception (sometimes leading to follow-up exceptions like `AssertionError: [node 0] Error: no RPC connection`) and printing out a stack trace, which can be confusing and at a first glance suggests that the node crashed or some alike.
Improve this by checking and printing out *all* of the missing release binaries and failing with an explicit error in this case. Also add an info on how to download previous releases binaries. Noticed while testing #30328.
Can be tested by e.g.
```
$ rm -rf ./releases
$ ./test/get_previous_releases.py -b
$ rm -rf ./releases/v28.0/
$ ./build/test/functional/wallet_migration.py
```
master:
<details>
<summary>Long test fail output</summary>
```
...
2024-12-10T18:48:54.067000Z TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 590, in start_nodes
node.start(extra_args[i], *args, **kwargs)
File "/home/thestack/bitcoin/test/functional/test_framework/test_node.py", line 257, in start
self.process = subprocess.Popen(self.args + extra_args, env=subp_env, stdout=stdout, stderr=stderr, cwd=cwd, **kwargs)
File "/usr/lib/python3.10/subprocess.py", line 971, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.10/subprocess.py", line 1863, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/thestack/bitcoin/releases/v28.0/bin/bitcoind'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 131, in main
self.setup()
File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 315, in setup
self.setup_network()
File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 409, in setup_network
self.setup_nodes()
File "/home/thestack/bitcoin/./build/test/functional/wallet_migration.py", line 54, in setup_nodes
self.start_nodes()
File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 595, in start_nodes
self.stop_nodes()
File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 610, in stop_nodes
node.stop_node(wait=wait, wait_until_stopped=False)
File "/home/thestack/bitcoin/test/functional/test_framework/test_node.py", line 396, in stop_node
self.stop(wait=wait)
File "/home/thestack/bitcoin/test/functional/test_framework/test_node.py", line 215, in __getattr__
assert self.rpc_connected and self.rpc is not None, self._node_msg("Error: no RPC connection")
AssertionError: [node 0] Error: no RPC connection
2024-12-10T18:48:54.118000Z TestFramework (INFO): Stopping nodes
Traceback (most recent call last):
File "/home/thestack/bitcoin/./build/test/functional/wallet_migration.py", line 1097, in <module>
WalletMigrationTest(__file__).main()
File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 159, in main
exit_code = self.shutdown()
File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 331, in shutdown
self.stop_nodes()
File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 610, in stop_nodes
node.stop_node(wait=wait, wait_until_stopped=False)
File "/home/thestack/bitcoin/test/functional/test_framework/test_node.py", line 396, in stop_node
self.stop(wait=wait)
File "/home/thestack/bitcoin/test/functional/test_framework/test_node.py", line 215, in __getattr__
assert self.rpc_connected and self.rpc is not None, self._node_msg("Error: no RPC connection")
AssertionError: [node 0] Error: no RPC connection
[node 0] Cleaning up leftover process
...
```
</details>
PR:
```
...
2025-01-01T20:26:27.999000Z TestFramework (INFO): PRNG seed is: 4570383538468804512
2025-01-01T20:26:28.000000Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_lz66_zcu
2025-01-01T20:26:28.003000Z TestFramework (ERROR): Binary not found: /home/thestack/bitcoin/releases/v28.0/bin/bitcoind
2025-01-01T20:26:28.003000Z TestFramework (ERROR): Binary not found: /home/thestack/bitcoin/releases/v28.0/bin/bitcoin-cli
2025-01-01T20:26:28.003000Z TestFramework (INFO): Previous releases binaries can be downloaded via `test/get_previous_releases.py -b`.
2025-01-01T20:26:28.003000Z TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 131, in main
self.setup()
File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 315, in setup
self.setup_network()
File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 409, in setup_network
self.setup_nodes()
File "/home/thestack/bitcoin/./build/test/functional/wallet_migration.py", line 50, in setup_nodes
self.add_nodes(self.num_nodes, versions=[
File "/home/thestack/bitcoin/test/functional/test_framework/test_framework.py", line 537, in add_nodes
raise AssertionError("At least one release binary is missing")
AssertionError: At least one release binary is missing
2025-01-01T20:26:28.061000Z TestFramework (INFO): Stopping nodes
...
```
ACKs for top commit:
fjahr:
re-ACK 1ea7e45a1f
kevkevinpal:
ACK [1ea7e45](1ea7e45a1f)
maflcko:
lgtm ACK 1ea7e45a1f
achow101:
ACK 1ea7e45a1f
pablomartin4btc:
tACK 1ea7e45a1f
Tree-SHA512: b621c3ce044ca8fc8715a4f4b1f96a8592a470c319a64444cced9ba692d315cfd4885a066679bf377b19136fa3530d9cff6f18894a45aa9c716d39b12719baa0
On FreeBSD, the `shasum` utility is provided by the `perl5` port, which
is not part of the base system and must be installed separately.
Note that this requirement is currently not documented in
`depends/README.md`.
This change switches to using the `sha256sum` utility, which is included
in the base system.
When the behavior was changes in a previous commit (caching `GetSerializeSize` and avoiding `AutoFile.tell`), (static)asserts were added to make sure the behavior was kept - to make sure reviewers and CI validates it.
We can safely remove them now.
Logs were also slightly modernized since they were trivial to do.
Co-authored-by: Anthony Towns <aj@erisian.com.au>
Co-authored-by: Hodlinator <172445034+hodlinator@users.noreply.github.com>
For consistency `UNDO_DATA_DISK_OVERHEAD` was also extracted to avoid the constant's ambiguity.
Asserts were added to help with the review - they are removed in the next commit.
Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
This change resolves an issue where a missing space caused the value of
the `build_AR` variable to be concatenated with the "NM=" string. This
resulted in subsequent calls to `${AR}` and `${NM}` failing.
Example logs before the change:
```
2025-01-07T11:58:33Z Verification progress: 99%
2025-01-07T11:58:33Z Verification: No coin database inconsistencies in last 6 blocks (18905 transactions)
2025-01-07T11:58:33Z block index 31892ms
2025-01-07T11:58:33Z Setting NODE_NETWORK on non-prune mode
2025-01-07T11:58:33Z block tree size = 878086
2025-01-07T11:58:33Z nBestHeight = 878085
```
Removed redundant duration as well since it can be recovered from the timestamps.
Co-authored-by: TheCharlatan <seb.kung@gmail.com>
Co-authored-by: MarcoFalke <*~=`'#}+{/-|&$^_@721217.xyz>
Similarly, `WriteBlockToDisk` wasn't really extracting a meaningful subset of the `SaveBlockToDisk` functionality, it's tied closely to the only caller (needs the header size twice, recalculated block serializes size, returns multiple branches, mutates parameter).
The inlined code should only differ in these parts (modernization will be done in other commits):
* renamed `blockPos` to `pos` in `SaveBlockToDisk` to match the parameter name;
* changed `return false` to `return FlatFilePos()`.
Also removed remaining references to `SaveBlockToDisk`.
Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
`UndoWriteToDisk` wasn't really extracting a meaningful subset of the `WriteUndoDataForBlock` functionality, it's tied closely to the only caller (needs the header size twice, recalculated undo serializes size, returns multiple branches, modifies parameter, needs documentation).
The inlined code should only differ in these parts (modernization will be done in other commits):
* renamed `_pos` to `pos` in `WriteUndoDataForBlock` to match the parameter name;
* inlined `hashBlock` parameter usage into `hasher << block.pprev->GetBlockHash()`;
* changed `return false` to `return FatalError`;
* capitalize comment.
Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
Done in separate commit to simplify review.
Also renames benchmarks, since they're not strictly tests.
Co-authored-by: Hodlinator <172445034+hodlinator@users.noreply.github.com>
(total_cache / 4) + (1 << 23) is at least 8 MiB and nMaxCoinsDBCache is
also 8 MiB, so the minimum between the two will always be
nMaxCoinsDBCache. This is just a simplification and not changing the
result of the calculation.
Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
7c123c08dd miner: add package feerate vector to CBlockTemplate (ismaelsadeeq)
Pull request description:
This PR enables `BlockAssembler` to add all selected packages' fee and virtual size to a vector, and then return the vector as a member of `CBlockTemplate` struct.
This PR is the first step in the https://github.com/bitcoin/bitcoin/issues/30392 project.
The packages' vsize and fee are used in #30157 to select a percentile fee rate of the top block in the mempool.
ACKs for top commit:
rkrux:
tACK 7c123c08dd
ryanofsky:
Code review ACK 7c123c08dd. Changes since last review are rebasing due to a test conflict, giving the new field a better name and description, resolving the test conflict, and renaming a lot of test variables. The actual code change is still one-line change.
glozow:
reACK 7c123c08dd
Tree-SHA512: 767b0b3d4273cf1589fd2068d729a66c7414c0f9574b15989fbe293f8c85cd6c641dd783cde55bfabab32cd047d7d8a071d6897b06ed4295c0d071e588de0861
Trying to immediately shut down a node after a startup failure without
waiting for the RPC to be fully up will in most cases just fail and lead
to an RPC error.
Also, it is confusing to sidestep the existing fallback to kill any
leftover nodes on a test failure.
So just rely on the fallback.
Printing to stderr instead of stdout makes the test_runner.py fail on
leftover processes. This is desired and fine, because a leftover process
should only happen on a test failure anyway.
- The package feerates are ordered by the sequence in which
packages are selected for inclusion in the block template.
- The commit also tests this new behaviour.
Co-authored-by: willcl-ark <will@256k1.dev>
For the task `MSan, depends (Cirrus CI)` we build a custom libc++ for
which we already use `-DLIBCXX_HARDENING_MODE=debug`. Compile it also
with `_LIBCPP_ABI_BOUNDED_*` to enable further checks.
Docs at: https://libcxx.llvm.org/Hardening.html#abi-options
If the `releases` directory exists, but still only a subset of the
necessary previous release binaries are available, the test fails by
throwing an exception (sometimes leading to follow-up exceptions like
"AssertionError: [node 0] Error: no RPC connection") and printing out
a stack trace, which can be confusing and at a first glance suggests
that the node crashed or some alike.
Improve this by checking and printing out *all* of the missing release
binaries and failing with an explicit error in this case. Also add an
info on how to download previous releases binaries.
Noticed while testing #30328.
Can be tested by e.g.
$ ./test/get_previous_releases.py -b
$ rm -rf ./releases/v28.0/
$ ./build/test/functional/wallet_migration.py
3e0a992a3f doc: Clarify comments about endianness after #30526 (Ryan Ofsky)
Pull request description:
This is a documentation-only change following up on suggestions made in the #30526 review.
Motivation for this change is that I was recently reviewing #31583, which reminded me how confusing the arithmetic blob code was and made me want to write better comments.
ACKs for top commit:
achow101:
ACK 3e0a992a3f
TheCharlatan:
ACK 3e0a992a3f
Sjors:
ACK 3e0a992a3f
BrandonOdiwuor:
LGTM ACK 3e0a992a3f
Tree-SHA512: 90d5582a25a51fc406d83ca6b8c4e5e4d3aee828a0497f4b226b2024ff89e29b9b50d0ae8ddeac6abf2757fe78548d58cf3dd54df4b6d623f634a2106048091d
04249682e3 test: use Mining interface in miner_tests (Sjors Provoost)
Pull request description:
Needed for both #31283 and #31564.
By using the Mining interface in `miner_tests.cpp` we increase its coverage in unit tests.
ACKs for top commit:
achow101:
ACK 04249682e3
ryanofsky:
Code review ACK 04249682e3, just minor suggested changes (renames, comments, BOOST_REQUIREs) since last review and some more extra clarifications and checks added to the CreateNewBlock_validity test. The CreateNewBlock_validity changes seem clear and easy to understand now.
vasild:
ACK 04249682e3
tdb3:
ACK 04249682e3
Tree-SHA512: 2761cb7555d759670e40d8f37b96a079f8e12a588ac43313b9e63c69afd478321515873a8896ea56784f0100dac4476b0c0e0ef8b5418f8aea24d9965cace4d4
GetPrivKey() needs the same handling of all keyids for xonly keys that
ToPrivateString() does. Refactor that into GetPrivKey() and reuse it in
ToPrivateString() to resolve this.
fa397177ac util: Add missing types in make_secure_unique (MarcoFalke)
Pull request description:
The return type of `std::forward` depends on the template type, and can not be recovered from the args. Attempting to do so will result in a compile failure. For example, `make_secure_unique<std::string>(std::string{});` does not compile on current master, but does with this pull.
Another example would be `make_secure_unique<std::pair<std::string, std::unique_ptr<int>>>(std::string{}, std::make_unique<int>(21));`
ACKs for top commit:
hodlinator:
ACK fa397177ac
hebasto:
ACK fa397177ac.
TheCharlatan:
ACK fa397177ac
Tree-SHA512: cc902c1111c929a79a6f806b5097136a465e8c727474176bad30a5777ebbb30bedb0bd35273b43bf839d2c00492500ddec724bd17349250451f6b329cb71e6f2
Now that we track all announcers of an orphan, it's not helpful to
consider an orphan provided by a peer that didn't send us this parent.
It can only hurt our chances of finding the right orphan when there are
multiple candidates.
Adapt the 2 tests in p2p_opportunistic_1p1c.py that looked at 1p1c
packages from different peers. Instead of checking that the right peer
is punished, we now check that the package is not submitted. We can't
use the functional test to see that the package was not considered
because the behavior is indistinguishable (except for the logs).
This means we no longer return parents we already have in the
m_unique_parents result from MempoolRejectedTx.
We need to separate the loop that checks AlreadyHave parents from the
loop that adds parents as announcements, because we may do the latter
loop multiple times for different peers.
Add ability to add and track multiple announcers per orphan transaction,
erasing announcers but not the entire orphan.
The tx creation code in orphanage_tests needs to be updated so that each
tx is unique, because the CountOrphans() check assumes that calling
EraseForPeer necessarily means its orphans are deleted.
Unused for now.
