This commit removes the single-parameter contructor of CMessageHeader
and replaces it with a default constructor.
The single parameter contructor isn't used anywhere except for tests.
There is no reason to initialize a CMessageHeader with a particular
messagestart. This messagestart should always be replaced when
deserializing an actual message header so that we can run checks on it.
The default constructor initializes it to zero, just like the command
and checksum.
This also removes a parameter of a V1TransportDeserializer constructor,
as it was only used for this purpose.
This removes the m_valid_checksum member from CNetMessage. Instead,
GetMessage() returns an Optional.
Additionally, GetMessage() has been given an out parameter to be used to
hold error information. For now it is specifically a uint32_t used to
hold the raw size of the corrupt message.
The checksum check is now done in GetMessage.
This is intended to only be used for logging.
This will allow log messages in the following commits to keep recording
the peer's ID, even when logging is moved into V1TransportDeserializer.
m_valid implies the block solution has been checked, which is not the
case. It only means the txs could be parsed. C++17 comes with
std::optional, so just use that instead.
Using the zmq notifications to avoid excessive mempool polling can be difficult
given the current notifications available. It announces all transactions
being added to mempool or included in blocks, but announces no evictions
and gives no indication if the transaction is in the mempool or a block.
Block notifications for zmq are also substandard, in that it only announces
block tips, while all block transactions are still announced.
This commit adds a unified stream which can be used to closely track mempool:
1) getrawmempool to fill out mempool knowledge
2) if txhash is announced, add or remove from set
based on add/remove flag
3) if blockhash is announced, get block txn list,
remove from those transactions local view of mempool
4) if we drop a sequence number, go to (1)
The mempool sequence number starts at the value 1, and
increments each time a transaction enters the mempool,
or is evicted from the mempool for any reason, including
block inclusion. The mempool sequence number is published
via ZMQ for any transaction-related notification.
These features allow for ZMQ/RPC consumer to track mempool
state in a more exacting way, without unnecesarily polling
getrawmempool. See interface_zmq.py::test_mempool_sync for
example usage.
fa6bb0ce5d Assert that RPCArg names are equal to CRPCCommand ones (rawtransaction) (MarcoFalke)
fa80c81487 Assert that RPCArg names are equal to CRPCCommand ones (blockchain) (MarcoFalke)
Pull request description:
This is split out from #18531 to just touch some RPC methods. Description from the main pr:
### Motivation
RPCArg names in the rpc help are currently only used for documentation. However, in the future they could be used to teach the server the named arguments. Named arguments are currently registered by the `CRPCCommand`s and duplicate the RPCArg names from the documentation. This redundancy is fragile, and has lead to errors in the past (despite having linters to catch those kind of errors). See section "bugs found" for a list of bugs that have been found as a result of the changes here.
### Changes
The changes here add an assert in the `CRPCCommand` constructor that the RPCArg names are identical to the ones in the `CRPCCommand`.
### Future work
> Here or follow up, makes sense to also assert type of returned UniValue?
Sure, but let's not get ahead of ourselves. I am going to submit any further works as follow-ups, including:
* Removing the CRPCCommand arguments, now that they are asserted to be equal and thus redundant
* Removing all python regex linters on the args, now that RPCMan can be used to generate any output, including the cli.cpp table
* Auto-formatting and sanity checking the RPCExamples with RPCMan
* Checking passed-in json in self-check. Removing redundant checks
* Checking returned json against documentation to avoid regressions or false documentation
* Compile the RPC documentation at compile-time to ensure it doesn't change at runtime and is completely static
### Bugs found
* The assert identified issue #18607
* The changes itself fixed bug #19250
ACKs for top commit:
fjahr:
utACK fa6bb0ce5d
tryphe:
utACK fa6bb0ce5d. Reducing data duplication is nice. Code changes are minimal and concise.
Tree-SHA512: deb0edc3f999baf055526eaa199b98c500635e12502dece7aa3cad5319db330eb5ee7459a5c8f040a83671a7f20c560c19a2026fb76c8416f138aa332727cbce
In addition to adding more specificity to the log statement about the type of
connection, this change also consolidates two statements into one. Previously,
the second one should have never been hit, since block-relay connections would
match the "!IsInboundConn()" condition and return early.
ddefb5c0b7 p2p: Use the greatest common version in peer logic (Hennadii Stepanov)
e084d45562 p2p: Remove SetCommonVersion() from VERACK handler (Hennadii Stepanov)
8d2026796a refactor: Rename local variable nSendVersion (Hennadii Stepanov)
e9a6d8b13b p2p: Unify Send and Receive protocol versions (Hennadii Stepanov)
Pull request description:
On master (6fef85bfa3) `CNode` has two members to keep protocol version:
- `nRecvVersion` for received messages
- `nSendVersion` for messages to send
After exchanging with `VERSION` and `VERACK` messages via protocol version `INIT_PROTO_VERSION`, both nodes set `nRecvVersion` _and_ `nSendVersion` to _the same_ value which is the greatest common protocol version.
This PR:
- replaces two `CNode` members, `nRecvVersion` `nSendVersion`, with `m_greatest_common_version`
- removes duplicated getter and setter
There is no change in behavior on the P2P network.
ACKs for top commit:
jnewbery:
ACK ddefb5c0b7
naumenkogs:
ACK ddefb5c0b7
fjahr:
Code review ACK ddefb5c0b7
amitiuttarwar:
code review but untested ACK ddefb5c0b7
benthecarman:
utACK `ddefb5c`
Tree-SHA512: 5305538dbaa5426b923b0afd20bdef4f248d310855d1d78427210c00716c67b7cb691515c421716b6157913e453076e293b10ff5fd2cd26a8e5375d42da7809d
0d04784af1 Refactor the functional test (Gleb Naumenko)
83ad65f31b Address nits in ADDR caching (Gleb Naumenko)
81b00f8780 Add indexing ADDR cache by local socket addr (Gleb Naumenko)
42ec558542 Justify the choice of ADDR cache lifetime (Gleb Naumenko)
Pull request description:
This is a follow-up on #18991 which does 3 things:
- improves privacy of a node listening to multiple addresses via adding cache index by local socket addr (suggested [here](https://github.com/bitcoin/bitcoin/pull/18991#issuecomment-668219345))
- documents on the choice of 24h cache lifetime
- addresses nits from #18991
ACKs for top commit:
jnewbery:
utACK 0d04784af1
vasild:
ACK 0d04784
jonatack:
Code review ACK 0d04784
Tree-SHA512: bb65a34dd1ce2811186d3e4469bc33e8399cebaaa494ce13041c7cff23275870e4176a719f7a72f8d779c49f8b2344bf4fa1aeb3ea4e2626d5ae76514f00a750
[META] This is a followup to "validation: Move FindFilesToPrune{,Manual}
to BlockManager" removing comments and assertions meant only to
show that the change is correct.
[META] No behaviour change is intended in this commit.
[META] This commit should be followed up by removing the comments and
assertions meant only to show that the change is correct.
Also stop FindFilesToPrune{,Manual} from unnecessary reaching for
::ChainActive() by passing in the necessary information.
Recognizing addresses from those networks allows us to accept and gossip
them, even though we don't know how to connect to them (yet).
Co-authored-by: eriknylund <erik@daychanged.com>
6fe2ef2acb scripted-diff: Rename SendMessage to SendZmqMessage. (Daniel Kraft)
a3ffb6ebeb Replace zmqconfig.h by a simple zmqutil. (Daniel Kraft)
7f2ad1b9ac Use std::unique_ptr for CZMQNotifierFactory. (Daniel Kraft)
b93b9d5456 Simplify and fix notifier removal on error. (Daniel Kraft)
e15b1cfc31 Various cleanups in zmqnotificationinterface. (Daniel Kraft)
Pull request description:
This contains various small code cleanups that make the ZMQ code easier to read and maintain (at least in my opinion). The only functional change is that a potential memory leak is fixed that would have occured when a notifier is removed from the `notifiers` list after its callback function returned `false` (which is likely not relevant in practice but still a bug).
ACKs for top commit:
instagibbs:
utACK 6fe2ef2acb
hebasto:
re-ACK 6fe2ef2acb, only the latest commit got a scripted-diff since my [previous](https://github.com/bitcoin/bitcoin/pull/13686#pullrequestreview-487649808) review.
Tree-SHA512: 8206f8713bf3698d7cd4cb235f6657dc1c4dd920f50a8c5f371a559dd17ce5ab6d94d6281165eef860a22fc844a6bb25489ada12c83ebc780efd7ccdc0860f70
23c35bf005 [test] add get_vsize util for more programmatic testing (gzhao408)
2233a93a10 [rpc] Return fee and vsize from testmempoolaccept (codeShark149)
Pull request description:
From #19093 and resolves #19057.
Difference from #19093: return `vsize` and `fees` object (similar to `getmempoolentry`) when the test accept is successful. Updates release-notes.md.
ACKs for top commit:
jnewbery:
utACK 23c35bf005
fjahr:
utACK 23c35bf
instagibbs:
reACK 23c35bf005
Tree-SHA512: dcb81b7b817a4684e9076bc5d427a6f2d549d2edc66544e718260c4b5f8f1d5ae1d47b754175e9f0c8a3bd8371ce116c2dca0583588d513a7d733d5d614f2b04
d26f0648f1 Tell users how to load or create a wallet when no wallet is loaded (Andrew Chow)
1bee1e6269 Do not create default wallet (Andrew Chow)
Pull request description:
Instead of automatically creating and loading a default wallet, users should instead explicitly create their wallet or load it on start.
Builds on #19754 which provides the `load_on_startup` behavior for the GUI.
ACKs for top commit:
jnewbery:
Manual test and very light code review ACK d26f0648f1
ryanofsky:
Code review ACK d26f0648f1. Just suggested changes to first commit (reusing MakeWalletDatabase and adding release notes), no changes to second commit
jonatack:
ACK d26f0648f1 light code review, debug build, ran tests, did manual testing with testnet, rebased on master, on linux debian.
Tree-SHA512: 091d785aef64736f7df661c576e815a87f3d029cfa32f3a75ba86fc25795f10b022ab3ae15c5b61a10b8cee16f5650f15cd79cbd6127e5e3ccbef631966d3c30
fc7f84a9ca tests: Add fuzzing harness for Keccak and SHA3_256 (practicalswift)
Pull request description:
Add fuzzing harness for Keccak and SHA3_256.
See [`doc/fuzzing.md`](https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md) for information on how to fuzz Bitcoin Core. Don't forget to contribute any coverage increasing inputs you find to the [Bitcoin Core fuzzing corpus repo](https://github.com/bitcoin-core/qa-assets).
Happy fuzzing :)
ACKs for top commit:
laanwj:
uACK fc7f84a9ca
elichai:
utACK :) fc7f84a9ca
Tree-SHA512: 01e1610e1c178d5f42578e2dd5644a4165596db34cf5037d574a5285e0ace4b06dc33ab81a308595246117537fe175294efd4bfc174ffc2e8eac98f0ec9dd3e9
a8a64acaf3 [BroadcastTransaction] Remove unsafe move operator (Amiti Uttarwar)
125c038126 [p2p] Remove dead code (Amiti Uttarwar)
fc66d0a65c [p2p] Check for nullptr before dereferencing pointer (Adam Jonas)
cb79b9dbf4 [mempool] Revert unbroadcast set to tracking just txid (Amiti Uttarwar)
Pull request description:
Addresses some outstanding review comments from #18044
- reverts unbroadcast txids to a set instead of a map (simpler, communicates intent better, takes less space, no efficiency advantages of map)
- adds safety around two touchpoints (check for nullptr before dereferencing pointer, remove an inaccurate std::move operator)
- removes some dead code
Links to comments on wtxid PR: [1](https://github.com/bitcoin/bitcoin/pull/18044#discussion_r460495254) [2](https://github.com/bitcoin/bitcoin/pull/18044#discussion_r460496023) [3](https://github.com/bitcoin/bitcoin/pull/18044#discussion_r463532611)
thanks to jnewbery & adamjonas for flagging these ! !
ACKs for top commit:
sdaftuar:
utACK a8a64acaf3
naumenkogs:
utACK a8a64acaf3
jnewbery:
utACK a8a64acaf3
Tree-SHA512: 7be669cb30cc17fb9e06b50e636ef7887c6a27354697987e4e4d38dba4b8f50e175647587430cd9bc3295bec01ce8b1e6639a50a4249d8fff9b1ca1b9ead3277
[META] This is a pure refactor commit.
Move PruneBlockFile to BlockManager because:
1. PruneOneBlockFile only acts on BlockManager
2. Eliminates the need for callers (FindFilesToPrune{,Manual}) to have a
reference to the larger ChainstateManager, just a reference to
BlockManager is enough. See following commits.
916d3596c4 help: Generate checkpoint height from chainparams (Luke Dashjr)
Pull request description:
Not sure if this is worth putting in Core, but might as well until checkpoints are removed entirely.
ACKs for top commit:
laanwj:
re-ACK 916d3596c4
Tree-SHA512: d8eb26b570ee730fdd75ca916507134db5f2f68987a911e33544b7f1c9ccfd1c76b9c9db63056971956b6daf16910f17ecfc197481c2f7b0773afdfbf7d381cf
fc9278d162 build: AX_PTHREAD serial 27 (fanquake)
15c27c4441 build: split PTHREAD_* flags out of AM_LDFLAGS (fanquake)
68e3e22944 scripted-diff: add FUZZ_SUITE_LDFLAGS_COMMON (fanquake)
afecde8046 build: add PTHREAD_LIBS to LDFLAGS configure output (fanquake)
Pull request description:
TLDR: Split pthread flags out of ldflags, and stop using them when building libconsensus.
Building libconsensus on Linux using Clang currently warns. i.e:
```bash
./autogen.sh
./configure --disable-tests --disable-bench --with-utils=no --with-daemon=no --with-gui=no --disable-wallet --with-libs=yes CC=clang CXX=clang++
make V=1 -j6
... -Wl,-z -Wl,relro -Wl,-z -Wl,now -pthread -Wl,-soname -Wl,libbitcoinconsensus.so.0 -o .libs/libbitcoinconsensus.so.0.0.0
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
```
Besides wanting to quiet the warnings, after digging into this it seemed we could clean up how we are passing around the pthread flags. I also learnt a bit more about how libtools builds shared libraries, and that passing `-pthread` on the link line wouldn't be enough to link against pthreads anyways, due to libtools usage of -nostdlib (see [related discussion where we build DLLs](476436b2de/configure.ac (L603))).
This can be demonstrated with a patch to libconsensus:
```patch
diff --git a/src/script/bitcoinconsensus.cpp b/src/script/bitcoinconsensus.cpp
index 15e204062..10bf3582f 100644
--- a/src/script/bitcoinconsensus.cpp
+++ b/src/script/bitcoinconsensus.cpp
@@ -10,6 +10,8 @@
#include <script/interpreter.h>
#include <version.h>
+#include <pthread.h>
+
namespace {
/** A class that deserializes a single CTransaction one time. */
@@ -127,3 +129,10 @@ unsigned int bitcoinconsensus_version()
// Just use the API version for now
return BITCOINCONSENSUS_API_VER;
}
+
+void *func_pthread(void *x) { return x; }
+
+void f() {
+ pthread_t t;
+ pthread_create(&t,0,func_pthread,0);
+}
```
After building, you'll find you have a `libbitcoinconsensus.so` using pthread symbols, but which isn't linked against libpthread:
```bash
ldd -r src/.libs/libbitcoinconsensus.so
linux-vdso.so.1 (0x00007ffe49378000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f553cee7000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f553cda2000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f553cd88000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f553cbc5000)
/lib64/ld-linux-x86-64.so.2 (0x00007f553d15d000)
undefined symbol: pthread_create (src/.libs/libbitcoinconsensus.so)
```
This libtool behaviour has been known about for some time, i.e this [thread from 2005](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=25460), describes the same issue. The suggestion from libtool maintainers at the time is to add `-lpthread` to LDFLAGS.
Also worth noting is that some of the users in those threads were also using the `AX_PTHREADS` macro, same as us, to determine how to compile with/link against pthreads. This macro has [recently been updated](https://git.savannah.gnu.org/gitweb/?p=autoconf-archive.git;a=commitdiff;h=2fb904589643eb6ca6122f834891b58d1d51b347), with reference to this issue. You can compare the output from the version we currently use, to the new version:
```bash
# our ax_pthread macro:
PTHREAD_CFLAGS = -pthread
PTHREAD_LIBS =
PTHREAD_CC = gcc / clang
# the new ax_pthread macro
PTHREAD_CFLAGS = -pthread
PTHREAD_LIBS = -lpthread
PTHREAD_CC = gcc / clang
```
Note that as part of this PR I've also added `PTHREAD_LIBS` to the split out flags. Although we weren't using it anywhere previously (and wouldn't have seemed to matter for the most part, given it was likely empty for most builders), the macro assumes it's use. i.e:
> NOTE: You are assumed to not only compile your program with these flags,
> but also to link with them as well. For example, you might link with
> $PTHREAD_CC $CFLAGS $PTHREAD_CFLAGS $LDFLAGS ... $PTHREAD_LIBS $LIBS
ACKs for top commit:
laanwj:
Code review ACK fc9278d162
hebasto:
re-ACK fc9278d162, only rebased and renamed s/`AM_PTHREAD_FLAGS`/`PTHREAD_FLAGS`/ since my [previous](https://github.com/bitcoin/bitcoin/pull/19558#pullrequestreview-473487730) review..
kallewoof:
ACK fc9278d162
Tree-SHA512: 7c0a5b0f0de4f54b1d7dce0e69020b341c37a383bb7c715867cc96c648774a557b1ddb42eb1b676f7bb2b822b69795bec14dc6272362d80662a21f10cb80331c
Windows headers define SendMessage as a macro, which leads to problems
with the method name "SendMessage". To circumvent this, we rename the
method to "SendZmqMessage".
-BEGIN VERIFY SCRIPT-
sed -i 's/SendMessage/SendZmqMessage/g' src/zmq/zmqpublishnotifier.*
-END VERIFY SCRIPT-
