Andrew Chow a97a89244e Merge bitcoin/bitcoin#28368: Fee Estimator updates from Validation Interface/CScheduler thread ... 91504cbe0d rpc: `SyncWithValidationInterfaceQueue` on fee estimation RPC's (ismaelsadeeq) 714523918b tx fees, policy: CBlockPolicyEstimator update from `CValidationInterface` notifications (ismaelsadeeq) dff5ad3b99 CValidationInterface: modify the parameter of `TransactionAddedToMempool` (ismaelsadeeq) 91532bd382 tx fees, policy: update `CBlockPolicyEstimator::processBlock` parameter (ismaelsadeeq) bfcd401368 CValidationInterface, mempool: add new callback to `CValidationInterface` (ismaelsadeeq) 0889e07987 tx fees, policy: cast with static_cast instead of C-Style cast (ismaelsadeeq) a0e3eb7549 tx fees, policy: bugfix: move `removeTx` into reason != `BLOCK` condition (ismaelsadeeq) Pull request description: This is an attempt to #11775 This Pr will enable fee estimator to listen to ValidationInterface notifications to process new transactions added and removed from the mempool. This PR includes the following changes: - Added a new callback to the Validation Interface `MempoolTransactionsRemovedForConnectedBlock`, which notifies listeners about the transactions that have been removed due to a new block being connected, along with the height at which the transactions were removed. - Modified the `TransactionAddedToMempool` callback parameter to include additional information about the transaction needed for fee estimation. - Updated `CBlockPolicyEstimator` to process transactions using` CTransactionRef` instead of `CTxMempoolEntry.` - Implemented the `CValidationInterface` interface in `CBlockPolicyEstimater` and overridden the `TransactionAddedToMempool`, `TransactionRemovedFromMempool`, and `MempoolTransactionsRemovedForConnectedBlock` methods to receive updates from their notifications. Prior to this PR, the fee estimator updates from the mempool, i.e whenever a new block is connected all transactions in the block that are in our mempool are going to be removed using the `removeForBlock` function in `txmempool.cpp`. This removal triggered updates to the fee estimator. As a result, the fee estimator would block mempool's `cs` until it finished updating every time a new block was connected. Instead of being blocked only on mempool tx removal, we were blocking on both tx removal and fee estimator updating. If we want to further improve fee estimation, or add heavy-calulation steps to it, it is currently not viable as we would be slowing down block relay in the process This PR is smaller in terms of the changes made compared to #11775, as it focuses solely on enabling fee estimator updates from the validationInterface/cscheduler thread notifications. I have not split the validation interface because, as I understand it, the rationale behind the split in #11775 was to have `MempoolInterface` signals come from the mempool and `CValidationInterface` events come from validation. I believe this separation can be achieved in a separate refactoring PR when the need arises. Also left out some commits from #11775 - Some refactoring which are no longer needed. - Handle reorgs much better in fee estimator. - Track witness hash malleation in fee estimator I believe they are a separate change that can come in a follow-up after this. ACKs for top commit: achow101: ACK 91504cbe0d TheCharlatan: Re-ACK 91504cbe0d willcl-ark: ACK 91504cbe0d Tree-SHA512: 846dfb9da57a8a42458827b8975722d153907fe6302ad65748d74f311e1925557ad951c3d95fe71fb90ddcc8a3710c45abb343ab86b88780871cb9c38c72c7b1		2023-12-01 15:07:23 -05:00
..
bench	Merge bitcoin/bitcoin#26762: bugfix: Make CCheckQueue RAII-styled (attempt 2)	2023-11-30 14:28:46 -05:00
common	Merge bitcoin/bitcoin#28486: test, bench: Initialize and terminate use of Winsock properly	2023-11-29 17:14:34 +00:00
compat
config
consensus
crc32c
crypto
index
init
interfaces
ipc
kernel	Merge bitcoin/bitcoin#28368: Fee Estimator updates from Validation Interface/CScheduler thread	2023-12-01 15:07:23 -05:00
leveldb
logging
minisketch
node	Merge bitcoin/bitcoin#28368: Fee Estimator updates from Validation Interface/CScheduler thread	2023-12-01 15:07:23 -05:00
policy
primitives	Merge bitcoin/bitcoin#28766: Improve peformance of CTransaction::HasWitness (28107 follow-up)	2023-11-28 08:44:41 -05:00
qt	Merge bitcoin/bitcoin#26762: bugfix: Make CCheckQueue RAII-styled (attempt 2)	2023-11-30 14:28:46 -05:00
rpc	Merge bitcoin/bitcoin#28368: Fee Estimator updates from Validation Interface/CScheduler thread	2023-12-01 15:07:23 -05:00
script
secp256k1
support
test	Merge bitcoin/bitcoin#28368: Fee Estimator updates from Validation Interface/CScheduler thread	2023-12-01 15:07:23 -05:00
univalue
util
wallet	Remove unused version.h include	2023-11-30 11:28:19 +01:00
zmq
.bear-tidy-config
.clang-format
.clang-tidy
addrdb.cpp
addrdb.h
addresstype.cpp
addresstype.h
addrman.cpp
addrman.h
addrman_impl.h
arith_uint256.cpp
arith_uint256.h
attributes.h
banman.cpp
banman.h
base58.cpp
base58.h
bech32.cpp
bech32.h
bip324.cpp
bip324.h
bitcoin-chainstate.cpp	Merge bitcoin/bitcoin#26762: bugfix: Make CCheckQueue RAII-styled (attempt 2)	2023-11-30 14:28:46 -05:00
bitcoin-cli-res.rc
bitcoin-cli.cpp
bitcoin-tx-res.rc
bitcoin-tx.cpp
bitcoin-util-res.rc
bitcoin-util.cpp
bitcoin-wallet-res.rc
bitcoin-wallet.cpp
bitcoind-res.rc
bitcoind.cpp
blockencodings.cpp
blockencodings.h
blockfilter.cpp
blockfilter.h
chain.cpp
chain.h
chainparams.cpp
chainparams.h
chainparamsbase.cpp
chainparamsbase.h
chainparamsseeds.h
checkqueue.h
clientversion.cpp
clientversion.h
coins.cpp
coins.h
compressor.cpp
compressor.h
core_io.h
core_memusage.h
core_read.cpp
core_write.cpp
cuckoocache.h
dbwrapper.cpp
dbwrapper.h
deploymentinfo.cpp
deploymentinfo.h
deploymentstatus.cpp
deploymentstatus.h
dummywallet.cpp
external_signer.cpp	Remove unused version.h include	2023-11-30 11:28:19 +01:00
external_signer.h
flatfile.cpp
flatfile.h
hash.cpp
hash.h
headerssync.cpp
headerssync.h
httprpc.cpp
httprpc.h
httpserver.cpp
httpserver.h
i2p.cpp
i2p.h
indirectmap.h
init.cpp	Merge bitcoin/bitcoin#28368: Fee Estimator updates from Validation Interface/CScheduler thread	2023-12-01 15:07:23 -05:00
init.h
key.cpp
key.h
key_io.cpp
key_io.h
logging.cpp
logging.h
Makefile.am	Merge bitcoin/bitcoin#28368: Fee Estimator updates from Validation Interface/CScheduler thread	2023-12-01 15:07:23 -05:00
Makefile.bench.include
Makefile.crc32c.include
Makefile.leveldb.include
Makefile.minisketch.include
Makefile.qt.include
Makefile.qt_locale.include
Makefile.qttest.include
Makefile.test.include
Makefile.test_fuzz.include
Makefile.test_util.include
Makefile.univalue.include
mapport.cpp
mapport.h
memusage.h
merkleblock.cpp	refactor: Use Txid in CMerkleBlock	2023-11-28 17:49:41 +01:00
merkleblock.h	refactor: Use Txid in CMerkleBlock	2023-11-28 17:49:41 +01:00
net.cpp
net.h	Rename version.h to node/protocol_version.h	2023-11-30 11:28:31 +01:00
net_permissions.cpp
net_permissions.h
net_processing.cpp	Merge bitcoin/bitcoin#28579: refactor: Remove redundant checks in compat/assumptions.h	2023-11-28 16:51:28 -05:00
net_processing.h
net_types.cpp
net_types.h
netaddress.cpp
netaddress.h
netbase.cpp
netbase.h
netgroup.cpp
netgroup.h
netmessagemaker.h
noui.cpp
noui.h
outputtype.cpp
outputtype.h
pow.cpp
pow.h
prevector.h
protocol.cpp
protocol.h
psbt.cpp	Remove unused version.h include	2023-11-30 11:28:19 +01:00
psbt.h
pubkey.cpp
pubkey.h
random.cpp
random.h
randomenv.cpp
randomenv.h
rest.cpp
rest.h
reverse_iterator.h
scheduler.cpp
scheduler.h
serialize.h	Remove unused SER_NETWORK, SER_DISK	2023-11-30 11:28:17 +01:00
shutdown.cpp
shutdown.h
signet.cpp
signet.h
span.h
streams.cpp
streams.h	Remove unused CDataStream	2023-11-30 11:27:54 +01:00
sync.cpp
sync.h
threadsafety.h
timedata.cpp
timedata.h
tinyformat.h
torcontrol.cpp
torcontrol.h
txdb.cpp
txdb.h
txmempool.cpp	Merge bitcoin/bitcoin#28368: Fee Estimator updates from Validation Interface/CScheduler thread	2023-12-01 15:07:23 -05:00
txmempool.h
txorphanage.cpp
txorphanage.h
txrequest.cpp
txrequest.h
uint256.cpp
uint256.h
undo.h
validation.cpp	Merge bitcoin/bitcoin#28368: Fee Estimator updates from Validation Interface/CScheduler thread	2023-12-01 15:07:23 -05:00
validation.h	Merge bitcoin/bitcoin#26762: bugfix: Make CCheckQueue RAII-styled (attempt 2)	2023-11-30 14:28:46 -05:00
validationinterface.cpp
validationinterface.h
versionbits.cpp
versionbits.h
walletinitinterface.h
warnings.cpp
warnings.h