0
0
Fork 0
mirror of https://github.com/bitcoin/bitcoin.git synced 2025-02-15 11:36:00 -05:00
Commit graph

1275 commits

Author SHA1 Message Date
josibake
1580e3be83
fuzz: add ConstructPubKeyBytes function
Today, this code only has one spot where it needs well-formed pubkeys,
but future PRs will want to reuse this code.

Add a function which creates a well-formed byte array that can be turned
into a pubkey. It is not required that the pubkey is valid, just that it
can be recognized as a compressed or uncompressed pubkey.

Note: while the main intent of this commit is to wrap the existing
logic into a function, it also switches to `PickValueFromArray` so that
we are only choosing one of 0x04, 0x06, or 0x07. The previous code,
`ConsumeIntegralInRange` would have also picked 0x05, which is not
definied in the context of compressed vs uncompressed keys.

See https://bitcoin.stackexchange.com/questions/57855/c-secp256k1-what-do-prefixes-0x06-and-0x07-in-an-uncompressed-public-key-signif
for more details.
2023-08-30 17:45:51 +02:00
Vasil Dimov
5086a99b84
net: remove Sock default constructor, it's not necessary 2023-08-25 14:42:07 +02:00
Antoine Poinsot
10546a569c
wallet: accurately account for the size of the witness stack
When estimating the maximum size of an input, we were assuming the
number of elements on the witness stack could be encode in a single
byte. This is a valid approximation for all the descriptors we support
(including P2WSH Miniscript ones), but may not hold anymore once we
support Miniscript within Taproot descriptors (since the max standard
witness stack size of 100 gets lifted).

It's a low-hanging fruit to account for it correctly, so just do it now.
2023-08-25 12:40:12 +02:00
Antoine Poinsot
fa7c46b503
descriptor: introduce a method to get the satisfaction size
In the wallet code, we are currently estimating the size of a signed
input by doing a dry run of the signing logic. This is unnecessary as
all outputs we are able to sign for can be represented by a descriptor,
and we can derive the size of a satisfaction ("signature") from the
descriptor itself directly.
In addition, this approach does not scale: getting the size of a
satisfaction through a dry run of the signing logic is only possible for
the most basic scripts.

This commit introduces the computation of the size of satisfaction per
descriptor. It's a bit intricate for 2 main reasons:
- We want to conserve the behaviour of the current dry-run logic used by
  the wallet that sometimes assumes ECDSA signatures will be low-r,
  sometimes not (when we don't create them).
- We need to account for the witness discount. A single descriptor may
  sometimes benefit of it, sometimes not (for instance `pk()` if used as
  top-level versus if used inside `wsh()`).
2023-08-25 12:40:11 +02:00
Antoine Poinsot
4ab382c2cd
miniscript: make GetStackSize independent of P2WSH context
It was taking into account the P2WSH script push in the number of stack
elements.
2023-08-25 11:56:09 +02:00
Andrew Chow
c9273f68f6
Merge bitcoin/bitcoin#28287: rpc, test: add sendmsgtopeer rpc and a test for net-level deadlock situation
b3a93b409e test: add functional test for deadlock situation (Martin Zumsande)
3557aa4d0a test: add basic tests for sendmsgtopeer to rpc_net.py (Martin Zumsande)
a9a1d69391 rpc: add test-only sendmsgtopeer rpc (Martin Zumsande)

Pull request description:

  This adds a `sendmsgtopeer` rpc (for testing only) that allows a node to send a message (provided in hex) to a peer.
  While we would usually use a `p2p` object instead of a node for this in the test framework, that isn't possible in situations where this message needs to trigger an actual interaction of multiple nodes.

  Use this rpc to add test coverage for the bug fixed in #27981 (that just got merged):
  The test lets two nodes (almost) simultaneously send a single large (4MB) p2p message to each other, which would have caused a deadlock previously (making this test fail), but succeeds now.

  As can be seen from the discussion in #27981, it was not easy to reproduce this bug without `sendmsgtopeer`. I would imagine that `sendmsgtopeer` could also be helpful in various other test constellations.

ACKs for top commit:
  ajtowns:
    ACK b3a93b409e
  sipa:
    ACK b3a93b409e
  achow101:
    ACK b3a93b409e

Tree-SHA512: 6e22e72402f3c4dd70cddb9e96ea988444720f7a164031df159fbdd48056c8ac77ac53def045d9208a3ca07437c7c8e34f8b4ebc7066c0a84d81cd53f2f4fa5f
2023-08-24 17:34:57 -04:00
Pieter Wuille
8a3b6f3387 refactor: make Transport::ReceivedBytes just return success/fail 2023-08-23 20:13:49 -04:00
Pieter Wuille
bb4aab90fd net: move message conversion to wire bytes from PushMessage to SocketSendData
This furthers transport abstraction by removing the assumption that a message
can always immediately be converted to wire bytes. This assumption does not hold
for the v2 transport proposed by BIP324, as no messages can be sent before the
handshake completes.

This is done by only keeping (complete) CSerializedNetMsg objects in vSendMsg,
rather than the resulting bytes (for header and payload) that need to be sent.
In SocketSendData, these objects are handed to the transport as permitted by it,
and sending out the bytes the transport tells us to send. This also removes the
nSendOffset member variable in CNode, as keeping track of how much has been sent
is now a responsability of the transport.

This is not a pure refactor, and has the following effects even for the current
v1 transport:

* Checksum calculation now happens in SocketSendData rather than PushMessage.
  For non-optimistic-send messages, that means this computation now happens in
  the network thread rather than the message handler thread (generally a good
  thing, as the message handler thread is more of a computational bottleneck).
* Checksum calculation now happens while holding the cs_vSend lock. This is
  technically unnecessary for the v1 transport, as messages are encoded
  independent from one another, but is untenable for the v2 transport anyway.
* Statistics updates about per-message sent bytes now happen when those bytes
  are actually handed to the OS, rather than at PushMessage time.
2023-08-23 20:13:49 -04:00
Pieter Wuille
009ff8d650 fuzz: add bidirectional fragmented transport test
This adds a simulation test, with two V1Transport objects, which send messages
to each other, with sending and receiving fragmented into multiple pieces that
may be interleaved. It primarily verifies that the sending and receiving side
are compatible with each other, plus a few sanity checks.
2023-08-23 20:13:45 -04:00
Pieter Wuille
fb2c5edb79 net: make V1Transport implicitly use current chainparams
The rest of net.cpp already uses Params() to determine chainparams in many
places (and even V1Transport itself does so in some places).

Since the only chainparams dependency is through the message start characters,
just store those directly in the transport.
2023-08-23 19:56:24 -04:00
Pieter Wuille
0de48fe858 net: abstract sending side of transport serialization further
This makes the sending side of P2P transports mirror the receiver side: caller provides
message (consisting of type and payload) to be sent, and then asks what bytes must be
sent. Once the message has been fully sent, a new message can be provided.

This removes the assumption that P2P serialization of messages follows a strict structure
of header (a function of type and payload), followed by (unmodified) payload, and instead
lets transports decide the structure themselves.

It also removes the assumption that a message must always be sent at once, or that no
bytes are even sent on the wire when there is no message. This opens the door for
supporting traffic shaping mechanisms in the future.
2023-08-23 19:56:24 -04:00
Pieter Wuille
649a83c7f7 refactor: rename Transport class receive functions
Now that the Transport class deals with both the sending and receiving side
of things, make the receive side have function names that clearly indicate
they're about receiving.

* Transport::Read() -> Transport::ReceivedBytes()
* Transport::Complete() -> Transport::ReceivedMessageComplete()
* Transport::GetMessage() -> Transport::GetReceivedMessage()
* Transport::SetVersion() -> Transport::SetReceiveVersion()

Further, also update the comments on these functions to (among others) remove
the "deserialization" terminology. That term is better reserved for just the
serialization/deserialization between objects and bytes (see serialize.h), and
not the conversion from/to wire bytes as performed by the Transport.
2023-08-23 19:56:24 -04:00
Pieter Wuille
93594e42c3 refactor: merge transport serializer and deserializer into Transport class
This allows state that is shared between both directions to be encapsulated
into a single object. Specifically the v2 transport protocol introduced by
BIP324 has sending state (the encryption keys) that depends on received
messages (the DH key exchange). Having a single object for both means it can
hide logic from callers related to that key exchange and other interactions.
2023-08-23 19:56:24 -04:00
Martin Zumsande
a9a1d69391 rpc: add test-only sendmsgtopeer rpc
This rpc can be used when we want a node to send a message, but
cannot use a python P2P object, for example for testing of low-level
net transport behavior.
2023-08-22 13:28:15 -04:00
Ryan Ofsky
94a98fbd1d assumeutxo cleanup: Move IsInitialBlockDownload & NotifyHeaderTip to ChainstateManager
This change makes IsInitialBlockDownload and NotifyHeaderTip functions no
longer tied to individual Chainstate objects. It makes them work with the
ChainstateManager object instead so code is simpler and it is no longer
possible to call them incorrectly with an inactive Chainstate.

This change also makes m_cached_finished_ibd caching easier to reason about,
because now there is only one cached value instead of two (for background and
snapshot chainstates) so the cached IBD state now no longer gets reset when a
snapshot is loaded.

There should be no change in behavior because these functions were always
called on the active ChainState objects.

These changes were discussed previously
https://github.com/bitcoin/bitcoin/pull/27746#discussion_r1246868905 and
https://github.com/bitcoin/bitcoin/pull/27746#discussion_r1237552792 as
possible followups for that PR.
2023-08-18 12:52:30 -04:00
Pieter Wuille
bdcbc8594c fuzz: support std::byte in Consume{Fixed,Variable}LengthByteVector 2023-08-17 15:31:56 -04:00
Pieter Wuille
7d1cd93234 crypto: require key on ChaCha20 initialization 2023-08-17 15:31:27 -04:00
Pieter Wuille
3da636e08b crypto: refactor ChaCha20 classes to use Span<std::byte> interface 2023-08-17 15:26:34 -04:00
fanquake
7ef2d4ee4d
Merge bitcoin/bitcoin#28244: Break up script/standard.{h/cpp}
91d924ede1 Rename script/standard.{cpp/h} to script/solver.{cpp/h} (Andrew Chow)
bacdb2e208 Clean up script/standard.{h/cpp} includes (Andrew Chow)
f3c9078b4c Clean up things that include script/standard.h (Andrew Chow)
8bbe257bac MOVEONLY: Move datacarrier defaults to policy.h (Andrew Chow)
7a172c76d2 Move CTxDestination to its own file (Andrew Chow)
145f36ec81 Move Taproot{SpendData/Builder} to signingprovider.{h/cpp} (Andrew Chow)
86ea8bed54 Move CScriptID to script.{h/cpp} (Andrew Chow)
b81ebff0d9 Remove ScriptHash from CScriptID constructor (Andrew Chow)
cba69dda3d Move MANDATORY_SCRIPT_VERIFY_FLAGS from script/standard.h to policy/policy.h (Anthony Towns)

Pull request description:

  Some future work needs to touch things in script/standard.{h/cpp}, however it is unclear if it is safe to do so as they are included in several different places that could effect standardness and consensus. It contains a mix of policy parameters, consensus parameters, and utilities only used by the wallet. This PR breaks up the various components and renames the files to clearly separate everything.

  * `CTxDestination` is moved to a new file `src/addresstype.{cpp/h}`
  * `TaprootSpendData` and `TaprootBuilder` (and their utility functions and structs) are moved to `SigningProvider` as these are used only during signing.
  * `CScriptID` is moved to `script/script.h` to be next to `CScript`.
  * `MANDATORY_SCRIPT_VERIFY_FLAGS` is moved to `interpreter.h`
  * The parameters `DEFAULT_ACCEPT_DATACARRIER` and `MAX_OP_RETURN_RELAY` are moved to `policy.h`
  * `standard.{cpp/h}` is renamed to `solver.{cpp/h}` since that's all that's left in the file after the above moves

ACKs for top commit:
  Sjors:
    ACK 91d924ede1
  ajtowns:
    ACK 91d924ede1
  MarcoFalke:
    ACK 91d924ede1 😇
  murchandamus:
    ACK 91d924ede1
  darosior:
    Code review ACK 91d924ede1.
  theStack:
    Code-review ACK 91d924ede1

Tree-SHA512: d347439890c652081f6a303d99b2bde6c371c96e7f4127c5db469764a17d39981f19884679ba883e28b733fde6142351dd8288c7bc61c379b7eefe7fa7acca1a
2023-08-17 12:54:16 +01:00
fanquake
a62f5ee86c
Merge bitcoin/bitcoin#27675: p2p: Drop m_recently_announced_invs bloom filter
fb02ba3c5f mempool_entry: improve struct packing (Anthony Towns)
1a118062fb net_processing: Clean up INVENTORY_BROADCAST_MAX constants (Anthony Towns)
6fa49937e4 test: Check tx from disconnected block is immediately requestable (glozow)
e4ffabbffa net_processing: don't add txids to m_tx_inventory_known_filter (Anthony Towns)
6ec1809d33 net_processing: drop m_recently_announced_invs bloom filter (Anthony Towns)
a70beafdb2 validation: when adding txs due to a block reorg, allow immediate relay (Anthony Towns)
1e9684f39f mempool_entry: add mempool entry sequence number (Anthony Towns)

Pull request description:

  This PR replaces the `m_recently_announced_invs` bloom filter with a simple sequence number tracking the mempool state when we last considered sending an INV message to a node. This saves 33kB per peer (or more if we raise the rate at which we relay transactions over the network, in which case we would need to increase the size of the bloom filter proportionally).

  The philosophy here (compare with #18861 and #19109) is that we consider the rate limiting on INV messages to only be about saving bandwidth and not protecting privacy, and therefore after you receive an INV message, it's immediately fair game to request any transaction that was in the mempool at the time the INV message was sent. We likewise consider the BIP 133 feefilter and BIP 37 bloom filters to be bandwidth optimisations here, and treat transactions as requestable if they would have been announced without those filters. Given that philosophy, tracking the timestamp of the last INV message and comparing that against the mempool entry time allows removal of each of `m_recently_announced_invs`, `m_last_mempool_req` and `UNCONDITIONAL_RELAY_DELAY` and associated logic.

ACKs for top commit:
  naumenkogs:
    ACK fb02ba3c5f
  amitiuttarwar:
    review ACK fb02ba3c5f
  glozow:
    reACK fb02ba3c5f

Tree-SHA512: cbba5ee04c86df26b6057f3654c00a2b45ec94d354f4f157a769cecdaa0b509edaac02b3128afba39b023e82473fc5e28c915a787f84457ffe66638c6ac9c2d4
2023-08-17 10:52:06 +01:00
Andrew Chow
cd43a8444b
Merge bitcoin/bitcoin#27460: rpc: Add importmempool RPC
fa776e61cd Add importmempool RPC (MarcoFalke)
fa20d734a2 refactor: Add and use kernel::ImportMempoolOptions (MarcoFalke)
fa8866990d doc: Clarify the getmempoolinfo.loaded RPC field documentation (MarcoFalke)
6888886cec Remove Chainstate::LoadMempool (MarcoFalke)

Pull request description:

  Currently it is possible to import a mempool by placing it in the datadir and starting the node. However this has many issues:

  * Users aren't expected to fiddle with the datadir, possibly corrupting it
  * An existing mempool file in the datadir may be overwritten
  * The node needs to be restarted
  * Importing an untrusted file this way is dangerous, because it can corrupt the mempool

  Fix all issues by adding a new RPC.

ACKs for top commit:
  ajtowns:
    utACK fa776e61cd
  achow101:
    ACK fa776e61cd
  glozow:
    reACK fa776e61cd

Tree-SHA512: fcb1a92d6460839283c546c47a2d930c363ac1013c4c50dc5215ddf9fe5e51921d23fe0abfae0a5a7631983cfc7e2fff3788b70f95937d0a989a203be4d67546
2023-08-15 10:15:22 -04:00
fanquake
5606d7f5a8
Merge bitcoin/bitcoin#28267: crypto: BIP324 ciphersuite follow-up
93cb8f0380 refactor: add missing headers for BIP324 ciphersuite (stratospher)
d22d5d925c crypto: BIP324 ciphersuite follow-up (stratospher)

Pull request description:

  follow-up to #28008.
  * move `dummy_tag` variable in FSChaCha20Poly1305 crypto_tests outside of the loop to be reused every time
  * use easy to read `cipher.last()` in `AEADChaCha20Poly1305::Decrypt()`
  * comment for initiator in `BIP324Cipher::Initialize()`
  * systematically damage ciphertext with bit positions in bip324_tests
  * use 4095 max bytes for `aad` in bip324 fuzz test

ACKs for top commit:
  fanquake:
    ACK 93cb8f0380 - thanks for following up here.

Tree-SHA512: 361f3e226d3168fdef69a2eebe6092cfc04ba14ce009420222e762698001eaf8be69a1138dab0be237964509c2b96a41a0b4db5c1df43ef75062f143c5aa741a
2023-08-15 11:11:55 +01:00
fanquake
e38c225261
Merge bitcoin/bitcoin#28215: fuzz: fix a couple incorrect assertions in the coins_view target
e417c988f6 fuzz: coins_view: remove an incorrect assertion (Antoine Poinsot)
c5f6b1db56 fuzz: coins_view: correct an incorrect assertion (Antoine Poinsot)

Pull request description:

  The `coins_view` fuzz target would assert in two places that the cache is consistent with the backend. But it's never the case (that's the whole point of using a cache).

  The only reason this didn't result in a crash was that we would never actually hit these assertions. I ran into this while introducing a new target with an in-memory `CCoinsViewDB` as the backend view (see https://github.com/bitcoin/bitcoin/pull/28216) which made the code paths with those assertions actually reachable.

ACKs for top commit:
  dergoegge:
    Code review ACK e417c988f6

Tree-SHA512: 5847bb2744a2f2831dace62d32b79cc491bf54e2af4ce425411d245d566622d9aff816d9be5ec8e830d10851c13f2500bf4f0c004d88b4d7cca1d483ef8960a6
2023-08-15 11:05:42 +01:00
stratospher
93cb8f0380 refactor: add missing headers for BIP324 ciphersuite 2023-08-15 07:30:48 +05:30
Andrew Chow
91d924ede1 Rename script/standard.{cpp/h} to script/solver.{cpp/h}
Since script/standard only contains things that are used by the Solver
and its callers, rename the files to script/solver.
2023-08-14 17:39:49 -04:00
Andrew Chow
f3c9078b4c Clean up things that include script/standard.h
Remove standard.h from files that don't use anything in it, and include
it in files that do.
2023-08-14 17:38:27 -04:00
Andrew Chow
7a172c76d2 Move CTxDestination to its own file
CTxDestination is really our internal representation of an address and
doesn't really have anything to do with standard script types, so move
them to their own file.
2023-08-14 17:38:27 -04:00
Andrew Chow
86ea8bed54 Move CScriptID to script.{h/cpp}
CScriptID should be next to CScript just as CKeyID is next to CPubKey
2023-08-14 17:38:27 -04:00
stratospher
d22d5d925c crypto: BIP324 ciphersuite follow-up
follow-up to #28008.
* move `dummy_tag` variable in FSChaCha20Poly1305 crypto_tests
outside of the loop to be reused every time
* use easy to read `cipher.last()` in `AEADChaCha20Poly1305::Decrypt()`
* comment for initiator in `BIP324Cipher::Initialize()`
* systematically damage ciphertext with bit positions in bip324_tests
* use 4095 max bytes for aad in bip324 fuzz test
2023-08-14 09:03:21 +05:30
Antoine Poinsot
e417c988f6
fuzz: coins_view: remove an incorrect assertion
Again, this was not hit because the default implementation of
`CCoinsView` return `false` for `GetCoin`.
2023-08-11 18:11:07 +02:00
fanquake
b2ec0326fd
Merge bitcoin/bitcoin#28008: BIP324 ciphersuite
1c7582ead6 tests: add decryption test to bip324_tests (Pieter Wuille)
990f0f8da9 Add BIP324Cipher, encapsulating key agreement, derivation, and stream/AEAD ciphers (Pieter Wuille)
c91cedf281 crypto: support split plaintext in ChaCha20Poly1305 Encrypt/Decrypt (Pieter Wuille)
af2b44c76e bench: add benchmark for FSChaCha20Poly1305 (Pieter Wuille)
aa8cee9334 crypto: add FSChaCha20Poly1305, rekeying wrapper around ChaCha20Poly1305 (Pieter Wuille)
0fee267792 crypto: add FSChaCha20, a rekeying wrapper around ChaCha20 (Pieter Wuille)
9ff0768bdc crypto: add the ChaCha20Poly1305 AEAD as specified in RFC8439 (Pieter Wuille)
9fd085a1a4 crypto: remove outdated variant of ChaCha20Poly1305 AEAD (Pieter Wuille)

Pull request description:

  Depends on #27985 and #27993, based on and partially replaces #25361, part of #27634. Draft while dependencies are not merged.

  This adds implementations of:
  * The ChaCha20Poly1305 AEAD from [RFC8439 section 2.8](https://datatracker.ietf.org/doc/html/rfc8439#section-2.8), including test vectors.
  * The FSChaCha20 stream cipher as specified in [BIP324](https://github.com/bitcoin/bips/blob/master/bip-0324.mediawiki#rekeying-wrappers-fschacha20poly1305-and-fschacha20), a rekeying wrapper around ChaCha20.
  * The FSChaCha20Poly1305 AEAD as specified in [BIP324](https://github.com/bitcoin/bips/blob/master/bip-0324.mediawiki#rekeying-wrappers-fschacha20poly1305-and-fschacha20), a rekeying wrapper around ChaCha20Poly1305.
  * A BIP324Cipher class that encapsulates key agreement, key derivation, and stream ciphers and AEADs for [BIP324 packet encoding](https://github.com/bitcoin/bips/blob/master/bip-0324.mediawiki#overall-packet-encryption-and-decryption-pseudocode).

  The ChaCha20Poly1305 and FSChaCha20Poly1305 implementations are new, taking advance of the improvements in #27993.

ACKs for top commit:
  jamesob:
    reACK 1c7582e
  theStack:
    ACK 1c7582ead6
  stratospher:
    tested ACK 1c7582e.

Tree-SHA512: 06728b4b95b21c5b732ed08faf40e94d0583f9d86ff4db3b92dd519dcd9fbfa0f310bc66ef1e59c9e49dd844ba8c5ac06e2001762a804fb5aa97027816045a46
2023-08-10 11:58:59 +02:00
MarcoFalke
fa776e61cd
Add importmempool RPC
test_importmempool_union contributed by glozow

Co-authored-by: glozow <gloriajzhao@gmail.com>
2023-08-07 11:33:34 +02:00
MarcoFalke
fa20d734a2
refactor: Add and use kernel::ImportMempoolOptions
This allows optional named arguments with default values.
2023-08-07 11:32:34 +02:00
MarcoFalke
6888886cec
Remove Chainstate::LoadMempool
The 3-line function is only called once outside of tests, so it is
clearer to inline it.
2023-08-07 10:59:15 +02:00
Antoine Poinsot
c5f6b1db56
fuzz: coins_view: correct an incorrect assertion
It is incorrect to assert that `cache.HaveCoin()` will always be `true`
if `backend.HaveCoin()` is. The coin could well have been marked as
spent in the cache but not yet flushed, in which case `cache.HaveCoin()`
would return `false`.

Note this was never hit because `exists_using_have_coin_in_backend` is
currently never `true` (it's the default implementation of `CCoinsView`.
However this might change if we were to add a target where the backend
is a `CCoinsViewDB`.
2023-08-04 13:51:30 +02:00
fanquake
61849f0464
Merge bitcoin/bitcoin#27918: fuzz: addrman, avoid ConsumeDeserializable when possible
025fda0a76 fuzz: addrman, avoid `ConsumeDeserializable` when possible (brunoerg)

Pull request description:

  Using specific functions like `ConsumeService`, `ConsumeAddress` and `ConsumeNetAddr` may be more effective than using `ConsumeDeserializable`. They always return some value while `ConsumeDeserializable` may return `std::nullopt`.

  E.g.: In this part of the code, if `op_net_addr` is `std::nullopt`,  we basically generated the addresses (if so) unnecessarily, because we won't be able to use them:
  ```cpp
  std::vector<CAddress> addresses;
  LIMITED_WHILE(fuzzed_data_provider.ConsumeBool(), 10000) {
      const std::optional<CAddress> opt_address = ConsumeDeserializable<CAddress>(fuzzed_data_provider);
      if (!opt_address) {
          break;
      }
      addresses.push_back(*opt_address);
  }
  const std::optional<CNetAddr> opt_net_addr = ConsumeDeserializable<CNetAddr>(fuzzed_data_provider);
  if (opt_net_addr) {
      addr_man.Add(addresses, *opt_net_addr, std::chrono::seconds{ConsumeTime(fuzzed_data_provider, 0, 100000000)});
  }
  ```

  Also, if we are not calling `Add` effectively, it would also be affect other functions that may "depend" on it.

ACKs for top commit:
  dergoegge:
    Code review ACK 025fda0a76

Tree-SHA512: 02450bec0b084c15ba0cd1cbdfbac067c8fea4ccf27be0c86d54e020f029a6c749a16d8e0558f9d6d35a7ca9db8916f180c872f09474702b5591129e9be0d192
2023-08-03 17:32:46 +01:00
Anthony Towns
1e9684f39f mempool_entry: add mempool entry sequence number 2023-08-03 13:42:45 +10:00
fanquake
e5a9f2fb62
Merge bitcoin/bitcoin#28194: test: python E721 and flake8 updates
bee2d57a65 script: update flake8 to 6.1.0 (Jon Atack)
38c3fd846b test: python E721 updates (Jon Atack)

Pull request description:

  Update our functional tests per [E721](https://www.flake8rules.com/rules/E721.html) enforced by [flake8 6.1.0](https://flake8.pycqa.org/en/latest/release-notes/6.1.0.html), and update our CI lint task to use that release.  This makes the following linter output on current master with flake8 6.1.0 green.

  ```
  $ ./test/lint/lint-python.py ; ./test/lint/lint-spelling.py
  test/functional/p2p_invalid_locator.py:35:16: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
  test/functional/test_framework/siphash.py:34:12: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
  test/functional/test_framework/siphash.py:64:12: E721 do not compare types, for exact checks use `is` / `is not`, for instance checks use `isinstance()`
  src/test/fuzz/descriptor_parse.cpp:88: occurences ==> occurrences
  ^ Warning: codespell identified likely spelling errors. Any false positives? Add them to the list of ignored words in test/lint/spelling.ignore-words.txt
  ```

ACKs for top commit:
  MarcoFalke:
    lgtm ACK bee2d57a65

Tree-SHA512: f3788a543ca98e44eeeba1d06c32f1b11eec95d4aef068aa1b6b5c401261adfa3fb6c6d6c769f3fe6839d78e74a310d5c926867e7c367d6513a53d580fd376f3
2023-08-01 09:42:07 +01:00
Ryan Ofsky
f4f1d6d230
Merge bitcoin/bitcoin#27746: Rework validation logic for assumeutxo
a733dd79e2 Remove unused function `reliesOnAssumedValid` (Suhas Daftuar)
d4a11abb19 Cache block index entry corresponding to assumeutxo snapshot base blockhash (Suhas Daftuar)
3556b85022 Move CheckBlockIndex() from Chainstate to ChainstateManager (Suhas Daftuar)
0ce805b632 Documentation improvements for assumeutxo (Ryan Ofsky)
768690b7ce Fix initialization of setBlockIndexCandidates when working with multiple chainstates (Suhas Daftuar)
d43a1f1a2f Tighten requirements for adding elements to setBlockIndexCandidates (Suhas Daftuar)
d0d40ea9a6 Move block-storage-related logic to ChainstateManager (Suhas Daftuar)
3cfc75366e test: Clear block index flags when testing snapshots (Suhas Daftuar)
272fbc370c Update CheckBlockIndex invariants for chains based on an assumeutxo snapshot (Suhas Daftuar)
10c05710ce Add wrapper for adding entries to a chainstate's block index candidates (Suhas Daftuar)
471da5f6e7 Move block-arrival information / preciousblock counters to ChainstateManager (Suhas Daftuar)
1cfc887d00 Remove CChain dependency in node/blockstorage (Suhas Daftuar)
fe86a7cd48 Explicitly track maximum block height stored in undo files (Suhas Daftuar)

Pull request description:

  This PR proposes a clean up of the relationship between block storage and the chainstate objects, by moving the decision of whether to store a block on disk to something that is not chainstate-specific.  Philosophically, the decision of whether to store a block on disk is related to validation rules that do not require any UTXO state; for anti-DoS reasons we were using some chainstate-specific heuristics, and those have been reworked here to achieve the proposed separation.

  This PR also fixes a bug in how a chainstate's `setBlockIndexCandidates` was being initialized; it should always have all the HAVE_DATA block index entries that have more work than the chain tip.  During startup, we were not fully populating `setBlockIndexCandidates` in some scenarios involving multiple chainstates.

  Further, this PR establishes a concept that whenever we have 2 chainstates, that we always know the snapshotted chain's base block and the base block's hash must be an element of our block index. Given that, we can establish a new invariant that the background validation chainstate only needs to consider blocks leading to that snapshotted block entry as potential candidates for its tip. As a followup I would imagine that when writing net_processing logic to download blocks for the background chainstate, that we would use this concept to only download blocks towards the snapshotted entry as well.

ACKs for top commit:
  achow101:
    ACK a733dd79e2
  jamesob:
    reACK a733dd79e2 ([`jamesob/ackr/27746.5.sdaftuar.rework_validation_logic`](https://github.com/jamesob/bitcoin/tree/ackr/27746.5.sdaftuar.rework_validation_logic))
  Sjors:
    Code review ACK a733dd79e2.
  ryanofsky:
    Code review ACK a733dd79e2. Just suggested changes since the last review. There are various small things that could be followed up on, but I think this is ready for merge.

Tree-SHA512: 9ec17746f22b9c27082743ee581b8adceb2bd322fceafa507b428bdcc3ffb8b4c6601fc61cc7bb1161f890c3d38503e8b49474da7b5ab1b1f38bda7aa8668675
2023-07-31 16:18:20 -04:00
Jon Atack
bee2d57a65 script: update flake8 to 6.1.0
and touch up the spelling returned by lint-spelling.py
2023-07-31 12:14:06 -06:00
fanquake
44b05bf3fe
Merge bitcoin/bitcoin#28091: fuzz: use ConnmanTestMsg in connman
ecfe507e07 fuzz: use `ConnmanTestMsg` in `connman` (brunoerg)

Pull request description:

  Fixes #27980

  Using `ConnmanTestMsg` we can add nodes and be
  more effective fuzzing functions like `DisconnectNode`,
  `FindNode`, `GetNodeStats` and other ones.

ACKs for top commit:
  MarcoFalke:
    review ACK ecfe507e07
  dergoegge:
    utACK ecfe507e07

Tree-SHA512: 97c363b422809f2e9755c082d1102237347abfab72c7baca417bd8975f8a595ddf3a085f8353dbdb9f17fb98fbfe830792bfc0b83451168458018faf6c239efa
2023-07-31 11:43:39 +01:00
fanquake
42a9110899
Merge bitcoin/bitcoin#28162: refactor: Revert additional univalue check in ParseSighashString
06199a995f refactor: Revert addition of univalue sighash string check (TheCharlatan)
0b47c16215 doc: Correct release-notes for sighashtype exceptions (TheCharlatan)

Pull request description:

  This is a follow up for #28113.

  The string type check is already done by the rpc parser / RPCHelpMan. Re-doing it is adding dead code. Instead, throwing an exception when the assumption does not hold is the already correct behavior. Pointed out in this [comment](https://github.com/bitcoin/bitcoin/pull/28113/files#r1274568557).

  Also correct the release note for the correct sighashtype exception change. There is no change in the handling of non-string sighashtype arugments. Pointed out in this [comment](https://github.com/bitcoin/bitcoin/pull/28113/files#r1274567555).

ACKs for top commit:
  MarcoFalke:
    lgtm ACK 06199a995f
  jonatack:
    Tested ACK 06199a995f
  stickies-v:
    ACK 06199a995f

Tree-SHA512: 3faa6b3d2247624c0973df8d79c09fbf1f90ffb99f1be484e359b528f485c31affea45976759bd206e4c81cbb54ebba5ad0ef4127d1deacbfe2a58153fcc94ee
2023-07-28 12:29:55 +01:00
Andrew Chow
cbf385058b
Merge bitcoin/bitcoin#27888: Fuzz: a more efficient descriptor parsing target
131314b62e fuzz: increase coverage of the descriptor targets (Antoine Poinsot)
90a24741e7 fuzz: add a new, more efficient, descriptor parsing target (Antoine Poinsot)
d60229ede5 fuzz: make the parsed descriptor testing into a function (Antoine Poinsot)

Pull request description:

  The current descriptor parsing fuzz target requires valid public or private keys to be provided. This is unnecessary as we are only interested in fuzzing the descriptor parsing logic here (other targets are focused on fuzzing keys serializations). And it's pretty inefficient, especially for formats that need a checksum (`xpub`, `xprv`, WIF).

  This introduces a new target that mocks the keys as an index in a list of precomputed keys. Keys are represented as 2 hex characters in the descriptor. The key type (private, public, extended, ..) is deterministically based on this one-byte value. Keys are deterministically generated at target initialization. This is much more efficient and also largely reduces the size of the seeds.
  TL;DR: for instance instead of requiring the fuzzer to generate a `pk(xpub6DdBu7pBoyf7RjnUVhg8y6LFCfca2QAGJ39FcsgXM52Pg7eejUHLBJn4gNMey5dacyt4AjvKzdTQiuLfRdK8rSzyqZPJmNAcYZ9kVVEz4kj)` to parse a valid descriptor, it just needs to generate a `pk(03)`.

  Note we only mock the keys themselves, not the entire descriptor key expression. As we want to fuzz the real code that parses the rest of the key expression (origin, derivation paths, ..).

  This is a target i used for reviewing #17190 and #27255, and figured it was worth PR'ing on its own since the added complexity for mocking the keys is minimal and it could help prevent introducing bugs to the descriptor parsing logic much more efficiently.

ACKs for top commit:
  MarcoFalke:
    re-ACK 131314b62e  🐓
  achow101:
    ACK 131314b62e

Tree-SHA512: 485a8d6a0f31a3a132df94dc57f97bdd81583d63507510debaac6a41dbbb42fa83c704ff3f2bd0b78c8673c583157c9a3efd79410e5e79511859e1470e629118
2023-07-27 13:48:12 -04:00
TheCharlatan
06199a995f
refactor: Revert addition of univalue sighash string check
This check is already done by the rpc parser. Re-doing it is adding dead
code. Instead, throwing an exception when the assumption does not hold
is the already correct behavior.

To make the fuzz test more accurate and not swallow all runtime errors,
add a check that the passed in UniValue sighash argument is either a
string or null.

Co-authored-by: stickies-v <stickies-v@protonmail.com>
2023-07-27 09:36:05 +02:00
Pieter Wuille
990f0f8da9 Add BIP324Cipher, encapsulating key agreement, derivation, and stream/AEAD ciphers
Co-authored-by: dhruv <856960+dhruv@users.noreply.github.com>
2023-07-26 17:09:23 -04:00
Pieter Wuille
0fee267792 crypto: add FSChaCha20, a rekeying wrapper around ChaCha20
This adds the FSChaCha20 stream cipher as specified in BIP324, a
wrapper around the ChaCha20 stream cipher (specified in RFC8439
section 2.4) which automatically rekeys every N messages, and
manages the nonces used for encryption.

Co-authored-by: dhruv <856960+dhruv@users.noreply.github.com>
2023-07-26 16:55:05 -04:00
Pieter Wuille
9fd085a1a4 crypto: remove outdated variant of ChaCha20Poly1305 AEAD
Remove the variant of ChaCha20Poly1305 AEAD that was previously added in
anticipation of BIP324 using it. BIP324 was updated to instead use rekeying
wrappers around otherwise unmodified versions of the ChaCha20 stream cipher
and the ChaCha20Poly1305 AEAD as specified in RFC8439.
2023-07-26 16:51:51 -04:00
Andrew Chow
1ed8a0f8d2
Merge bitcoin/bitcoin#28113: kernel: Remove UniValue from kernel library
6960c81cbf kernel: Remove Univalue from kernel library (TheCharlatan)
10eb3a9faa kernel: Split ParseSighashString (TheCharlatan)

Pull request description:

  Besides the build system changes, this is a mostly move-only change for moving the few UniValue-related functions out of kernel files.

  UniValue is not required by any of the kernel components and a JSON library should not need to be part of a consensus library.

ACKs for top commit:
  achow101:
    ACK 6960c81cbf
  theuni:
    Re-ACK 6960c81cbf
  stickies-v:
    re-ACK 6960c81cbf

Tree-SHA512: d92e4cb4e12134c94b517751bd746d39f9b8da528ec3a1c94aaedcce93274a3bae9277832e8a7c0243c13df0397ca70ae7bbb24ede200018c569f8d81103c1da
2023-07-25 18:13:16 -04:00
TheCharlatan
6960c81cbf
kernel: Remove Univalue from kernel library
It is not required by any of the kernel components.
A JSON library should not need to be part of a consensus library.
2023-07-25 17:40:07 +02:00
TheCharlatan
10eb3a9faa
kernel: Split ParseSighashString
This split is done in preparation for the next commit where the
dependency on UniValue in the kernel library is removed.
2023-07-25 17:40:02 +02:00