[Bitcoin-development] Squashing redundant tx data in blocks on the wire

[Kaz Wesley]

> the FEC still lets you fill in the missing transactions without knowing in advance all that will be missing.

I don't see why we need to solve that problem, since the protocol
already involves exchanging the information necessary to determine
(with some false positives) what a peer is missing, and needs to
continue doing so regardless of how blocks are transmitted.

Set reconciliation does have the benefit of eliminating a subset of
those false positives and offering a finer-grained mechanism for
defining what a node can choose to forget from its mempool than
remember-last-N, but if we implement it for block transmission I don't
see why we wouldn't also use it to synchronize mempool txes, and if
mempools are synchronized we don't actually need to do it as part of
block-transmission to get those benefits.

As far as I can tell, channel memory sparseblocks achieve most of the
possible bandwidth savings, and when FEC-based mempool synchronization
is implemented its benefits can be applied to the sparseblocks by
resetting the channel memory to the mutual mempool state each time
mempool differences are exchanged. Am I missing a benefit to doing FEC
at block forwarding time that can't be realized by FEC-based mempool
synchronization, implemented separately from channel-memory based
index-coding?


On Thu, Jul 31, 2014 at 2:51 PM, Gregory Maxwell <gmaxwell at gmail.com> wrote:
> On Thu, Jul 31, 2014 at 2:41 PM, Kaz Wesley <keziahw at gmail.com> wrote:
>>> the need to have transmitted the transaction list [..] first
>>
>> 32 bits per transaction is at least double the communication overhead
>> of the simple approach, and only offers a bound on the probability of
>> needing a round trip.
>
> "(e.g. from a reconciliation step first)" the list can be communicated
> in the space roughly equal to the size of the difference in sets plus
> coding the permutation from the permissible orderings.   If you did
> have some "simple approach" that guaranteed that some transactions
> would be present, then you could code those with indexes... the FEC
> still lets you fill in the missing transactions without knowing in
> advance all that will be missing.   (Also, the suggestion on the
> network block coding page of using part of a cryptographic permutation
> as the key means that for unknown transactions the transmission of the
> new unknown keys is always goodput— doesn't add overhead)
>
> It's "only a bound" but you can pick whatever bound you want,
> including— if you send data equal to the missing amount, then it'll be
> always successful, but no bandwidth savings.   Though if the transport
> is unordered (e.g. UDP or non-blocking SCTP) even sending 100% of the
> missing amount can save time by eliminating a round trip that might
> otherwise be needed for a retransmission.