[Bitcoin-ml] Malleability fix proposal`

Tue Aug 22 11:13:54 UTC 2017

On Tuesday, 22 August 2017 12:07:11 CEST Tomas via bitcoin-ml wrote:
> As a change minimalist, I am a bit sceptical about this huge list. Given
> the immutable nature  of the blockchain, every change is an addition to
> the specification. We should change only what is really needed.

FlexTrans makes a very small change.
The spec is tiny.
The confusion arrives from the fact that the *effects* are huge.

This is rather like when the world started using sgml or xml instead of 
binary formats.
The spec is actually much much simpler than the binary formats, because we 
change the foundation.

When I implemented the reading of transactions in a different language for 
the first time there were many little pitfalls. The usage of different types 
of ways to encode an number, for instance. Having two fields for a byte-array 
(size and actual data) in your spec doesn't help either. I remember being 
confused about the txid being in need of reversing.
A pitfall like the simple fact that the vast majority of transactions don't 
use nLockTime, yet every single one is obligated to include it.

None of these are *changes* in the spec.
The are solutions that come for free when changing from a binary stream to a 
tagged format.

> The primary thing we want to fix is malleability which doesn't need a
> new format.

Why do we want to fix malleability primarily?

> Adding a completely different serialization format has a huge cost as
> every implementation and tool must understand both formats. You seem to
> suggest that the ability to add custom key-value pairs merits such a
> major addition, but I am not so sure.

As a software developer my experience is that its actually cheaper (over the 
long term) to have code that does something like;

if version = 4:
   parsev4()
else
  parsev1()

compared to your suggestion that actually changes the parsing code with an 
exception in a couple of places.
The code to completely separate out the two formats is going to be 
magnitudes easier to maintain over the next decades than the one that just 
adds more exceptions. This is for a very simple reason; the old parsev1() 
method will be frozen from modifications. Therefore will not attract any 
bugs.

So my point is unchanged, adding the support for a completely new version, 
which is a pretty small spec, using widely known concepts, this is not a 
major addition.
It is a small change to the lower level of stream processing which gives a 
huge amount of benefits higher up in the layers.

> What changes are you thinking of that require adding new
> transaction-level key/value pairs?

I think the most mentioned one is a new version of script.
But I think the counter argument is stronger.

If we go for small changes now, which make the format more complex until the 
end of time, that is only worth it if we know we will never make any 
additions to transactions in the lifetime of Bitcoin Cash.
So, are you arguing that we never will add anything to transactions?

But to answer your question, transactions can be extended with new features 
of any kind. From the idea pitched to me by a miner that a webwallet 
includes a “created-by”:”bitcrust” like tag could help in giving revenue 
sharing from miners to those that help Bitcoin grow the most.
To suggestions that we still have room for improvement in the concept of 
locking fund on-chain. A CSV replacement can be made in the next years.

Again, Bitcoin will hopefully exist for decades, my ability to come up with 
ways to extend it is irrelevant. History has shown various additions. We 
will see more.

> Key/value pairs that are not required for validation are non-normative
> and can simply be added using OP_RETURN today. Key/value pairs that
> *are* used for validation are changes to the protocol and must be
> understood by all, and could as well be extended to the plain old
> binary.

This is the core difference between a tagged system and a binary system.
Sure, you can hard fork and state that you add a new set of bytes to the 
transaction format.
But then 100% of the transactions using that version need to add it. Whether 
they use this new feature or not. That's the way that this format works. And 
we can see that in the fact that the vast majority of the transactions today 
do not use the nLockTime, but are obligated to add those 4 bytes anyway.
Adding features that way is not sustainable.

>>  Naturally, all full nodes need to upgrade.  
> > A wallet needs to be modified to both create and receive such
> > transactions.  As long as the wallet doesn’t receive such transactions
> > they don’t need to upgrade.
> 
> I think it is of high importance that everyone agrees on what
> transactions are valid and what are not. Surely creating the situation
> where some wallets are ignorant of their outputs being spend is *much*
> worse than a simple hardfork requiring them to upgrade? Do we really
> want to introduce transactions that are only understood by some?

I’m not sure if I understand how your objection is related to what you 
replied to.
There are never any wallets ignorant of outputs being spent. That would be 
impossible...
You misunderstood my point, I’m thinking. Let me be clear; any transaction a 
certain wallet sees will be 100% understood by that wallet. 
The trick is more like this; instead of just sending you a Word document, I 
actually ask what you support and I can send a Word doc, an LibreOffice doc or 
a PDF as a result of that conversation. And if you receive a WordPerfect 
doc, you get a warning instead of validation error.
The difference is that we don’t have to assume 100% of the world always 
supports the exact same format.
As a sidenote, users can still be asked to upgrade. Its just not as black/
white anymore.

>  We can upgrade the scripting language indefinitely with opcode
> redefinition and addition. This is a really elegant method

Lets look at what you are proposing, if we apply this to the next decades.
We will have a long list of rules of this kind;

if (op == 23) {
  if (block-height < 1234 && block-height > 100)  op_fooBar()
  if (block-height >= 1234 && block-height < 2546)  op_fooBar2() 
  else if (block-version == 3) op_other()
  else  another_op()
}
if (op == 
      etc.

This is not maintainable, and I can guarantee that clients will have 
problems maintaining the ability to do a full sync since genesis with this 
approach.

You can see that in most protocols (and Windows APIs as well) a different 
approach.
A simple
  if version == 4: parsev4()
  else parsev1()

Code duplication is Ok when your intention is to never change the old copy 
(the parsev1()).

The actual format for FlexTrans is immensely more simple than the current 
format for transactions.
https://github.com/bitcoinclassic/transactions/blob/master/
Transaction.cpp#L79-L107

We have supporting code in many languages in the bindings project, I don't 
know rust but I doubt it will be hard to add.
https://github.com/bitcoinclassic/cmf-bindings/
The people that work on hardware wallets told me specifically that its easy 
to add a new format of the kind that FT introduces.

-- 
Tom Zander
Blog: https://zander.github.io
Vlog: https://vimeo.com/channels/tomscryptochannel