[Lightning-dev] type,len,value standard

[Rusty Russell]

Conner Fromknecht <conner at lightning.engineering> writes:
> Hi ZmnSCPxj,
>
> Thanks for writing this up! I had started an email, but you beat me to it :)
>
>> 1.  For a sequence of `type,len,value`, each `type` must be unique. --
>> accepted.
>
> To add to this, it seemed that there was some agreement that repeated fields
> should be serialized under a single root key, since a receiver can't know if a
> field is allowed to have duplicates if they don't understand the field.

If a receiver doesn't understand it, it doesn't have to enforce any
rules on it *at all*.  So it depends on individual cases.

>> For a sequence of `type,len,value`, the `type`s must be in ascending order
>> -- not explicitly accepted or rejected.  It would be easier to check
>> uniqueness > (the previous rule we accepted) here for a naive parser (keep
>> track of some "minimum allowed type" that initializes at zero, check current
>> type >= this, update to current type + 1) if `type`s are in ascending order.
>
> Yep ascending makes sense to me, for the reasons you stated.

Again, writer MUST, with no constraint on reader.

>> 1, `type` - one byte or two?
>
> I'd lean towards one, if a message has 256 optional fields, it might be time to
> consider a new message type altogether.

Exactly.

>> 3. `type` - does "it's OK to be odd" apply?  i.e. if an even `type` that is
>> not known is found, crash and burn.  But intent of this system is for future
>> expansion for optional fields, so...?
>
> Perhaps this depends on context:
>  - for gossip messages, I think the primary concern is not breaking signature
>      validation, and that these would need to remain optional for backwards
>      compatibility.

Yes, "not OK" is message dependent.  It doesn't break parsing (if you
want that, use a new message type), but it does break *usage*; "you
can't use this information".

TLV makes sense when there are many many options, at cost of a couple of
bytes per option.  We agreed to consider it for new additions, but
sometimes a simple option bitmap will be a win.

>  - for link-level messages, we have a little more control. I imagined the fields
>    would be gated by feature bit negotiation, and deviating from
>    unsupported/required would result in being disconnected.

It's nice if you can simplify your code by always sending it, even if
the remote doesn't understand it.

>> 5. `len` - one byte or two? I believe we tend to use two bytes for various
>> lengths.
>
> Maybe varint? One byte is not enough for all lengths, but two seems excessive
> for uint8 or even uint32.

Yes, it's a nice compromise IMHO.

>> 6.  BOLT - I propose making a separate BOLT for `type,len,value`, which other
>> messages and so on simply refer to.
>
> Indeed, are you thinking we'd use this to add new fields proposed in 1.1?

Well, it's a very individual message thing.  We would specify it in
general, then each place where it's used (eg. gossip) would have their
own types.

> In addition to the above, do we also want to flesh out what sub-TLV structures
> would look like? Or perhaps that isn't necessary, if we can continue adding more
> root-level keys.

I think that the general TLV definition just belongs in BOLT #1, since
it's so short.

Cheers,
Rusty.