[Lightning-dev] Every node must be aware of all other nodes - scalability problem?

[Christian Decker]

Welcome to the mailing list Oliver :-)

> It seems to me by reading BOLT #7 that every node in the lightning network
> must be aware of every other. That is necessary to choose a complete route
> to send a transaction for example.

Yes, Bolt 7 is a purposefully simplistic gossiping protocol that does
not scale infinitely. It's simple because we need something to get
started and it is not intended to work forever, just long enough for us
to come up with something better.

> If the lightning network grows large, one could imagine multiple network
> nodes per person, so say 7e10 nodes. For a fully connected graph, there
> must also be 7e10 channels, and likely many more.
>
> That means each node, upon joining the network, must download, keep in
> local storage, and keep updates on at least 35TB of data to be able to send
> payments elsewhere on the network.

I should note that you will absolutely need more than n-1 channels if you
have n nodes, otherwise you're just creating a line-graph, that would
not be very useful :-)

Rusty had some back-of-the-envelope calculations about the raw size of
the data that a node has to handle for 1 million nodes [1], and they
come to about 120 MB, without updates. And yes the initial sync is also
very simplistic, but we are already starting to think about better, more
fine-grained sync protocols to reduce that upfront download when
joining, and you can disable the initial sync already.

> Surely there needs to be some kind of method whereby a peer can keep track
> of only the nearby nodes, and have some kind of routing table for groups of
> more distant nodes, which need not individually be known. A DHT is an
> example of that. Designing such a system where no intermediate node knows
> both sender and receiver sounds hard, but possible...
>
> Why aren't we doing that?

We are also thinking about more advanced path finding algorithms that
reduce the need for the complete information on the node (some of them
also mentioned in that blog post). There's just a lot to implement
generally for Lightning, so we punted on the routing problem a bit,
since it is a luxury problem: before we hit that wall we first have to
have a network that is successful enough to need a better solution.

Cheers,
Christian

[1] https://medium.com/@rusty_lightning/lightning-routing-rough-background-dbac930abbad