[bitcoin-dev] Huge wallets make Bitcoin Core unusable (Daemon+CLI & Qt)

Sat Aug 20 14:16:53 UTC 2022

Hi dear devs,

1. THE ISSUE - DAEMON+CLI
========================
I had a wallet in a server production since 2017 (5 years old) and when it
reached about 273 MB, 2.079.337 transactions and 446.503 generated
addresses, the performance started to degrade exponentially.

Most of the commands, e.g. "getbalance", "walletpassphrase" and
"getreceivedbyaddress" started to timeout (more than 15 minutes delay -
default timeout). The CPU was 100% used (all 32 cores - with 150 load avg)
and the machine became almost unusable breaking everything else, with the
default config of 16 RPC threads and 15 min timeout and some attempt calls
per mi

Increasing the timeout and/or the RPC threads in the config file turns
things even worse.

Putting the wallet.dat in a very fast SSD disk and increasing the size of
the cache (I tried with 8GB) have improved but I'm not sure if it is enough.

2. TEST ON BITCOIN QT
====================
If you try to load the wallet in the "bitcoin-qt" everything gets stuck,
even the system (OS/UI) doesn't respond anymore. You click on a button and
receive the message "window doesn't respond, wait or terminate?" - if you
wait it releases after a while but it is slow and hard to use the wallet
anyway.

3. WHY IS THIS SO BAD?
=====================
This is bad because the standard client becomes almost useless for the
wallet feature:

3.1) the wallet Qt already is not so popular among end users. It doesn't
look modern, slow to first sync and hard to use. That's why people prefer
to use Electrum or Wasabi - I personally don't care but it's the sad truth;

3.2) it becomes useless now also for servers in production, forcing them to
use third party solutions for huge wallets. Even if you split in 10 wallets
it will just delay 10 times more each to degrade, postponing the problem
but not eliminating it. Not to mention the slow and daily degradation.

4. SHOULD WE GIVE UP THE WALLET FEATURE?
========================================
Then, Bitcoin Core becomes just a reference implementation and blocks
relayers, but as an application wallet itself turns into a really bad
choice. --- It leads me to the following question: if we won't invest time
on improving this, shouldn't we remove the wallet feature at all? Why keep
a wallet feature that is not useful for the end user nor the production
server? Is it useful for what then?

5. THE CURRENT "SOLUTION" IS BAD
===============================
Currently, the only "solution" for huge wallets is shameful: create a new
one and send the funds there from time to time. But when is the right time
exactly? The performance degrades suddenly or gets worse slowly for each
new address and/or tx?. And besides not being an elegant solution and "not
in the handbook", it also can break a lot of things like monitoring old
addresses and also can lead to privacy concerns unifying lots of inputs in
a big and expensive tx.

6. OTHER USER CASES?
====================
I think this could also become an issue if we have LN nodes that use the
Bitcoin Core wallet infrastructure behind to open / close many channels for
a long time.

7. FINAL THOUGHTS
=================
If moving the wallet from a HDD to a SSD improved a lot, maybe just caching
the entire wallet in memory could improve even more, but I'm afraid some
code optimization is also necessary.

8. SOME QUESTIONS
==================
8.1) Can we "optimize" a huge wallet without moving the funds to a new one?
Like a "fsck" or eqv?

8.2) Can we improve the cache usage somehow? Putting the entire wallet in
memory, for example?

8.3) Is it possible to reduce the wallet size (273 MB is too much for a HD
wallet)?

8.4) Can we tell the CLI to ignore old addresses? What if I need to watch
only the last 30 days?

8.5) How to improve the I/O treatment and/or CPU usage in the main thread
on Bitcoin-Qt to avoid window freezing on big and huge wallets?

8.6) In the last case (if it was not possible to optimize the wallet or the
CLI & Qt), can the CLI just warn the user like: "the wallet is becoming too
big and slow, execute the command 'archive'". And then, the command
"archive" could rename the current wallet to something like
"wallet.dat.archive_until_20220818", create a new "wallet.dat" and move the
funds automatically? Also, would it be nice to have an
"autoarchivehugewallets=1" in the file config?

9. POSSIBLE RELATED AND TESTS
=============================

[1] https://github.com/bitcoin/bitcoin/issues/15015
[2] https://github.com/bitcoin/bitcoin/issues/15148
[3] https://github.com/bitcoin/bitcoin/issues/16874
[4] https://github.com/bitcoin/bitcoin/pull/17135
[5] https://github.com/bitcoin/bitcoin/pull/18160
[6] https://github.com/bitpay/bitcore-node/issues/463
[7] https://github.com/RavenProject/Ravencoin/issues/499
[8] https://github.com/sugarchain-project/sugarchain/issues/106
[9]
https://bitcoin.stackexchange.com/questions/111844/loadwallet-takes-too-much-and-times-out
[10] https://bitcoin.stackexchange.com/a/45713/1761

ANOTHER POSSIBLE BUG
======================

Even if my node is 100% sync:
2022-08-20T13:11:43Z UpdateTip: new
best=00000000000000000005bba0593c2be0f1d322223501591d2b31b544e3af3d0b
height=750300 version=0x2fffe000 log2_work=93.687081 tx=758181489
date='2022-08-20T13:11:16Z' progress=1.000000 cache=4.6MiB(34964txo)

After a "loadwallet" command I am getting an old / wrong balance. The
wallet is already empty because I had moved the funds to a new one 3 or 4
days ago but it is still showing the old  balance. I didn't receive any
warning message saying the need for a rescan or something like that.

I am trying the "rescanblockchain" command but it is running and it taking
a looooooooooooooong time.

Best regards,

Felipe.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/bitcoin-dev/attachments/20220820/07f9887b/attachment.html>