<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi Jeff<div class=""><br class=""></div><div class="">There where previous discussions about similar approaches [1] [2].</div><div class=""><br class=""></div><div class="">I’m not sure if compression should be built into the protocol.</div><div class="">My humble understanding of it, is, that it should be built into different layers.</div><div class=""><br class=""></div><div class="">If bandwidth is a concern, then on the fly gzip compression like apaches mod_deflate could be something. But I expect fast propagation is often more important then a ~30% bandwidth reduction.</div><div class="">Bandwidth may be a concern for historical blocks transmission. If you continue the proposal, I think you should focus on historical blocks.</div><div class=""><br class=""></div><div class="">If disk space is a concern, then the database layer should handle the compression.</div><div class=""><br class=""></div><div class="">Thanks</div><div class="">—</div><div class=""></jonas></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">[1] <a href="https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-November/011692.html" class="">https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-November/011692.html</a></div><div class="">[2] <a href="https://github.com/bitcoin/bitcoin/pull/6973" class="">https://github.com/bitcoin/bitcoin/pull/6973</a></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div><blockquote type="cite" class=""><div class="">Am 26.11.2017 um 16:11 schrieb Jeff Johnson via bitcoin-dev <<a href="mailto:bitcoin-dev@lists.linuxfoundation.org" class="">bitcoin-dev@lists.linuxfoundation.org</a>>:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="">I'm new to this mailing list and apologize if this has been suggested before. I was directed from the Bitcoin core github to this mailing list for suggestions.</div><div class=""><br class=""></div>I'd just like to post a possible solution that increases the amount of data in a block without actually increasing the size on disk or the size in memory or the size transmitted over the Internet. Simply applying various compression algorithms, I was able to achieve about a 50% compression ratio. Here are my findings on a recent Bitcoin block using max compression for all methods:<div class=""><br class=""></div><div class="">Raw block</div><div class="">998,198 bytes</div><div class=""><br class=""></div><div class="">Gzip</div><div class="">521,212 bytes (52% ratio)</div><div class="">(needs 2MB to decompress).</div><div class=""><br class=""></div><div class="">LZMA</div><div class="">415,308 bytes (41% ratio)</div><div class=""> (1MB dictionary, needs 3MB to decompress)</div><div class=""><br class=""></div><div class="">- ZStandard: 469,179 bytes (47% ratio)</div><div class="">(1MB memory to decompress)</div><div class=""><br class=""></div><div class="">- LZ4: 641,063 bytes (64% ratio)</div><div class="">(32-64K to decompress)</div><div class=""><br class=""></div><div class="">The compression time on my modest laptop (2 years old) was "instant". I ran all from the command line and did not notice any lag as I pressed enter to do the compression, so easily less than a second. But compression time doesn't matter, decompression time is what matters as blocks will be decompressed billions of times more than they will be compressed. Decompression speed for LZ4 is the fastest of the above methods, at 3.3GB / second, slightly less than half the speed of memcpy, see char at (<a href="https://github.com/lz4/lz4" class="">https://github.com/lz4/lz4</a>).</div><div class=""><br class=""></div><div class="">If decompression speed, CPU and memory usage is a concern, LZ4 is a no brainer. You basically get a 33% larger block size for "free". But ZStandard, in my opinion, makes the most sense as it offers greater than 50% compression ratio with a very good decompression ratio of 900MB / second.</div><div class=""><br class=""></div><div class="">If this were implemented in the Bitcoin protocol, there would need to be a place to specify the compression type in a set of bits somewhere, so that future compression algorithms could potentially be added.<br class=""></div><div class=""><br class=""></div><div class="">Miners could do nothing and keep sending blocks as is, and these blocks would have "no compression" as the type of compression, just as today. Or they could opt in to compress blocks and choose how many transactions they want to stuff into the block, keeping the compressed size under the limit.</div><div class=""><br class=""></div><div class="">The bitcoin client code would also need to be able to handle the appropriate compression bits, and limits of signature data, etc. modified to deal with the compression.</div><div class=""><br class=""></div><div class="">I understand schnorr signatures are on the roadmap as a 25% compression gain which is great, I suspect that schnorr signatures would compress even further when compressed with the above compression methods.</div><div class=""><br class=""></div><div class="">Here is a link to the block that I compressed: <a href="https://mega.nz/#!YPIF2KTa!4FxxLvqzjqIftrkhXwSC2h4G4Dolk8dLteNUolEtq98" class="">https://mega.nz/#!YPIF2KTa!4FxxLvqzjqIftrkhXwSC2h4G4Dolk8dLteNUolEtq98</a><br class=""></div><div class=""><span class="gmail-key gmail-file-link-info" style="font-size:13px;outline:none;color:rgb(217,0,7);font-family:source_sans_proregular,Arial;white-space:pre-wrap"><br class=""></span></div><div class="">Thanks for reading, best wishes to all.</div><div class=""><br class=""></div><div class="">-- Jeff Johnson<br class=""></div></div>
_______________________________________________<br class="">bitcoin-dev mailing list<br class=""><a href="mailto:bitcoin-dev@lists.linuxfoundation.org" class="">bitcoin-dev@lists.linuxfoundation.org</a><br class="">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev<br class=""></div></blockquote></div><br class=""></div></body></html>