<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">If that were true then we wouldn't need
to gzip large files before sending them over the internet. Data
compression generally helps transmission speed as long as the
amount of compression is high enough and the time it takes is low
enough to make it worthwhile. On a corporate LAN it's generally
not worthwhile unless you're dealing with very large files, but
over a corporate WAN or the internet where network latency can be
high it is IMO a worthwhile endevor.<br>
<br>
<br>
<br>
On 11/11/2015 10:49 AM, Marco Pontello wrote:<br>
</div>
<blockquote
cite="mid:CAE0pACK1-xQC4MsdbM46_Z0TQvZTrZKw4e8xFt3X=PmW7pmGJQ@mail.gmail.com"
type="cite">
<div dir="ltr">A random thought: aren't most communication over a
data link already compressed, at some point?<br>
<div>When I used a modem, we had the V.42bis protocol. Now,
nearly all ADSL connections using PPPoE, surely are. And so
on.</div>
<div>I'm not sure another level of generic, data agnostic kind
of compression will really give us some real-life practical
advantage over that.</div>
<div><br>
</div>
<div>Something that could take advantage of of special knowledge
of the specific data, instead, would be an entirely different
matter.</div>
<div><br>
</div>
<div>Just my 2c.</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Nov 11, 2015 at 7:35 PM, Peter
Tschipper via bitcoin-dev <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:bitcoin-dev@lists.linuxfoundation.org"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Here are the latest results on compression ratios for
the first 295,000 blocks, compressionlevel=6. I think
there are more than enough datapoints for statistical
significance. <br>
<br>
Results are very much similar to the previous test.
I'll work on getting a comparison between how much time
savings/loss in time there is when syncing the
blockchains: compressed vs uncompressed. Still, I think
it's clear that serving up compressed blocks, at least
historical blocks, will be of benefit for those that
have bandwidth caps on their internet connections.<br>
<br>
The proposal, so far is fairly simple:<br>
1) compress blocks with some compression library:
currently zlib but I can investigate other possiblities<br>
2) As a fall back we need to advertise compression as a
service. That way we can turn off compression AND
decompression completely if needed.<br>
3) Do the compression at the datastream level in the
code. CDataStream is the obvious place.<br>
<br>
<br>
Test Results:<span class=""><br>
<br>
range = block size range<br>
ubytes = average size of uncompressed blocks<br>
cbytes = average size of compressed blocks<br>
ctime = average time to compress<br>
dtime = average time to decompress<br>
cmp_ratio% = compression ratio<br>
datapoints = number of datapoints taken<br>
<br>
range ubytes cbytes ctime dtime
cmp_ratio% datapoints<br>
</span> 0-250b 215 189 0.001
0.000 12.40 91280<br>
250-500b 438 404 0.001 0.000
7.85 13217<br>
500-1KB 761 701 0.001 0.000
7.86 11434<br>
1KB-10KB 4149 3547 0.001 0.000
14.51 52180<br>
10KB-100KB 41934 32604 0.005 0.001
22.25 82890<br>
100KB-200KB 146303 108080 0.016 0.001
26.13 29886<br>
200KB-300KB 243299 179281 0.025 0.002
26.31 25066<br>
300KB-400KB 344636 266177 0.036 0.003
22.77 4956<br>
400KB-500KB 463201 356862 0.046 0.004
22.96 3167<br>
500KB-600KB 545123 429854 0.056 0.005
21.15 366<br>
600KB-700KB 647736 510931 0.065 0.006
21.12 254<br>
700KB-800KB 746540 587287 0.073 0.008
21.33 294<br>
800KB-900KB 868121 682650 0.087 0.008
21.36 199<br>
900KB-1MB 945747 726307 0.091 0.010
23.20 304<span class=""><br>
<br>
On 10/11/2015 8:46 AM, Jeff Garzik via bitcoin-dev
wrote:<br>
</span></div>
<div>
<div class="h5">
<blockquote type="cite">
<div dir="ltr">Comments:
<div><br>
</div>
<div>1) cblock seems a reasonable way to extend
the protocol. Further wrapping should probably
be done at the stream level.</div>
<div><br>
</div>
<div>2) zlib has crappy security track record.</div>
<div><br>
</div>
<div>3) A fallback path to non-compressed is
required, should compression fail or crash.</div>
<div><br>
</div>
<div>4) Most blocks and transactions have runs of
zeroes and/or highly common bit-patterns, which
contributes to useful compression even at
smaller sizes. Peter Ts's most recent numbers
bear this out. zlib has a dictionary (32K?)
which works well with repeated patterns such as
those you see with concatenated runs of
transactions.</div>
<div><br>
</div>
<div>5) LZO should provide much better
compression, at a cost of CPU performance and
using a less-reviewed, less-field-tested
library.</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Nov 10, 2015 at
11:30 AM, Tier Nolan via bitcoin-dev <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:bitcoin-dev@lists.linuxfoundation.org"
target="_blank">bitcoin-dev@lists.linuxfoundation.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote"><span>On Tue, Nov
10, 2015 at 4:11 PM, Peter Tschipper <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:peter.tschipper@gmail.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:peter.tschipper@gmail.com">peter.tschipper@gmail.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex"> <span></span><span></span>There
are better ways of sending new
blocks, that's certainly true but
for sending historical blocks and
seding transactions I don't think
so. This PR is really designed to
save bandwidth and not intended to
be a huge performance improvement in
terms of time spent sending.<span><br>
</span></blockquote>
<div><br>
</div>
</span>
<div>If the main point is for historical
data, then sticking to just blocks is
the best plan.<br>
<br>
</div>
<div>Since small blocks don't compress
well, you could define a "cblocks"
message that handles multiple blocks
(just concatenate the block messages
as payload before compression). <br>
<br>
The sending peer could combine blocks
so that each cblock is compressing at
least 10kB of block data (or whatever
is optimal). It is probably worth
specifying a maximum size for network
buffer reasons (either 1MB or 1 block
maximum).<br>
<br>
</div>
<div>Similarly, transactions could be
combined together and compressed
"ctxs". The inv messages could be
modified so that you can request
groups of 10-20 transactions. That
would depend on how much of an
improvement compressed transactions
would represent. <br>
<br>
</div>
<div>More generally, you could define a
message which is a compressed message
holder. That is probably to complex
to be worth the effort though.<br>
</div>
<span>
<div><br>
</div>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000"><span>
<blockquote type="cite">
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Tue, Nov 10, 2015 at 5:40
AM, Johnathan Corgan via
bitcoin-dev <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:bitcoin-dev@lists.linuxfoundation.org" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a></a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div dir="ltr"><span>
<div
style="font-size:small">On
Mon, Nov 9, 2015
at 5:58 PM,
gladoscc via
bitcoin-dev <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:bitcoin-dev@lists.linuxfoundation.org"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a></a>></span>
wrote:<br>
</div>
</span>
<div
class="gmail_extra">
<div
class="gmail_quote"><span>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div dir="ltr">I
think 25%
bandwidth
savings is
certainly
considerable,
especially for
people running
full nodes in
countries like
Australia
where internet
bandwidth is
lower and
there are data
caps.</div>
</blockquote>
<div><br>
</div>
</span>
<div>
<div
style="font-size:small;display:inline">
This
reinforces the
idea that such
trade-off
decisions
should be be
local and
negotiated
between peers,
not a required
feature of the
network P2P.</div>
</div>
</div>
<span>
<div><br>
</div>
-- <br>
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Johnathan
Corgan<br>
Corgan Labs -
SDR Training
and
Development
Services</div>
<div><a
moz-do-not-send="true"
href="http://corganlabs.com" target="_blank"><a class="moz-txt-link-freetext" href="http://corganlabs.com">http://corganlabs.com</a></a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</span></div>
</div>
<br>
_______________________________________________<br>
bitcoin-dev mailing list<br>
<a
moz-do-not-send="true"
href="mailto:bitcoin-dev@lists.linuxfoundation.org" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a></a><br>
<a
moz-do-not-send="true"
href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev"
rel="noreferrer"
target="_blank"><a class="moz-txt-link-freetext" href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</a></a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
bitcoin-dev mailing list
<a moz-do-not-send="true" href="mailto:bitcoin-dev@lists.linuxfoundation.org" target="_blank">bitcoin-dev@lists.linuxfoundation.org</a>
<a moz-do-not-send="true" href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" target="_blank">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</a>
</pre>
</blockquote>
<br>
</span></div>
</blockquote>
</span></div>
<br>
</div>
</div>
<br>
_______________________________________________<br>
bitcoin-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:bitcoin-dev@lists.linuxfoundation.org"
target="_blank">bitcoin-dev@lists.linuxfoundation.org</a><br>
<a moz-do-not-send="true"
href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev"
rel="noreferrer" target="_blank">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
bitcoin-dev mailing list
<a moz-do-not-send="true" href="mailto:bitcoin-dev@lists.linuxfoundation.org" target="_blank">bitcoin-dev@lists.linuxfoundation.org</a>
<a moz-do-not-send="true" href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" target="_blank">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</a>
</pre>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
bitcoin-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a><br>
<a moz-do-not-send="true"
href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev"
rel="noreferrer" target="_blank">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail_signature">Try the Online TrID File Identifier<br>
<a moz-do-not-send="true"
href="http://mark0.net/onlinetrid.aspx" target="_blank">http://mark0.net/onlinetrid.aspx</a></div>
</div>
</blockquote>
<br>
</body>
</html>