<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">If that were true then we wouldn't need
      to gzip large files before sending them over the internet.  Data
      compression generally helps transmission speed as long as the
      amount of compression is high enough and the time it takes is low
      enough to make it worthwhile.  On a corporate LAN it's generally
      not worthwhile unless you're dealing with very large files, but
      over a corporate WAN or the internet where network latency can be
      high it is IMO a worthwhile endevor.<br>
      <br>
      <br>
      <br>
      On 11/11/2015 10:49 AM, Marco Pontello wrote:<br>
    </div>
    <blockquote
cite="mid:CAE0pACK1-xQC4MsdbM46_Z0TQvZTrZKw4e8xFt3X=PmW7pmGJQ@mail.gmail.com"
      type="cite">
      <div dir="ltr">A random thought: aren't most communication over a
        data link already compressed, at some point?<br>
        <div>When I used a modem, we had the V.42bis protocol. Now,
          nearly all ADSL connections using PPPoE, surely are. And so
          on.</div>
        <div>I'm not sure another level of generic, data agnostic kind
          of compression will really give us some real-life practical
          advantage over that.</div>
        <div><br>
        </div>
        <div>Something that could take advantage of of special knowledge
          of the specific data, instead, would be an entirely different
          matter.</div>
        <div><br>
        </div>
        <div>Just my 2c.</div>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">On Wed, Nov 11, 2015 at 7:35 PM, Peter
          Tschipper via bitcoin-dev <span dir="ltr">&lt;<a
              moz-do-not-send="true"
              href="mailto:bitcoin-dev@lists.linuxfoundation.org"
              target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a></a>&gt;</span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div bgcolor="#FFFFFF" text="#000000">
              <div>Here are the latest results on compression ratios for
                the first 295,000 blocks, compressionlevel=6.  I think
                there are more than enough datapoints for statistical
                significance.  <br>
                <br>
                Results are very much similar to the previous test.  
                I'll work on getting a comparison between how much time
                savings/loss in time there is when syncing the
                blockchains: compressed vs uncompressed.  Still, I think
                it's clear that serving up compressed blocks, at least
                historical blocks, will be of benefit for those that
                have bandwidth caps on their internet connections.<br>
                <br>
                The proposal, so far is fairly simple:<br>
                1) compress blocks with some compression library:
                currently zlib but I can investigate other possiblities<br>
                2) As a fall back we need to advertise compression as a
                service.  That way we can turn off compression AND
                decompression completely if needed.<br>
                3) Do the compression at the datastream level in the
                code.  CDataStream is the obvious place.<br>
                <br>
                <br>
                Test Results:<span class=""><br>
                  <br>
                  range = block size range<br>
                  ubytes = average size of uncompressed blocks<br>
                  cbytes = average size of compressed blocks<br>
                  ctime = average time to compress<br>
                  dtime = average time to decompress<br>
                  cmp_ratio% = compression ratio<br>
                  datapoints = number of datapoints taken<br>
                  <br>
                  range       ubytes    cbytes    ctime    dtime   
                  cmp_ratio%    datapoints<br>
                </span> 0-250b      215            189    0.001   
                0.000    12.40             91280<br>
                250-500b    438            404    0.001    0.000   
                7.85             13217<br>
                500-1KB     761            701    0.001    0.000   
                7.86               11434<br>
                1KB-10KB    4149    3547    0.001    0.000     
                14.51             52180<br>
                10KB-100KB  41934    32604    0.005    0.001   
                22.25         82890<br>
                100KB-200KB 146303    108080    0.016    0.001   
                26.13    29886<br>
                200KB-300KB 243299    179281    0.025    0.002   
                26.31    25066<br>
                300KB-400KB 344636    266177    0.036    0.003   
                22.77    4956<br>
                400KB-500KB 463201    356862    0.046    0.004   
                22.96    3167<br>
                500KB-600KB 545123    429854    0.056    0.005   
                21.15    366<br>
                600KB-700KB 647736    510931    0.065    0.006   
                21.12    254<br>
                700KB-800KB 746540    587287    0.073    0.008   
                21.33    294<br>
                800KB-900KB 868121    682650    0.087    0.008   
                21.36    199<br>
                900KB-1MB   945747    726307    0.091    0.010   
                23.20    304<span class=""><br>
                  <br>
                  On 10/11/2015 8:46 AM, Jeff Garzik via bitcoin-dev
                  wrote:<br>
                </span></div>
              <div>
                <div class="h5">
                  <blockquote type="cite">
                    <div dir="ltr">Comments:
                      <div><br>
                      </div>
                      <div>1) cblock seems a reasonable way to extend
                        the protocol.  Further wrapping should probably
                        be done at the stream level.</div>
                      <div><br>
                      </div>
                      <div>2) zlib has crappy security track record.</div>
                      <div><br>
                      </div>
                      <div>3) A fallback path to non-compressed is
                        required, should compression fail or crash.</div>
                      <div><br>
                      </div>
                      <div>4) Most blocks and transactions have runs of
                        zeroes and/or highly common bit-patterns, which
                        contributes to useful compression even at
                        smaller sizes.  Peter Ts's most recent numbers
                        bear this out.  zlib has a dictionary (32K?)
                        which works well with repeated patterns such as
                        those you see with concatenated runs of
                        transactions.</div>
                      <div><br>
                      </div>
                      <div>5) LZO should provide much better
                        compression, at a cost of CPU performance and
                        using a less-reviewed, less-field-tested
                        library.</div>
                      <div><br>
                      </div>
                      <div><br>
                      </div>
                      <div><br>
                      </div>
                      <div><br>
                      </div>
                    </div>
                    <div class="gmail_extra"><br>
                      <div class="gmail_quote">On Tue, Nov 10, 2015 at
                        11:30 AM, Tier Nolan via bitcoin-dev <span
                          dir="ltr">&lt;<a moz-do-not-send="true"
                            href="mailto:bitcoin-dev@lists.linuxfoundation.org"
                            target="_blank">bitcoin-dev@lists.linuxfoundation.org</a>&gt;</span>
                        wrote:<br>
                        <blockquote class="gmail_quote" style="margin:0
                          0 0 .8ex;border-left:1px #ccc
                          solid;padding-left:1ex">
                          <div dir="ltr"><br>
                            <div class="gmail_extra"><br>
                              <div class="gmail_quote"><span>On Tue, Nov
                                  10, 2015 at 4:11 PM, Peter Tschipper <span
                                    dir="ltr">&lt;<a
                                      moz-do-not-send="true"
                                      href="mailto:peter.tschipper@gmail.com"
                                      target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:peter.tschipper@gmail.com">peter.tschipper@gmail.com</a></a>&gt;</span>
                                  wrote:<br>
                                  <blockquote class="gmail_quote"
                                    style="margin:0 0 0
                                    .8ex;border-left:1px #ccc
                                    solid;padding-left:1ex"> <span></span><span></span>There
                                    are better ways of sending new
                                    blocks, that's certainly true but
                                    for sending historical blocks and
                                    seding transactions I don't think
                                    so.  This PR is really designed to
                                    save bandwidth and not intended to
                                    be a huge performance improvement in
                                    terms of time spent sending.<span><br>
                                    </span></blockquote>
                                  <div><br>
                                  </div>
                                </span>
                                <div>If the main point is for historical
                                  data, then sticking to just blocks is
                                  the best plan.<br>
                                  <br>
                                </div>
                                <div>Since small blocks don't compress
                                  well, you could define a "cblocks"
                                  message that handles multiple blocks
                                  (just concatenate the block messages
                                  as payload before compression).  <br>
                                  <br>
                                  The sending peer could combine blocks
                                  so that each cblock is compressing at
                                  least 10kB of block data (or whatever
                                  is optimal).  It is probably worth
                                  specifying a maximum size for network
                                  buffer reasons (either 1MB or 1 block
                                  maximum).<br>
                                  <br>
                                </div>
                                <div>Similarly, transactions could be
                                  combined together and compressed
                                  "ctxs".  The inv messages could be
                                  modified so that you can request
                                  groups of 10-20 transactions.  That
                                  would depend on how much of an
                                  improvement compressed transactions
                                  would represent. <br>
                                  <br>
                                </div>
                                <div>More generally, you could define a
                                  message which is a compressed message
                                  holder.  That is probably to complex
                                  to be worth the effort though.<br>
                                </div>
                                <span>
                                  <div><br>
                                     </div>
                                  <blockquote class="gmail_quote"
                                    style="margin:0 0 0
                                    .8ex;border-left:1px #ccc
                                    solid;padding-left:1ex">
                                    <div bgcolor="#FFFFFF"
                                      text="#000000"><span>
                                        <blockquote type="cite">
                                          <div class="gmail_extra"><br>
                                            <div class="gmail_quote">On
                                              Tue, Nov 10, 2015 at 5:40
                                              AM, Johnathan Corgan via
                                              bitcoin-dev <span
                                                dir="ltr">&lt;<a
                                                  moz-do-not-send="true"
href="mailto:bitcoin-dev@lists.linuxfoundation.org" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a></a>&gt;</span>
                                              wrote:<br>
                                              <blockquote
                                                class="gmail_quote"
                                                style="margin:0 0 0
                                                .8ex;border-left:1px
                                                #ccc
                                                solid;padding-left:1ex">
                                                <div dir="ltr"><span>
                                                    <div
                                                      style="font-size:small">On

                                                      Mon, Nov 9, 2015
                                                      at 5:58 PM,
                                                      gladoscc via
                                                      bitcoin-dev <span
                                                        dir="ltr">&lt;<a
moz-do-not-send="true"
                                                          href="mailto:bitcoin-dev@lists.linuxfoundation.org"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a></a>&gt;</span>
                                                      wrote:<br>
                                                    </div>
                                                  </span>
                                                  <div
                                                    class="gmail_extra">
                                                    <div
                                                      class="gmail_quote"><span>
                                                        <div> </div>
                                                        <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0
                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div dir="ltr">I
                                                          think 25%
                                                          bandwidth
                                                          savings is
                                                          certainly
                                                          considerable,
                                                          especially for
                                                          people running
                                                          full nodes in
                                                          countries like
                                                          Australia
                                                          where internet
                                                          bandwidth is
                                                          lower and
                                                          there are data
                                                          caps.</div>
                                                        </blockquote>
                                                        <div><br>
                                                        </div>
                                                      </span>
                                                      <div>
                                                        <div
                                                          style="font-size:small;display:inline">​
                                                          This
                                                          reinforces the
                                                          idea that such
                                                          trade-off
                                                          decisions
                                                          should be be
                                                          local and
                                                          negotiated
                                                          between peers,
                                                          not a required
                                                          feature of the
                                                          network P2P.​</div>
                                                         </div>
                                                    </div>
                                                    <span>
                                                      <div><br>
                                                      </div>
                                                      -- <br>
                                                      <div>
                                                        <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div dir="ltr">
                                                          <div dir="ltr">
                                                          <div dir="ltr">
                                                          <div dir="ltr">
                                                          <div>Johnathan
                                                          Corgan<br>
                                                          Corgan Labs -
                                                          SDR Training
                                                          and
                                                          Development
                                                          Services</div>
                                                          <div><a
                                                          moz-do-not-send="true"
href="http://corganlabs.com" target="_blank"><a class="moz-txt-link-freetext" href="http://corganlabs.com">http://corganlabs.com</a></a><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                        </div>
                                                      </div>
                                                    </span></div>
                                                </div>
                                                <br>
_______________________________________________<br>
                                                bitcoin-dev mailing list<br>
                                                <a
                                                  moz-do-not-send="true"
href="mailto:bitcoin-dev@lists.linuxfoundation.org" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a></a><br>
                                                <a
                                                  moz-do-not-send="true"
href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev"
                                                  rel="noreferrer"
                                                  target="_blank"><a class="moz-txt-link-freetext" href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</a></a><br>
                                                <br>
                                              </blockquote>
                                            </div>
                                            <br>
                                          </div>
                                          <br>
                                          <fieldset></fieldset>
                                          <br>
                                          <pre>_______________________________________________
bitcoin-dev mailing list
<a moz-do-not-send="true" href="mailto:bitcoin-dev@lists.linuxfoundation.org" target="_blank">bitcoin-dev@lists.linuxfoundation.org</a>
<a moz-do-not-send="true" href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" target="_blank">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</a>
</pre>
                                        </blockquote>
                                        <br>
                                      </span></div>
                                  </blockquote>
                                </span></div>
                              <br>
                            </div>
                          </div>
                          <br>
_______________________________________________<br>
                          bitcoin-dev mailing list<br>
                          <a moz-do-not-send="true"
                            href="mailto:bitcoin-dev@lists.linuxfoundation.org"
                            target="_blank">bitcoin-dev@lists.linuxfoundation.org</a><br>
                          <a moz-do-not-send="true"
                            href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev"
                            rel="noreferrer" target="_blank">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</a><br>
                          <br>
                        </blockquote>
                      </div>
                      <br>
                    </div>
                    <br>
                    <fieldset></fieldset>
                    <br>
                    <pre>_______________________________________________
bitcoin-dev mailing list
<a moz-do-not-send="true" href="mailto:bitcoin-dev@lists.linuxfoundation.org" target="_blank">bitcoin-dev@lists.linuxfoundation.org</a>
<a moz-do-not-send="true" href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev" target="_blank">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</a>
</pre>
                  </blockquote>
                  <br>
                </div>
              </div>
            </div>
            <br>
            _______________________________________________<br>
            bitcoin-dev mailing list<br>
            <a moz-do-not-send="true"
              href="mailto:bitcoin-dev@lists.linuxfoundation.org">bitcoin-dev@lists.linuxfoundation.org</a><br>
            <a moz-do-not-send="true"
              href="https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev"
              rel="noreferrer" target="_blank">https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev</a><br>
            <br>
          </blockquote>
        </div>
        <br>
        <br clear="all">
        <div><br>
        </div>
        -- <br>
        <div class="gmail_signature">Try the Online TrID File Identifier<br>
          <a moz-do-not-send="true"
            href="http://mark0.net/onlinetrid.aspx" target="_blank">http://mark0.net/onlinetrid.aspx</a></div>
      </div>
    </blockquote>
    <br>
  </body>
</html>