[Bitcoin-development] getmemorypool BIP process

[Geir Harald Hansen]

On 04.03.2012 02:04, Luke-Jr wrote:
> On Saturday, March 03, 2012 6:51:34 PM Geir Harald Hansen wrote:
>> Long polling as currently implemented in pools has a race condition.
>> Does the miner reconnect first or does another block change happen
>> first? "Double" block changes are common with merged mining and I'm
>> doing all sorts of tricks in my pool backend to reduce this problem.
> 
> How would you suggest addressing this? I presume if a share solves blocks on 
> multiple chains, you just longpoll once when that's successful?

If the client doesn't tell you which block change it saw last, you can't
avoid this problem completely.

>> How about another entry "longpollid" in long poll responses. The last
>> seen longpollid should be included by the client in future long poll
>> requests. This enables the server to see if the client has missed any
>> block changes. The ID could perhaps be submitted in an HTTP header
>> (X-LongPollID?) if we wish to keep the JSON-RPC params empty, or params
>> could hold an object with a key "longpollid". Could be a string or
>> number, like "workid".
> 
> Hmm, the problem is that adding any parameters to getmemorypool itself breaks 
> compatibility with bitcoind 0.5, and using HTTP headers makes it HTTP-specific 
> again. Any ideas?

I think long polling and reconnecting to LP is an HTTP-specific problem
anyway? So it may be best to go with an HTTP header. See below for an
idea for a capabilities reporting system, though, which would allow
putting this data into JSON.

>> Another useful value in the getmemorypool response would be "height", so
>> the miner can include the correct height in the coinbase. I would like
>> that in bitcoind as well. One JSON-RPC call instead of two, and no race
>> condition between getmemorypool and getblocknumber.
> 
> Good catch. Should this be required (since it might be necessary for future 
> Bitcoin blocks), or just "should" for compatibility?

I think "should" makes sense.

>> It should be explained how target vs. fulltarget works.
> 
> What is unclear about this?

Reading through it again it is mostly clear except for one thing, how to
submit "share/merkle" style. It doesn't say exactly how to submit this data.

It says merkle tree, but a merkle branch should suffice. Also, I believe
the coinbase must always be the first transaction of a block? So we
won't need the transaction's index in the merkle tree (it's always index
zero). This would then be just an array of sha256 hashes making up the
merkle branch for the coinbase.

I'd prefer if block header, coinbase tx and merkle branch were submitted
in 3 pieces. When merged mining and submitting a block generating proof
to the aux backend you put an auxiliary proof of work between the block
header and transactions. So the header and transactions are not
necessarily adjacent. Also, there's no merkle branch in a regular
bitcoin block.

>> Perhaps some things should be optional for a client to implement?
> 
> Doing this safely needs some way for clients to communicate capabilities to 
> the server, which has the problem of passing parameters to getmemorypool.

Let's add an optional "servercapabilities" key on the server side, the
value being a space-delimited list of what's supported by the server. If
that list contains "clientcapabilities" then the client may, instead of
sending an empty request, send an object where one of the keys is
"clientcapabilities", and the value is a space-delimited list of extensions.

Those names are a bit long, maybe "caps" is better.

Ok, this is not entirely clean. For the first request the client won't
know what the server supports. For subsequent requests I guess always
assume the last caps the server reported.

>> I think "noncerange" is of limited use and there's a good chance of getting
>> the endianness wrong.
> 
> There is no mining hardware to date that exhausts even half the nonce space, 
> so I'd really prefer to see this as a required feature on the miner side. On 
> the other hand, it's merely an extension for getwork, so I can see the problem 
> so long as we're using getwork proxies.

Yes, you can end up getting half a nonce range from an upstream server,
and not being able to make use of it because your downstream clients
don't support it.

A single GPU can make full use of the noncerange if you allow the
timestamp to slip a few seconds into the past. I believe most miners do
that today.

Also worth including are the "hostlist" and "switchto" getwork
extensions. I think those can fit right into the JSON response, to keep
with being transport-agnostic, but otherwise look like they do for getwork.

Regards,
Geir H. Hansen