[Bitcoin-development] Revisiting the BIPS process, a proposal

[Jeremy Spilman]

Thanks Christian, this is a really interesting bit of history. My own  
personal experience from when I wrote my own client and BCCAPI-ish server  
was that the protocol specification on the Wiki was hugely valuable, and  
rarely sent me astray. Supplement that with the occasional questions on  
#bitcoin-dev, and then just coding, coding, coding and getting unit tests  
to pass.

Nothing compares (IMO) to stepping through your own code watching the unit  
tests run, scripts evaluate, calculating transaction hashes for the  
different SIGHASH modes, and finally getting your first transaction into  
the block chain. I really appreciated the .json files holding the unit  
test data, which were easy to load into my own test harness, the tables on  
the Wiki showing what the stack should look like at each point in a script  
execution, and the diagrams showing transaction signing.

Bitcoin takes some time to "grok" when you first approach; more than a  
day, less than a month, and really no amount of reading documentation or  
specs will get you to that "ah ha" moment. When the fog lifts and the  
blockchain, scripting, signing, and wallet handling really click, suddenly  
the bitcoind code (and many other great public sources in just about any  
language you could want) actually does starts to feel fairly simple and  
obvious. But it certainly doesn't start out that way on day one.

I think the majority of client code development is actually people writing  
'agents' not end-user P2P wallets, and they tend to be written to connect  
to a single bitcoind acting as a proxy to the network. Even some end-user  
wallets work this way! As such, I spent very little time in my own client  
writing P2P protocol code, no peer discovery code, no anti-DoS, etc.  
Clients like this also don't pose much systemic risk, because they don't  
mine, they don't connect directly to external nodes, etc. They can  
certainly be used to "cause trouble" though, but so can  
'sendrawtransaction'.

I chose to speak the P2P protocol to bitcoind versus using some of the  
other options like ZeroMQ, but it still didn't take long to get headers,  
blocks, and transactions downloading. I remember getting stuck on the very  
first version message, because of missing the checksum and user-agent or  
something caused the latest bitcoind to just ignore me. A little wireshark  
capture of the exchange between two working bitcoind instances cleared it  
right up. I didn't mind the leg work, I don't think everything needs to be  
spoon fed, and it's certainly not purposefully obfuscated. Maybe one  
exception is the mix-matched endianness will throw you off, especially if  
you are developing on LE! Anyway, I have huge respect for how much effort  
it takes to keep even small bits of documentation up-to-date. For as  
"slow" as bitcoin moves, it's actually moving incredibly fast.

Finally, the bitcoind console and debug logs, as well sites like  
blockchain.info and blockexplorer.com are hugely helpful for debugging raw  
and live transactions for when you get stuck. There's a surprisingly large  
tooling and support ecosystem out there.

Moral of the story, I think, is everything you need is there. No, it's not  
all in one place. Yes, it takes time to find it and assimilate all that  
knowledge. It also really helps that the community is extremely willing to  
help and answer technical questions, and point you in the right direction,  
even when you're working on your own private client code. The IRC channel  
can certainly be intimidating because it seems like every time I hit enter  
to send a question, gmaxwell's respond 300ms later would invoke an  
immediate forehead slap and a groan of "shit, I knew/should have known  
that, now I feel dumb" ;-) but if you're working on bitcoin, you better  
get used to not being the smartest person in the room! The responses I got  
were never arrogant or disparaging, but they were straight to-the-point  
and surprisingly high quality. Ain't no slouches in that channel, yes you  
will have to bring your A-game and you are expected to have "tried first"  
before just asking. I have fairly limited experience working on open  
source projects, but I'm extremely happy with my experience with the  
Bitcoin community and found writing Bitcoin code hugely enjoyable.

The flip side to helping people implement their own clients, agents, or  
even miners, is helping people to contribute pulls requests, or at the  
very highest echelon, a BIP. If you haven't written any significant  
Bitcoin code, you might want to consider investing in that first before  
submitting a BIP. :-)

For a BIP to be valuable, often it requires widespread or even consensus  
adoption. BIPs are probably not the place to toss just any old 'good idea'  
because BIPs impose a cost on all active developers. I want to read and  
understand 'all the BIPs' because for the most part they are actually  
essential, like, how to handle duplicate transactions in BIP30 - if you  
don't read BIP30 you very likely totally miss that, until your code throws  
exceptions while processing block 91842.

And perhaps the hardest kind of BIP of all is the "lets get wallets to add  
this user-facing feature" where it has no bearing on the blockchain or  
transaction processing, it doesn't make the network more resilient or add  
crucial functionality for increasing scalability. Kind of like JPK's HD  
wallet encryption proposal, which I love, and I tried to contribute to in  
the forums, but I can totally understand the headwinds for making progress  
on BIPs like that one and BIP39. No one is against it per-say, it's just  
much harder to articulate and justify the NEED for everyone to implement,  
test, and support this new not-yet-standard, nice-to-have feature. For  
those kinds of BIPs you probably have to go out and get some wallets to  
implement it, or implement it yourself, to prove the value and kick start  
critical mass before you will even get enough support for getting a BIP  
number assigned. IMO, it's not a Bad Thing.

TL;DR; The current support systems worked very well for me. I was able to  
accomplish all my goals, and I would even say it was a pleasure. Keep a  
high bar for assigning BIP numbers. And I hope to be able to jump back in  
and do more with Bitcoin soon.

Thanks all, sorry if I'm rambling,
Jeremy Spilman

On Thu, 24 Oct 2013 04:11:05 -0700, Christian Decker  
<decker.christian at gmail.com> wrote:

> I'd like to add some historical background about how the "protocol
> specification" came to be in the first place.
>
> A bit over three years [1] ago I started an attempt to document the
> network protocol, by reverse engineering it from the satoshi
> client. My goal, back then, was to enable like-minded engineers to
> create alternative clients and move away from the client-monoculture
> that is still predominant today. It was clear from the beginning that
> it would merely be a reverse engineering effort, and that it would
> likely lag a bit behind the changes in the main client. It was meant
> as a help for engineers that are not well versed in C/C++ to enable
> them to contribute by creating new clients, but the satoshi client
> would always be the de-facto standard.
>
> With the move from Google Code to the Bitcoin.it wiki somehow this
> notion of it being a reverse engineering effort was lost and people
> started assuming that if the behavior of the satoshi client did not
> match the protocol description it was a bug on the client
> side. Instead it is because the reverse engineering of the protocol is
> incorrect or simply missing some details. Although the protocol
> description is far more complete than it was back when we started, I
> still don't feel comfortable giving it the name specification.
>
> I still believe that a client monoculture is bad for the system as a
> whole, because a single bug might bring down the whole network. Giving
> people the necessary tools to implement new clients brings
> stability. I do understand the criticism that writing a specification
> might hinder future development as it restricts the possible changes
> to the protocol, but isn't this already the case as long as we have
> legacy versions of the client participating in the network? I would
> also argue that having a specification allows an application
> independent review of the protocol to identify possible improvements
> and bugs.
>
> I think the protocol description has an important place in the
> development of Bitcoin, so much so that we pushed a long time ago to
> separate protocol version from the client version. I would love to see
> the protocol specification becoming official part of the bitcoin
> github repository, which would ideally be maintained alongside the
> satoshi client to keep it up to date.
>
> Regards,
> Christian Decker
>
> [1] https://bitcointalk.org/index.php?topic=231
> --
> Christian Decker
>
>