-
Notifications
You must be signed in to change notification settings - Fork 3
Bitcoin Mining Pool Developer's Reference
This document contains my notes about how mining pools work, and what specific extensions have been implemented to make the pool more efficient. It's most useful for other developers interested in hand-crafting their own pools or attempting to understand the implementation details of Bitpool.
This section details the core essentials of mining pools, before any other bells and whistles (such as long polling) have been added. Implement this first, because the pool won't function otherwise.
A mining pool involves injecting a middleware piece in between the miner and the Bitcoin daemon. To do this, you need to understand what the miner's requests look like.
Line breaks have been added to the examples to improve readability.
A miner starts by sending a Getwork request. This JSON-RPC request looks like:
{
"id": 1,
"method": "getwork",
"params": []
}
The bitcoin server will then respond with "work". The JSON response for an initial Getwork request looks like:
{
"result": {
"midstate":"...",
"data":"...",
"hash1":"...",
"target":"..."
},
"error": null,
"id": 1
}
The "..." for each value represents real data, which is too lengthy to display here. Each of these is a hex String with the following dimensions:
- the
midstatefield is 64 characters long - the
datafield is 256 characters long - the
hash1field is 128 characters long - the
targetfield is 64 characters long
To retrieve a real Getwork response from your Bitcoin daemon, use the following command:
curl --user user:pass --data-binary '{ "id":"curltest", "method":"getwork", "params":[] }' \
-H 'content-type: text/plain;' http://localhost:8332/ -i
Note the errors field in the response object. If no error has occurred, this value will be null. However, if something goes horribly wrong, the error field will contain a JSON object. The entire response will then look similar to:
{
"result": null,
"error": {
"code": -1,
"message": "value is type int, expected str"
}
"id": 1
}
The code field is a numeric error code, while the message field is a human-readable String value. (Note: if using the Ruby bitcoin-client gem, errors will be raised as runtime errors, so you can rescue them like any other Ruby error rather than parsing out this object.)
When the client has found a potential completion hash, it will send the data up to the server with a request very similar to its initial Getwork. This request is also frequently referred to, confusingly, as Getwork. I prefer to call it a Getwork Completion just so there's no confusion. The only difference between a Getwork and a Getwork Completion is that the latter will send a single parameter, which is the hash itself, sent as a string.
{
"id": 1,
"method":"getwork",
"params": ["a791feeb00ae7fce4c7147b7bcbb525616a93552e7e96c45b48cc56aed3311c1"]
}
The Getwork Completion response from the Bitcoin daemon is very simply a true or false value. If the value is true, then the hash was accepted and a new block has been created! If the value is false then the hash was rejected. An example response is as follows:
{
"version": "1.1",
"id": 1,
"error": null,
"result": false
}
A mining pool is essentially a glorified proxy. It is a piece of middleware that sits between the Bitcoin daemon and the individual miners. It receives the Getwork and Getwork Completion requests from the miners, delegates them into the daemon, and analyzes the daemon's response. Usually, the mining pool will alter the response somewhat.
When an initial Getwork is received by the mining pool, it is dispatched into the daemon. As you've seen, the result will include a target. The target represents the difficulty of the hash that needs to be generated. The lower the target, the more difficult (which directly results in a higher difficulty) the hash will be to generate.
When a hash is successfully generated whose Checkwork value is less than the target value, the miner has successfully produced a Block.
Pools usually send payouts in direct relation to the amount of work actually performed, so that faster machines requiring more power to operate receive a greater portion of the profits than their slower, cheaper counterparts. In order to measure the participation of each miner in the pool, a unit of measurement is required. This unit is usually called a Share. Shares are identical to Blocks, except that the pool arbitrarily raises the target field so that a valid Share is much easier to produce than an actual Block. In this way, any given Share may or may not be a valid Block.
After determining that the share is valid by calculating its Checkwork and making sure that the result is less than the value of the pool-predetermined target, the share is sent to the Bitcoin daemon to see if it is actually a valid Block.
Only the mining pool need be aware of the actual, real target. Usually, this information is not sent to the miner. Instead, as mentioned, the target field of the daemon's response is overwritten by a specially-tailored target. Most mining pools set the target to the maximum achievable value, equal in hex to 0x00000000ffff0000000000000000000000000000000000000000000000000000. This allows the miners to generate hashes using the lowest possible difficulty: 1. (As a side note, you can always calculate the current difficulty by dividing the maximum target by the current, real target.)
Different pool operators may increase the difficulty (by lowering the target) to force the miners to produce more difficult hashes; this has the effect of reducing bandwidth consumption, since it will take more processing power (and therefore, more time) to produce a valid Share.
Pools can also calculate how close they are to a likely Block based on how many Shares have been received. Given a difficulty of 1805700.83619367, the pool can expect to receive one Block per every 1805700.83619367 Shares. Obviously, altering the target downward will have an effect upon this formula since the pool will be receiving fewer shares overall, and pool operators should take that into account.
Sometimes, the server will receive a Getwork Completion referring to a block which has already been solved, or a Share that is identical to one already in the database. These completions are invalid because they are either outdated or someone else has already solved them. They are referred to as "stale". Stale Shares arise due to network latency, machine slowness, and general bad luck. They tend to be encountered most often when a Block has been completed but the miner has not yet received the latest Getwork.
Stale Shares are not saved, and their owners do not receive credit for them, since the work has already been completed or may in fact not even be pertinent to the current Block.
Usually, mining pools also alter the response from a Getwork Completion. For most Shares, the response from the Bitcoin daemon will be false, to indicate that the Share was not a valid Block. However, the mining pool will usually switch this to true, to indicate to the miner that the Share was itself accepted by the pool. This way, false values can represent stale or otherwise invalid shares.
When a Share is validated and the Bitcoin daemon confirms that the Share is in fact a valid Block, the pool must make a note of who owns how many shares in the Block, making exceptions for the possibility that some Shares may in fact belong to the next Block. (This is easily determined via a getblockcount JSON-RPC call to the Bitcoin daemon.) Any Share whose height (that is, the block count at the time of the generation of the Share) is less than or equal to the height of the freshly-minted Block should be counted towards the current Block. Credits are awarded to the various accounts in proportion to the number of Shares each account holds.
After creating the appropriate Credits, the corresponding Shares should be deleted so that they are not mistakenly credited again when the next Block has been found.
In addition, the mining pool might create an additional Credit for the account which actually produced the winning hash. This is entirely implementation-dependent and may or may not be used in determining payouts or searching for suspicious activity.
There are a number of extensions that have been made throughout the Bitcoin mining community which have helped to make mining pools considerably more efficient.
If a miner supports any or all of these extensions, it should advertise as much by sending an X-Mining-Extensions header with its requests, which contains a space-delimited list of extension names. Examples:
X-Mining-Extensions: hostlist
X-Mining-Extensions: hostlist longpoll noncerange reject-reason rollntime switchto
Below, each extension is detailed. Items marked with an asterisk (*) have not yet been implemented in Bitpool.
One of the major weaknesses of a mining pool is that the server can go down -- or more likely, be brought down by an attack. In an attempt to work around this issue, a pool server may return a hostlist. The idea is that if the primary server should go down, the miner can try one of the other servers listed in the hostlist until it comes back up.
The hostlist is signified by the X-Host-List header, whose value is a JSON array containing a set of generic objects. Following is an example of X-Host-List:
X-Host-List: [{"host":"server.tld","port":8332,"ttr":0},{"host":"backup.tld","port":8332,"ttr":20}]
A miner is expected to attempt to connect to the various hosts in left-to-right order until it can maintain a connection.
The ttr field stands for "time to return". If set to zero, the miner can continue using the host indefinitely. Otherwise, the ttr is a timeout in minutes, after which the miner is expected to return to a host whose ttr is equal to 0.
Traditionally, miners will send a new Getwork request after a set delay, usually between 5 and 10 seconds. Long polling improves performance, reduces bandwidth consumption and helps mitigate the occurrence of Stale Shares.
The process of Long Polling is outlined below:
- The miner initiates a Getwork request as per usual.
- If the mining pool supports Long Polling, its response should contain an
X-Long-Pollingheader. The value of this header is the location of the Long Polling host. This may be a relative path, or a full URL including the port number. - The miner opens a second connection to the server, this time on the path referred to by the
X-Long-Pollingheader. The miner should use the same HTTP Basic Auth credentials as were used for the primary mining connection. - If a Block is solved, all miners listening on the Long Polling connection will receive a Getwork response for the next Block. The format of this response is identical to one received on the corresponding primary connection. Upon receiving this response, miners should immediately discard all work being performed, and instead begin working on this new Getwork response. The miner should also reopen the Long Polling connection in preparation for the completion of the next Block.
- If 60 seconds have elapsed without a response from the server, or if the miner exhausts all of its nonce space, the miner should perform a new Getwork request via the main connection in order to allow for new transactions which may have been added to the Block. Then the miner should reopen the Long Polling connection if it has been closed.
- Most mining pools will disable the timeout on the Long Polling connection so that it cannot time out automatically. However, if it does, the Long Polling connection should be reopened as soon as possible. It would also be wise to perform a new Getwork request on the primary connection to make sure the latest work hasn't been missed.
Nonce range allows the mining pool to try to guess how much work is adequate for a given miner, depending upon the miner's hash rate. The mining pool can break a single Getwork response into multiple chunks, sending the same response to multiple different miners with the addition of an X-Nonce-Range header. This header is used to inform the miner of just how many hashes to generate before stopping midstream and requesting new work. By splitting a single Getwork amongst numerous miners, the mining pool can:
- reduce the number of true Getwork requests made to the daemon which reduces overall bandwidth consumption
- iterate through all the nonces in a given Getwork response more quickly by performing hash generation in parallel
- greatly improve the efficiency of generating work when combined with
X-Roll-Ntime
If a nonce range is supported by the miner, it should send an additional header, X-Mining-Hashrate, whose value is the miner's expected hash rate in full hashes per second (an integer).
The server may then use this information to calculate a nonce range in the form of two 32-bit numbers, given in big endian format and inserted into the Getwork response as the "noncerange" field. The first number represents the starting nonce and the second represents the ending nonce. The miner will then iterate through the given range.
If used in combination with X-Roll-Ntime, the miner should iterate through the entire nonce range, increment the timestamp by 1 second, and start over at the first nonce again. This process is repeated until it is time to send a new Getwork request.
Otherwise, the miner should request new work after the nonce range has been exhausted.
Example JSON response including the noncerange field:
{
"result": {
"midstate":"...",
"data":"...",
"hash1":"...",
"target":"...",
"noncerange": "000000001fffffff" // representing nonces 0 through 536,870,911
},
"error": null,
"id": 1
}
When a share is rejected, the server may include an X-Reject-Reason header indicating the reason why it was rejected.
Example:
X-Reject-Reason: Stale share; block already completed
Part of the seed information that miners use to calculate hashes stems from the current timestamp. In order to reduce bandwidth consumption, particularly when network latency is high, the miner may increment the timestamp locally instead of querying the server for new information. The assumption is made that no transactions have been added to the block, so the only data to actually change will be the timestamp. The mining pool usually imposes some limit on exactly how much the timestamp can be changed, in order to prevent cheating by the miners.
If the mining pool allows rolling, it should return the X-Roll-Ntime header in the response for Getwork requests. Note that the server may choose to omit the header as needed; if the X-Roll-Ntime header is missing for a particular Getwork request, the miner may not roll the work even if it was allowed to do so for previous Getworks.
The value of the X-Roll-Ntime header has not been formally defined. Most mining pools seem to use a simple Y or N value (though it seems the miner may choose not to honor the N value, since only the presence of the header need be tested). However, the Bitcoin Wiki also says the value may be set to expire=N, where N is the integer number of seconds for which the server is willing to accept rolled work.
Examples:
X-Roll-Ntime: Y
X-Roll-Ntime: N
X-Roll-Ntime: expire=30
When the mining pool is about to go down for maintenance or is having other network-related issues, it is helpful to be able to switch some or all of its load over to another server. This is performed by sending the X-Switch-To header. The value of the header is a single JSON object representing the host to switch to, and the duration in minutes to stay on the alternate server before returning to the primary server.
X-Switch-To: {"host":"temporary.tld","port":8332,"ttr":10}
If this header is present, the miner is expected to make the switch immediately after the current Getwork and its corresponding Getwork Completion have been processed.