Connect with us

Hi, what are you looking for?

Cryptography

Bitcoind does not like ECDSA (r, s) pair produced by OpenSSL

I am writing transactions manually and have stumbled across a rather bizarre situation.

Cyber attack with unrecognizable hooded hacker using virtual reality, digital glitch effect.

The following content was written by pgmforever on October 18, 2018, 09:08:44 AM in the thread Bitcoind does not like ECDSA (r, s) pair produced by OpenSSL. All content is owned by the author of the bitcointalk.org post. (original)


I am writing transactions manually and have stumbled across a rather bizarre situation.

Only one in a few of the transactions I broadcast to bitcoind is accepted, otherwise I get a
Code:
REJECT_NONSTANDARD
(Non-canonical DER signature).

So I got my hands dirty and tracked the rejection to be originating from this line: https://github.com/bitcoin/bitcoin/blob/9c5f0d542d1db507b3b9c87bd9de6d0d758d51c1/src/script/interpreter.cpp#L163

I read about DER encoding and checked how IsValidSignatureEncoding is enforcing it, but I do not know why OpenSSL generates not-DER-compliant (r, s) values?

How should I overcome this? I am thinking of something along the lines of (pseudocode):

Code:
Pair (r, s);
do
{
   (r, s) = sign(hash, pvtkey);
} while (r[0] >= 128 || s[0] >= 128); // where r[0], s[0] should be the very first byte of each value

But isn’t that kind of redundant? Can I give OpenSSL any flag to produce a valid DER (R, S) pair in the first place?

The following content was written by bob123 on October 18, 2018, 11:23:56 AM in the thread Bitcoind does not like ECDSA (r, s) pair produced by OpenSSL. All content is owned by the author of the bitcointalk.org post. (original)


Did you choose the correct elliptic curve to make the calculations ? Bitcoin is using secp256k1.

To generate a private key using openssl:
Code:
openssl ecparam -genkey -name secp256k1 -rand /dev/urandom -out $PRIVATE_KEY

To generate the public key related to the previously generated private key:
Code:
openssl ec -in $PRIVATE_KEY -pubout -out $PUBLIC_KEY

The following content was written by pgmforever on October 18, 2018, 11:33:54 AM in the thread Bitcoind does not like ECDSA (r, s) pair produced by OpenSSL. All content is owned by the author of the bitcointalk.org post. (original)


Did you choose the correct elliptic curve to make the calculations ? Bitcoin is using secp256k1.

To generate a private key using openssl:
Code:
openssl ecparam -genkey -name secp256k1 -rand /dev/urandom -out $PRIVATE_KEY

To generate the public key related to the previously generated private key:
Code:
openssl ec -in $PRIVATE_KEY -pubout -out $PUBLIC_KEY

I’m using the OpenSSL headers and linking against the shared library, not generating stuff from the command line, but as I said for like maybe 1 in 5 signatures everything is fine (because I am lucky enough that the produced signature’s S value has not its first byte >= 128).

I read some other threads which said that if that happens then you need to prepend an additional 0x00 byte in front of S but after doing that now I’m getting SCRIPT_ERR_SIG_HIGH_S heheh… so now I’m trying to get S = N-S if S >= N/2 where N is secp256k1’s order… any idea if that’s the way to go?

The following content was written by Coding Enthusiast on October 19, 2018, 04:57:14 AM in the thread Bitcoind does not like ECDSA (r, s) pair produced by OpenSSL. All content is owned by the author of the bitcointalk.org post. (original)


I read about DER encoding and checked how IsValidSignatureEncoding is enforcing it, but I do not know why OpenSSL generates not-DER-compliant (r, s) values?

2 questions.
Are you doing the DER encoding yourself or is OpenSSL does it for you? If it is the later then it is doing it correctly.
And what do you mean by “non-DER-compliant”? DER is just an encoding, you have some data that you encode using a form of TLV encoding scheme. This way of encoding can handle any kind of value. From 32 byte integers (r and s) to anything else like booleans, strings,…

Quote
How should I overcome this? I am thinking of something along the lines of (pseudocode):

Code:
Pair (r, s);
do
{
   (r, s) = sign(hash, pvtkey);
} while (r[0] >= 128 || s[0] >= 128); // where r[0], s[0] should be the very first byte of each value

But isn’t that kind of redundant? Can I give OpenSSL any flag to produce a valid DER (R, S) pair in the first place?

If OpenSSL is doing the DER-encoding then let it be like that. If you are DER-encoding yourself then first you have to understand why that initial 0 is appended to the result.
For humans when we want to represent sign we put a symbol behind the number showing its sign: -2 or +2
For computer there is no “sign” under the hood so the way it does it is with manipulating the bits (zeros and ones). The way it works for big numbers (like r and s here) is that the computer looks at the highest-order bit of the last byte (little endian order) if that is set the number is negative, if it is not set then the number is positive.
1 byte is 8 bits: 0000 0000 you look at the bold bit and if it was set then you decide about the sign.
127 = 0111 1111
128 = 1000 0000
129 = 1000 0001

So if the last byte (or first byte since DER encodes in big endian) is bigger than or equal to 128 and you want the number to be positive (which is the case with r and s) you append a new byte (0) to it to make the computer treat that number as positive.

So this code you posted is wrong because if the first byte of the byte array of ‘r’ or ‘s’ is bigger than 128 that doesn’t make them invalid that you want to try again. They can still be valid, you just have to append a first 0 byte to make it positive.

What you need to check are listed in sec1 pdf section 4.1.3 (!=0 & In bitcoin there is an additional check for s value only and that is to check if s is bigger than curve order. In which case you use s’=N-s and because of the way used arithmetic works s’ is valid.
So here technically you always* end up with a value for s that is small, hence doesn’t have its highest-order bit of the first byte (big endian order) set so you don’t need that appended 0.

* I haven’t really thought about this much but I believe it may be possible to find a smaller s (eg. 31 bytes instead of 32) and that s is lower than N/2 but can have its highest-order bit set (>=128 will fail and reject this valid s) which is another reason why the posted code is wrong.

The following content was written by pgmforever on October 19, 2018, 01:20:57 PM in the thread Bitcoind does not like ECDSA (r, s) pair produced by OpenSSL. All content is owned by the author of the bitcointalk.org post. (original)


First off thanks for your help

I am doing the encoding myself but 2 days ago when I wrote the question I figured if I followed the “<30><02><02><S>” scheme and still get rejected there's something wrong in the S/R produced by openssl, hence why I came up with the shitty code above Smiley. Obviously, I was wrong.

I made some progress in the meantime and out of the 10-20 txes I managed to broadcast not any of them was rejected, but this might be a coincidence so for extra checking I am posting yet another pseudocode of how I do things now:

Code:
signature = ecdsa_sign(txHash, privateKey);

// enforce a low S
curveOrder = ecdsa_get_order(secp256k1Group);
halfCurveOrder = curveOrder / 2; // actual division done with openssl's right shift BIGNUM function as well as comparison and subtraction operations below
if (signature->s > halfCurveOrder)
{
    signature->s = curveOrder – signature->s;
}

// check for LSBit of the MSByte for R
if (signature->r[0] >= 128)
{
    signature->r = 0x00 + signature->r; // appending by having a new buffer memcpy(buff, &zero, 1); memcpy(buff + 1, rBuffer, rLength)
}

// the same check/appending done for S — with this Im not sure it needs to be done since I should have a low S already??
// …

return <30><4 + rLength + sLength><02><02><S>

Does this look better? Am I being lucky that my transactions made it or the code is actually correct? Thanks

The following content was written by Coding Enthusiast on October 19, 2018, 06:12:11 PM in the thread Bitcoind does not like ECDSA (r, s) pair produced by OpenSSL. All content is owned by the author of the bitcointalk.org post. (original)


I don’t see anything wrong.
But I don’t know two things, your full code and how OpenSSL works. For instance I am not sure appending that 0 is even needed. Usually when these functions return a value like this they also take care of that initial 0 for you. So your second if may never be true!

The following content was written by pgmforever on October 22, 2018, 09:00:33 AM in the thread Bitcoind does not like ECDSA (r, s) pair produced by OpenSSL. All content is owned by the author of the bitcointalk.org post. (original)


Well, there is some built-in OpenSSL function that takes a signature and DER-encodes it, but for the function that just signs there is no encoding whatsoever so my second “if” is needed and is actually hit like 50% of the time.

Haven’t gotten any other rejected TX so I suppose everything is fine, the only question still open is whether that 0-byte “if” is needed for S as well (third “if” in the pseudocode) or just for R (I have them both right now but for S didn’t get any hits inside the if so far).

The following content was written by gmaxwell on October 24, 2018, 12:53:10 AM in the thread Bitcoind does not like ECDSA (r, s) pair produced by OpenSSL. All content is owned by the author of the bitcointalk.org post. (original)


The DER encoder in OpenSSL is correct, but the above examples apparently weren’t using it.  The result of “I made my own DER encoder” is usually wrong… it’s not hard to encode DER correctly but it takes a little effort that people don’t bother to put in.

Even once you get the encoding right, you’ll still have to obey the lowS rule: https://bitcoin.stackexchange.com/questions/59820/sign-a-tx-with-low-s-value-using-openssl

The following content was written by pgmforever on October 31, 2018, 02:40:27 PM in the thread Bitcoind does not like ECDSA (r, s) pair produced by OpenSSL. All content is owned by the author of the bitcointalk.org post. (original)


it’s not hard to encode DER correctly but it takes a little effort that people don’t bother to put in.

You’re right, it was not hard at all to encode it correctly once I got all the facts right. Been using this “own encoder” for a while now and had no other issues whatsoever.

I encountered some other weird situation in the meantime if you want to take a look — https://bitcointalk.org/index.php?topic=5060723

Regards

Author

You May Also Like

Tutorials

KuCoin is a popular digital currency exchange that has a large daily trading volume in the market. Registration on this site is very easy...

Bitcoin

Losing your Bitcoin private  keys can be devastating and at the same time frustrating. This is because you wouldn’t have access to your precious...

Tutorials

Earlier Bitcoin Wallet used to be a bunch of private keys. In order to use new address, user had to generate new private key...

Tutorials

The aim of this thread is to give a complete overview for anyone who finds Bitcoins in any storage format.