bug: invalid payload format: decoding failed: Incorrect base64 string

[fryorcraken]

Problem

JSON RPC API Does not accept valid base64 strings.

Impact

Cannot use v0.15.0 in the latest js-waku CI.

To reproduce

Do Json RPC Query:

post_waku_v2_relay_v1_message [
  '/waku/2/default-waku/proto',
  {
    payload: 'Ew==',
    contentTopic: '/test/1/waku-store/utf8',
    timestamp: 1676508406436000000n
  }
] 

Get an error:

RPC Response:  {"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"post_waku_v2_relay_v1_message raised an exception","data":"invalid payload format: decoding failed: Incorrect base64 string"}} +1ms

Expected behavior

A message is sent with said payload

Screenshots/logs

  waku:nwaku RPC Query:  post_waku_v2_relay_v1_message [
  '/waku/2/default-waku/proto',
  {
    payload: 'AA==',
    contentTopic: '/test/1/waku-store/utf8',
    timestamp: 1676508406416000000n
  }
] +0ms
  waku:nwaku RPC Response:  {"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"post_waku_v2_relay_v1_message raised an exception","data":"invalid payload format: decoding failed: Incorrect base64 string"}} +2ms
  waku:nwaku RPC Query:  post_waku_v2_relay_v1_message [
  '/waku/2/default-waku/proto',
  {
    payload: 'AQ==',
    contentTopic: '/test/1/waku-store/utf8',
    timestamp: 1676508406418000000n
  }
] +0ms
  waku:nwaku RPC Response:  {"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"post_waku_v2_relay_v1_message raised an exception","data":"invalid payload format: decoding failed: Incorrect base64 string"}} +1ms
  waku:nwaku RPC Query:  post_waku_v2_relay_v1_message [
  '/waku/2/default-waku/proto',
  {
    payload: 'Ag==',
    contentTopic: '/test/1/waku-store/utf8',
    timestamp: 1676508406419000000n
  }
] +0ms
  waku:nwaku RPC Response:  {"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"post_waku_v2_relay_v1_message raised an exception","data":"invalid payload format: decoding failed: Incorrect base64 string"}} +1ms
  waku:nwaku RPC Query:  post_waku_v2_relay_v1_message [
  '/waku/2/default-waku/proto',
  {
    payload: 'Aw==',
    contentTopic: '/test/1/waku-store/utf8',
    timestamp: 1676508406420000000n
  }
] +0ms
  waku:nwaku RPC Response:  {"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"post_waku_v2_relay_v1_message raised an exception","data":"invalid payload format: decoding failed: Incorrect base64 string"}} +1ms
  waku:nwaku RPC Query:  post_waku_v2_relay_v1_message [
  '/waku/2/default-waku/proto',
  {
    payload: 'BA==',
    contentTopic: '/test/1/waku-store/utf8',
    timestamp: 1676508406421000000n
  }
] +0ms
  waku:nwaku RPC Response:  {"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"post_waku_v2_relay_v1_message raised an exception","data":"invalid payload format: decoding failed: Incorrect base64 string"}} +1ms
  waku:nwaku RPC Query:  post_waku_v2_relay_v1_message [
  '/waku/2/default-waku/proto',
  {
    payload: 'BQ==',
    contentTopic: '/test/1/waku-store/utf8',
    timestamp: 1676508406422000000n
  }
] +0ms
  waku:nwaku RPC Response:  {"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"post_waku_v2_relay_v1_message raised an exception","data":"invalid payload format: decoding failed: Incorrect base64 string"}} +1ms
  waku:nwaku RPC Query:  post_waku_v2_relay_v1_message [
  '/waku/2/default-waku/proto',
  {
    payload: 'Bg==',
    contentTopic: '/test/1/waku-store/utf8',
    timestamp: 1676508406423000000n
  }
] +0ms
  waku:nwaku RPC Response:  {"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"post_waku_v2_relay_v1_message raised an exception","data":"invalid payload format: decoding failed: Incorrect base64 string"}} +1ms
  waku:nwaku RPC Query:  post_waku_v2_relay_v1_message [
  '/waku/2/default-waku/proto',
  {
    payload: 'Bw==',
    contentTopic: '/test/1/waku-store/utf8',
    timestamp: 1676508406424000000n
  }
] +0ms
  waku:nwaku RPC Response:  {"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"post_waku_v2_relay_v1_message raised an exception","data":"invalid payload format: decoding failed: Incorrect base64 string"}} +1ms
  waku:nwaku RPC Query:  post_waku_v2_relay_v1_message [
  '/waku/2/default-waku/proto',
  {
    payload: 'CA==',
    contentTopic: '/test/1/waku-store/utf8',
    timestamp: 1676508406425000000n
  }
] +0ms
  waku:nwaku RPC Response:  {"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"post_waku_v2_relay_v1_message raised an exception","data":"invalid payload format: decoding failed: Incorrect base64 string"}} +1ms
  waku:nwaku RPC Query:  post_waku_v2_relay_v1_message [
  '/waku/2/default-waku/proto',
  {
    payload: 'CQ==',
    contentTopic: '/test/1/waku-store/utf8',
    timestamp: 1676508406427000000n
  }
] +1ms
  waku:nwaku RPC Response:  {"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"post_waku_v2_relay_v1_message raised an exception","data":"invalid payload format: decoding failed: Incorrect base64 string"}} +0ms
  waku:nwaku RPC Query:  post_waku_v2_relay_v1_message [
  '/waku/2/default-waku/proto',
  {
    payload: 'Cg==',
    contentTopic: '/test/1/waku-store/utf8',
    timestamp: 1676508406428000000n
  }
] +1ms
  waku:nwaku RPC Response:  {"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"post_waku_v2_relay_v1_message raised an exception","data":"invalid payload format: decoding failed: Incorrect base64 string"}} +0ms
  waku:nwaku RPC Query:  post_waku_v2_relay_v1_message [
  '/waku/2/default-waku/proto',
  {
    payload: 'Cw==',
    contentTopic: '/test/1/waku-store/utf8',
    timestamp: 1676508406429000000n
  }
] +1ms
  waku:nwaku RPC Response:  {"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"post_waku_v2_relay_v1_message raised an exception","data":"invalid payload format: decoding failed: Incorrect base64 string"}} +0ms
  waku:nwaku RPC Query:  post_waku_v2_relay_v1_message [
  '/waku/2/default-waku/proto',
  {
    payload: 'DA==',
    contentTopic: '/test/1/waku-store/utf8',
    timestamp: 1676508406430000000n
  }
] +1ms
  waku:nwaku RPC Response:  {"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"post_waku_v2_relay_v1_message raised an exception","data":"invalid payload format: decoding failed: Incorrect base64 string"}} +0ms
  waku:nwaku RPC Query:  post_waku_v2_relay_v1_message [
  '/waku/2/default-waku/proto',
  {
    payload: 'DQ==',
    contentTopic: '/test/1/waku-store/utf8',
    timestamp: 1676508406431000000n
  }
] +1ms
  waku:nwaku RPC Response:  {"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"post_waku_v2_relay_v1_message raised an exception","data":"invalid payload format: decoding failed: Incorrect base64 string"}} +0ms
  waku:nwaku RPC Query:  post_waku_v2_relay_v1_message [
  '/waku/2/default-waku/proto',
  {
    payload: 'Dg==',
    contentTopic: '/test/1/waku-store/utf8',
    timestamp: 1676508406431000000n
  }
] +0ms
  waku:nwaku RPC Response:  {"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"post_waku_v2_relay_v1_message raised an exception","data":"invalid payload format: decoding failed: Incorrect base64 string"}} +1ms
  waku:nwaku RPC Query:  post_waku_v2_relay_v1_message [
  '/waku/2/default-waku/proto',
  {
    payload: 'Dw==',
    contentTopic: '/test/1/waku-store/utf8',
    timestamp: 1676508406432000000n
  }
] +0ms
  waku:nwaku RPC Response:  {"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"post_waku_v2_relay_v1_message raised an exception","data":"invalid payload format: decoding failed: Incorrect base64 string"}} +2ms
  waku:nwaku RPC Query:  post_waku_v2_relay_v1_message [
  '/waku/2/default-waku/proto',
  {
    payload: 'EA==',
    contentTopic: '/test/1/waku-store/utf8',
    timestamp: 1676508406434000000n
  }
] +0ms
  waku:nwaku RPC Response:  {"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"post_waku_v2_relay_v1_message raised an exception","data":"invalid payload format: decoding failed: Incorrect base64 string"}} +0ms
  waku:nwaku RPC Query:  post_waku_v2_relay_v1_message [
  '/waku/2/default-waku/proto',
  {
    payload: 'EQ==',
    contentTopic: '/test/1/waku-store/utf8',
    timestamp: 1676508406434000000n
  }
] +0ms
  waku:nwaku RPC Response:  {"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"post_waku_v2_relay_v1_message raised an exception","data":"invalid payload format: decoding failed: Incorrect base64 string"}} +1ms
  waku:nwaku RPC Query:  post_waku_v2_relay_v1_message [
  '/waku/2/default-waku/proto',
  {
    payload: 'Eg==',
    contentTopic: '/test/1/waku-store/utf8',
    timestamp: 1676508406435000000n
  }
] +0ms
  waku:nwaku RPC Response:  {"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"post_waku_v2_relay_v1_message raised an exception","data":"invalid payload format: decoding failed: Incorrect base64 string"}} +1ms
  waku:nwaku RPC Query:  post_waku_v2_relay_v1_message [
  '/waku/2/default-waku/proto',
  {
    payload: 'Ew==',
    contentTopic: '/test/1/waku-store/utf8',
    timestamp: 1676508406436000000n
  }
]

Note taht https://base64.guru/converter/decode/hex is able to correctly decode the payload.

nwaku logs:
nwaku_Waku_Store_Generator.log

nwaku version/commit hash

v0.15.0

Additional context

[fryorcraken]

Resolved by using base64 url encoding instead of standard base64

[alrevuelta]

Can reproduce it with

curl http://localhost:8545 -d '{"jsonrpc":"2.0","method":"post_waku_v2_relay_v1_message","params":["/waku/2/default-waku/proto", {"payload": "Ew==", "contentTopic": "my", "ephemeral": false}],"id":1}' -H 'Content-Type: application/json'

And works with:

curl http://localhost:8545 -d '{"jsonrpc":"2.0","method":"post_waku_v2_relay_v1_message","params":["/waku/2/default-waku/proto", {"payload": "Ew", "contentTopic": "my", "ephemeral": false}],"id":1}' -H 'Content-Type: application/json'

Looks like nwaku doesn't accept the padding =:

• Ew== doesnt work
• Ew works

Unsure if a bug or feature.

[fryorcraken]

Re-opening as we agreed we should use standard base64 with padding for both encoding and decoding

[LNSD]

Looks like nwaku doesn't accept the padding =:

• Ew== doesnt work
• Ew works

To understand what is wrong with the base64 encoding (in the JSON RPC context), I need to know the unencoded version of that "test vector".

How did you encode that (using the base64 CLI utility)? What is the unencoded version of Ew==/Ew?

[LNSD]

As pointed out in the Discord discussion, PR #1572 fixed the issue. This can be marked as resolved. The fix will be included as part of the v0.16.0 release.

[alrevuelta]

Looks like nwaku doesn't accept the padding =:
Ew== doesnt work
Ew works

To understand what is wrong with the base64 encoding (in the JSON RPC context), I need to know the unencoded version of that "test vector".
How did you encode that (using the base64 CLI utility)? What is the unencoded version of Ew==/Ew?

Note that I did not encoded anything, I directly typed that.

But can be easily replicated using https://www.base64encode.org/
asds -> YXNkcw==