/consumers/:username_or_id endpoints incorrectly uri decodes usernames

[mandrewpowers]

Summary

When requesting a consumer by username (email in our case) params are uri decoded and when + is used it decodes to a space. I'm questioning whether or not the param should actually be decoded or not.

Note: I added logs to the request, see below.

Steps To Reproduce

1. Create consumer with + in username Consumer is successfully created with +
2. Request user through /consumers/:username
3. Receive 404

Additional Details & Logs

• Version: 0.10.1
• Debug-level startup logs

2017/05/22 14:30:13 [verbose] Kong: 0.10.1
2017/05/22 14:30:13 [debug] ngx_lua: 10007
2017/05/22 14:30:13 [debug] nginx: 1011002
2017/05/22 14:30:13 [debug] Lua: LuaJIT 2.1.0-beta2
2017/05/22 14:30:13 [verbose] reading config file at /etc/kong/kong.conf
2017/05/22 14:30:13 [debug] admin_listen = "0.0.0.0:8001"
2017/05/22 14:30:13 [debug] admin_listen_ssl = "0.0.0.0:8444"
2017/05/22 14:30:13 [debug] admin_ssl = true
2017/05/22 14:30:13 [debug] anonymous_reports = true
2017/05/22 14:30:13 [debug] cassandra_consistency = "ONE"
2017/05/22 14:30:13 [debug] cassandra_contact_points = {"127.0.0.1"}
2017/05/22 14:30:13 [debug] cassandra_data_centers = {"dc1:2","dc2:3"}
2017/05/22 14:30:13 [debug] cassandra_keyspace = "kong"
2017/05/22 14:30:13 [debug] cassandra_lb_policy = "RoundRobin"
2017/05/22 14:30:13 [debug] cassandra_port = 9042
2017/05/22 14:30:13 [debug] cassandra_repl_factor = 1
2017/05/22 14:30:13 [debug] cassandra_repl_strategy = "SimpleStrategy"
2017/05/22 14:30:13 [debug] cassandra_ssl = false
2017/05/22 14:30:13 [debug] cassandra_ssl_verify = false
2017/05/22 14:30:13 [debug] cassandra_timeout = 5000
2017/05/22 14:30:13 [debug] cassandra_username = "kong"
2017/05/22 14:30:13 [debug] cluster_listen = "0.0.0.0:7946"
2017/05/22 14:30:13 [debug] cluster_listen_rpc = "127.0.0.1:7373"
2017/05/22 14:30:13 [debug] cluster_profile = "wan"
2017/05/22 14:30:13 [debug] cluster_ttl_on_failure = 3600
2017/05/22 14:30:13 [debug] custom_plugins = {}
2017/05/22 14:30:13 [debug] database = "postgres"
2017/05/22 14:30:13 [debug] dns_hostsfile = "/etc/hosts"
2017/05/22 14:30:13 [debug] dns_resolver = {}
2017/05/22 14:30:13 [debug] log_level = "notice"
2017/05/22 14:30:13 [debug] lua_code_cache = "on"
2017/05/22 14:30:13 [debug] lua_package_cpath = ""
2017/05/22 14:30:13 [debug] lua_package_path = "?/init.lua;./kong/?.lua"
2017/05/22 14:30:13 [debug] lua_socket_pool_size = 30
2017/05/22 14:30:13 [debug] lua_ssl_verify_depth = 1
2017/05/22 14:30:13 [debug] mem_cache_size = "128m"
2017/05/22 14:30:13 [debug] nginx_daemon = "on"
2017/05/22 14:30:13 [debug] nginx_optimizations = true
2017/05/22 14:30:13 [debug] nginx_worker_processes = "auto"
2017/05/22 14:30:13 [debug] pg_database = "kong"
2017/05/22 14:30:13 [debug] pg_host = "127.0.0.1"
2017/05/22 14:30:13 [debug] pg_password = "******"
2017/05/22 14:30:13 [debug] pg_port = 5432
2017/05/22 14:30:13 [debug] pg_ssl = false
2017/05/22 14:30:13 [debug] pg_ssl_verify = false
2017/05/22 14:30:13 [debug] pg_user = "kong"
2017/05/22 14:30:13 [debug] prefix = "/usr/local/kong/"
2017/05/22 14:30:13 [debug] proxy_listen = "0.0.0.0:80"
2017/05/22 14:30:13 [debug] proxy_listen_ssl = "0.0.0.0:8443"
2017/05/22 14:30:13 [debug] serf_path = "serf"
2017/05/22 14:30:13 [debug] ssl = true
2017/05/22 14:30:13 [debug] upstream_keepalive = 60
2017/05/22 14:30:13 [verbose] prefix in use: /usr/local/kong
2017/05/22 14:30:13 [verbose] preparing nginx prefix directory at /usr/local/kong
2017/05/22 14:30:13 [verbose] saving serf identifier to /usr/local/kong/serf/serf.id
2017/05/22 14:30:13 [debug] searching for OpenResty 'resty' executable
2017/05/22 14:30:13 [debug] /usr/local/openresty/bin/resty -V: 'nginx version: openresty/1.11.2.2'
2017/05/22 14:30:13 [debug] found OpenResty 'resty' executable at /usr/local/openresty/bin/resty
2017/05/22 14:30:13 [verbose] saving serf shell script handler to /usr/local/kong/serf/serf_event.sh
2017/05/22 14:30:13 [verbose] SSL enabled, no custom certificate set: using default certificate
2017/05/22 14:30:13 [verbose] default SSL certificate found at /usr/local/kong/ssl/kong-default.crt
2017/05/22 14:30:13 [verbose] Admin SSL enabled, no custom certificate set: using default certificate
2017/05/22 14:30:13 [verbose] admin SSL certificate found at /usr/local/kong/ssl/admin-kong-default.crt
2017/05/22 14:30:13 [warn] ulimit is currently set to "1024". For better performance set it to at least "4096" using "ulimit -n"
2017/05/22 14:30:13 [verbose] running datastore migrations
2017/05/22 14:30:13 [verbose] migrations up to date
2017/05/22 14:30:13 [verbose] serf agent not running, deleting /usr/local/kong/pids/serf.pid
2017/05/22 14:30:13 [debug] searching for 'serf' executable
2017/05/22 14:30:13 [debug] serf version: 'Serf v0.7.0'
2017/05/22 14:30:13 [debug] found 'serf' executable at serf
2017/05/22 14:30:13 [debug] starting serf agent: nohup serf agent -profile 'wan' -bind '0.0.0.0:7946' -log-level 'err' -rpc-addr '127.0.0.1:7373' -event-handler 'member-join,member-leave,member-failed,member-update,member-reap,user:kong=/usr/local/kong/serf/serf_event.sh' -node 'dev_0.0.0.0:7946_470b634076b94e2aa6a0bb7bce7673f7' > /usr/local/kong/logs/serf.log 2>&1 & echo $! > /usr/local/kong/pids/serf.pid
2017/05/22 14:30:13 [verbose] waiting for serf agent to be running
2017/05/22 14:30:14 [debug] sending signal to pid at: /usr/local/kong/pids/serf.pid
2017/05/22 14:30:14 [debug] kill -0 `cat /usr/local/kong/pids/serf.pid` >/dev/null 2>&1
2017/05/22 14:30:14 [verbose] serf agent started
2017/05/22 14:30:14 [verbose] auto-joining serf cluster
2017/05/22 14:30:14 [verbose] no other nodes found in the cluster
2017/05/22 14:30:14 [verbose] registering serf node in datastore
2017/05/22 14:30:14 [verbose] cluster joined and node registered in datastore
2017/05/22 14:30:14 [debug] searching for OpenResty 'nginx' executable
2017/05/22 14:30:14 [debug] /usr/local/openresty/nginx/sbin/nginx -v: 'nginx version: openresty/1.11.2.2'
2017/05/22 14:30:14 [debug] found OpenResty 'nginx' executable at /usr/local/openresty/nginx/sbin/nginx
2017/05/22 14:30:14 [debug] starting nginx: /usr/local/openresty/nginx/sbin/nginx -p /usr/local/kong -c nginx.conf
2017/05/22 14:30:14 [debug] nginx started
2017/05/22 14:30:14 [info] Kong started

• Kong error logs:

-- Normal startup logs as always, relevant logs from local change follow...
2017/07/21 12:24:03 [info] 32002#0: *295 [lua] consumers.lua:20: filter(): ### raw self.params.username_or_id = [email protected], client: 192.168.56.1, server: kong_admin, request: "GET /consumers/[email protected] HTTP/1.1", host: "localhost:8001"
2017/07/21 12:24:03 [info] 32002#0: *295 [lua] consumers.lua:22: filter(): ### unescaped self.params.username_or_id = apowers [email protected], client: 192.168.56.1, server: kong_admin, request: "GET /consumers/[email protected] HTTP/1.1", host: "localhost:8001"

==> /usr/local/kong/logs/admin_access.log <==
192.168.56.1 - - [21/Jul/2017:12:24:03 -0400] "GET /consumers/[email protected] HTTP/1.1" 404 35 "-" "-"

• Kong configuration:

API: Single node.js app that dumps headers as json for testing
Plugins:

basic-auth (anonymous consumer id set)
jwt (anonymous consumer id set)
acl (whitelist set as 'internal')

Config:

pg_host = 127.0.0.1
pg_port = 5432
pg_user = kong
pg_password = kong
pg_database = kong

• Operating System: Ubuntu 16.04.2 LTS

[p0pr0ck5]

@tehbiga please see #2651 as well as this comment: #2681 (comment)

Form fields with + characters need to be either encoded, or sent in a content type that doesn't escape +.

[mandrewpowers]

@p0pr0ck5 That is POST data, not URL parameters. Content-Type does not affect this whatsoever.

I am doing GET http://localhost:8001/consumers/[email protected] after I successfully create the user with the +.

[p0pr0ck5]

Bah, silly blind me :)

You should still be able to retrieve the credential by encoding the + char as %2b? Does that work in your case?

[mandrewpowers]

Unfortunately I tried that in my REST client and through NodeJS. Either the client sends the request with the + for me or openresty /nginx is already decoding it. I'll try with a CURL request really quick.

Edit: CURL seems to leave my %2B alone so I will have to see if I can get nodejs module to do the same. Unfortuantely it seems that every REST client I try to make the request in converts %2B to a +... I tried Postman and a few Firefox plugins.

Sorry for the noise, I'm surprised that the REST clients are modifying my requests :\

Edit 2: Here are my CURLs for fun:

$ curl http://localhost:8001/consumers/f9c01810-695d-4ce8-a807-2a7989185c24
{
    "created_at": 1500583896000,
    "id": "f9c01810-695d-4ce8-a807-2a7989185c24",
    "username": "[email protected]"
}

$ curl http://localhost:8001/consumers/f9c01810-695d-4ce8-a807-2a7989185c24
{"username":"[email protected]","created_at":1500583896000,"id":"f9c01810-695d-4ce8-a807-2a7989185c24"}

$ curl http://localhost:8001/consumers/[email protected]
{"message":"Not found"}

$ curl http://localhost:8001/consumers/apowers%[email protected]
{"username":"[email protected]","created_at":1500583896000,"id":"f9c01810-695d-4ce8-a807-2a7989185c24"}```

[mandrewpowers]

Oh wait, haha @p0pr0ck5 I just realized that only /consumers/:username does this:

$ curl http://localhost:8001/consumers/[email protected]/basic-auth
{
    "data": [],
    "total": 0
}

$ curl http://localhost:8001/consumers/[email protected]/jwt
{
    "data": [
        {
            "algorithm": "HS256",
            "consumer_id": "f9c01810-695d-4ce8-a807-2a7989185c24",
            "created_at": 1500583896000,
            "id": "a3151385-1874-433d-a31b-e6e1ded1e312",
            "key": "KEY",
            "secret": "SECRET"
        }
    ],
    "total": 1
}

$ curl http://localhost:8001/consumers/[email protected]/acls
{
    "data": [
        {
            "consumer_id": "f9c01810-695d-4ce8-a807-2a7989185c24",
            "created_at": 1500583896000,
            "group": "external",
            "id": "36ad93e5-fdf9-4d36-b692-c2edd7a7e131"
        }
    ],
    "total": 1
}

For now I will work around the one endpoint, but using %2B on anything besides /consumers/:username 404's.