mesos-dns domain 502 bad gateway

[samek]

Hi,
We're using mesos to manage our cluster. We also use mesos-dns for service discovery.

I've started kong in docker.
Created 2 apis (pointing to samek api server) one directly with target_url 10.0.0.200:3100 and one with (through haproxy) mesos-dns name haproxy.marathon.mesos

I always get 502 bad gateway when using that domain.

Conf:
curl localhost:8001/apis/ | jq .
{
"data": [
{
"created_at": 1432203540000,
"name": "testapi2",
"target_url": "http://haproxy.marathon.mesos:95/",
"id": "34942843-819f-4afe-ca40-17ed1f2bfc7e",
"public_dns": "test2"
},
{
"created_at": 1432203430000,
"name": "testapi",
"target_url": "http://10.0.0.200:31000/",
"id": "657ed867-7138-4686-c684-9ffba1de09fb",
"public_dns": "test"
}
]
}

curl -i -X GET --url http://localhost:8000/ --header 'Host: test2'
HTTP/1.1 502 Bad Gateway
Server: openresty/1.7.10.2rc0
Date: Thu, 21 May 2015 10:31:28 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 182
Connection: keep-alive
Via: kong/0.2.1

<title>502 Bad Gateway</title>

502 Bad Gateway

---

openresty/1.7.10.2rc0

and direct with ip:
curl -i -X GET --url http://localhost:8000/ --header 'Host: test'
HTTP/1.1 200 OK
Date: Thu, 21 May 2015 10:31:43 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Server: nginx/1.4.6 (Ubuntu)
X-Powered-By: PHP/5.5.9-1ubuntu4.7
Cache-Control: private, must-revalidate
pragma: no-cache
expires: -1
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
Access-Control-Allow-Headers: Origin, Content-Type, Accept, Authorization, X-Request-With, X-Requested-With
Access-Control-Allow-Credentials: true
Set-Cookie: laravel_session=eyJpdiI6InBybWJWRFhETWw5Z0dtZCs5TkRsZVE9PSIsInZhbHVlIjoiYXI4OERcL0ljd1pXeHh6aVAzamN6MmxjVUtqMUVXZEphc3dwcGlkQjRmS1N6MGh0UHgrXC81U0dLYVJtQjVxQVR0MXM3TUxZMlJ0Vk43M3ljcWl5aXRDdz09IiwibWFjIjoiOGU5YjRiZjdiNTgwYjA2MjRiN2IzNDRmMDI5OGU2ZjI4YzM5MzY1ZTkwOWU2NGFlZGYyNjFkNjM5MWJlMzVmMCJ9; expires=Thu, 21-May-2015 12:31:43 GMT; Max-Age=7200; path=/; httponly
Via: kong/0.2.1

"api?"

---

If I go to the kong docker I can get the url which is reported as bad gateway.

curl -i "http://haproxy.marathon.mesos:95/"
HTTP/1.1 200 OK
Server: nginx/1.4.6 (Ubuntu)
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Powered-By: PHP/5.5.9-1ubuntu4.7
Cache-Control: no-cache
Date: Thu, 21 May 2015 10:33:34 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
Access-Control-Allow-Headers: Origin, Content-Type, Accept, Authorization, X-Request-With, X-Requested-With
Access-Control-Allow-Credentials: true
Set-Cookie: laravel_session=eyJpdiI6IklqdWlcL3BFWGhFaU5DY1pqd1hiRVpBPT0iLCJ2YWx1ZSI6Ik5ydTRtR3VJWW81bFM0czkycUpCUmdyMXBTQ28wWUFKUTR4VFpIK0VkSUJjUkYrVEd2UkNVazBaSFViTFF4XC9zanNXbmgzdXFmZGFPeWZ1TGRHNjRZZz09IiwibWFjIjoiNzhmOWZhNmRiNmFkNDU3MDlhN2U1NTJiMTFlZjgyZjBkMTljNmJjYmNmM2FiZjQ5NjVhYjQ3YWEwNjNmNGM4MiJ9; expires=Thu, 21-May-2015 12:33:34 GMT; Max-Age=7200; path=/; httponly

"api?"

So does Kong have a problem resolving domain .mesos since it's not a "valid TLD" one in the real world?

[subnetmarco]

What version of Kong are you using? You can find it out by running kong version. We had some DNS problems that were fixed with 0.2.1.

[subnetmarco]

I see from the logs you are using 0.2.1. Kong will try to resolve a DNS using the local /etc/hosts file, and fallback to 8.8.8.8 if it can't find an entry. Maybe the hosts file has no entry for haproxy.marathon.mesos?

[samek]

I'm running docker mashape/kong:latest
Which is:
kong version
Kong version: 0.2.1

Is there anything else I could run in order to debug it?

[subnetmarco]

Most likely the problem is missing DNS entries in /etc/hosts inside of the Docker instance, that needs to be configured in a way to that the system knows how to resolve haproxy.marathon.mesos.

I am not familiar with Mesos, but how are *.mesos DNS addresses resolved? What DNS server are you using usually to resolve these addresses?

[samek]

Well mesos-dns is dynamic based on tasks that are running on mesos.
It's a DNS like any other.
So If a task is started A record will be in DNS.

All our hosts are using mesos-dns and there for it's also in the dockers resolv.conf.

I can call curl haproxy.marathon.mesos from inside of the kong docker and get response.

[samek]

@thefosk So what you're saying is kong ignores /etc/resolv.conf and if an entry isn't in /etc/hosts it goes directly to google dns 8.8.8.8 ?

[subnetmarco]

Can you try to log into the Kong instance, start Kong and then execute: nslookup -port=8053 haproxy.marathon.mesos 127.0.0.1.

Kong is using a local dnsmasq server to resolve DNS addresses to bypass a nginx limitation when resolving URLs in a proxy_pass directive.

[samek]

Ok,

nslookup -port=8053 24ur.com 127.0.0.1

nslookup: parse of /etc/resolv.conf failed

The problem was that we have
options timeout:0.5 attempts:3 in resolv.conf
When I removed this line it started to work.

[subnetmarco]

@samek So do you confirm it seems more like a bad configuration in resolv.conf rather than a Kong-specific problem?

[samek]

Yes although this configuration works with everything else just dnsmasq.

So bottom line if the resolv conf has basic stuff in it, it works.

Thanks

Sent from my iPhone

On 26 May 2015, at 22:14, Marco Palladino [email protected] wrote:

@samek So do you confirm it seems more like a bad configuration in resolv.conf rather than a Kong-specific problem?

—
Reply to this email directly or view it on GitHub.

[subnetmarco]

I will try to replicate the problem. Can you tell me the output of dnsmasq -v?

[subnetmarco]

It seems to be working with dnsmasq 2.72

[sonicaghi]

@samek is it working? can we close the issue?

[samek]

It does.
The problem was some extra options in my resolve.conf like i said.

Thanks.

On 19 Jun 2015, at 03:17, Augusto Marietti [email protected] wrote:

@samek https://github.com/samek is it working? can we close the issue?

—
Reply to this email directly or view it on GitHub #256 (comment).

[sonicaghi]

cool thx. Will close it then.

[samek]

would you guys accept changes to kong/cli/utils/signal.lua

I would add options to dnsmasq ( -o and -N )

Why ?

The problem is that (again mesos-dns) in our resolv.conf we have multiple dns servers first one is mesos-dns and other's are fall back (like 8.8.8.8 ..) and since we're using mesos domain's it should really try asking the first one for the answer not any one.

Right now what's happening is ... It'll ask random one and if it's not the first one It will neg-cache it.

We could also add an option to kong.yml dns-forwarders or something and it would use those if entered instead of resolv.conf.

If we take the system level on how name servers are used it is ordered so one would expect that they would also be in kong.

[subnetmarco]

I think -N and -o should also be defaults maybe in Kong, because they kinda make sense.

@samek In your tests, setting this parameters makes it work?

[samek]

I'll let you know in couple of hours when i test the build.

Sent from my iPhone

On 22 Jul 2015, at 18:51, Marco Palladino [email protected] wrote:

I think -N and -o should also be defaults maybe in Kong, because they kinda make sense.

@samek In your tests, setting this parameters makes it work?

—
Reply to this email directly or view it on GitHub.

[thibaultcha]

Side note: the thing is signal.lua is becoming more and more a garbage file. This part needs to be refactored. We have bits of the file dealing with nginx (the actual signals), configuration default, configuration validation, and dnsmasq. signal.lua was never intended to do all this.

[samek]

@thefosk It does work.

I've started 5 nodes and no gw timeout. Tomorrow I'll give it some traffic.

anyway All I've added is those two options.

git diff
diff --git a/kong/cli/utils/signal.lua b/kong/cli/utils/signal.lua
index 46aee31..e89ab1d 100644
--- a/kong/cli/utils/signal.lua
+++ b/kong/cli/utils/signal.lua
@@ -195,7 +195,7 @@ local function start_dnsmasq(kong_config)

   -- Start the dnsmasq
   local file_pid = kong_config.nginx_working_dir..(stringy.endswith(kong_config.nginx_working_dir, "/")     and "" or "/")..constants.CLI.DNSMASQ_PID
-  local res, code = IO.os_execute(cmd.." -p "..kong_config.dnsmasq_port.." --pid-file="..file_pid)
+  local res, code = IO.os_execute(cmd.." -p "..kong_config.dnsmasq_port.." --pid-file="..file_pid.." -N -o ")
   if code ~= 0 then
     cutils.logger:error_exit(res)
   else

[subnetmarco]

@samek sounds good - I have addressed this issue and all the other tests seem to be working too.

In your environment, can you please also try to use just -o (without -N) ?

[samek]

It works ok also without -N, but the nature of using mesos is that things move around.

It could very well happen that the upstream address would not exist for example we decide to shut down the upstreams then kong would cache that neg answer even though it might be again be present when we put it back live. I would also have -N in it, just in case.

[subnetmarco]

@samek the new Kong 0.4.1 has these flags enabled by default, you can use that instead of your patched Kong.

[samek]

Marco, thanks i saw it today btw.

Sent from my iPhone

On 05 Aug 2015, at 00:58, Marco Palladino [email protected] wrote:

@samek the new Kong 0.4.1 has these flags enabled by default, you can use that instead of your patched Kong.

—
Reply to this email directly or view it on GitHub.