No 'Access-Control-Allow-Origin' header is present

[jasongrishkoff]

Thanks for creating this!

I'm using mup to deploy this to a Digital Ocean droplet. My setup is as follows:

• Main website: https://www.domain.com/
• Kadira: https://kadira.domain.com/

I keep encountering this issue: "Failed to load https://kadira.domain.com/simplentp/sync: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'https://www.domain.com' is therefore not allowed access."

I've tried jiggling with the settings in server/config/security.js -- am I on the right path?

[lmachens]

Hey,
I don't think it is a Kadira or Meteor issue.
You have to set the CORS header (Access-Control-Allow-Origin) on your Main website server.

https://enable-cors.org/server_meteor.html
https://en.wikipedia.org/wiki/Same-origin_policy
https://forums.meteor.com/t/how-to-fix-cors-access-control-allow-origin-error-in-meteor-app/1896/2

[jasongrishkoff]

That's strange -- the language in the error specifically says that the CORS header is missing on the requested resource, which is the one hosting the Kadira package. All I've done on the Digital Ocean droplet was to mup setup && mup deploy this package with SSL.

Furthermore, I've been able to use other docker packages on Github for Kadira without issue in the past from this subdomain. I'm eager to get yours up and running though because it seems to have solved some of the issues the others have (trimming mongodb/indexing/etc.).

I'll keep digging and let you know what I come up with!

[lmachens]

You might be right, but I don't have any problems.
Take a look at https://github.com/lmachens/meteor-apm-server/blob/master/apm/server/engine/lib/utils.js And change the 'Access-Control-Allow-Origin': origin, to 'Access-Control-Allow-Origin': *, and investigate the content of origin. If this fixes your problem, the origin might not be right.

[jasongrishkoff]

Adding the following to the server/config/security.js file got rid of the CORS issue:

WebApp.rawConnectHandlers.use(function(req, res, next) {                                                               
    res.setHeader("Access-Control-Allow-Origin", "*");                                                                 
    return next();                                                                                                     
});

Now I'm just trying to figure out why none of the incoming data is saving to mongodb...

[jehartzog]

@jasongrishkoff Your solution helped with the CORS issue, but I'm seeing the same issue with data not saving mongodb. Where you able to figure that one out as well? Thanks in advance! 🙇

[lmachens]

Did you configure a mongo replica set?

[jehartzog]

Wow I can't believe I got a response so quickly on xmas :D. Happy holidays!

I did set up a full replica set on mlab and verified that my user info went to the DB and double checked my appId/secret. I'm no longer getting the CORS issue, or any errors at all, but actual app data isn't flowing into the DB.

Just like with @jasongrishkoff, I used MUP to deploy my kadira.mysite.com (running on AWS) while my main site www.mysite.com is on Galaxy.

Any tips on where I can look into to keep troubleshooting? This repo has a lot of potential and I'd like to help out if I can at least get it up and running.

[lmachens]

:D Happy holidays!
Can you show me your mup.js?
And please check if there is any data in the raw collections like RawSystemMetrics.

[jehartzog]

Sure, my mup.js is pretty simple.

module.exports = {
  servers: {
    one: {
      host: 'kadira.mysite.com',
      username: 'ubuntu',
      pem: './mypem',
    }
  },

  app: {
    name: 'apm',
    path: '../apm',
    servers: {
      one: {},
    },
    buildOptions: {
      serverOnly: true,
    },
    env: {
      ROOT_URL: 'https://kadira.mysite.com',
      MONGO_URL: 'mongodb://myreplicamongoconnection',
    },
    docker: {
      image: 'abernix/meteord:node-8.4.0-base',
    },
    enableUploadProgressBar: true
  },

  proxy: {
    domains: 'kadira.mysite.com',
    ssl: {
      forceSSL: true,
      letsEncryptEmail: '[email protected]',
    }
  },

  // mongo: {
  //   version: '3.4.1',
  //   servers: {
  //     one: {}
  //   }
  // }
};

I actually don't see any collection called RawSystemMetrics, here is what my freshly created replica set looks like:

[lmachens]

That is strange. Did you check server logs with mup logs?
And please add MONGO_OPLOG_URL.
Maybe the ports are not open on your server (11011 for the engine).

[jehartzog]

I added the MONGO_OPLOG_URL, double checked that I opened up ports 11011 and 7007, but still no change. I'm still working it but not even sure where to look at this point, I'll probably start adding more logs statements to the kadira source and re-deploy to try to figure out. Any pointers would be very much appreciated :D.

The logs were not very useful, nothing unexpected, I pasted them below.

[kadira.skoolerstutoring.com]=> Starting meteor app on port:80
[kadira.skoolerstutoring.com]Kadira Alertsman started
[kadira.skoolerstutoring.com]Fetchman started on port: 7007
[kadira.skoolerstutoring.com]starting apm-engine on port 11011
[kadira.skoolerstutoring.com]   fetched 0 of rawErrorMetrics in 1 ms'
[kadira.skoolerstutoring.com]   writing completed in 0 ms
[kadira.skoolerstutoring.com]very strange! - no entries found
[kadira.skoolerstutoring.com]   fetched 0 of rawMethodsMetrics in 1 ms'
[kadira.skoolerstutoring.com]   writing completed in 0 ms
[kadira.skoolerstutoring.com]very strange! - no entries found

[jehartzog]

I'm still troubleshooting, but I noticed that apm/server/engine/lib/middlewares/simplentp.js is supposed to return a timestamp, but is somehow returning an entire HTML meteor page.

I think the quick fix above to add the CORS header to every response is masking whatever issue that is going on rather than fixing the underlying issue.

[jasongrishkoff]

@jehartzog I had noticed the same -- all responses to the /sync request were returning garbled HTML. It didn't seem right. And this was with MONGO_OPLOG_URL. Those were the exact same logs I was seeing on my side.

[jehartzog]

Thanks @jasongrishkoff, I'm at a bit of a loss at where to look now. I've dumped the incoming requests to logs and verified that my kadira server is receiving at least the /sync requests from my main app and sending a response, but it looks like that response isn't correct. And I can't yet verify whether or not if my kadira app is getting any of the actual meteor app data network requests.

The next step is to go through the code base and figure out where something unexpected is happening, but with this size of code base that's a bit of a task to figure out. If anybody has tips on where to start looking, that will be greatly appreciated :D.

[lmachens]

Do you have the same problems when you run the apm server on your dev machine? Which monitoring package are you using? I recommend https://github.com/lmachens/kadira

[jehartzog]

I ran the entire setup (my app + APM server) locally on my dev machine, it returned the exact same behavior as above. I didn't get the CORS error since everything was on localhost, but the sync timestamp was still HTML gibberish and no app data flowed to the apm server.

I tried using https://atmospherejs.com/lmachens/kadira but it crashed 20 seconds after starting up:

'TypeError: traces.forEach is not a function',
2017-12-27 10:41:19-05:00 ' at TracerStore.collectTraces (packages/lmachens_kadira.js:1216:10)',
2017-12-27 10:41:19-05:00 ' at MethodsModel.buildPayload (packages/lmachens_kadira.js:1540:45)',
2017-12-27 10:41:19-05:00 ' at Object.Kadira._buildPayload (packages/lmachens_kadira.js:2398:43)',
2017-12-27 10:41:19-05:00 ' at packages/lmachens_kadira.js:2446:26'

I then locally installed https://github.com/lmachens/kadira, but had the same error.

[lmachens]

So weired. I have more time next week and will take a deeper look at it. Maybe you can find out the problems. My setup is a mongodb 3.4 single replica set with a meteor 1.6 client. Servers OS is ubuntu.

[ivan046]

It's really weird. We run Kadira from this project for about 2 weeks in 2 different production environments with meteor 1.6.0.1 apps, and the only error we had was

TypeError: MongoOplog is not a constructor

and it was fixed with "const oplogConn = new MongoOplog" ---> "const oplogConn = MongoOplog"
Also we built docker images for this APM and everything works fine. Looks like this issue has roots in some other place then in the APM code.
I'll make a PR with our additions during holidays.

[jehartzog]

Thanks for the responses, I'm still working on it to isolate what is going on. Here is the setup I tried everything on:

mLab shared replica set running 3.4.10
meteor 1.6.0.1 app
meteorhacks:[email protected]
Ubuntu 16.04.3 LTS (fresh ec2 instance)
deployed using mup v1.3.7, using docker image abernix/meteord:node-8.4.0-base

I also tried it locally and saw the exact same issues, so perhaps it's something with my app, although it seems I'm not the only one seeing this issue.

I'll try reverting to some previous versions to see if that will fix it.

[ivan046]

Did you use mup locally, too? If yes, try without it.

[jehartzog]

Nope, mup for the EC2 ubuntu production/staging deployment.

After the tip to run locally, I ran the two meteor projects locally on different ports. Now that I think about it, I didn't set up a local replica set, I just used the default meteor create mongodb instance, so I may look into doing that.

[ivan046]

Yap, will not work without RS. Use official mongo:3.4 image with storage engine wiredtiger and replicaset name options. Then init one-member RS and APM runs perfect with it.

[jehartzog]

Update: Local deployment works, I had misconfigured my settings.json file when I set my endpoint by not setting port 11011. That explains why my time sync was getting a html response @jasongrishkoff .

Fixing my .json config by adding 11011 as the port as below helped.

  "kadira": {
    "appId": "...",
    "appSecret": "...",
    "options": {
      "endpoint": "https://kadira.mysite.com:11011" 
    }

Sorry for wasting your all time with this step that's listed in the documentation, my bad. It now works fully on development using the meteorhacks:kadira package (the lmachens:kadira still crashes 20 secs after launch).

Turns out if the kadira client tries to auth on a webserver port 80, it doesn't throw an error message so it's not obvious what is going on.

After adding the correct port now throws an expected error message Kadira: authentication failed - check your appId & appSecret since my current MUP reverse proxy settings aren't set up to pass through port 11011. That's not an issue with this package though, so I'll keep going on that.

I'll keep trying to set it up to work with MUP and pass on any big 'gotchas' if I find any.

[jasongrishkoff]

@jehartzog Makes sense -- I too tried 11011 but got the same authentication failed error. Once you get that mup.js configured, it's be great if you could share what you put in the proxy section to open that port :)

[jehartzog]

Okay, up and running at last. Here are the takeaways, @lmachens let me know if you want me to put together a PR to add this to documentation for you.

Here are the steps you need to take to use MUP to deploy this to a remote server:

In your Meteor app, you must use set your endpoint to use http and port 11011:

  "kadira": {
    "appId": "myappid",
    "appSecret": "myappsekrit",
    "options": {
      "endpoint": "http://kadira.myapp.com:11011"
    }
  },

In your mup.js file for this Kadira project, a few things are important:

1. You can't use the reverse proxy setup, since I didn't see any way with MUP to pass through a random port to a specific docker without modifying MUP.
2. You need to send the docker argument -p 11011:11011 to allow the apm engine to receive data. @jasongrishkoff

module.exports = {
  servers: {
    one: {
      host: 'kadira.myapp.com',
      username: 'ubuntu',
      pem: './mypem',
    }
  },

  app: {
    name: 'apm',
    path: '../apm',
    servers: {
      one: {},
    },
    buildOptions: {
      serverOnly: true,
    },
    env: {
      ROOT_URL: 'https://kadira.myapp.com',
      MONGO_URL: 'mongodb://localhost/meteor',
      // ENGINE_PORT: 22022, // If you set this, it changes the Kadira engine port
    },
    ssl: {
      autogenerate: {
        email: '[email protected]',
        domains: 'kadira.myapp.com',
      },
    },
    docker: {
      image: 'abernix/meteord:node-8.4.0-base',
      args: [
        '-p 11011:11011',
      ],
    },
    enableUploadProgressBar: true,
  },

  // This was the reverse proxy setup, this will not work since it does not pass port 11011
  // proxy: {
  //   domains: 'kadira.myapp.com',
  //   ssl: {
  //     forceSSL: true,
  //     letsEncryptEmail: '[email protected]',
  //   },
  // },

  mongo: {
    version: '3.4.1',
    servers: {
      one: {},
    },
  },
};

A few notes about what I've found so far:

1. Using the MUP created mongodb instance works. I thought that it would not as the PR to create a single node replica set hasn't been released yet, but my server is up and running just fine with a MUP created non-replica set local mongodb instance.

2. The current setup still has issues with the http endpoint. Clients on your app will receive in console a constant stream of Mixed Content: The page at 'https://new.myapp.com' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://kadira.myapp.com:11011/simplentp/sync'. I could fix this by throwing another proxy/webbalancer in front of the kadira server, but that's a step for later.

[lmachens]

Good to here that you found a solution.

It is true, that it is not easy to set up the reverse proxy with mup for multiple ports. You need 1 port for the UI (3000) and 1 port for the engine (11011).

Take a look at https://github.com/dbendelman/nginx-proxy which allows multiple ports.
It shouldn't be too complicated to set up the nginx-proxy and letsencrypt.

My mup.js:

env: {
...
LETSENCRYPT_HOST: 'apm-engine.domain.com,apm.domain.com',
LETSENCRYPT_EMAIL: '[email protected]',
VIRTUAL_HOST: 'apm-engine.domain.com:11011,apm.domain.com:3000' // only works with dbendelman/nginx-proxy
},
 docker: {
      image: 'abernix/spaceglue:node-8.9.0', // I like the spaceglue image. but meteord should work fine too
      imagePort: 3000,
      args: ['-p 3000', '-p 11011', '--link=mongodb:mongodb'], // Mongo runs on the same server but is not configured by mup
    },

The replica set is required for the alerts system. If you don't need it, it might work with a non-replica mongodb instance.

Tell me what you think. We should update the README 👍

[jasongrishkoff]

@lmachens Looks like your mup version allows https, right?

[lmachens]

I don't use the mup proxy settings, so https is not configured by mup.

[jehartzog]

Alright I got everything up and running just fine over https completely with a single meteor app + local mongo replica set using MUP plus a couple dirty script hacks. I used the reverse proxy for MUP set up with two domains:
mup.js

module.exports = {
  servers: {
    one: {
      host: 'myip',
      username: 'ubuntu',
      pem: './mypem',
    }
  },

  app: {
    name: 'myapp-apm',
    path: '../apm',
    servers: {
      one: {},
    },
    buildOptions: {
      serverOnly: true,
    },
    env: {
      ROOT_URL: 'https://apm.mydomain.com',
      MONGO_URL: 'mongodb://mongodb/meteor',
      MONGO_OPLOG_URL: 'mongodb://mongodb/local',
    },
    docker: {
      image: 'abernix/meteord:node-8.4.0-base',
      args: [
        '-p 11011:11011',
      ],
    },
    enableUploadProgressBar: true,
  },

  proxy: {
    domains: 'apm.mydomain.com,apm-engine.mydomain.com', // both of these domains point to the same ip as the server above
    ssl: {
      forceSSL: true,
      letsEncryptEmail: '[email protected]',
    },
  },

  mongo: {
    version: '3.4.1',
    servers: {
      one: {},
    },
  },
};

These are commands are needed to fix up the default MUP deployment and need to be re-preformed when the app is deployed/restarted :(.

This is needed because this PR hasn't been released yet.

ssh apm.mydomain.com

docker exec mongodb sh -c 'echo "replSet=meteor" >> /mongodb.conf'
(restart the mongo instance using mup stop, start, then...)
docker exec mongodb mongo --eval 'rs.initiate({_id: "meteor", members: [{_id: 0, host: "127.0.0.1:27017"}]});'

This one is needed due to lack of good Virtual Host support with the built in MUP proxy. It's ugly, and I'm not a great regex writer, but all it does is change the first upstream server (which on my build was apm-engine.mydomain.com) from 80 to 11011, sending all the traffic to the correct engine port.

ssh apm.mydomain.com

docker exec mup-nginx-proxy sed -i '0,/server.*:80/ s/\(server.*:\)80/\111011/1' /etc/nginx/conf.d/default.conf && docker exec mup-nginx-proxy nginx -s reload

[lmachens]

@jehartzog There was a bug in https://github.com/lmachens/kadira. It should be fixed now in 2.30.6.
Does mup works for you now?

[jehartzog]

So the bug I reported in your kadira fork preventing me from even running it locally, so I didn't even try it with MUP.

I went ahead and tried 2.30.6 and it worked just fine. I fully expect it will work will with MUP too. I'm not ready to deploy it to my staging/production yet, but I will do so eventually and raise another issue at that repo if it doesn't work for any reason.

It's funny when I was looking into what you did with the Kadira fork, it included pulling in the jquery removal PR which I had moved over to the meteor-apm-agent package.

I really like the work you've done so far and I'll be using both your packages in my production app. Thanks for keeping them up to date and let me know if you need any assist!

[jasongrishkoff]

@jehartzog have followed your steps but authentication keeps failing on port 11011. Will keep digging into this, but wondering if I'm missing something here.

I copy your mup.js, changing the domains and server info. Then I do mup config && mup deploy. Then I ssh into the server and execute:

docker exec mongodb sh -c 'echo "replSet=meteor" >> /mongodb.conf'
docker mongodb rstart
docker exec mongodb mongo --eval 'rs.initiate({_id: "meteor", members: [{_id: 0, host: "127.0.0.1:27017"}]});'

Followed by:
docker exec mup-nginx-proxy sed -i '0,/server.*:80/ s/\(server.*:\)80/\111011/1' /etc/nginx/conf.d/default.conf && docker exec mup-nginx-proxy nginx -s reload

I can access the https domains for both kadira.domain.com and kadira-engine.domain.com, including as part of my Kadira config (it authenticates just fine). As soon as port 11011 enters the scene things go south :(

The logs seem like everything's setting up fine as well, so I'm thinking this has something to do with mup-nginx-proxy:

[138.197.75.12]Kadira Alertsman started
[138.197.75.12]Fetchman started on port: 7007
[138.197.75.12]starting apm-engine on port 11011
[138.197.75.12]very strange! - no entries found

[jehartzog]

The sed regex to edit nginx default.conf is a bit hacky and probably dependent on your exact config. You're better off going into the docker instance and manually looking at the config to make the exact change.

docker exec -it mup-nginx-proxy /bin/bash
cat /etc/nginx/conf.d/default.conf

You should see two upstream configurations, such as:

# apm-engine.yourdomain.com
upstream apm-engine.yourdomain.com {
                                ## Can be connect with "bridge" network
                        # yourapp-apm
                        server 172.yourinternalserverip.0:80;
}

Change the last line to server 172.yourinternalserverip.0:11011; so that it forwards incoming traffic to that domain to the correct port. Note that with that setup, your endpoint for Meteor should be configured to send data to default https port, since you're using nginx config to forward to port 11011:

    "options": {
      "endpoint": "https://apm-engine.yourdomain.com"
    }

The other domain, with an identical upstream config, will be for the other domain apm.yourdomain.com. Leave this one alone, you'll use that one to view the apm website.

Overall I'm not happy with this setup, it's obviously hacked together and brittle, but it works for now :/.

[jasongrishkoff]

Wow, it's working! Exciting stuff! Thanks for the quick response and your help. I actually think your original sed worked because when I looked at the config it was port 11011 as you said. I think where I was going wrong was by trying to include port 11011 in my Meteor config when connecting to Kadira.

[cunneen]

@jehartzog @jasongrishkoff : Would it do any damage to whack the regex into a cron script on the docker host machine, to run e.g. every 5 minutes ?

I mean this one:

docker exec mup-nginx-proxy sed -i '0,/server.*:80/ s/\(server.*:\)80/\111011/1' /etc/nginx/conf.d/default.conf && docker exec mup-nginx-proxy nginx -s reload

It's still brittle and hacky, but at least it could survive restarts and it could also be automated in the sample setup as part of the post-setup hook.

[jehartzog]

@cunneen that would probably break things as if you run the sed regex command twice, it will overwrite the config used to serve the apm page with port 11011. You could maybe do more regex/bash hacking to work around this, but I much prefer spending time finding a better solution using mup nginx config. I will look into this sometime next week as discussed here.

[cunneen]

Thanks @jehartzog. That sounds like a better approach.

[cunneen]

For the benefit of anyone still looking for an automated workaround, I've had some success with adding the attached shell script and cron job.

/home/ubuntu/fix_nginx_proxy_port.sh

#!/usr/bin/env bash
##################################################
# Workaround meteor-apm-server / nginx proxy bug 
# https://github.com/lmachens/meteor-apm-server/issues/14
#
# To Install:
# 1. Put this file onto the docker host machine as /home/ubuntu/fix_nginx_proxy_port.sh
# 2. Add a crontab entry:
#    crontab -e
# Add the following line (without leading spaces):
#    */30 * * * * docker exec -i mup-nginx-proxy bash < /home/ubuntu/fix_nginx_proxy_port.sh > /home/ubuntu/fix_nginx_proxy_port.log
#################################################

TMPFILE=`tempfile`;

awk '
BEGIN{in_block=0;} 
/upstream apm-engine/{
	in_block=1; 
	print $0;
	next;
} 
/server (.*):80/{
	if(in_block == 1) { 
		newval = $0;
		gsub(/:80;$/,":11011;",newval);
		print newval;
	} else {
		print $0;
	}
	next;
}
/[\}]/{
	in_block=0; 
	print $0;
	next;
}
{print $0;}
' /etc/nginx/conf.d/default.conf > ${TMPFILE} && \
mv -f ${TMPFILE} /etc/nginx/conf.d/default.conf && \
nginx -s reload

[jehartzog]

@cunneen Thanks a ton for posting this, my halfway solution would sometimes result in the fix dropping off, which would lead to no monitoring.

I used your instructions and they worked like a charm, no modifications needed.