Log objects rather than JSON stringified objects #1922

spenthil · 2016-05-26T19:33:21Z

To easily configure CloudWatch (AWS), each log entry should be on a single line. This makes VERBOSE logging do just that.

In our case - when connecting the parse-server app, we also configure the logger to stringify JSON:

logger.configure({
  level: 'silly',
  transports: [
    new (winston.transports.Console)({
      json: true,
      stringify: true
    })
  ]
});

We can then pipe straight to CloudWatch which ingests the JSON without issue.

ghost · 2016-05-26T20:11:54Z

@spenthil updated the pull request.

codecov-io · 2016-05-26T20:20:12Z

Current coverage is 91.87%

Merging #1922 into master will decrease coverage by <.01%

2 files (not in diff) in src were modified. more
- Misses +2
- Hits -2

@@             master      #1922   diff @@
==========================================
  Files            91         91          
  Lines          6399       6400     +1   
  Methods        1094       1094          
  Messages          0          0          
  Branches       1342       1342          
==========================================
- Hits           5881       5880     -1   
- Misses          518        520     +2   
  Partials          0          0

Powered by Codecov. Last updated by 103839c...eb9677b

facebook-github-bot · 2016-05-26T20:52:51Z

@spenthil updated the pull request.

facebook-github-bot · 2016-05-26T20:59:01Z

@spenthil updated the pull request.

ghost · 2016-05-26T22:14:46Z

@spenthil updated the pull request.

ghost · 2016-05-27T00:10:52Z

@spenthil updated the pull request.

ghost · 2016-05-27T00:16:51Z

@spenthil updated the pull request.

ghost · 2016-05-27T00:23:52Z

@spenthil updated the pull request.

flovilmart · 2016-05-27T11:54:34Z

src/PromiseRouter.js

@@ -155,14 +155,18 @@ export default class PromiseRouter {
 function makeExpressHandler(promiseHandler) {
  return function(req, res, next) {
    try {
-      log.verbose(req.method, maskSensitiveUrl(req), req.headers,
-                  JSON.stringify(maskSensitiveBody(req), null, 2));
+      log.verbose('Got request', {


Can we remove the 'Got request'? or remplace by REQUEST or something more easily greppable

flovilmart · 2016-05-27T11:59:11Z

@spenthil there as some reconfigure that can be happening (upon log rotation for example). I like your idea of exposing the global logger like that :)

See:

I'm all willing to simplify all that. Maybe even drop the logsFolder thing, the log rotation stuff for basic logging.

What do you think?

ghost · 2016-05-27T16:34:59Z

@spenthil updated the pull request.

ghost · 2016-05-31T19:45:00Z

@spenthil updated the pull request.

ghost · 2016-06-01T01:08:49Z

@spenthil updated the pull request.

spenthil · 2016-06-01T01:14:28Z

The dashboard only pulls out the message field from the logged object - since this logs objects as meta (winston concept), they don't appear. This is only an issue with VERBOSE log messages. I've updated the PR so some information is included in parse dashboard (http://files.spenthil.com/wZTQ). The rest of the fields are present in the files.

@drew-gross is this a reasonable compromise? This results in the information being minimally logged twice - some stuff unstructured (in message for parse-dashboard) and everything as structured (JSON field).

drew-gross · 2016-06-01T01:28:21Z

Yes I'd say thats fairly reasonable. Maybe you could include an update to the docs as well to make it clear what the default is and how to get more info? @flovilmart I'll let you do the final go-ahead/merge on this one.

ghost · 2016-06-01T01:33:51Z

@spenthil updated the pull request.

spenthil · 2016-06-01T01:57:49Z

@drew-gross: updated docs

ghost · 2016-06-01T01:57:53Z

@spenthil updated the pull request.

drew-gross · 2016-06-06T19:53:41Z

Not having the full request and response body is interfering with my debugging flow a lot more than I thought :( I may have to revert this.

…nity#1922)" This reverts commit bea655e.

… stringified objects (parse-community#1922)"

…tringified objects (parse-community#1922)"

… stringified objects (parse-community#1922)"

* Add unique indexing * Add unique indexing for username/email * WIP * Finish unique indexes * Notes on how to upgrade to 2.3.0 safely * index on unique-indexes: c454180 Revert "Log objects rather than JSON stringified objects (#1922)" * reconfigure username/email tests * Start dealing with test shittyness * Remove tests for files that we are removing * most tests passing * fix failing test * Make specific server config for tests async * Fix more tests * fix more tests * Fix another test * fix more tests * Fix email validation * move some stuff around * Destroy server to ensure all connections are gone * Fix broken cloud code * Save callback to variable * no need to delete non existant cloud * undo * Fix all tests where connections are left open after server closes. * Fix issues caused by missing gridstore adapter * Update guide for 2.3.0 and fix final tests * use strict * don't use features that won't work in node 4 * Fix syntax error * Fix typos * Add duplicate finding command * Update 2.3.0.md

… stringified objects (parse-community#1922)"

* Remove adaptiveCollection * Remove an adaptiveCollection use * Remove an adaptiveCollection * make adaptiveCollection private * Remove collection from mongoadapter * Move schema collection usage into mongo adapter * stop relying on mongo format for removing join tables * reduce usage of schemaCollection * remove uses of _collection * Move CLP setting into mongo adapter * remove all uses of schemaCollection * make schemaCollection private * remove transform from schemaCollection * rename some stuff * Tweak paramaters and stuff * reorder some params * reorder find() arguments * finishsh touching up argument order * Accept a database adapter as a parameter * First passing test with postgres! * Actually use the provided className * index on unique-indexes: c454180 Revert "Log objects rather than JSON stringified objects (#1922)" * Start dealing with test shittyness * Make specific server config for tests async * Fix email validation * Fix broken cloud code * Save callback to variable * undo * Fix tests * Setup travis * fix travis maybe * try removing db user * indentation? * remove postgres version setting * sudo maybe? * use postgres username * fix check for _PushStatus * excludes * remove db=mongo * allow postgres to fail * Fix allow failure * postgres 9.4 * Remove mongo implementations and fix test * Fix test leaving behind connections

… stringified objects (parse-community#1922)" reconfigure username/email tests Fix broken cloud code Save callback to variable undo Fix all tests where connections are left open after server closes. Fix issues caused by missing gridstore adapter remove uses of _collection reorder find() arguments Accept a database adapter as a parameter sudo maybe? use postgres username reorder find() arguments Build objects with default fields correctly Don't tell adapter about ACL WIP

WIP Notes on how to upgrade to 2.3.0 safely index on unique-indexes: c454180 Revert "Log objects rather than JSON stringified objects (parse-community#1922)" reconfigure username/email tests Start dealing with test shittyness most tests passing Make specific server config for tests async Fix more tests Save callback to variable undo remove uses of _collection reorder some params reorder find() arguments finishsh touching up argument order Accept a database adapter as a parameter First passing test with postgres! Fix tests Setup travis sudo maybe? use postgres username reorder find() arguments Build objects with default fields correctly Don't tell adapter about ACL WIP Passing postgres test with user Fix up createdAt, updatedAt, nad _hashed_password handling

… stringified objects (parse-community#1922)" reconfigure username/email tests Fix broken cloud code Save callback to variable undo Fix all tests where connections are left open after server closes. Fix issues caused by missing gridstore adapter remove uses of _collection reorder find() arguments Accept a database adapter as a parameter sudo maybe? use postgres username reorder find() arguments Build objects with default fields correctly Don't tell adapter about ACL WIP

WIP Notes on how to upgrade to 2.3.0 safely index on unique-indexes: c454180 Revert "Log objects rather than JSON stringified objects (parse-community#1922)" reconfigure username/email tests Start dealing with test shittyness most tests passing Make specific server config for tests async Fix more tests Save callback to variable undo remove uses of _collection reorder some params reorder find() arguments finishsh touching up argument order Accept a database adapter as a parameter First passing test with postgres! Fix tests Setup travis sudo maybe? use postgres username reorder find() arguments Build objects with default fields correctly Don't tell adapter about ACL WIP Passing postgres test with user Fix up createdAt, updatedAt, nad _hashed_password handling

… stringified objects (parse-community#1922)" reconfigure username/email tests Fix broken cloud code Save callback to variable undo Fix all tests where connections are left open after server closes. Fix issues caused by missing gridstore adapter remove uses of _collection reorder find() arguments Accept a database adapter as a parameter sudo maybe? use postgres username reorder find() arguments Build objects with default fields correctly Don't tell adapter about ACL WIP

WIP Notes on how to upgrade to 2.3.0 safely index on unique-indexes: c454180 Revert "Log objects rather than JSON stringified objects (parse-community#1922)" reconfigure username/email tests Start dealing with test shittyness most tests passing Make specific server config for tests async Fix more tests Save callback to variable undo remove uses of _collection reorder some params reorder find() arguments finishsh touching up argument order Accept a database adapter as a parameter First passing test with postgres! Fix tests Setup travis sudo maybe? use postgres username reorder find() arguments Build objects with default fields correctly Don't tell adapter about ACL WIP Passing postgres test with user Fix up createdAt, updatedAt, nad _hashed_password handling

facebook-github-bot added the GH Review: review-needed label May 26, 2016

spenthil force-pushed the logging branch from 1227d13 to b45e5d6 Compare May 26, 2016 20:09

spenthil force-pushed the logging branch from b45e5d6 to c73ee66 Compare May 26, 2016 20:52

spenthil force-pushed the logging branch from c73ee66 to 0b328e1 Compare May 26, 2016 20:52

spenthil force-pushed the logging branch from 0b328e1 to c31e48f Compare May 26, 2016 22:14

spenthil force-pushed the logging branch from c31e48f to cb41cd5 Compare May 27, 2016 00:09

spenthil force-pushed the logging branch from cb41cd5 to 433d8a3 Compare May 27, 2016 00:11

spenthil force-pushed the logging branch from 433d8a3 to 0434f95 Compare May 27, 2016 00:21

flovilmart reviewed May 27, 2016
View reviewed changes

spenthil force-pushed the logging branch from 0434f95 to ec85959 Compare May 27, 2016 16:34

flovilmart added GH Review: needs-revision and removed GH Review: review-needed labels May 28, 2016

spenthil force-pushed the logging branch from ec85959 to 2072406 Compare May 31, 2016 19:43

ghost added GH Review: review-needed and removed GH Review: needs-revision labels May 31, 2016

spenthil force-pushed the logging branch from 2072406 to cbe9b97 Compare June 1, 2016 01:08

spenthil force-pushed the logging branch from cbe9b97 to 0dfffed Compare June 1, 2016 01:32

Log objects rather than JSON stringified objects

b47bf39

spenthil force-pushed the logging branch from 0dfffed to b47bf39 Compare June 1, 2016 01:57

flovilmart added GH Review: accepted and removed GH Review: review-needed labels Jun 2, 2016

TylerBrock merged commit bea655e into parse-community:master Jun 2, 2016

drew-gross mentioned this pull request Jun 6, 2016

Revert "Log objects rather than JSON stringified objects" #1995

Merged

drew-gross added a commit to drew-gross/parse-server that referenced this pull request Jun 6, 2016

Revert "Log objects rather than JSON stringified objects (parse-commu…

c454180

…nity#1922)" This reverts commit bea655e.

drew-gross added a commit to drew-gross/parse-server that referenced this pull request Jun 6, 2016

index on unique-indexes: c454180 Revert "Log objects rather than JSON…

8342f3d

… stringified objects (parse-community#1922)"

drew-gross added a commit to drew-gross/parse-server that referenced this pull request Jun 6, 2016

WIP on unique-indexes: c454180 Revert "Log objects rather than JSON s…

d93d9a9

…tringified objects (parse-community#1922)"

drew-gross added a commit to drew-gross/parse-server that referenced this pull request Jun 8, 2016

index on unique-indexes: c454180 Revert "Log objects rather than JSON…

f0353e6

… stringified objects (parse-community#1922)"

drew-gross added a commit to drew-gross/parse-server that referenced this pull request Jun 10, 2016

index on unique-indexes: c454180 Revert "Log objects rather than JSON…

2e58107

… stringified objects (parse-community#1922)"

drew-gross added a commit to drew-gross/parse-server that referenced this pull request Jun 11, 2016

index on unique-indexes: c454180 Revert "Log objects rather than JSON…

f960990

… stringified objects (parse-community#1922)"

drew-gross added a commit to drew-gross/parse-server that referenced this pull request Jun 12, 2016

index on unique-indexes: c454180 Revert "Log objects rather than JSON…

2abccbb

… stringified objects (parse-community#1922)"

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Log objects rather than JSON stringified objects #1922

Log objects rather than JSON stringified objects #1922

spenthil commented May 26, 2016 •

edited

Loading

ghost commented May 26, 2016

codecov-io commented May 26, 2016 •

edited

Loading

facebook-github-bot commented May 26, 2016

facebook-github-bot commented May 26, 2016

ghost commented May 26, 2016

ghost commented May 27, 2016

ghost commented May 27, 2016

ghost commented May 27, 2016

flovilmart May 27, 2016 •

edited

Loading

spenthil May 31, 2016

flovilmart commented May 27, 2016

ghost commented May 27, 2016

ghost commented May 31, 2016

ghost commented Jun 1, 2016

spenthil commented Jun 1, 2016

drew-gross commented Jun 1, 2016

ghost commented Jun 1, 2016

spenthil commented Jun 1, 2016

ghost commented Jun 1, 2016

drew-gross commented Jun 6, 2016

Log objects rather than JSON stringified objects #1922

Log objects rather than JSON stringified objects #1922

Conversation

spenthil commented May 26, 2016 • edited Loading

ghost commented May 26, 2016

codecov-io commented May 26, 2016 • edited Loading

Current coverage is 91.87%

facebook-github-bot commented May 26, 2016

facebook-github-bot commented May 26, 2016

ghost commented May 26, 2016

ghost commented May 27, 2016

ghost commented May 27, 2016

ghost commented May 27, 2016

flovilmart May 27, 2016 • edited Loading

Choose a reason for hiding this comment

spenthil May 31, 2016

Choose a reason for hiding this comment

flovilmart commented May 27, 2016

ghost commented May 27, 2016

ghost commented May 31, 2016

ghost commented Jun 1, 2016

spenthil commented Jun 1, 2016

drew-gross commented Jun 1, 2016

ghost commented Jun 1, 2016

spenthil commented Jun 1, 2016

ghost commented Jun 1, 2016

drew-gross commented Jun 6, 2016

spenthil commented May 26, 2016 •

edited

Loading

codecov-io commented May 26, 2016 •

edited

Loading

flovilmart May 27, 2016 •

edited

Loading