Sync Gateway crashes while processing webhook events

[ashvindersingh]

Scenario:

• Install the 1.1 release SG
• Configure with following event handler:
  "event_handlers": {"document_changed":[{"handler":"webhook","max_processes" : 500,"wait_for_process" : "600000","url":"http://testServer:port","timeout":60}] }
• Create and replicate 10K documents
• Create 10 revisions for each document.

After sending webhook events for 5K documents. the SG crashes. All the documents have been replicated to SG.

Here is the snippet from the SG logs:

2015-04-28T18:05:29.894-07:00 WARNING: Error attempting to post to url http://172.23.105.219:9091: Post http://172.23.105.219:9091: dial tcp 172.23.105.219:9091: too many open files -- db.(_Webhook).HandleEvent() at event_handler.go:97
2015-04-28T18:05:29.894-07:00 WARNING: Error attempting to post to url http://172.23.105.219:9091: Post http://172.23.105.219:9091: dial tcp 172.23.105.219:9091: too many open files -- db.(_Webhook).HandleEvent() at event_handler.go:97

2015-04-28T18:05:29.894-07:00 CRUD: Stored doc "api-revision1_9998" / "1-46d14cc5eea4aa91c14b208ca7698687"
2015-04-28T18:05:29.894-07:00 CRUD: Stored doc "api-revision1_9998" / "1-46d14cc5eea4aa91c14b208ca7698687"

2015-04-28T18:05:29.894-07:00 WARNING: Error attempting to post to url http://172.23.105.219:9091: Post http://172.23.105.219:9091: dial tcp 172.23.105.219:9091: too many open files -- db.(_Webhook).HandleEvent() at event_handler.go:97
2015-04-28T18:05:29.894-07:00 WARNING: Error attempting to post to url http://172.23.105.219:9091: Post http://172.23.105.219:9091: dial tcp 172.23.105.219:9091: too many open files -- db.(_Webhook).HandleEvent() at event_handler.go:97

2015-04-28T18:05:29.894-07:00 CRUD: Stored doc "api-revision1_9999" / "1-46d14cc5eea4aa91c14b208ca7698687"
2015-04-28T18:05:29.894-07:00 CRUD: Stored doc "api-revision1_9999" / "1-46d14cc5eea4aa91c14b208ca7698687"

2015-04-28T18:05:29.894-07:00 WARNING: Error attempting to post to url http://172.23.105.219:9091: Post http://172.23.105.219:9091: dial tcp 172.23.105.219:9091: too many open files -- db.(_Webhook).HandleEvent() at event_handler.go:97
2015-04-28T18:05:29.894-07:00 WARNING: Error attempting to post to url http://172.23.105.219:9091: Post http://172.23.105.219:9091: dial tcp 172.23.105.219:9091: too many open files -- db.(_Webhook).HandleEvent() at event_handler.go:97

2015-04-28T18:05:29.895-07:00 HTTP+: #208: --> 201 (15.8 ms)
2015-04-28T18:05:29.895-07:00 HTTP+: #208: --> 201 (15.8 ms)

2015-04-28T18:05:32.646-07:00 HTTP: #209: PUT /db/_local/110322b00ab9dc4b74f89d2dd523a982f207535c
2015-04-28T18:05:32.646-07:00 HTTP: #209: PUT /db/_local/110322b00ab9dc4b74f89d2dd523a982f207535c

2015-04-28T18:05:32.646-07:00 HTTP+: #209: --> 201 (0.2 ms)
2015-04-28T18:05:32.646-07:00 HTTP+: #209: --> 201 (0.2 ms)

2015/04/28 18:05:32 http: Accept error: accept tcp [::]:9888: too many open files; retrying in 5ms
2015/04/28 18:05:32 http: Accept error: accept tcp [::]:9888: too many open files; retrying in 5ms

verify replicated num-docs=10000

not ok 10006 failed to get db info from http://localhost:9888/db/_all_docs: Error: read ECONNRESET

---

file:   /Users/ashvinder/mygit/sync-gateway-tests/node_modules/coax/node_modules/hoax/lib/hoax-core.js
line:   16
column: 9
stack:  

[adamcfraser]

Can you share a link to your testing code?

[ashvindersingh]

Hi Adam, I'm using sync-gateway-tests repo
to run the test:

• please install https://github.com/couchbaselabs/sync-gateway-tests
• copy the file https://gist.github.com/ashvindersingh/db45a789408df221f276 to /tests/ folder
• run with node tests/crash_sg_with_webhooks.js

[ashvindersingh]

Also please overwrite the sync-gateway-tests/config/admin_party.json file in with this file:
https://gist.github.com/ashvindersingh/dbd2ae3b0c328a209c4f

[adamcfraser]

Reviewed in detail with Ashvinder, but we weren't able to reproduce a SG crash. Ashvinder took action item to identify better steps to reproduce, and to provide the full SG log if he's able to repro.

[ashvindersingh]

The crash does not occur with latest release of 1.1.0-12.

[ashvindersingh]

Verified the issue does not occur by generating 10 k documents with 10 revisions each.