Varnish retries/timeout is not honoured

[strk]

I've installed a local version of 2.5.1 and I have a varnish server that requires a password (or so it seems by netcat'ing into it). As a result any editing on cartodb tables hangs forever.

Given varnish_management section of app_config.yml requests 5 retries and a timeout of 5 it looks like those settings are not being honoured.

I'd love to look further into the issue, but this would be better done after the varnish trigger is centralized, as requested in bug #197

[strk]

Errata corrige: it's not "forever" as eventually the statement_timeout kicks in (5 minutes later):

Timing is on.
cartodb_dev_user_1_db=> insert into test_1037 (the_geom) select CDB_LatLng(0,0);
WARNING:  Varnish connection error: 'error' object has no attribute 'errno'
CONTEXT:  PL/Python function "update_timestamp"
ERROR:  plpy.Error: canceling statement due to statement timeout
CONTEXT:  Traceback (most recent call last):
  PL/Python function "update_timestamp", line 15, in <module>
    plpy.warning('Varnish connection error: ' +  str(err))
PL/Python function "update_timestamp"
Time: 300000.631 ms

[strk]

NOTE: shutting down Varnish drops the blockage and indeed is non-critical if configuration asks so.

[strk]

Centralization of varnish invalidation landed in 2.5.5 so there's now a cdb_invalidate_varnish function defined in app/models/user.rb by the User.create_function_invalidate_varnish method that embeds those settings.

I'll have to check of the function definition got the correct settings, and I guess we'll need to document what's needed when those settings change (the function definition should be recreated...).

In any case this bug would best be handled by adding an automated testcase for it, as varnish invalidation is a spot that's bound to change soon (easier done with a safety net)

[strk]

I'm running Varnish server 3.0.2 and it does not greet new connections, while it looks like varnish.py expects to read a "Type 'quit' to close CLI session" message, thus the hang.

[strk]

See #138 in case it's related

[krdyke]

For what it's worth, I was experiencing a similar issue and upgrading the python-varnish lib to v0.2.1 seemed to resolve the problem with hanging (which at least in my case was a result of the update_timestamp() trigger).

The docs indicate that v0.2.1 should be used with Varnish version 3+

https://github.com/justquick/python-varnish

[strk]

I confirm I have varnish server 3.0.4 installed and python-varnish 0.1.2 as requested by python_requirements.txt.
Not sure how to deal with this. The current README doesn't mention that varnish < 3.0 is required, only says "2.1+".

Ideally we'd support both old and new varnish, but this requires a change in python_requirements.txt based on the version we want to use. The version could be autodetected in a ./configure step, but not if the varnish server is on another machine.

@lbosque how do you think we should deal with this ? I'm happy to draft a ./configure script to generate python_requirements.txt based on --with-varnish-version switch

[strk]

For the record, this command brought me a recent and working python-varnish:
sudo pip install --upgrade python-varnish
We could document this in the README and be done...

[strk]

Errata corrige: upgrading python-varnish to 0.2.1 doesn't really "work", it just fixes the timeout issue, but still fails with:

WARNING:  Varnish fetch error: Bad response code: 101 Unknown request. (purge obj.http.X-Cache-Channel ~ "^cartodb_dev_user_1_db:(.*b.*)|(cdb_tablemetadata)|(table)$")
CONTEXT:  PL/Python function "cdb_invalidate_varnish"

Which suggests me upgrade of python-varnish would also require re-writing the "cdb_invalidate_varnish" function.

[xiangdao]

I was having a similar timeout issue and upgrading python-varnish to 0.2.1 would result in same error message as @strk.

My workaround is to disable admin access authentication by NOT specify -S /path/to/secret_file when starting varnishd.

My understanding is by default, the varnish daemon running on my instance is expecting client to authenticated according to an existing secret file, when conducting admin operations such as purge the cache. However, the trigger function created by CartoDB (app/models/table.rb) doesn’t take any secret into consideration. When the request comes through, the server returns a random string expecting client to ping back the correct SHA value, while the client is waiting to see some text (“Type 'quit' to close CLI session.\n\n") which indicates the connection has been established. Because the trigger function was created after table creation, each new table import works fine, but if I try to update the table, e.g adding a column, it would timeout.

I have to point out the CartoDB repo I'm working on was fairly old (v2.3.1). And it seems cartodb is moving away from this update_timestamp() function and use trigger on CDB_TableMetadata instead. So probably my solution is out of date and not necessary for later release.

[aeneaswiener]

@xiangdao, I can confirm that not specifying a secret file (via -S parameter on Varnish startup) has fixed this particular issue for my setup with varnish-3.0.2.