Add a warning in documentation about memory consumption on varnish

[joelwurtz]

Hey,

We use the user context cache for some months on a heavy load media website. And we were having since the beginning inconsistence on random use case (disconnected user, blank esi, bad hash, ....).

Over the months inconsistence grew up until this week where it was happening at a faster peace.

We are using the following features from varnish / fos http cache :

• User context cache (and hash is cached for a 900s ttl)
• Tag cache
• ESI
• Grace mode

After some brainfuck debugging we discover that Varnish has a specific configuration for memory used when dealing with header (sess_workspace in varnish 3 , workspace_client / workspace_backend in varnish 4)

Using all of this features makes varnish hit the limit pretty often of the default configuration (64k memory per request / session by default), specially when doing regex against a big header (like a regsuball on the Cookie in our case).

Maybe it will be a good thing to add a warning about this in documentation at the top of varnish reference to alert user on this.

One of the good way to monitor this use case is to check the LostHeader tag in varnishlog / varnishtop which indicates that varnish had to drop a Header because of workspace memory overflow.

[dbu]

this is interesting! yes, i agree its good to add such things to the documentation. should we add a Troubleshooting main section? i think adding it to the Caching Proxy Configuration > Varnish Configuration page is overloading that page. it could however have a note somewhere to this one. do you think the cookie case is the most likely to cause troubles? then the cross-reference note should go somewhere near http://foshttpcache.readthedocs.org/en/latest/varnish-configuration.html#cleaning-the-cookie-header

[ddeboer]

Interesting indeed! @joelwurtz Are you using many different tags for single responses? I can imagine that makes the header size grow substantially as well.

[dbu]

@joelwurtz do you plan to work on this?

[joelwurtz]

Don't have time, but will review if someone else works on it

[andrerom]

Not run into this with xkey, however what we have had customers running into is the header limit. 8k by default* and so on. Tried xkey since you reported this @joelwurtz?

* Side for #451: It's 16k on Fastly where it's not configurable. Not had anyone hit that yet, but kind of waiting for it to happen

[joelwurtz]

No, did not try it, not using varnish ATM so difficult to have some feedback on this unfortunetaly.

[dbu]

note that we do split requests for tag invalidation to not explode the header size. and since #424 we even split the response header into multiple lines if needed. at least varnish and symfony cache support this.

[dbu]

also we optimized a bit to avoid adding the same tag multiple times #423

[dbu]

we did another fix for user context to make the cache much more efficient in #497