"error 37 from memcached_set: SUCCESS" when setting value > 1 MB

[jwhitlock]

With libmemcached 1.0.18 (OS X, with homebrew patches), I get an unexpected error value when attempting to set a value greater than 1 MB:

>>> import pylibmc 
>>> mc = pylibmc.Client(["127.0.0.1"])
>>> mc['foo'] = 'a' * 1024 * 1024
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pylibmc/src/pylibmc/client.py", line 163, in __setitem__
    if not self.set(key, value):
pylibmc.Error: error 37 from memcached_set: SUCCESS

I'm expecting an error, because of the 1 MB limit, but I was hoping for a more specific error, and maybe one that includes the length of the value after serialization and compression (see django-pylibmc/django-pylibmc#29).

This sounds like a similar issue as #180, but without the binary protocol and behaviors used in that test case.

[lericson]

Hello @jwhitlock! Sadly, as is the case for #180, this is not something we can realistically do anything about. Perhaps it's time we rolled up our sleeves and started taking on active development of libmemcached. Until that time I'm afraid I can't do much for the sometimes confusing error messages and other behaviors in libmemcached. Other than that I am open to ideas!

[jwhitlock]

c and bzr and launchpad - that's a bit of a technical hill to climb, and I feel I'd need to get comfortable with _pylibmcmodule.c first. But might be worth it to nail down why these errors aren't as useful as they could be.

Another thing that might help is exposing the value serialization, so that I can add the length of the value to exceptions, with extra emphasis if they go over the 1 MB limit. That might be _PylibMC_SerializeValue, but it looks like key preparation might be non-trivial.

[austinpatrickbishop]

Is it possible to increase the max value size to be larger than 1MB? I could not find this in the docs

[lericson]

Sorry, no, it's not really something we defined a limit for. I'm not sure if it's from the protocol itself or just libmemcached, though.

[jwhitlock]

It's the famous 1MB upper limit in memcached itself. I had to dig into this a few years ago, when the project was hosted on code.google.com and there was a FAQ with the explanation of why. I seem to remember it boiling down to "if you need to cache something that big, you are doing it wrong", that TCP overhead would start to get ridiculous. It looks like you can increase it up to 128MB on the command line, if you have that server access, with something like memcached -I 128m. There may be a way to request the setting value over the API, if we wanted to do something smarter, but we'd have to guess at the overhead of storage structures, so it might not be worth it.

[austinpatrickbishop]

Understood, thank you.

[lericson]

It seems to me that we can't do much about this; closing for lack of means of resolution.

[boxbeatsy]

Hi all, sorry for posting on a closed thread, but does anyone know if this "error 37 from memcached_set: SUCCESS" error with libmemcached persists?

I see references across the web of this error, but I can't reproduce it. If it still exists, I have to handle it gracefully in my application logic, but I'm wondering if it's actually been fixed in a recent update to libmemcached?

[jwhitlock]

Yes, it reproduces for me, running the above code, with:

• memcached 1.4.24
• pylibmc 1.5.1
• libmemcached 1.0.18_1 installed via Homebrew
• OS X El Capitan 10.11.6

[boxbeatsy]

Thanks! I ended up using django_pylibmc to configure memcached with my application, and it handles this error gracefully by default.

[jwhitlock]

django_pylibmc uses compression, and 1,000,000 copies of 'a' compresses to much less than 1 MB of data. You'll need to pass more than 1MB of random data to actually store 1 MB of data and trigger the error. In that case, the error is logged and False is returned. Your app will have to be OK with not caching your big value, maybe silently.

[lericson]

Hm, sounds like a poor strategy to silence an error. No warnings, nothing?

[jwhitlock]

django_pylibmc would like to do something more intelligent about the upper limit, like in django-pylibmc/django-pylibmc#29, but the feedback we get from pylibmc is a generic error that doesn't include the data we'd need to, for example, to switch to chunking a large value into pieces that memcached would like. An example would be an error that communicated that the stored item was too large to fit in a memcached bucket, and then we could chunk the value into half-sized chunks until it fit. Even better would be to communicate what the memcached bucket size is (1MB by default, but can be changed).

The best django_pylibmc can do by default is log the error, since Django doesn't expect cache calls to raise exceptions. If a client wants to do more than that, they can extend the class and try something else.

I opened this bug in the hope that pylibmc could raise a more specific error than "Error: Success!" I understand that may mean diving into libmemcached, or maybe even memcached, which is barely surviving the transition to GitHub. I'm not doing it, so I can't really expect anyone else to do it either.

[lericson]

[…] the feedback we get from pylibmc is a generic error that doesn't include the data we'd need […]

Right, I see the problem, it’s just that pylibmc is in essentially the same situation: we don’t have the information needed to give you a proper error, likely because memcached doesn’t even send an error but rather just closes the connection or something. I’m not sure but I would suspect it to be this way given how the error comes up.

The best django_pylibmc can do by default is log the error https://github.com/django-pylibmc/django-pylibmc/blob/master/django_pylibmc/memcached.py#L145-L147 […]

Isn't that enough, though?
I opened this bug in the hope that pylibmc could raise a more specific error than "Error: Success!" I understand that may mean diving into libmemcached, or maybe even memcached, which is barely surviving the transition to GitHub. I'm not doing it, so I can't really expect anyone else to do it either.

I'd think the libmemcached people should be interested in fixing this, but you’re right, it’s maybe not the funnest of bugs to go at.

[lericson]

So curiosity got the better of me, here are my findings:

$ nc -cv 127.0.0.1 11211
found 0 associations
found 1 connections:
     1: flags=82<CONNECTED,PREFERRED>
    outif lo0
    src 127.0.0.1 port 51561
    dst 127.0.0.1 port 11211
    rank info not available
    TCP aux info available

Connection to 127.0.0.1 port 11211 [tcp/*] succeeded!
set a 0 0 1
h
STORED

Works well.

set a 0 0 2147483648
CLIENT_ERROR bad command line format
set a 0 0 2147483647
CLIENT_ERROR bad command line format
set a 0 0 1073741823
SERVER_ERROR object too large for cache

Don't work so well. The first one is 2^31, the second one is one less, the third one is 2^30 - 1. It turns out the breakpoint is at exactly 2^30 - 16 -- presumably 16 bytes of bookkeeping is included in the size limit.

[lericson]

Surprisingly, this is a very detectable and reportable error. It should in fact be triggering pylibmc.ClientError and pylibmc.ServerError respectively.

[lericson]

Also, the error pylibmc triggers for 2^31 byte values:

pylibmc.Error: error 26 from memcached_set: Socket is not connected

Whereas for 2^30 byte values, we get

pylibmc.Error: error 37 from memcached_set: SUCCESS

[lericson]

It's an issue with pylibmc!

[lericson]

Well, maybe I got your hopes up with my last message. However, as it turns out, I can really only fix the latter of the previous two cases, the midpoint case. It also turns out my math was off somehow, it seems memcached gives a server error when 2**20 - 72 < len(value) < 2**31 - 2.

It also seems that when memcached sends a SERVER_ERROR, it also disconnects you. This is why we get "socket is not connected."

[lericson]

So I think this is all I can do for now. The following program now works with libmemcached 1.0.2 and above:

import pylibmc

v = 'h'*(2**30)
mc = pylibmc.Client(['127.0.0.1:11211'])

try:
    mc.set('a', v)
except pylibmc.TooBig:
    print('(sad trumpet) value was too big')