Segmentation fault (core dumped) when adding GeoJSON polygon in Aerospike

[anuranjit]

I am getting Segmentation fault (core dumped) when i am trying to put GeoJSON in aerospike in persistent namspace. It happens very frequently.
I also create an Index on the bbox field.

Below is the sample code ( This runs in a loop of 300000)

     geo_gson = GeoJSON({
                    "type": "Polygon",
                    "coordinates": [[
                        [min_lng, max_lat],
                        [max_lng, max_lat],
                        [max_lng, min_lat],
                        [min_lng, min_lat],
                        [min_lng, max_lat],
                    ]]
                })
                bins = {
                    "row": row, 
                    "col": col,
                    "city": city,
                    "bbox": geo_gson,
                    "min_lat": min_lat,
                    "min_lng": min_lng,
                    "max_lat": max_lat,
                    "max_lng": max_lat
                }
          aerospike_client.put(("namespace", "set", "mykey"), bins)

I am using aerospike server 3.8.3 There is no error in aerospike log. This just fails without any error information.

{'city': 1L, 'max_lng': 12.66811406299999, 'min_lng': 77.229080252800131, 'bbox': '{"type": "Polygon", "coordinates": [[[77.229080252800131, 12.66811406299999], [77.239080252800136, 12.66811406299999], [77.239080252800136, 12.65811406299999], [77.229080252800131, 12.65811406299999], [77.229080252800131, 12.66811406299999]]]}', 'max_lat': 12.66811406299999, 'min_lat': 12.65811406299999, 'col': 26, 'row': 46}
This is the bin for which it crashes (client)

If I limit the float precision to 6 decimal places it works.

Using python client version 2.0.5

GDB stack trace

        Program received signal SIGSEGV, Segmentation fault.
        0x00007ffff203cd8f in as_geojson_len (string=string@entry=0x4a0af78) at src/main/aerospike/as_geojson.c:89
        89	src/main/aerospike/as_geojson.c: No such file or directory.
        (gdb) bt full
        #0  0x00007ffff203cd8f in as_geojson_len (string=string@entry=0x4a0af78) at src/main/aerospike/as_geojson.c:89
        No locals.
        #1  0x00007ffff202940f in as_command_value_size (val=0x4a0af78, buffer=buffer@entry=0x7ffffffdcba0) at src/main/aerospike/as_command.c:112
        No locals.
        #2  0x00007ffff201ef35 in as_command_bin_size (buffer=0x7ffffffdcba0, bin=0x4a0af68) at src/include/aerospike/as_command.h:210
        No locals.
        #3  aerospike_key_put (as=0x1ac0f20, err=err@entry=0x7fffffffced0, policy=0x1ac2010, key=key@entry=0x7fffffffcd40, rec=rec@entry=0x7fffffffcdf0)
        at src/main/aerospike/aerospike_key.c:358
        i = <optimized out>
        status = <optimized out>
        bins = 0x4a0aec0
        n_bins = 8
        buffers = 0x7ffffffdcb70
        n_fields = 3
        size = 146
        cmd = <optimized out>
        p = <optimized out>
        cn = {node = 0x7fffffffcdf0, ns = 0x1f20311d7 <error: Cannot access memory at address 0x1f20311d7>, digest = 0x7fffdd3ba6e0 "\002",
        replica = (unknown: 4294823080), write = 255}
        msg = {proto = {version = 0 '\000', type = 0 '\000', sz = 0, data = 0x7ffffffdcc88 "T\320\063\335\377\177"}, m = {header_sz = 84 'T',
        info1 = 208 '\320', info2 = 51 '3', info3 = 221 '\335', unused = 255 '\377', result_code = 127 '\177', generation = 4118806528,
        record_ttl = 2147481577, transaction_ttl = 3240624128, n_fields = 149, n_ops = 0, data = 0x7ffffffdcc9e ""}}
        #4  0x00007ffff1fed590 in AerospikeClient_Put_Invoke (self=self@entry=0x99ecd0, py_key=0x7fffdd3ff4b0, py_bins=0x7fffdd3ba910,
        py_meta=<optimized out>, py_policy=0x0, serializer_option=1) at src/main/client/put.c:104
        _save = 0x9440a0
        err = {code = AEROSPIKE_OK,
        message = "\000\000\000\000\340\025\221", '\000' <repeats 13 times>, "\350\027=\335\377\177\000\000\002\000\000\000\002\000\000\000\370\247;\335\377\177\000\000b\336\006\362\377\177", '\000' <repeats 18 times>, "\201\230M\000\000\000\000\000\070\375\177\363\377\177\000\000\377\377\377\377\377\377\377\377\330\302;\335\377\177\000\000\240@\224\000\000\000\000\000\240@\224\000\000\000\000\000`\275\237\004\000\000\000\000@;<\335\377\177\000\000^\336\006\362\377\177\000\000\000\000\000\000\000\000\000\000\020\266\237\004\000\000\000\000\\\336\006\362\377\177\000\000\275\222S\000\000\000\000\000\240@\224\000\000\000\000\000\220\267\237\004\000\000\000\000\020\266\237\004\000\000\000\000\240@\224\000\000\000\000\000"...,
        func = 0x0, file = 0x0, line = 0}
        write_policy = {timeout = 0, retry = 0, compression_threshold = 0, key = AS_POLICY_KEY_DIGEST, gen = AS_POLICY_GEN_IGNORE,
        exists = AS_POLICY_EXISTS_IGNORE, commit_level = AS_POLICY_COMMIT_LEVEL_ALL, durable_delete = false}
        write_policy_p = 0x0
        key = {_free = false, ns = "<persitent_namespace>\000\000\000\000\000\000\210i\302\356\377\377\377\377\000\000\000\000\377\377\377\377\060",
        set = "latlng\000\300\212`\365\377\177\000\000\220\001\222\000\000\000\000\000\220\001\222\000\000\000\000\000\070\375\177\363\377\377\377\377\000\000\000\000\377\377\377\377\060\230\257\365\377\177\000\000\260i\302\356\377\177\000", <incomplete sequence \320>, value = {integer = {_ = {
        type = AS_STRING, free = false, count = 1}, value = 140737352632832}, string = {_ = {type = AS_STRING, free = false, count = 1},
        free = false, value = 0x7fffdd43cbec "1.cluster.3.89", len = 14}, bytes = {_ = {type = AS_STRING, free = false, count = 1},
        capacity = 4159244800, size = 32767, value = 0x7fffdd43cbec "1.cluster.3.89", free = 14, type = AS_BYTES_UNDEF}},
        valuep = 0x7fffffffcda8, digest = {init = true, value = ";\324\024Rj\316{\257\t\225P\235\060͏0\313\306P\006"}}
        ---Type <return> to continue, or q <return> to quit---
        rec = {_ = {_ = {type = AS_REC, free = false, count = 1}, data = 0x7fffffffcdf0, hooks = 0x7ffff2296ec0 <as_record_rec_hooks>}, key = {
        _free = false, ns = "\000\372\367\377\177\000\000\002", '\000' <repeats 16 times>, "\030=\335\377\177\000\000\252",
        set = "\000V\000\000\000\000\000\240@\224", '\000' <repeats 13 times>, "\070\211`\365\377\177\000\000\000\000\000\000\000\000\000\000\002\000\000\000\377\177", '\000' <repeats 18 times>, value = {integer = {_ = {type = 16, free = 104, count = 32767}, value = 0}, string = {_ = {
        type = 16, free = 104, count = 32767}, free = false,
        value = 0x7fff00000005 <error: Cannot access memory at address 0x7fff00000005>, len = 0}, bytes = {_ = {type = 16, free = 104,
        count = 32767}, capacity = 0, size = 0, value = 0x7fff00000005 <error: Cannot access memory at address 0x7fff00000005>,
        free = false, type = AS_BYTES_UNDEF}}, valuep = 0x0, digest = {init = false, value = '\000' <repeats 19 times>}}, gen = 0, ttl = 0,
        bins = {_free = true, capacity = 8, size = 8, entries = 0x4a0aec0}}
        key_initialised = true
        record_initialised = true
        static_pool = {bytes_pool = {{_ = {type = AS_UNDEF, free = false, count = 0}, capacity = 0, size = 0, value = 0x0, free = false,
        type = AS_BYTES_UNDEF} <repeats 4096 times>}, current_bytes_id = 0}
        __func__ = "AerospikeClient_Put_Invoke"
        #5  0x00007ffff1fed806 in AerospikeClient_Put (self=0x99ecd0, args=<optimized out>, kwds=<optimized out>) at src/main/client/put.c:182
        py_key = 0x7fffdd3ff4b0
        py_bins = 0x7fffdd3ba910
        py_meta = 0x0
        py_policy = 0x0
        py_serializer_option = 0x0
        serializer_option = <optimized out>
        kwlist = {0x7ffff206ba78 "key", 0x7ffff2078584 "bins", 0x7ffff206de66 "meta", 0x7ffff2070059 "policy", 0x7ffff206fe07 "serializer", 0x0}
        #6  0x000000000052f936 in PyEval_EvalFrameEx ()
        No symbol table info available.
        #7  0x000000000055c594 in PyEval_EvalCodeEx ()
        No symbol table info available.
        #8  0x000000000052ca8d in PyEval_EvalFrameEx ()
        No symbol table info available.
        #9  0x000000000055c594 in PyEval_EvalCodeEx ()
        No symbol table info available.
        #10 0x000000000052ca8d in PyEval_EvalFrameEx ()
        No symbol table info available.
        #11 0x000000000052cf32 in PyEval_EvalFrameEx ()
        No symbol table info available.
        #12 0x000000000055c594 in PyEval_EvalCodeEx ()
        No symbol table info available.
        #13 0x00000000005b7392 in PyEval_EvalCode ()
        No symbol table info available.
        #14 0x0000000000469663 in ?? ()
        No symbol table info available.
        #15 0x00000000004699e3 in PyRun_FileExFlags ()
        No symbol table info available.
        ---Type <return> to continue, or q <return> to quit---
        #16 0x0000000000469f1c in PyRun_SimpleFileExFlags ()
        No symbol table info available.
        #17 0x000000000046ab81 in Py_Main ()
        No symbol table info available.
        #18 0x00007ffff7815f45 in __libc_start_main (main=0x46ac3f <main>, argc=4, argv=0x7fffffffdd88, init=<optimized out>, fini=<optimized out>,
        rtld_fini=<optimized out>, stack_end=0x7fffffffdd78) at libc-start.c:287
        result = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, -7425880562582933243, 5720405, 140737488346496, 0, 0, 7425880563629961477,
        7425899199553483013}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x5bb9c0 <__libc_csu_init>, 0x7fffffffdd88}, data = {
        prev = 0x0, cleanup = 0x0, canceltype = 6011328}}}
        not_first_call = <optimized out>
        #19 0x000000000057497e in _start ()

[aerospikerobertmarks]

Hi, Thanks for reporting this, we are looking into the causes of the issue.

I've been able to reproduce this for versions up to 2.0.8 of the Python Client, but have been unable to reproduce it after that. Are you still experiencing it for the current version: 2.0.13 ?

Is that stack trace from the Python Client version 2.0.5?

Also if you could let us know what version of Python your are using it would be helpful.
python --version should print out that information.

Thank you again for letting us know about the issue and sending the code which triggers the problem; it is very useful!

[anuranjit]

@aerospikerobertmarks

1: The issue was happening in 2.0.13 also but not as consistent.
2: Yes stack trace is from 2.0.5
3: Python version 2.7.12

[anuranjit]

@aerospikerobertmarks

Here is a stack trace from 2.0.13.
One thing, before putting data to aerospike, I am scanning and deleting all the keys in the same set.
I ran the script after restarting aerospike server version 3.8.3. But i don't thinkt the error is related to this

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
strlen () at ../sysdeps/x86_64/strlen.S:106
106	../sysdeps/x86_64/strlen.S: No such file or directory.
(gdb) bt full
#0  strlen () at ../sysdeps/x86_64/strlen.S:106
No locals.
#1  0x00007fffd42f7d33 in as_geojson_len (string=0x5fc3868) at src/main/aerospike/as_geojson.c:89
No locals.
#2  0x00007fffd42deb65 in as_command_value_size (val=0x5fc3868, buffer=buffer@entry=0x7ffffffdc790) at src/main/aerospike/as_command.c:112
No locals.
#3  0x00007fffd42d3f5d in as_command_bin_size (buffer=0x7ffffffdc790, bin=0x5fc3858) at src/include/aerospike/as_command.h:211
No locals.
#4  aerospike_key_put (as=0x9faed0, err=err@entry=0x7ffffffdcab0, policy=0x9fafd8, key=key@entry=0x7ffffffdc920, rec=rec@entry=0x7ffffffdc9d0) at src/main/aerospike/aerospike_key.c:361
        i = <optimized out>
        status = <optimized out>
        bins = 0x5fc37b0
        n_bins = 8
        buffers = 0x7ffffffdc760
        n_fields = 3
        size = 146
        cmd = <optimized out>
        p = <optimized out>
        cn = {node = 0x100000000, ns = 0x0, digest = 0x7ffffffdc880 "\240\034\241", replica = (unknown: 3199811668)}
        msg = {proto = {version = 32 ' ', type = 201 '\311', sz = 2147483645, data = 0x7ffffffdc868 "\260\"\351\367\377\177"}, m = {header_sz = 176 '\260', info1 = 34 '"', info2 = 233 '\351', 
            info3 = 247 '\367', unused = 255 '\377', result_code = 127 '\177', generation = 3479175168, record_ttl = 153, transaction_ttl = 1114112, n_fields = 0, n_ops = 0, data = 0x7ffffffdc87e ""}}
#5  0x00007fffd429ed90 in AerospikeClient_Put_Invoke (self=self@entry=0xa11ca0, py_key=0x7fffce4a7370, py_bins=0x7fffc200ce88, py_meta=<optimized out>, py_policy=0x0, serializer_option=1)
    at src/main/client/put.c:104
        _save = 0x9850a0
        err = {code = AEROSPIKE_OK, message = '\000' <repeats 1023 times>, func = 0x0, file = 0x0, line = 0}
        write_policy = {timeout = 0, retry = 0, sleep_between_retries = 0, compression_threshold = 0, key = AS_POLICY_KEY_DIGEST, gen = AS_POLICY_GEN_IGNORE, exists = AS_POLICY_EXISTS_IGNORE, 
          commit_level = AS_POLICY_COMMIT_LEVEL_ALL, retry_on_timeout = false, durable_delete = false}
        write_policy_p = 0x0
        key = {_free = false, ns = "<persistent_namespace>", '\000' <repeats 22 times>, set = "latlng", '\000' <repeats 57 times>, value = {integer = {_ = {type = AS_STRING, free = false, count = 1}, value = 0}, string = {
              _ = {type = AS_STRING, free = false, count = 1}, free = false, value = 0x7fffc2006d3c "18.cluster.22.20", len = 16}, bytes = {_ = {type = AS_STRING, free = false, count = 1}, capacity = 0, 
              size = 0, value = 0x7fffc2006d3c "18.cluster.22.20", free = 16, type = AS_BYTES_UNDEF}}, valuep = 0x7ffffffdc988, digest = {init = true, 
            value = "ӧs\371:\037+\233\005\347\065\\\333\373\021@\324\344", <incomplete sequence \343>}}
        rec = {_ = {_ = {type = AS_REC, free = false, count = 1}, data = 0x7ffffffdc9d0, hooks = 0x7fffd4555ec0 <as_record_rec_hooks>}, key = {_free = false, ns = '\000' <repeats 31 times>, 
            set = '\000' <repeats 63 times>, value = {integer = {_ = {type = AS_UNDEF, free = false, count = 0}, value = 0}, string = {_ = {type = AS_UNDEF, free = false, count = 0}, free = false, value = 0x0, 
                len = 0}, bytes = {_ = {type = AS_UNDEF, free = false, count = 0}, capacity = 0, size = 0, value = 0x0, free = false, type = AS_BYTES_UNDEF}}, valuep = 0x0, digest = {init = false, 
              value = '\000' <repeats 19 times>}}, gen = 0, ttl = 0, bins = {_free = true, capacity = 8, size = 8, entries = 0x5fc37b0}}
        key_initialised = true
        record_initialised = true
        static_pool = <error reading variable static_pool (value of type `as_static_pool' requires 131080 bytes, which is more than max-value-size)>
        __func__ = "AerospikeClient_Put_Invoke"
#6  0x00007fffd429f023 in AerospikeClient_Put (self=0xa11ca0, args=<optimized out>, kwds=<optimized out>) at src/main/client/put.c:182
        py_key = 0x7fffce4a7370
        py_bins = 0x7fffc200ce88
        py_meta = 0x0
        py_policy = 0x0
        py_serializer_option = 0x0
        serializer_option = <optimized out>
        kwlist = {0x7fffd4329058 "key", 0x7fffd43370b4 "bins", 0x7fffd432b846 "meta", 0x7fffd432dc79 "policy", 0x7fffd432da27 "serializer", 0x0}
#7  0x00000000004cb26d in PyEval_EvalFrameEx ()
No symbol table info available.
#8  0x00000000004c22e5 in PyEval_EvalCodeEx ()
No symbol table info available.
#9  0x00000000004ca3d2 in PyEval_EvalFrameEx ()
No symbol table info available.
#10 0x00000000004c22e5 in PyEval_EvalCodeEx ()
No symbol table info available.
#11 0x00000000004c9c59 in PyEval_EvalFrameEx ()
No symbol table info available.
#12 0x00000000004c994f in PyEval_EvalFrameEx ()
---Type <return> to continue, or q <return> to quit---
No symbol table info available.
#13 0x00000000004c22e5 in PyEval_EvalCodeEx ()
No symbol table info available.
#14 0x00000000004c2089 in PyEval_EvalCode ()
No symbol table info available.
#15 0x00000000004f1e6f in ?? ()
No symbol table info available.
#16 0x00000000004ec802 in PyRun_FileExFlags ()
No symbol table info available.
#17 0x00000000004eaf81 in PyRun_SimpleFileExFlags ()
No symbol table info available.
#18 0x000000000049e36a in Py_Main ()
No symbol table info available.
#19 0x00007ffff7811830 in __libc_start_main (main=0x49dcd0 <main>, argc=4, argv=0x7fffffffd9f8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffd9e8)
    at ../csu/libc-start.c:291
        result = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, 3163349192360878488, 4840400, 140737488345584, 0, 0, -3163349851613142632, -3163331321019735656}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x4, 
              0x49dcd0 <main>}, data = {prev = 0x0, cleanup = 0x0, canceltype = 4}}}
        not_first_call = <optimized out>
#20 0x000000000049dbf9 in _start ()

[anuranjit]

@aerospikerobertmarks Any fix for this?

[aerospikerobertmarks]

@anuranjita We believe that we have identified the issue, and will have a new release with a fix later today, or tomorrow.

[aerospikerobertmarks]

@anuranjit Version 2.1.0 of the Python client fixes that issue. If you continue to see it after that release please let us know.

Thank you again for reporting the issue and providing the stack traces, and code examples.