LN_NEWUOA_BOUND: ***buffer overflow detected***

[jschueller]

on fedora 38 the following example fails when compiled with -D_FORTIFY_SOURCE=3 (fedora default flags):

#!/usr/bin/env python

from __future__ import print_function
import nlopt
import numpy as np


def myfunc(x, grad):
    if grad.size > 0:
        grad[0] = 0.0
        grad[1] = 0.5 / np.sqrt(x[1])
    return np.sqrt(x[1])

def myconstraint(x, grad, a, b):
    if grad.size > 0:
        grad[0] = 3 * a * (a*x[0] + b)**2
        grad[1] = -1.0
    return (a*x[0] + b)**3 - x[1]

opt = nlopt.opt(nlopt.LN_NEWUOA_BOUND, 2)
opt.set_lower_bounds([-500, 0])
opt.set_upper_bounds([500] * 2)
opt.set_min_objective(myfunc)
#opt.add_inequality_constraint(lambda x, grad: myconstraint(x,grad, 2, 0), 1e-8)
#opt.add_inequality_constraint(lambda x, grad: myconstraint(x,grad, -1, 1), 1e-8)
opt.set_xtol_rel(1e-4)
x0 = [1.234, 5.678]
x = opt.optimize(x0)
minf = opt.last_optimum_value()
print('optimum at ', x)
print('minimum value = ', minf)
print('result code = ', opt.last_optimize_result())
print('nevals = ', opt.get_numevals())
print('initial step =', opt.get_initial_step(x0))

to reproduce:

cmake -DCMAKE_BUILD_TYPE=None -DCMAKE_CXX_FLAGS="-O2 -g" -DCMAKE_C_FLAGS="-O2 -fexceptions -g -pipe -Wall -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3" -DCMAKE_INSTALL_PREFIX=$PWD/install . 

Program received signal SIGABRT, Aborted.
0x00007ffbdee22b94 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x00007ffbdee22b94 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x00007ffbdedd1aee in raise () from /lib64/libc.so.6
#2  0x00007ffbdedba87f in abort () from /lib64/libc.so.6
#3  0x00007ffbdedbb60f in __libc_message.cold () from /lib64/libc.so.6
#4  0x00007ffbdeeb6b29 in __fortify_fail () from /lib64/libc.so.6
#5  0x00007ffbdeeb5364 in __chk_fail () from /lib64/libc.so.6
#6  0x00007ffbde86b6e0 in memset (__len=16, __ch=0, __dest=0x55dc8d0ed7b8) at /usr/include/bits/string_fortified.h:59
#7  trsapp_ (ub=<optimized out>, lb=<optimized out>, xbase=<optimized out>, crvmin=<synthetic pointer>, hs=0x55dc8d0ed828, hd=0x55dc8d0ed818, g=0x55dc8d0ed808, d__=0x55dc8d0ed7f8, step=<optimized out>, delta=0x7fffcb1c6478, pq=<optimized out>, hq=<optimized out>, gq=<optimized out>, xpt=<optimized out>, xopt=0x55dc8d0ed608, npt=0x7fffcb1c6458, n=0x7fffcb1c645c) at /tmp/nlopt/src/algs/newuoa/newuoa.c:184
#8  newuob_ (w=0x55dc8d0ed7f8, vlag=0x55dc8d0ed7c0, d__=<optimized out>, ndim=0x7fffcb1c6460, zmat=0x55dc8d0ed738, bmat=0x55dc8d0ed6b8, pq=0x55dc8d0ed6c8, hq=0x55dc8d0ed6b0, gq=0x55dc8d0ed6a0, fval=0x55dc8d0ed678, xpt=0x55dc8d0ed600, xnew=0x55dc8d0ed618, xopt=0x55dc8d0ed608, xbase=<optimized out>, calfun_data=0x55dc8d403d40, calfun=0x7ffbde885be0 <f_noderiv>, minf=0x55dc8d43d698, stop=0x7fffcb1c65d0, ub=0x55dc8d1db080, lb=0x55dc8d47ef10, rhobeg=<synthetic pointer>, x=0x55dc8cfaded8, npt=0x7fffcb1c6458, n=0x7fffcb1c645c) at /tmp/nlopt/src/algs/newuoa/newuoa.c:1858
#9  newuoa (n=<optimized out>, n@entry=2, npt=<optimized out>, x=<optimized out>, x@entry=0x55dc8cfadee0, lb=0x55dc8d47ef10, ub=0x55dc8d1db080, rhobeg=<optimized out>, stop=0x7fffcb1c65d0, minf=0x55dc8d43d698, calfun=0x7ffbde885be0 <f_noderiv>, calfun_data=0x55dc8d403d40) at /tmp/nlopt/src/algs/newuoa/newuoa.c:2571
#10 0x00007ffbde887325 in nlopt_optimize_ (minf=0x55dc8d43d698, x=<optimized out>, opt=0x55dc8d403d40) at /tmp/nlopt/src/api/optimize.c:711
#11 nlopt_optimize (opt=<optimized out>, x=<optimized out>, opt_f=0x55dc8d43d698) at /tmp/nlopt/src/api/optimize.c:883
#12 0x00007ffbde8d9bc8 in nlopt::opt::optimize (opt_f=@0x55dc8d43d698: inf, x=..., this=0x55dc8d43d640) at /tmp/nlopt/nlopt.hpp:330
#13 nlopt::opt::optimize (x0=std::vector of length 2, capacity 2 = {...}, this=0x55dc8d43d640) at /tmp/nlopt/nlopt.hpp:341
#14 _wrap_opt_optimize__SWIG_1 (nobjs=2, self=<optimized out>, swig_obj=0x7fffcb1c6720) at /tmp/nlopt/src/swig/CMakeFiles/nlopt_python.dir/nloptPYTHON_wrap.cxx:8865
#15 _wrap_opt_optimize (self=<optimized out>, args=<optimized out>) at /tmp/nlopt/src/swig/CMakeFiles/nlopt_python.dir/nloptPYTHON_wrap.cxx:8926
#16 0x00007ffbdf1402b8 in cfunction_call () from /lib64/libpython3.11.so.1.0
#17 0x00007ffbdf151029 in _PyObject_Call () from /lib64/libpython3.11.so.1.0
#18 0x00007ffbdf13052a in _PyEval_EvalFrameDefault () from /lib64/libpython3.11.so.1.0
#19 0x00007ffbdf1284aa in _PyEval_Vector () from /lib64/libpython3.11.so.1.0
#20 0x00007ffbdf1acf5c in PyEval_EvalCode () from /lib64/libpython3.11.so.1.0
#21 0x00007ffbdf1ca823 in run_eval_code_obj () from /lib64/libpython3.11.so.1.0
#22 0x00007ffbdf1c6e2a in run_mod () from /lib64/libpython3.11.so.1.0
#23 0x00007ffbdf1dcbe2 in pyrun_file () from /lib64/libpython3.11.so.1.0
#24 0x00007ffbdf1dc398 in _PyRun_SimpleFileObject () from /lib64/libpython3.11.so.1.0
#25 0x00007ffbdf1dbff8 in _PyRun_AnyFileObject () from /lib64/libpython3.11.so.1.0
#26 0x00007ffbdf1d5f1c in Py_RunMain () from /lib64/libpython3.11.so.1.0
#27 0x00007ffbdf19cf5b in Py_BytesMain () from /lib64/libpython3.11.so.1.0
#28 0x00007ffbdedbbb4a in __libc_start_call_main () from /lib64/libc.so.6
#29 0x00007ffbdedbbc0b in __libc_start_main_impl () from /lib64/libc.so.6
#30 0x000055dc8b102095 in _start ()

[jschueller]

it turns out the address is valid and it is a compiler false positive

[jschueller]

see #563