R memory safety issues

[david-cortes]

From a similar issue in lightgbm: microsoft/LightGBM#4597

The R interface for XGBoost throws R errors from C++ exception objects' char pointers:

xgboost/R-package/src/xgboost_R.cc

Line 28 in e48e05e

error(e.what()); \

R errors trigger a C long jump, and if done from a catch clause, will leak the C++ exception object under GCC and CLANG.

Additionally, some functions will call R's asChar and retrieve a pointer from it through CHAR:

xgboost/R-package/src/xgboost_R.cc

Line 294 in e48e05e

CHECK_CALL(XGBoosterSetParam(R_ExternalPtrAddr(handle),

I'm not 100% sure about it (asChar is a bit different from other functions), but I think that:

• The result of that asChar needs to be PROTECTed just like other R objects, otherwise it can lead to use-after-free errors if it does any coercion from the original object.
• Calling asChar can also produce a long jump if it ends up allocating a new object as it coerces the input, and if that allocation fails, would lead to leaking C++ objects (e.g.

  xgboost/R-package/src/xgboost_R.cc

  Line 344 in e48e05e

  vec_names.push_back(std::string(CHAR(asChar(VECTOR_ELT(evnames, i)))));

  ).