Seed value ignored? Reproducibility not given.

[jakob-r]

We observed different behaviors depending on the OS.
Given this code:

seed1 = 2
seed2 = 3
set.seed(1)
library(xgboost)
d = matrix(runif(1000), nrow = 200)
y = 3 + d %*% runif(ncol(d), min = 0.1, max = 5) + rnorm(nrow(d), sd = 10)
d = xgb.DMatrix(data = d, label = y)
set.seed(seed1)
m = xgb.train(data = d, nrounds = 3, objective = "reg:linear", lambda = 0.4, alpha = 0.2, booster = "gblinear")
p1 = predict(m, newdata = d)
set.seed(seed2)
m = xgb.train(data = d, nrounds = 3, objective = "reg:linear", lambda = 0.4, alpha = 0.2, booster = "gblinear")
p2 = predict(m, newdata = d)
all(p1 == p2)

On OSX this will be always true. Even if seed1 and seed2 are different.
On Linux this will be always false. Even if seed1 and seed2 are the same.

For Windows in test I made some time ago it seemed like it has the same behavior as OSX.

[tqchen]

This is not random seed problem. This is because gblinear uses a multi-threading coordinate descent and with each thread eagerly update the parameter without sync with others for efficiency reason. This difference is due to running order of different threads. setting nthread=1 will disable the behavior.

For gbtree, most updates are synced and usually there is no such behavior

[tqchen]

@jakob-r As explained, this was not problem of seed, but due to undermined behavior in multi-threading, I think this issue could be closed

[dataforager]

Don't think multi-threading is the error. Seeds were set below either by passing 'seed' param to xgb.train or by using R's set.seed() function.

Independently verified (by extracting value of .Random.seed) after calling set.seed() that seed was indeed being changed (it was).

nthread was set to 1 for both training/test runs with the same model parameters. Predicted probabilities appear below and are identical for two different seed values.

Predicted probabilities (seed=1):
0.4745588005
0.9879690409
0.5989014506
0.9906733632
0.5989014506
0.9928959012
0.1146880165
0.9928619266
0.9917168021
0.9958292842

Predicted probabilities (seed = 2):
0.4745588005
0.9879690409
0.5989014506
0.9906733632
0.5989014506
0.9928959012
0.1146880165
0.9928619266
0.9917168021
0.9958292842

[dataforager]

Can confirm seeing exact same behavior (with exact same parameter settings), though different predicted probabilities in Python version.

Predicted probabilities (seed = 1):
0.141121
0.98446
0.805141
0.949947
0.805141
0.979856
0.511622
0.990588
0.985136
0.987054

Predicted probabilities (seed = 2):
0.141121
0.98446
0.805141
0.949947
0.805141
0.979856
0.511622
0.990588
0.985136
0.987054

[vaughnkoch]

I'm seeing this behavior as well on macOS, using nthread=1 and different values of seed, including seed=0, which according to the docs should disable a specific seed. The outputs from the engine are unchanged.

Versions:
macOS 10.12.6
Python 3.6.4
xgboost==0.71