How does the fusion ensemble differentiate the members of the ensemble?

[jmusiel]

The base estimators are all trained on the same data, is that correct?

However mini batch gradient descent is used separately on each of them from what I can tell, so my understanding is that this should cause some stochastic differences between the estimators. Is this sufficient?

Are the estimators are initialized the same way? Or do they also start with separate random weights?

[xuyxu]

Hi @jmusiel, thanks for your good question.

Are the estimators are initialized the same way? Or do they also start with separate random weights?

I have checked estimators in fusion after initialization, the truth is that each estimator has a separate random weights even after setting the random seed using torch.manual_seed.

However mini batch gradient descent is used separately on each of them from what I can tell, so my understanding is that this should cause some stochastic differences between the estimators. Is this sufficient?

Stochastic differences during the optimization in combination with separate random weights are sufficient for training a ensemble with high diversity (Another example is voting). However, in my practice, fusion is typically not the best choice, since it suffers from the severe over-fitting problem.

[jmusiel]

Ah, thank you for your quick response, this is very helpful!

However, in my practice, fusion is typically not the best choice, since it suffers from the severe over-fitting problem.

Can you elaborate a bit about what you mean? Do the individual members of the ensemble each over-fit or is it that the ensemble as a whole tends to over-fit?

For achieving a relatively high-diversity ensemble in practice, what would be a better choice to alleviate the over-fitting issue? Does one of the other simpler ensembles (voting/bagging) do a better job? Or is it better to use one of the more sophisticated ensembles?

[xuyxu]

Can you elaborate a bit about what you mean? Do the individual members of the ensemble each over-fit or is it that the ensemble as a whole tends to over-fit?

The entire ensemble system tends to over-fit. You can check our experiment ResNet@CIFAR-100. A single ResNet-18 is already a model with very high model capacity. If we use fusion to combine several ResNet-18, the over-fitting problem becomes more severe. For example, with more estimators added, the testing acc of fusion becomes lower (blue line in the last figure).

What would be a better choice to alleviate the over-fitting issue

It depends on the performance of your base estimator and the hardness of your problem:

• Weak base estimator + Hard Problem: Gradient Boosting
• Strong base estimator + Easy Problem: Voting or Bagging

If further taking training budgets into the consideration, Snapshot Ensemble and Fast Geometric Ensembling could be better. However, their effectiveness is not as good as voting.