TransAct configuration problem (training issues of AUC=0.5)

[zhujiem]

Hi, Thanks for the open source code of TransAct!

Recently, I have integreted TransAct to our FuxiCTR library. But I found some confusing results when applying grid search on the hyper-parameters. That is, for some configurations, the model outputs AUC=0.5. It is especially common when setting k_cols > 1 and concat_max_pooling=True. Do you have this problem in your experiments?

For example:
When applying grid search on the following hyperparameters, we get six settings.

    transformer_layers: [1, 2, 3]
    k_cols: [1, 2]
    concat_max_pooling: True

We obtain the following results, where only the first setting (transformer_layers=1, k_cols=1, and concat_max_pooling=True) produce reasonable result.

[test] gAUC: 0.694292 - AUC: 0.695478 -   logloss: 0.425579
[test] gAUC: 0.516050 - AUC: 0.515230 -   logloss: 2.712645
[test] gAUC: 0.500612 - AUC: 0.500910 -   logloss: 2.712645
[test] gAUC: 0.500153 - AUC: 0.500072 -   logloss: 2.712645
[test] gAUC: 0.500000 - AUC: 0.500000 -   logloss: 2.712645
[test] gAUC: 0.499978 - AUC: 0.499977 -   logloss: 2.712645

When I set concat_max_pooling=False, and keep the other hyperparameters the same:

    transformer_layers: [1, 2, 3]
    k_cols: [1, 2]
    concat_max_pooling: False

The results become normal in all cases.

[test] gAUC: 0.693382 - AUC: 0.693303 - logloss: 0.431847
[test] gAUC: 0.691872 - AUC: 0.695574 - logloss: 0.424100
[test] gAUC: 0.690954 - AUC: 0.686386 - logloss: 0.429123
[test] gAUC: 0.688613 - AUC: 0.686696 - logloss: 0.435287
[test] gAUC: 0.685512 - AUC: 0.687692 - logloss: 0.436102
[test] gAUC: 0.685075 - AUC: 0.688416 - logloss: 0.423795