How can we create the embeddings?

[rasbt]

Hey there, thanks for sharing the paper and the code. In fact, it's really awesome that you made the paper fully-reproducible!

1. One the other hand, it's a lot of code :). I was wondering if you have some minimal examples on how to create the PLEs and periodic activation functions so that I can add them as data transforms to my existing neural network training scripts. I am currently working on an applied project and am interested in giving your method a try. (This would be on existing networks though, so I would only need the embedding code.)

2. I was also wondering if you tried the numeric embeddings on the GBDTs, just out of curiosity.

[Yura52]

Hey @rasbt !

1. As of now, we do not have minimal examples. Ideally, the corresponding module should become a part of the rtdl library, but, unfortunately, I cannot say if/when it will happen. However, I can point to the key lines related to PLEs:

• first, you need to use the "bins" option. Use the "tree option" to use the target-aware variation of PLE. As for the hyperparameters, I recommend tuning them. If it is not possible, then you can take some inspiration from the configurations tuned for the datasets used in the paper (you can find them in the exp/ directory).
• this block is about pre-computing bin edges, bin values etc. (the "print" statement is a bit outdated, we should fix this)
• this block calculates PLEs during the training

2. We have not tried this. However, as far as I know, the modern GBDT libraries use histogram-based training by default. In terms of the paper, it literally means splitting the feature value ranges using (by default) quantiles (as in Equation 6) and encoding them using the "binary" strategy (see Section 5.3).

[Yura52]

Hi @rasbt ! Although quite some time has passed, just in case, sharing that I created an easy-to-use package with usage examples showing how to apply the embeddings (periodic, piecewise-linear, etc.) to a tabular neural network:

• The package
• The announcement with more details

P.S. More projects on tabular deep learning are available here.