[BUG] Exporting embedding tables to parquet does not match the TF variable weights

[gabrielspmoreira]

Bug description

When we export the embedding table of a categorical feature to a dataframe (embedding_table_df()), dataset (embedding_table_dataset()) or parquet file (export_embedding_table()) the values in the exported dataframe/parquet do not match the values of the original feature embedding weighs.

Steps/Code to reproduce bug

1. Disable skipping the test test_embedding_features_exporting_and_loading_pretrained_initializer() in tests\tf\features\test_embeddings.py
2. That test will fail because EmbeddingFeatures.embedding_table_dataset() uses under the hood cudf.from_dlpack(), which is not working as expected we reported in this issue #10754 of cudf repo.

Expected behavior

The values of the exported embedding table should match the embedding table weights.

[gabrielspmoreira]

This issue is blocked by issue #10754 of cudf repo.

[viswa-nvidia]

@gabrielspmoreira , what is the impact of this bug - Which example is this affecting ? @EvenOldridge for viz.

[EvenOldridge]

@gabrielspmoreira we may be assuming cudf behaves in a specific way. I believe that order of the list isn't guaranteed unless you specifically order it. There may be a way to force that.

[gabrielspmoreira]

Sure @EvenOldridge. The cudf team mentioned in my issue that cudf.from_dlpack() expects tensor to be in column-major (Fortran order). The cudf team placed a PR based on that to raise an exception if tensor is not in column-major.
From I what investigated, TF doesn't support encoding tensors in Fortran order (like numpy and cupy do).
So I fixed the issue on our side in this PR by converting from TF to cuPy using DLPack and then creating the cudf dataframe from cuPy.