cudf.core.subword_tokenizer.SubwordTokenizer.__call__ Tensorflow isssues

[RaeWallace10]

For 'return_tensors' I tried multiple tensoflow versions with 'tf' and it give the same error but it works just fine with pytorch 'pt'

from cudf.core.subword_tokenizer import SubwordTokenizer
cudf_tokenizer  = SubwordTokenizer('voc_hash.txt',
                                   do_lower_case=True)

Test_tokenizer_output = cudf_tokenizer(x_test1,
                                  max_length=seq_len,
                                  max_num_rows=len(x_test1),
                                  padding='max_length',
                                  return_tensors='tf',
                                  truncation=True)
print(Test_tokenizer_output)

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
/tmp/ipykernel_9305/4211951720.py in <module>
      4                                   padding='max_length',
      5                                   return_tensors='tf',
----> 6                                   truncation=True)
      7 print(Test_tokenizer_output)

~/miniconda3/envs/rtd/lib/python3.7/site-packages/cudf/core/subword_tokenizer.py in __call__(self, text, max_length, max_num_rows, add_special_tokens, padding, truncation, stride, return_tensors, return_token_type_ids)
    233         tokenizer_output = {
    234             k: _cast_to_appropriate_type(v, return_tensors)
--> 235             for k, v in tokenizer_output.items()
    236         }
    237 

~/miniconda3/envs/rtd/lib/python3.7/site-packages/cudf/core/subword_tokenizer.py in <dictcomp>(.0)
    233         tokenizer_output = {
    234             k: _cast_to_appropriate_type(v, return_tensors)
--> 235             for k, v in tokenizer_output.items()
    236         }
    237 

~/miniconda3/envs/rtd/lib/python3.7/site-packages/cudf/core/subword_tokenizer.py in _cast_to_appropriate_type(ar, cast_type)
     22 
     23     elif cast_type == "tf":
---> 24         from tf.experimental.dlpack import from_dlpack
     25 
     26     return from_dlpack(ar.astype("int32").toDlpack())

ModuleNotFoundError: No module named 'tf'

[beckernick]

Do you have tensorflow installed in your environment? What happens if you run from tf.experimental.dlpack import from_dlpack in your Python session?

[RaeWallace10]

from tf.experimental.dlpack import from_dlpack
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
/tmp/ipykernel_3648/1205705631.py in <module>
----> 1 from tf.experimental.dlpack import from_dlpack

ModuleNotFoundError: No module named 'tf'

My current TF version

print(tf.__version__)
2.5.0

[beckernick]

Are you able to directly access tf.experimental.dlpack.from_dlpack using the full module namespace? Perhaps Tensorflow changed their module structure, as the interface is still available in 2.5+ https://www.tensorflow.org/versions/r2.5/api_docs/python/tf/experimental/dlpack/from_dlpack

[beckernick]

@VibhuJawa , does the tf symbol exist automatically or does it need to be specified in our (or user) code?

[VibhuJawa]

This seems like a bug on our end here. We should have imported it from tensorflow rather than tf. Let me check and push a fix right away.

cudf/python/cudf/cudf/core/subword_tokenizer.py

Lines 23 to 24 in 2718443

	elif cast_type == "tf":
	from tf.experimental.dlpack import from_dlpack

@RaeWallace10 , For now it might be worthwhile to change to tensorflow tensor downstream.

import tensorflow as tf
from tf.experimental.dlpack import from_dlpack



test_tokenizer_output = cudf_tokenizer(x_test1,
                                  max_length=seq_len,
                                  max_num_rows=len(x_test1),
                                  padding='max_length',
                                  return_tensors='cp',
                                  truncation=True)
                               
                                                                    
test_tokenizer_output['input_ids'] =  from_dlpack( test_tokenizer_output['input_ids'].astype("int32").toDlpack())
test_tokenizer_output['attention_mask']  = from_dlpack ( test_tokenizer_output['attention_mask'].astype("int32").toDlpack())
test_tokenizer_output['metadata']  = from_dlpack( test_tokenizer_output['metadata'] .astype("int32").toDlpack())

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[github-actions]

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.

[davidwendt]

Looks like this has been fixed in #9968 here:

cudf/python/cudf/cudf/core/subword_tokenizer.py

Lines 23 to 24 in 7f2a16f

	elif cast_type == "tf":
	from tensorflow.experimental.dlpack import from_dlpack

Can we close this?