[Feat] Save and load cuckoo hashtable from hdfs #257
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rhdong
requested review from
bashimao and
Lifann
and removed request for
bashimao
luliyucoordinate
force-pushed
the
export_to_hdfs
branch
from
8e262b0 to
421fbec
Compare
acmore
reviewed
Jul 4, 2022
tensorflow_recommenders_addons/dynamic_embedding/core/kernels/lookup_impl/lookup_table_op_cpu.h
Outdated
Show resolved
Hide resolved
luliyucoordinate
force-pushed
the
export_to_hdfs
branch
2 times, most recently
from
4c9a513 to
cdd9ddf
Compare
rhdong
reviewed
Jul 9, 2022
thorneliu
reviewed
Jul 13, 2022
tensorflow_recommenders_addons/dynamic_embedding/core/kernels/lookup_impl/lookup_table_op_cpu.h
Outdated
Show resolved
Hide resolved
tensorflow_recommenders_addons/dynamic_embedding/core/kernels/lookup_impl/lookup_table_op_cpu.h
Show resolved
Hide resolved
tensorflow_recommenders_addons/dynamic_embedding/core/kernels/lookup_impl/lookup_table_op_cpu.h
Outdated
Show resolved
Hide resolved
thorneliu
approved these changes
Jul 14, 2022
thorneliu
previously approved these changes
Jul 15, 2022
|
Hi @luliyucoordinate, please squash the commits into one, I believe all of them belong to one feature. After that I will merge it, Thank you! |
code format and fix bracket issue [fix] Fixed bug which cause by hiredis updating failed to compile. add hadoop fix undefined symbol and skip test hdfs fix test error fix variable-sized object may not be initialized fix build for tf version >= 2.7.0 save to tmp file convert values to const load_from_hdfs adopts zero-copy method
luliyucoordinate
force-pushed
the
export_to_hdfs
branch
from
4579ee6 to
3e9103c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Because of the storage mechanism of tensorflow, when the embedding table is large, the worker will consume a lot of memory, and eventually OOM.
This PR provides some methods to save or load files from hdfs for dynamic embedding tables.
Type of change
Checklist:
How Has This Been Tested?
In the test file, you need to replace the
filepathwith the real hdfs path. In addition, you need to add the header filethird_party/hadoop/hdfs.hof hadoop in tensorflow.