Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove hard-coded pad token id in distilbert and albert #3965

Merged
merged 1 commit into from
May 12, 2020
Merged

Remove hard-coded pad token id in distilbert and albert #3965

merged 1 commit into from
May 12, 2020

Conversation

monologg
Copy link
Contributor

@monologg monologg commented Apr 25, 2020

As the config adds pad_token_id attribute, padding_idx set the value of config.pad_token_id in BertEmbedding. ( PR #3793 )

But it seems that not only the config of Bert, but also that of DistilBert and Albert has pad_token_id. (Distilbert config, Albert config)

But in Embedding class of Distilbert and Albert, it seems that padding_idx is still hard-coded. So I've fixed those parts.

@codecov-io
Copy link

codecov-io commented Apr 25, 2020

Codecov Report

Merging #3965 into master will decrease coverage by 0.01%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3965      +/-   ##
==========================================
- Coverage   78.45%   78.44%   -0.02%     
==========================================
  Files         111      111              
  Lines       18518    18518              
==========================================
- Hits        14528    14526       -2     
- Misses       3990     3992       +2     
Impacted Files Coverage Δ
src/transformers/modeling_albert.py 75.31% <100.00%> (ø)
src/transformers/modeling_distilbert.py 98.15% <100.00%> (ø)
src/transformers/file_utils.py 68.49% <0.00%> (-0.37%) ⬇️
src/transformers/modeling_utils.py 90.94% <0.00%> (-0.13%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 73d6a2f...9d92a30. Read the comment docs.

@julien-c
Copy link
Member

LGTM

@monologg
Copy link
Contributor Author

monologg commented May 7, 2020

@LysandreJik @VictorSanh

Hi:) Can you please check this PR? This one makes issue on Korean BERT. (which use pad_token_id=1 and unk_token_id=0)

I hope this PR will be applied on the next version of transformers library:)

@VictorSanh
Copy link
Contributor

lgtm!

Copy link
Member

@LysandreJik LysandreJik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@monologg
Copy link
Contributor Author

@julien-c

Can you merge this PR? Thank you so much:)

@julien-c julien-c merged commit 31e67dd into huggingface:master May 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants