-
Notifications
You must be signed in to change notification settings - Fork 27.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added resources on albert model #20697
Closed
JuheonChu
wants to merge
42
commits into
huggingface:main
from
JuheonChu:added-resources-on-ALBERT-model
Closed
Changes from all commits
Commits
Show all changes
42 commits
Select commit
Hold shift + click to select a range
e413124
Changed assert into 7-8 exceptions
210c0af
updated syntax error
e197dcb
updated error
6d862a2
updated file (Co-autho: Batese2001)
8d23a50
Successful test on test_modeling_distilbert.py
JuheonChu 1a6bcb7
Delete test_modeling_distilbert.ipynb
JuheonChu eef5428
Update modeling_distilbert.py
JuheonChu 8216cbc
Successful raising of exceptions with the conditions that are contrar…
e7135da
Successful raising of exceptions with the conditions that are contrar…
8ce829f
committing the reformatted distilbert model
0ad5940
reformatted distilbert model
155b696
reformatted distilbert model
e95bd4b
reformatted distilbert model
e7c695f
reformatted distilbert model with black
05e73a7
Changed comments that explain better about raising exceptions for not…
f0d9366
Changed comments that explain better about raising exceptions for not…
3903799
changed based on the feedback
79dc337
Changed line 833 based on the suggestion made from @younesbelkada
4254060
Changed line 833 based on the suggestion made from @younesbelkada draft2
14b8b03
reformatted file
0b4322f
First Commit
c62f727
Added text-classification and token classification
b5c1d51
Added text-classification and token classifications
df0dfe0
Added resources on question-answering
725ca57
Added resources on question-answering
770a444
Added resources on text-classification
f5026c5
Added resources on text-recognition and research
fbf6d39
reformatted the file
0f7f91c
reformatted the file
82d683f
upgraded tensorflow
b8b1cb6
retemplate albert.mdx
13d7939
retemplate albert.mdx
af4e4c6
Update docs/source/en/model_doc/albert.mdx
JuheonChu c219fb9
revert modeling_distilbert.py
70137c6
Merge branch 'added-resources-on-ALBERT-model' of https://github.com/…
e88b1c6
Accepted changes to the suggestion from @younesbelkada to revert
6a02453
Accepted changes to the suggestion from @younesbelkada to revert
1ef5292
Reformatted
23cf018
Reformatted
00b326f
Reformatted
8684466
Reformatted
cc39d10
Delete modeling_distilbert.py
JuheonChu File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -23,14 +23,14 @@ speed of BERT: | |
|
||
The abstract from the paper is the following: | ||
|
||
*Increasing model size when pretraining natural language representations often results in improved performance on | ||
_Increasing model size when pretraining natural language representations often results in improved performance on | ||
downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations, | ||
longer training times, and unexpected model degradation. To address these problems, we present two parameter-reduction | ||
techniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows | ||
that our proposed methods lead to models that scale much better compared to the original BERT. We also use a | ||
self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks | ||
with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and | ||
SQuAD benchmarks while having fewer parameters compared to BERT-large.* | ||
SQuAD benchmarks while having fewer parameters compared to BERT-large._ | ||
|
||
Tips: | ||
|
||
|
@@ -43,17 +43,60 @@ Tips: | |
This model was contributed by [lysandre](https://huggingface.co/lysandre). This model jax version was contributed by | ||
[kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/ALBERT). | ||
|
||
## Resources | ||
|
||
A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ALBERT. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource. | ||
|
||
<PipelineTag pipeline="introduce" /> | ||
- A blog post on [Getting Started with ALBERT](https://huggingface.co/albert-base-v2). | ||
- A blog post on [Albert-base-v2](https://huggingface.co/albert-base-v2). - A blog | ||
post on [albert-xlarge-v2](https://huggingface.co/albert-xlarge-v2). - A blog post | ||
on [Chinese Albert](https://huggingface.co/uer/albert-base-chinese-cluecorpussmall). | ||
- A blog post on [Albert-tiny](https://huggingface.co/clue/albert_chinese_tiny). | ||
- An article about [ALBERT: A Lite BERT for Self-supervised Learning of Language | ||
Representations](https://arxiv.org/abs/1909.11942) | ||
<PipelineTag pipeline="language-modeling_albert" />- An example code on | ||
[language-modeling](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling) | ||
that supports ALBERT model. | ||
|
||
<PipelineTag pipeline="text-classification" />- A notebook on how to [Fine-tune | ||
ALBERT for sentence-pair | ||
classification](https://github.com/NadirEM/nlp-notebooks/blob/master/Fine_tune_ALBERT_sentence_pair_classification.ipynb). | ||
- A Transformers ALBERT document on | ||
[AlbertForSequenceClassification](https://huggingface.co/docs/transformers/v4.24.0/en/model_doc/albert#transformers.AlbertForSequenceClassification) | ||
- A text-classification example code on | ||
[Pytorch](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification) | ||
that supports ALBERT model. - A text-classification example code on | ||
[Flax](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification) | ||
that supports ALBERT model. - A text-classification example code on | ||
[Tensorflow](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification). | ||
|
||
<PipelineTag pipeline="text-recognition" /> | ||
- A notebook on [Finetune T5 for Named Entity Recognition](https://github.com/ToluClassics/Notebooks/blob/main/T5_Ner_Finetuning.ipynb) | ||
<PipelineTag pipeline="token-classification" />- A course material on | ||
[EncoderModel](https://huggingface.co/course/chapter1/5). - A course material | ||
on[Creating your own dataset](https://huggingface.co/course/chapter5/5). - A | ||
Transformers ALBERT document on | ||
[AlbertForTokenClassification](https://huggingface.co/docs/transformers/v4.24.0/en/model_doc/albert#transformers.AlbertForTokenClassification). | ||
|
||
<PipelineTag pipeline="QuestionAnswering" />- A Tensorflow example code on | ||
[Question-Answering](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering) | ||
that supports ALBERT model. - A Flax example code on | ||
[Question-Answering](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering) | ||
- A PyTorch example code on | ||
[Question-Answering](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering) | ||
|
||
<PipelineTag pipeline="Research" />- A research project | ||
[bert-loses-patience](https://github.com/huggingface/transformers/tree/main/examples/research_projects/bert-loses-patience) | ||
that uses the ALBERT pretrained model. | ||
|
||
## AlbertConfig | ||
|
||
[[autodoc]] AlbertConfig | ||
|
||
## AlbertTokenizer | ||
|
||
[[autodoc]] AlbertTokenizer | ||
- build_inputs_with_special_tokens | ||
- get_special_tokens_mask | ||
- create_token_type_ids_from_sequences | ||
- save_vocabulary | ||
[[autodoc]] AlbertTokenizer - build_inputs_with_special_tokens - get_special_tokens_mask - create_token_type_ids_from_sequences - save_vocabulary | ||
|
||
## AlbertTokenizerFast | ||
|
||
|
@@ -67,104 +110,84 @@ This model was contributed by [lysandre](https://huggingface.co/lysandre). This | |
|
||
## AlbertModel | ||
|
||
[[autodoc]] AlbertModel | ||
- forward | ||
[[autodoc]] AlbertModel - forward | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You can leave this alone as well and allow the |
||
|
||
## AlbertForPreTraining | ||
|
||
[[autodoc]] AlbertForPreTraining | ||
- forward | ||
[[autodoc]] AlbertForPreTraining - forward | ||
|
||
## AlbertForMaskedLM | ||
|
||
[[autodoc]] AlbertForMaskedLM | ||
- forward | ||
[[autodoc]] AlbertForMaskedLM - forward | ||
|
||
## AlbertForSequenceClassification | ||
|
||
[[autodoc]] AlbertForSequenceClassification | ||
- forward | ||
[[autodoc]] AlbertForSequenceClassification - forward | ||
|
||
## AlbertForMultipleChoice | ||
|
||
[[autodoc]] AlbertForMultipleChoice | ||
|
||
## AlbertForTokenClassification | ||
|
||
[[autodoc]] AlbertForTokenClassification | ||
- forward | ||
[[autodoc]] AlbertForTokenClassification - forward | ||
|
||
## AlbertForQuestionAnswering | ||
|
||
[[autodoc]] AlbertForQuestionAnswering | ||
- forward | ||
[[autodoc]] AlbertForQuestionAnswering - forward | ||
|
||
## TFAlbertModel | ||
|
||
[[autodoc]] TFAlbertModel | ||
- call | ||
[[autodoc]] TFAlbertModel - call | ||
|
||
## TFAlbertForPreTraining | ||
|
||
[[autodoc]] TFAlbertForPreTraining | ||
- call | ||
[[autodoc]] TFAlbertForPreTraining - call | ||
|
||
## TFAlbertForMaskedLM | ||
|
||
[[autodoc]] TFAlbertForMaskedLM | ||
- call | ||
[[autodoc]] TFAlbertForMaskedLM - call | ||
|
||
## TFAlbertForSequenceClassification | ||
|
||
[[autodoc]] TFAlbertForSequenceClassification | ||
- call | ||
[[autodoc]] TFAlbertForSequenceClassification - call | ||
|
||
## TFAlbertForMultipleChoice | ||
|
||
[[autodoc]] TFAlbertForMultipleChoice | ||
- call | ||
[[autodoc]] TFAlbertForMultipleChoice - call | ||
|
||
## TFAlbertForTokenClassification | ||
|
||
[[autodoc]] TFAlbertForTokenClassification | ||
- call | ||
[[autodoc]] TFAlbertForTokenClassification - call | ||
|
||
## TFAlbertForQuestionAnswering | ||
|
||
[[autodoc]] TFAlbertForQuestionAnswering | ||
- call | ||
[[autodoc]] TFAlbertForQuestionAnswering - call | ||
|
||
## FlaxAlbertModel | ||
|
||
[[autodoc]] FlaxAlbertModel | ||
- __call__ | ||
[[autodoc]] FlaxAlbertModel - **call** | ||
|
||
## FlaxAlbertForPreTraining | ||
|
||
[[autodoc]] FlaxAlbertForPreTraining | ||
- __call__ | ||
[[autodoc]] FlaxAlbertForPreTraining - **call** | ||
|
||
## FlaxAlbertForMaskedLM | ||
|
||
[[autodoc]] FlaxAlbertForMaskedLM | ||
- __call__ | ||
[[autodoc]] FlaxAlbertForMaskedLM - **call** | ||
|
||
## FlaxAlbertForSequenceClassification | ||
|
||
[[autodoc]] FlaxAlbertForSequenceClassification | ||
- __call__ | ||
[[autodoc]] FlaxAlbertForSequenceClassification - **call** | ||
|
||
## FlaxAlbertForMultipleChoice | ||
|
||
[[autodoc]] FlaxAlbertForMultipleChoice | ||
- __call__ | ||
[[autodoc]] FlaxAlbertForMultipleChoice - **call** | ||
|
||
## FlaxAlbertForTokenClassification | ||
|
||
[[autodoc]] FlaxAlbertForTokenClassification | ||
- __call__ | ||
[[autodoc]] FlaxAlbertForTokenClassification - **call** | ||
|
||
## FlaxAlbertForQuestionAnswering | ||
|
||
[[autodoc]] FlaxAlbertForQuestionAnswering | ||
- __call__ | ||
[[autodoc]] FlaxAlbertForQuestionAnswering - **call** |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you revert these changes? 🙏
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does that mean deleting "_"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It means you can leave the asterisks
*
instead of using an underscore_