Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added resources on albert model #20697

Closed
wants to merge 42 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
e413124
Changed assert into 7-8 exceptions
Nov 18, 2022
210c0af
updated syntax error
Nov 22, 2022
e197dcb
updated error
Nov 22, 2022
6d862a2
updated file (Co-autho: Batese2001)
Nov 22, 2022
8d23a50
Successful test on test_modeling_distilbert.py
JuheonChu Nov 22, 2022
1a6bcb7
Delete test_modeling_distilbert.ipynb
JuheonChu Nov 22, 2022
eef5428
Update modeling_distilbert.py
JuheonChu Nov 22, 2022
8216cbc
Successful raising of exceptions with the conditions that are contrar…
Nov 22, 2022
e7135da
Successful raising of exceptions with the conditions that are contrar…
Nov 22, 2022
8ce829f
committing the reformatted distilbert model
Nov 24, 2022
0ad5940
reformatted distilbert model
Nov 24, 2022
155b696
reformatted distilbert model
Nov 24, 2022
e95bd4b
reformatted distilbert model
Nov 24, 2022
e7c695f
reformatted distilbert model with black
Nov 24, 2022
05e73a7
Changed comments that explain better about raising exceptions for not…
Nov 25, 2022
f0d9366
Changed comments that explain better about raising exceptions for not…
Nov 25, 2022
3903799
changed based on the feedback
Nov 26, 2022
79dc337
Changed line 833 based on the suggestion made from @younesbelkada
Nov 26, 2022
4254060
Changed line 833 based on the suggestion made from @younesbelkada draft2
Nov 26, 2022
14b8b03
reformatted file
Nov 26, 2022
0b4322f
First Commit
Dec 2, 2022
c62f727
Added text-classification and token classification
Dec 2, 2022
b5c1d51
Added text-classification and token classifications
Dec 2, 2022
df0dfe0
Added resources on question-answering
Dec 8, 2022
725ca57
Added resources on question-answering
Dec 8, 2022
770a444
Added resources on text-classification
Dec 8, 2022
f5026c5
Added resources on text-recognition and research
Dec 8, 2022
fbf6d39
reformatted the file
Dec 9, 2022
0f7f91c
reformatted the file
Dec 9, 2022
82d683f
upgraded tensorflow
Dec 9, 2022
b8b1cb6
retemplate albert.mdx
Dec 9, 2022
13d7939
retemplate albert.mdx
Dec 9, 2022
af4e4c6
Update docs/source/en/model_doc/albert.mdx
JuheonChu Dec 9, 2022
c219fb9
revert modeling_distilbert.py
Dec 9, 2022
70137c6
Merge branch 'added-resources-on-ALBERT-model' of https://github.com/…
Dec 9, 2022
e88b1c6
Accepted changes to the suggestion from @younesbelkada to revert
Dec 9, 2022
6a02453
Accepted changes to the suggestion from @younesbelkada to revert
Dec 9, 2022
1ef5292
Reformatted
Dec 9, 2022
23cf018
Reformatted
Dec 9, 2022
00b326f
Reformatted
Dec 9, 2022
8684466
Reformatted
Dec 9, 2022
cc39d10
Delete modeling_distilbert.py
JuheonChu Dec 9, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
117 changes: 70 additions & 47 deletions docs/source/en/model_doc/albert.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,14 @@ speed of BERT:

The abstract from the paper is the following:

*Increasing model size when pretraining natural language representations often results in improved performance on
_Increasing model size when pretraining natural language representations often results in improved performance on
downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations,
longer training times, and unexpected model degradation. To address these problems, we present two parameter-reduction
techniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows
that our proposed methods lead to models that scale much better compared to the original BERT. We also use a
self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks
with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and
SQuAD benchmarks while having fewer parameters compared to BERT-large.*
SQuAD benchmarks while having fewer parameters compared to BERT-large._
Comment on lines +26 to +33
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you revert these changes? 🙏

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does that mean deleting "_"?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It means you can leave the asterisks * instead of using an underscore _


Tips:

Expand All @@ -43,17 +43,60 @@ Tips:
This model was contributed by [lysandre](https://huggingface.co/lysandre). This model jax version was contributed by
[kamalkraj](https://huggingface.co/kamalkraj). The original code can be found [here](https://github.com/google-research/ALBERT).

## Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with ALBERT. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

<PipelineTag pipeline="introduce" />
- A blog post on [Getting Started with ALBERT](https://huggingface.co/albert-base-v2).
- A blog post on [Albert-base-v2](https://huggingface.co/albert-base-v2). - A blog
post on [albert-xlarge-v2](https://huggingface.co/albert-xlarge-v2). - A blog post
on [Chinese Albert](https://huggingface.co/uer/albert-base-chinese-cluecorpussmall).
- A blog post on [Albert-tiny](https://huggingface.co/clue/albert_chinese_tiny).
- An article about [ALBERT: A Lite BERT for Self-supervised Learning of Language
Representations](https://arxiv.org/abs/1909.11942)
<PipelineTag pipeline="language-modeling_albert" />- An example code on
[language-modeling](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling)
that supports ALBERT model.

<PipelineTag pipeline="text-classification" />- A notebook on how to [Fine-tune
ALBERT for sentence-pair
classification](https://github.com/NadirEM/nlp-notebooks/blob/master/Fine_tune_ALBERT_sentence_pair_classification.ipynb).
- A Transformers ALBERT document on
[AlbertForSequenceClassification](https://huggingface.co/docs/transformers/v4.24.0/en/model_doc/albert#transformers.AlbertForSequenceClassification)
- A text-classification example code on
[Pytorch](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification)
that supports ALBERT model. - A text-classification example code on
[Flax](https://github.com/huggingface/transformers/tree/main/examples/flax/text-classification)
that supports ALBERT model. - A text-classification example code on
[Tensorflow](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/text-classification).

<PipelineTag pipeline="text-recognition" />
- A notebook on [Finetune T5 for Named Entity Recognition](https://github.com/ToluClassics/Notebooks/blob/main/T5_Ner_Finetuning.ipynb)
<PipelineTag pipeline="token-classification" />- A course material on
[EncoderModel](https://huggingface.co/course/chapter1/5). - A course material
on[Creating your own dataset](https://huggingface.co/course/chapter5/5). - A
Transformers ALBERT document on
[AlbertForTokenClassification](https://huggingface.co/docs/transformers/v4.24.0/en/model_doc/albert#transformers.AlbertForTokenClassification).

<PipelineTag pipeline="QuestionAnswering" />- A Tensorflow example code on
[Question-Answering](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering)
that supports ALBERT model. - A Flax example code on
[Question-Answering](https://github.com/huggingface/transformers/tree/main/examples/flax/question-answering)
- A PyTorch example code on
[Question-Answering](https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering)

<PipelineTag pipeline="Research" />- A research project
[bert-loses-patience](https://github.com/huggingface/transformers/tree/main/examples/research_projects/bert-loses-patience)
that uses the ALBERT pretrained model.

## AlbertConfig

[[autodoc]] AlbertConfig

## AlbertTokenizer

[[autodoc]] AlbertTokenizer
- build_inputs_with_special_tokens
- get_special_tokens_mask
- create_token_type_ids_from_sequences
- save_vocabulary
[[autodoc]] AlbertTokenizer - build_inputs_with_special_tokens - get_special_tokens_mask - create_token_type_ids_from_sequences - save_vocabulary

## AlbertTokenizerFast

Expand All @@ -67,104 +110,84 @@ This model was contributed by [lysandre](https://huggingface.co/lysandre). This

## AlbertModel

[[autodoc]] AlbertModel
- forward
[[autodoc]] AlbertModel - forward
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can leave this alone as well and allow the forward method to be listed under the AlbertModel object. Same comment applies to all the other objects changed below :)


## AlbertForPreTraining

[[autodoc]] AlbertForPreTraining
- forward
[[autodoc]] AlbertForPreTraining - forward

## AlbertForMaskedLM

[[autodoc]] AlbertForMaskedLM
- forward
[[autodoc]] AlbertForMaskedLM - forward

## AlbertForSequenceClassification

[[autodoc]] AlbertForSequenceClassification
- forward
[[autodoc]] AlbertForSequenceClassification - forward

## AlbertForMultipleChoice

[[autodoc]] AlbertForMultipleChoice

## AlbertForTokenClassification

[[autodoc]] AlbertForTokenClassification
- forward
[[autodoc]] AlbertForTokenClassification - forward

## AlbertForQuestionAnswering

[[autodoc]] AlbertForQuestionAnswering
- forward
[[autodoc]] AlbertForQuestionAnswering - forward

## TFAlbertModel

[[autodoc]] TFAlbertModel
- call
[[autodoc]] TFAlbertModel - call

## TFAlbertForPreTraining

[[autodoc]] TFAlbertForPreTraining
- call
[[autodoc]] TFAlbertForPreTraining - call

## TFAlbertForMaskedLM

[[autodoc]] TFAlbertForMaskedLM
- call
[[autodoc]] TFAlbertForMaskedLM - call

## TFAlbertForSequenceClassification

[[autodoc]] TFAlbertForSequenceClassification
- call
[[autodoc]] TFAlbertForSequenceClassification - call

## TFAlbertForMultipleChoice

[[autodoc]] TFAlbertForMultipleChoice
- call
[[autodoc]] TFAlbertForMultipleChoice - call

## TFAlbertForTokenClassification

[[autodoc]] TFAlbertForTokenClassification
- call
[[autodoc]] TFAlbertForTokenClassification - call

## TFAlbertForQuestionAnswering

[[autodoc]] TFAlbertForQuestionAnswering
- call
[[autodoc]] TFAlbertForQuestionAnswering - call

## FlaxAlbertModel

[[autodoc]] FlaxAlbertModel
- __call__
[[autodoc]] FlaxAlbertModel - **call**

## FlaxAlbertForPreTraining

[[autodoc]] FlaxAlbertForPreTraining
- __call__
[[autodoc]] FlaxAlbertForPreTraining - **call**

## FlaxAlbertForMaskedLM

[[autodoc]] FlaxAlbertForMaskedLM
- __call__
[[autodoc]] FlaxAlbertForMaskedLM - **call**

## FlaxAlbertForSequenceClassification

[[autodoc]] FlaxAlbertForSequenceClassification
- __call__
[[autodoc]] FlaxAlbertForSequenceClassification - **call**

## FlaxAlbertForMultipleChoice

[[autodoc]] FlaxAlbertForMultipleChoice
- __call__
[[autodoc]] FlaxAlbertForMultipleChoice - **call**

## FlaxAlbertForTokenClassification

[[autodoc]] FlaxAlbertForTokenClassification
- __call__
[[autodoc]] FlaxAlbertForTokenClassification - **call**

## FlaxAlbertForQuestionAnswering

[[autodoc]] FlaxAlbertForQuestionAnswering
- __call__
[[autodoc]] FlaxAlbertForQuestionAnswering - **call**
Loading