-
Notifications
You must be signed in to change notification settings - Fork 197
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Negated antonym #255
Open
Mayukhga83
wants to merge
8
commits into
GEM-benchmark:main
Choose a base branch
from
Mayukhga83:negated_antonym
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Negated antonym #255
Changes from 6 commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
d2e2dea
Added NegatedAntonym Transformation
Mayukhga83 19765fe
Added negated_antonym_perturbation transformation
Mayukhga83 a6de4b3
Fixed requirements.txt
Mayukhga83 55bb3bf
Resolved maxoutput
Mayukhga83 e1e907b
Added Robustness and Keyword
Mayukhga83 a9afa1f
fixed test
Mayukhga83 f63287d
Keyword Corrected
Mayukhga83 05d311c
error with period and commas corrected, test new example added
Mayukhga83 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -2,6 +2,7 @@ checklist==0.0.11 | |
spacy==3.0.0 | ||
numpy | ||
|
||
|
||
# for back_translation | ||
torch | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
# Negated Antonym Perturbation | ||
This perturbation rephrases the adjectives and adverbs to their negated antonym | ||
|
||
Authors: Mayukh Das (Technical University of Braunschweig / [email protected]) | ||
|
||
## What type of a transformation is this? | ||
This transformation detects all type of adjectives and adverb and converts them to their negated antonym. | ||
Therfore the transformation retains the semantics of the original text as positives gets converted to negated negatives and negatives gets converted to negated positives. | ||
Example: I think you are prepared for the test --> I think you are not unprepared for the test | ||
## What tasks does it intend to benefit? | ||
This can act as a valid perturbation that retains the semantics. | ||
This will benefit for robustness of text classification like sentiment analysis, etc. | ||
|
||
|
||
## What are the limitations of this transformation? | ||
It is limited to only adjectives and adverbs (comparative and superlative) | ||
|
||
## Robustness Evaluation | ||
|
||
original accuracy: 96.0 | ||
dataset_name: 'imdb' | ||
model_name: 'aychang/roberta-base-imdb' | ||
no_of_examples: 250 | ||
accuracy after perturbation: 91.0 | ||
|
||
The accuracy drops by 5% when tested on imdb dataset for roberta | ||
|
||
## KeyWords | ||
tokenizer-required | ||
highly-meaning-preserving |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
from .transformation import * |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,68 @@ | ||
{ | ||
"type": "negated_antonym_transformation", | ||
"test_cases": [ | ||
{ | ||
"class": "NegatedAntonym", | ||
"inputs": { | ||
"sentence": "I think you are successful." | ||
}, | ||
"outputs": [{ | ||
"sentence": "I think you are not unsuccessful ." | ||
}] | ||
}, | ||
{ | ||
"class": "NegatedAntonym", | ||
"inputs": { | ||
"sentence": "I think you are prepared for the test." | ||
}, | ||
"outputs": [{ | ||
"sentence": "I think you are not unprepared for the test ." | ||
}] | ||
}, | ||
{ | ||
"class": "NegatedAntonym", | ||
"inputs": { | ||
"sentence": "He ran quickly." | ||
}, | ||
"outputs": [{ | ||
"sentence": "He ran not slowly ." | ||
}] | ||
}, | ||
{ | ||
"class": "NegatedAntonym", | ||
"inputs": { | ||
"sentence": "He plays the flute beautifully." | ||
}, | ||
"outputs": [{ | ||
"sentence": "He plays the flute not unattractively ." | ||
}] | ||
}, | ||
{ | ||
"class": "NegatedAntonym", | ||
"inputs": { | ||
"sentence": "I think you are joyful." | ||
}, | ||
"outputs": [{ | ||
"sentence": "I think you are not sorrowful ." | ||
}] | ||
}, | ||
{ | ||
"class": "NegatedAntonym", | ||
"inputs": { | ||
"sentence": "I think you are smarter than me." | ||
}, | ||
"outputs": [{ | ||
"sentence": "I think you are not stupid than me ." | ||
}] | ||
}, | ||
{ | ||
"class": "NegatedAntonym", | ||
"inputs": { | ||
"sentence": "The movie was bad." | ||
}, | ||
"outputs": [{ | ||
"sentence": "The movie was not good ." | ||
}] | ||
} | ||
] | ||
} |
40 changes: 40 additions & 0 deletions
40
transformations/negated_antonym_perturbation/transformation.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,40 @@ | ||
import nltk | ||
from nltk.corpus import wordnet | ||
import nltk.tokenize as nt | ||
|
||
from interfaces.SentenceOperation import SentenceOperation | ||
from tasks.TaskTypes import TaskType | ||
|
||
''' Class that converts adjectives and adverbs to its negated antonym''' | ||
|
||
|
||
class NegatedAntonym(SentenceOperation): | ||
tasks = [TaskType.TEXT_CLASSIFICATION, TaskType.TEXT_TO_TEXT_GENERATION] | ||
Mayukhga83 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
languages = ["en"] | ||
|
||
Mayukhga83 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
def __init__(self, seed=0): | ||
super().__init__(seed) | ||
|
||
def generate(self, sentence): | ||
return [self.Neg_Antonym(sentence)] | ||
|
||
def Neg_Antonym(self, sentence): | ||
|
||
tokenized_sent = nt.word_tokenize(sentence) | ||
pos_sentences = nltk.pos_tag(tokenized_sent) | ||
|
||
for i in range(len(pos_sentences)): | ||
antonyms = [] | ||
if pos_sentences[i][1] == 'JJ' or pos_sentences[i][1] == 'JJR' or pos_sentences[i][1] == 'JJS' or \ | ||
pos_sentences[i][1] == 'RB' or pos_sentences[i][1] == 'RBR' or pos_sentences[i][1] == 'RBS': | ||
for syn in wordnet.synsets(tokenized_sent[i]): | ||
|
||
for lm in syn.lemmas(): | ||
|
||
if lm.antonyms(): | ||
antonyms.append(lm.antonyms()[0].name()) | ||
|
||
if len(antonyms) != 0: | ||
tokenized_sent[i] = 'not ' + antonyms[0] | ||
|
||
return ' '.join(tokenized_sent) |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This transformation is unfortunately the same as #222 as it too uses negated antonyms. @Mayukhga83 is there anything different?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kaustubhdhole unfortunately yes except for the 'not' word as @marco-digio pointed. Secondly, this always does double negation for both even and odd number of adjective unlike #222 which only does it for even number of adjective words.
As the addition of 'not' gives a better robustness score than #222 (I have only checked for roberta-base-imdb) , It could be merged as a special case of the other PR if suited.