Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add data_validate_args #13

Merged
merged 2 commits into from
Sep 8, 2021
Merged

Add data_validate_args #13

merged 2 commits into from
Sep 8, 2021

Conversation

HotThoughts
Copy link
Contributor

@HotThoughts HotThoughts commented Sep 7, 2021 •
edited
Loading

Resolves #12

✅ The workflow seems correctly picked up the flag --fail-on-warning and failed the run.

@HotThoughts HotThoughts self-assigned this Sep 7, 2021
@github-actions
Copy link

github-actions bot commented Sep 7, 2021

Commit: 430619b
Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
Rasa Demo - ALL 0.8769 (0.00) 0.8743 (0.00) 0.7160 (0.00) 1.0000 (0.00)

@github-actions
Copy link

github-actions bot commented Sep 7, 2021

Commit: 430619b
Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
Rasa Demo - NLU 0.6693 (no data) 0.7211 (0.16) 0.5942 (-0.06) no data

Intent Cross-Validation Results

class support f1-score confused_with
macro avg 4995 0.41047157525178724 N/A
weighted avg 4995 0.62063923660556540 N/A
faq 880 0.82032854209445580 enter_data(33), chitchat(28)
chitchat 812 0.76849894291754750 faq(40), enter_data(19)
enter_data 759 0.73391494002181030 faq(27), affirm(12)
out_of_scope 410 0.16309012875536480 chitchat(128), enter_data(125)
affirm 224 0.68131868131868130 enter_data(21), chitchat(5)
technical_question 221 0.32500000000000000 enter_data(52), faq(48)
how_to_get_started 211 0.63473053892215570 faq(34), chitchat(26)
contact_sales 155 0.66832917705735660 faq(8), out_of_scope(3)
greet 147 0.64591439688715960 enter_data(29), chitchat(16)
signup_newsletter 141 0.82208588957055230 out_of_scope(2), need_help_broad(2)
install_rasa 108 0.74881516587677720 faq(10), nlu_info(7)
ask_which_events 107 0.82790697674418600 enter_data(5), chitchat(5)
deny 100 0.48587570621468923 affirm(16), enter_data(10)
human_handoff 69 0.58252427184466030 contact_sales(20), chitchat(9)
react_positive 65 0.16438356164383564 affirm(23), chitchat(19), enter_data(9)
nlu_info 62 0.50000000000000000 enter_data(8), faq(4)
switch 54 0.70476190476190480 enter_data(7), nlu_info(3)
ask_how_contribute 53 0.27272727272727276 chitchat(14), contact_sales(6)
react_negative 47 0.14545454545454545 enter_data(14), contact_sales(9), chitchat(8)
why_rasa 45 0.45070422535211270 faq(9), switch(6)
bye 42 0.46666666666666660 affirm(13), enter_data(6)
ask_question_in_forum 42 N/A faq(22), signup_newsletter(5), chitchat(3)
need_help_broad 41 0.34375000000000006 install_rasa(8), nlu_info(6)
thank 39 0.67567567567567570 contact_sales(4), affirm(4)
source_code 34 0.19047619047619047 chitchat(9), faq(4)
canthelp 26 0.06250000000000000 chitchat(10), enter_data(4), technical_question(3)
ask_why_contribute 21 N/A nlu_info(5), contact_sales(3), chitchat(3)
pipeline_recommendation 19 0.17391304347826086 enter_data(3), ask_which_events(3)
explain 16 0.10526315789473684 chitchat(8), enter_data(2), affirm(1)
broken 15 0.19047619047619044 faq(4), enter_data(3), contact_sales(2)
nlu_generation_tool_recommendation 14 N/A faq(5), enter_data(4), nlu_info(3)
restart 10 0.19047619047619050 why_rasa(2), faq(2)
book_demo 6 N/A contact_sales(2), deny(2), faq(2)

Entity Cross-Validation Results

entity support f1-score precision recall
micro avg 1498 0.72112885258076490 0.8125523012552301 0.64819759679572760
macro avg 1498 0.54890069388941810 0.6696198465999292 0.52814387839948930
weighted avg 1498 0.68618510234277650 0.7950963310770189 0.64819759679572760
product 554 0.69573459715639810 0.7325349301397206 0.66245487364620940
language 297 0.96013289036544850 0.9475409836065574 0.97306397306397310
job_function 160 0.57364341085271310 0.7551020408163265 0.46250000000000000
name 155 0.49549549549549554 0.8208955223880597 0.35483870967741940
nlu_part 94 0.89552238805970150 0.8411214953271028 0.95744680851063830
company 87 0.08695652173913043 0.8000000000000000 0.04597701149425287
current_api 61 0.84210526315789470 0.9056603773584906 0.78688524590163930
location 42 0.75362318840579710 0.9629629629629629 0.61904761904761910
user_type 19 0.73469387755102040 0.6000000000000000 0.94736842105263150
entity 16 N/A N/A N/A
install_type 13 N/A N/A N/A

@github-actions
Copy link

github-actions bot commented Sep 7, 2021

Commit: 430619b
Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
Rasa Demo - Configuration 1 0.8689 (0.00) 0.8862 (0.00) 0.8363 (0.00) 1.0000 (0.00)
Rasa Demo - Configuration 2 0.8605 (0.00) 0.8943 (0.00) 0.8335 (0.00) 1.0000 (0.00)

@github-actions
Copy link

github-actions bot commented Sep 7, 2021

Commit: ed4a17e
Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
Rasa Demo - ALL 0.8685 (0.00) 0.8659 (0.00) 0.7926 (0.00) 0.9975 (0.00)

@github-actions
Copy link

github-actions bot commented Sep 7, 2021

Commit: ed4a17e
Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
Rasa Demo - NLU 0.6747 (no data) 0.7179 (0.16) 0.5666 (-0.09) no data

Intent Cross-Validation Results

class support f1-score confused_with
macro avg 4995 0.42487967520737980 N/A
weighted avg 4995 0.63188683701501840 N/A
faq 880 0.80038666022232960 enter_data(21), chitchat(20)
chitchat 812 0.74778991159646380 faq(56), enter_data(19)
enter_data 759 0.75725900116144020 faq(37), chitchat(16)
out_of_scope 410 0.24116424116424112 chitchat(145), enter_data(95)
affirm 224 0.68148148148148160 enter_data(27), react_positive(11)
technical_question 221 0.27338129496402880 faq(58), chitchat(32)
how_to_get_started 211 0.61777777777777780 faq(30), chitchat(16)
contact_sales 155 0.82131661442006280 faq(16), enter_data(3)
greet 147 0.74825174825174820 enter_data(14), chitchat(10)
signup_newsletter 141 0.80120481927710840 enter_data(4), greet(2)
install_rasa 108 0.69955156950672650 faq(15), how_to_get_started(9)
ask_which_events 107 0.75925925925925930 chitchat(16), faq(6)
deny 100 0.43076923076923080 enter_data(30), chitchat(8)
human_handoff 69 0.56862745098039210 chitchat(15), faq(8)
react_positive 65 0.46153846153846160 chitchat(20), affirm(6)
nlu_info 62 0.75000000000000000 faq(6), chitchat(5)
switch 54 0.74576271186440670 faq(4), enter_data(3)
ask_how_contribute 53 0.39436619718309857 chitchat(14), faq(9)
react_negative 47 0.46808510638297873 enter_data(10), chitchat(10)
why_rasa 45 0.40625000000000000 faq(9), switch(7)
bye 42 0.26865671641791045 how_to_get_started(9), affirm(6)
ask_question_in_forum 42 0.24561403508771930 faq(26), contact_sales(6)
need_help_broad 41 0.04545454545454545 install_rasa(7), enter_data(6), technical_question(5)
thank 39 0.60975609756097570 greet(3), faq(3)
source_code 34 0.17391304347826090 chitchat(11), faq(9)
canthelp 26 N/A chitchat(16), signup_newsletter(3), technical_question(2)
ask_why_contribute 21 0.14814814814814814 why_rasa(5), chitchat(3)
pipeline_recommendation 19 N/A how_to_get_started(8), signup_newsletter(4), out_of_scope(2)
explain 16 N/A chitchat(8), faq(4), restart(2)
broken 15 0.10526315789473685 faq(5), enter_data(5), chitchat(2)
nlu_generation_tool_recommendation 14 N/A how_to_get_started(4), faq(3), enter_data(3)
restart 10 0.25000000000000000 how_to_get_started(4), react_negative(2)
book_demo 6 N/A ask_which_events(2), faq(2), chitchat(1)

Entity Cross-Validation Results

entity support f1-score precision recall
micro avg 1498 0.71790988262021960 0.8293963254593176 0.63284379172229640
macro avg 1498 0.51607447652780470 0.6524722136997665 0.48083495929563840
weighted avg 1498 0.67679065408984160 0.8017163125898689 0.63284379172229640
product 554 0.73110047846889950 0.7780040733197556 0.68953068592057760
language 297 0.95695364238410590 0.9413680781758957 0.97306397306397310
job_function 160 0.49579831932773116 0.7564102564102564 0.36875000000000000
name 155 0.39810426540284360 0.7500000000000000 0.27096774193548390
nlu_part 94 0.89340101522842630 0.8543689320388349 0.93617021276595750
company 87 0.08695652173913043 0.8000000000000000 0.04597701149425287
current_api 61 0.84403669724770640 0.9583333333333334 0.75409836065573770
location 42 0.71232876712328760 0.8387096774193549 0.61904761904761910
user_type 19 0.55813953488372090 0.5000000000000000 0.63157894736842100
entity 16 N/A N/A N/A
install_type 13 N/A N/A N/A

@github-actions
Copy link

github-actions bot commented Sep 7, 2021

Commit: ed4a17e
Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
Rasa Demo - Configuration 1 0.8919 (0.00) 0.8513 (0.00) 0.7755 (0.00) 1.0000 (0.00)
Rasa Demo - Configuration 2 0.8741 (0.00) 0.8671 (0.00) 0.7298 (0.00) 1.0000 (0.00)

tczekajlo
tczekajlo approved these changes Sep 8, 2021
View reviewed changes
Copy link
Contributor

@tczekajlo tczekajlo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🎸

@HotThoughts HotThoughts merged commit 63f1556 into main Sep 8, 2021
@HotThoughts HotThoughts deleted the data-validation-argument branch September 8, 2021 08:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tczekajlo tczekajlo tczekajlo approved these changes

Assignees

@HotThoughts HotThoughts

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Rasa data validation should accept arguments
2 participants
@HotThoughts @tczekajlo