-
Notifications
You must be signed in to change notification settings - Fork 1
/
datasets.yaml
169 lines (162 loc) · 5.93 KB
/
datasets.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
mmlu:
tasks:
mmlu:
task_type: classification
labels:
- A
- B
- C
- D
model_output_parser: KeywordParser
column_input: prompt
column_reference: target
description: Modified MMLU dataset from HuggingFace
metadata:
source: custom
loader_classname: MMLU
loader_filename: evaluate
n_shots: 5
bbh:
tasks:
bbh:
task_type: classification
model_output_parser: KeywordParser
column_input: prompt
column_reference: target
description: Modified BBH dataset from HuggingFace
metadata:
source: custom
loader_classname: BBH
loader_filename: evaluate
n_shots: 5
demo-random-generation:
tasks:
random-generation-prompts:
task_type: generation
column_input: prompt
description: Dataset for runtime inference stat measurement
metadata:
source: local
data:
- prompt: How to get to Vancouver from Seattle?
- prompt: Generate 3 male names in a list.
- prompt: Convert 4.33 x 10 to scientific notation.
- prompt: Suggest a vegetarian breakfast menu.
- prompt: What is the difference between vegetarian and vegan food?
- prompt: Generate an example of an artificial intelligence use case.
- prompt: Tell me five words that rhyme with 'shock'.
- prompt: List all Canadian provinces in alphabetical order.
- prompt: Given two sentences, find the most appropriate transition word to make the sentences flow smoothly.\nThe house was empty. The furniture was missing.
- prompt: |
Provide a list of action items from the following conversation.
John: Hi Alice, how are you?
Alice: I am good, how about yourself?
John: Good good. Hey, I wanted to know if the ticket number 55555 had been resolved?
Alice: Let me check that for you. Please hold on.
John: Okay.
Alice: Thanks for waiting, John. Looks like the team was able address the issue with the internet. It should work now.
John: Thanks!
- prompt: |
Provide a concise summary of the following conversation.
John: Hi Alice, how are you?
Alice: I am good, how about yourself?
John: Good good. Hey, I wanted to know if the ticket number 55555 had been resolved?
Alice: Let me check that for you. Please hold on.
John: Okay.
Alice: Thanks for waiting, John. Looks like the team was able address the issue with the internet. It should work now.
John: Thanks!
- prompt: |
Provide a concise summary of the following conversation.
John: Hi Alice, how are you?
Alice: I am good, how about yourself?
John: Good good. Hey, I wanted to know if the ticket number 55555 had been resolved?
Alice: Let me check that for you. Please hold on.
John: Okay.
Alice: Thanks for waiting, John. Looks like the team was able address the issue with the internet. It should work now.
John: Thanks!
John: Hi Alice, how are you?
Alice: I am good, how about yourself?
John: Good good. Hey, I wanted to know if the ticket number 55555 had been resolved?
Alice: Let me check that for you. Please hold on.
John: Okay.
Alice: Thanks for waiting, John. Looks like the team was able address the issue with the internet. It should work now.
John: Thanks!
John: Hi Alice, how are you?
Alice: I am good, how about yourself?
John: Good good. Hey, I wanted to know if the ticket number 55555 had been resolved?
Alice: Let me check that for you. Please hold on.
John: Okay.
Alice: Thanks for waiting, John. Looks like the team was able address the issue with the internet. It should work now.
John: Thanks!
John: Hi Alice, how are you?
Alice: I am good, how about yourself?
John: Good good. Hey, I wanted to know if the ticket number 55555 had been resolved?
Alice: Let me check that for you. Please hold on.
John: Okay.
Alice: Thanks for waiting, John. Looks like the team was able address the issue with the internet. It should work now.
John: Thanks!
John: Hi Alice, how are you?
Alice: I am good, how about yourself?
John: Good good. Hey, I wanted to know if the ticket number 55555 had been resolved?
Alice: Let me check that for you. Please hold on.
John: Okay.
Alice: Thanks for waiting, John. Looks like the team was able address the issue with the internet. It should work now.
John: Thanks!
demo-chat:
tasks:
random-questions:
task_type: generation
column_input: prompt
column_reference: response
description: Dataset for random generative tasks (single)
metadata:
source: local
data:
- prompt: How to get to Vancouver from Seattle?
response: Take a plane, bus, train or car.
- prompt: Generate 3 male names in a list.
response: One, Two, Three
- prompt: Convert 4.33 x 10 to scientific notation.
response: 4.33e2
demo-classification:
tasks:
random-mcq:
task_type: classification
model_output_parser: KeywordParser
labels:
- A
- B
- C
- D
- E
none_value: E
column_input: prompt
column_reference: response
description: Dataset for random generative tasks (numeric reference)
metadata:
source: local
data:
- prompt: |
Pick the correct answer for the following question:
1 + 1 = ?
A. 0
B. 1
C. 2
D. 3
response: C
- prompt: |
Answer the following multiple choice question:
2 + 2 = ?
A. 1
B. 2
C. 3
D. 4
response: D
- prompt: |
Select the correct response for the following multiple choice question:
1 + 2 = ?
A. 1
B. 2
C. 3
D. 4
response: C