bug fix in data

[xingdi-eric-yuan]

Some bugs in splitting options. I only looked at ruin_names but there might be similar things in other tasks.

It's quite easy to target, look for datapoints where 1) the "target" is not an option label; or 2) the length of options is different from BigBench.

The buggy data points are:

1. bbh:

   {"input": "Which of the following is a humorous edit of this artist or movie name: 'rita, sue and bob too'?\nOptions:\n(A) rita\n(B) sue and bob too\n(C) rita\n(D) sue and bob poo\n(E) rita\n(F) sue and box too\n(G) rita,y sue and bob too", "target": "rita, sue and bob poo"}

   bb:
   https://github.com/google/BIG-bench/blob/main/bigbench/benchmark_tasks/ruin_names/task.json#L1437

2. bbh:

   {"input": "Which of the following is a humorous edit of this artist or movie name: 'earth, wind, & fire'?\nOptions:\n(A) eareth\n(B) wind\n(C) & fire\n(D) earth\n(E) bind\n(F) & fire\n(G) earthm wind\n(H) & fire\n(I) dearth\n(J) wind\n(K) & fire", "target": "dearth, wind, & fire"}

   bb:
   https://github.com/google/BIG-bench/blob/main/bigbench/benchmark_tasks/ruin_names/task.json#L1743

It's quite difficult to see the diff in github, you can use this diff checker:
https://www.diffchecker.com/xMl0kTzr/

[xingdi-eric-yuan]

Similar thing in movie_recommendation

1. bbh:

   {"input": "Find a movie similar to Terminator 2 Judgment Day, The Fugitive, The Shawshank Redemption, Dead Man Walking:\nOptions:\n(A) Walk\n(B) Don't Run\n(C) Shaun the Sheep Movie\n(D) Rocky IV\n(E) Braveheart", "target": "(E)"}

   bb:
   https://github.com/google/BIG-bench/blob/main/bigbench/benchmark_tasks/movie_recommendation/task.json#L1423

2. bbh:

   {"input": "Find a movie similar to Minority Report, Shrek, Catch Me If You Can, Aladdin:\nOptions:\n(A) Monsters\n(B) Inc\n(C) Children of the Night\n(D) The Incredible Shrinking Man\n(E) Town & Country", "target": "Monsters, Inc"}

   bb:
   https://github.com/google/BIG-bench/blob/main/bigbench/benchmark_tasks/movie_recommendation/task.json#L2593

and a dozen+ more..

[xingdi-eric-yuan]

From these few examples, maybe the bug happens when there's a , in the option?

[Brian-Ckwu]

I stumbled across these bugs, too. Thank @xingdi-eric-yuan for addressing these first!

[RylanSchaeffer]

@suzgunmirac can this pull request be merged please?