Small bugs in the dataset
#4
by t1u1 - opened
From Figure 9 of the paper:
Which of the following best describes the seventh chord in the above example?
L:1/4
M:4/4
K:D
[FGBd]4 |] %1
A. Major seventh in third inversion
B. Dominant seventh in second inversion
C. Major/minor seventh in third inversion
D. Minor seventh in second inversion
Answer: A
Here, the word "above" is wrong, because the example is below not above. All the examples in that figure have the same bug.
I wonder whether the LLM gets confused or learns wrong meanings by such bugs.
Thanks a lot for reminding us about the bug!
This bug is introduced in our evaluation template code (will be open-sourced in our github repo soon).
We are planning a new test soon. We will rearrange the template and see how much it affects the result.
New version will look something like this:
L:1/4
M:4/4
K:D
[FGBd]4 |] %1
Which of the following best describes the seventh chord in the above example?
A. Major seventh in third inversion
B. Dominant seventh in second inversion
C. Major/minor seventh in third inversion
D. Minor seventh in second inversion
Answer: A