Pass@1 metric does not increase even with provided example dataset

[nasrullonurullaev]

I'm testing code performance using benchllama with my own dataset. However, the pass@1 metric does not change as expected. Even when using the dataset provided in the examples folder, the pass@1 score does not increase.

Could you provide guidance on how to properly test the code? Is there a specific setup required to make sure benchllama evaluates the results correctly?

Any help would be greatly appreciated. Thanks!