Meaning of 'loss' in QA pairs & why all values are identical

[Jellyfish042]

I'm generating QA pairs and have a question about the 'loss' value:

What does this 'loss' signify for the generated QA pairs?
All my generated QA pairs show the exact same 'loss' value. Is this expected?

Thanks!

[ChenZiHong-Gavin]

In GraphGen framework, we use ECE(Expected Calibration Error) to assigned a comprehension loss to each knowledge point in the knowledge graph, which quantifies the Trainee Model's understanding level of that concept (i.e., identifying its "blind spots"). This loss aids in structuring subgraphs during generation.

For generated QA pairs, 'loss' is the average comprehension loss of all knowledge points in the subgraph. This metric evaluates how well the Trainee Model comprehends the QA content.

Exception Case:
If the Trainee Model is not used or its deployed service lacks support for retrieving logprob values, the loss will default to a predefined value.

You may try vllm serve with v0.6.6post1 version (see #10).