the wrong loss func was chosen at evaluation

[peterjc123]

The loss func is always moe_loss_func as can be seen here. But the loss is only calculated when training, which can be seen here. We should fallback to the original loss func during evaluation.

[tgale96]

That sounds reasonable to me! We'd be happy to take a PR if you'd like to fix this in our Megatron-LM fork!

[peterjc123]

@tgale96 I've put up a fix. Please take a look when you have time. stanford-futuredata/Megatron-LM#6