Different from the output of the HF inference

[xcxhy]

Looking forward to your reply! I set the temperature=0.1, top-k=10, top-p=0.75，I think i infer the same prompt, I will get the same output, through test the hf and vllm inference, HF will get stable output then vllm sometimes get the different output. Is this normal? The parameters of the two inferences are the same and do not bring the same constraints.

[marscc]

I have encountered the same problem. use llama 13B model, set max_tokens=256, frequency_penalty=0.1, temperature=0.1, top-k=50, top-p=0.75，I tested on a set of 40 questions and found that the outputs for 15 questions were different from the outputs obtained using huggingface inference.

[zhuohan123]

The LLM inference process includes sampling, which is a random process. Because the implementation of HF and vLLM are different, it is normal to get different samples. However, if you perform argmax sampling (e.g., temperature=0), then you should be able to see the same results.

[tonystz]

even when I set temperature=0, the result is still not the same. Anyone has some suggestions?

[kaishxu]

same problem. I set temperature=0, and the result is still not the same.

[damandeep-hyprbots]

Same issue.

[p1x33l]

Same issue

[GhostofAdam]

I am using greedy search in decoding.
I find when the prompt lenght is less than 7(while 8 as block size), vllm and hf get different output texts, but they are same with longer prompts. I believe a bug is hidden behind this.

[AmazeQiu]

I've meet the same problem.
I use hf with this config:
temperature=0, top_p=1.0 ,top_k=40 ,num_beams=4, max_new_tokens=64
then use vllm with this config :
max_tokens = 64, use_beam_search = True, best_of = 4, temperature = 0 ,top_p = 1.0, top_k = -1
The speed is fast but the accuracy is pretty lower than that using hf. T _ T
I even change the source code to use "model.half().cuda()" to make it same with the way I use hf in my code.

[BaiMoHan]

Has this issue been resolved?

[Jerrrrykun]

I have encountered the same issue for temperature=0. Interestingly, some of outputs are same with HF inference but some are not, leading to decreases in performance.

[hongboshi1234]

There is no guarantee to get the exact the same output even for the same model weights and sampling parameters(say temp=0, or greedy). The reasons could be :
1 different serving framework, vllm sglang, hf or trtllm have different kernel implementations
2 even within the same framework, differnet batch size may trigger different kernel or gemm computations, resulting in slightly difference in the logits. those subtle difference might be cut off in lower precisions, then making two tokens have exactly the same logits.
3 precision difference. fp16, bf16, fp32, it's easy to make mistakes as different frameworks may read those setups differently

So if you see the difference. try several things
1 if both results looks reasonble to you. it's highly possible due to computation precision variance from kernels
2 if one looks obviously good than the other(you want to look at multiple cases), there might be some isseus with the setup

Another quick thing to check is to look at the logits between the HF and vllm. especially look at the logits where they starts to diverge