Fix paddle.incubate.nn.functional.fused_bias_dropout_residual_layer_norm big Tensor

[xingmingyyj]

PR Category

Operator Mechanism

PR Types

Bug fixes

Description

• 修复int溢出导致的访存越界问题
• batch较大时存在精度diff，未定位出明显bug

[accuracy error] backward  paddle.incubate.nn.functional.fused_bias_dropout_residual_layer_norm(x=Tensor([270000000, 2, 4],"float32"), residual=Tensor([270000000, 2, 4],"float32"), bias=None, ln_scale=Tensor([4],"float32"), ln_bias=None, dropout_rate=0.0, ln_epsilon=1e-05, training=True, mode="upscale_in_train", name=None, ) 
 Not equal to tolerance rtol=0.01, atol=0.01
Tensor-likes are not close!

Mismatched elements: 3 / 2160000000 (0.0%)
Greatest absolute difference: 0.03557777404785156 at index (122354108, 0, 3) (up to 0.01 allowed)
Greatest relative difference: 0.1643351912498474 at index (64014755, 1, 3) (up to 0.01 allowed)
ACTUAL: (shape=torch.Size([270000000, 2, 4]), dtype=torch.float32)

通过一些测试发现和做layernorm的feature size有一定关系

[accuracy error] backward  paddle.nn.functional.layer_norm(Tensor([270000000, 2, 3],"float32"), 3, weight=Tensor([3],"float32"), bias=Tensor([3],"float32"), epsilon=1e-05, ) 
 Not equal to tolerance rtol=0.01, atol=0.01
Tensor-likes are not close!

Mismatched elements: 21 / 1620000000 (0.0%)
Greatest absolute difference: 0.0194987952709198 at index (59252892, 1, 0) (up to 0.01 allowed)
Greatest relative difference: 1.9963128566741943 at index (112159141, 0, 0) (up to 0.01 allowed)
ACTUAL: (shape=torch.Size([270000000, 2, 3]), dtype=torch.float32)
tensor([[[ 0.4093,  0.2924, -0.7016],


[accuracy error] backward  paddle.nn.functional.layer_norm(Tensor([270000000, 2,3],"float32"), 3, weight=Tensor([3],"float32"), bias=Tensor([3],"float32"), epsilon=1e-05, ) 
 Not equal to tolerance rtol=0.01, atol=0.01
Tensor-likes are not close!

Mismatched elements: 61 / 1620000000 (0.0%)
Greatest absolute difference: 0.022376805543899536 at index (159520737, 0, 1) (up to 0.01 allowed)
Greatest relative difference: 2.069608688354492 at index (165861042, 0, 1) (up to 0.01 allowed)
ACTUAL: (shape=torch.Size([270000000, 2, 3]), dtype=torch.float32)
tensor([[[ 0.0337, -0.0248, -0.0089],
         [-0.3800,  0.1632,  0.2168]],

但paddle和torch的layer_norm实现在数学上等价。
Pcard-73263

[paddle-bot]

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[paddle-ci-bot]

Sorry to inform you that d9090b0's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

[xingmingyyj]

/re-run all-failed