pytorch - 💡(How to fix) Fix Softplus degenerates incorrectly [1 comments, 2 participants]

TL;DR

The issue can be resolved by adjusting the beta parameter in the F.softplus function to a smaller value, as the current value of 1e30 is causing the function to produce infinite values.

Guidance

• The beta parameter in F.softplus controls the steepness of the softplus function, and a very large value like 1e30 can lead to infinite outputs for small inputs.
• Try reducing the beta value to a smaller number, such as 1 or 10, to see if it produces the expected results.
• The threshold parameter is set to float('inf'), which means it will not have any effect on the output, so adjusting the beta value is the most likely solution.
• Verify the results by printing the output of F.softplus with the adjusted beta value and comparing it to the expected output.

Example

import torch
import torch.nn.functional as F

x = torch.tensor([0.5, 1.0, 2.0])
cpu = F.softplus(x, beta=1, threshold=20)
gpu = F.softplus(x.cuda(), beta=1, threshold=20).cpu()

print("CPU:", cpu.tolist())
print("GPU:", gpu.tolist())

Notes

The issue may be specific to the version of PyTorch being used, and adjusting the beta value may not work for all versions.

Recommendation

Apply workaround: Adjust the beta value in the F.softplus function to a smaller number, such as 1 or 10, to avoid producing infinite outputs.