pytorch - 💡(How to fix) Fix Adding Dynamic Tanh (DyT) element wise Operation (Paper from Meta)

🚀 The feature, motivation and pitch

An important paper from Meta(FAIR), MIT, NYU and Princeton in late 2025 shows transformers can perform equally if not better using a Dynamic tanh operation instead of a Layer Normalization. The PR for this issue will aim to add this technique in the nn folder and norms file with minimal code. There is a successor paper to this as well featuring Dynamic erf (Derf) function for which i will open another issue.

I am attaching the related links here: Paper: https://arxiv.org/pdf/2503.10622 Title: Transformers without Normalization Code: https://github.com/jiachenzhu/DyT Website: https://jiachenzhu.github.io/DyT/

cc. @msaroufim @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki @malfet @zou3519 @jerryzh168 @cyyever @apaszke @anijain2305 @suo @janeyx99

Alternatives

No response

Additional context

<img width="732" height="490" alt="Image" src="https://github.com/user-attachments/assets/9f9d7c63-f158-4eb7-a3e2-288f8dc18399" /> <img width="1576" height="348" alt="Image" src="https://github.com/user-attachments/assets/a5df82be-046b-4071-9e8a-091e1b0fe2c3" />

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki