pytorch - 💡(How to fix) Fix CTCLoss backward is inconsistent with its forward derivative with respect to the documented log_probs input. [1 participants]

import torch import torch.nn as nn import torch.nn.functional as F

torch.manual_seed(123)

T, N, C = 3, 1, 3 # time, batch, classes incl. blank log_probs = F.log_softmax(torch.randn(T, N, C), dim=-1).double() log_probs.requires_grad_(True)

targets = torch.tensor([[1]]) input_lens = torch.tensor([T]) target_lens = torch.tensor([1])

loss_fn = nn.CTCLoss(blank=0)

def f(x): return loss_fn(x, targets, input_lens, target_lens)

print(torch.autograd.gradcheck(f, (log_probs,), raise_exception=True))

TL;DR

The most likely fix is to verify the correctness of the CTCLoss implementation, specifically the backward method, to ensure it aligns with the derivative of CTCLoss.forward() with respect to the log_probs input.

Guidance

• Review the CTCLoss implementation, focusing on the backward method, to identify potential inconsistencies with the forward method's derivative.
• Verify that the log_probs input is correctly handled in the backward method, particularly for classes that are neither the blank class nor the target label.
• Check the PyTorch documentation and issue tracker for any known issues or updates related to CTCLoss and its gradient calculation.
• Consider testing the CTCLoss implementation with different inputs and scenarios to further isolate the issue.

Notes

The provided example suggests an inconsistency in the CTCLoss.backward() method, but without access to the PyTorch implementation details, it's challenging to provide a definitive fix. The issue may be related to a specific version of PyTorch or a particular use case.

Recommendation

Apply workaround: Until the CTCLoss implementation is verified or updated, consider using a custom or alternative loss function that correctly handles the log_probs input and its gradient calculation.