transformers - 💡(How to fix) Fix A nice UX for generating dynamic tensors that break torch.compile/torch.export [1 participants]

TL;DR

Implementing a dynamic tensor construction solution, either by utilizing the processor for per-sample processing loops or unifying the collator API, is likely the most effective fix.

Guidance

• Investigate the feasibility of using the processor to handle dynamic tensor construction, considering its existing role in per-sample processing loops.
• Explore unifying the collator API as a potential solution, examining how it could be integrated into the default path (processor -> collator -> inference).
• Review the related issue #45396 for additional context and motivation behind addressing dynamic tensor construction.
• Consider the trade-offs between the two potential solutions, weighing factors such as complexity, performance, and maintainability.

Notes

The ideal solution may depend on specific requirements and constraints not fully detailed in the issue, so careful evaluation of both options is necessary.

Recommendation

Apply workaround: Utilize the processor for dynamic tensor construction, as it appears to be the most direct and potentially efficient solution, given its existing involvement in per-sample processing loops.