System Info

PPDocLayoutV3 seems to have a numerical precision issue that snowballs into larger discrepancies. One cause of those precision issues is the function _cached_generate_anchors, which creates a binary mask. The mask may or may not contain the bottom-right boundary, depending on whether the chosen execution device is CPU or CUDA.

The function is duplicated at least 7 times over the transformers repository: https://grep.app/search?q=_cached_generate_anchors

I have not verified whether other models besides PPDocLayoutV3 actually trigger this issue. It requires specific widths and heights, for example 50. I have simplified the code a bit to make the error easier to spot:

The code below will result in the following different masks:

The root cause is a bug PyTorch, which has undefined floating point rounding modes: https://github.com/pytorch/pytorch/issues/186618

I would guess that the CUDA mask is probably the intended behavior due to symmetry, although I am not entirely certain.

Who can help?

Perhaps @yonigozlan knows more.

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

# This program creates two masks, one using CPU and one using CUDA. Boundary pixels are handled differently.
import torch

def _cached_generate_anchors(device):
    width = height = 50

    grid_y, grid_x = torch.meshgrid(
        torch.arange(end=height, device=device).float(),
        torch.arange(end=width, device=device).float(),
        indexing="ij",
    )

    grid_xy = torch.stack([grid_x, grid_y], -1)
    grid_xy = grid_xy.unsqueeze(0) + 0.5
    grid_xy[..., 0] /= width # <--- rounding error introduced here
    grid_xy[..., 1] /= height
    anchors = grid_xy
    eps = 1e-2
    valid_mask = ((anchors > eps) * (anchors < 1 - eps)).all(-1, keepdim=True) # <--- rounding error turned into larger error here
    return valid_mask

mask_cuda = _cached_generate_anchors("cuda")
mask_cpu = _cached_generate_anchors("cpu")

if not mask_cpu.equal(mask_cuda.cpu()):
    print("ERROR: Different masks on CPU and GPU")

import matplotlib.pyplot as plt
plt.subplot(1, 2, 1)
plt.title("CPU")
plt.imshow(mask_cuda.reshape(50, 50).detach().cpu(), cmap="gray")
plt.subplot(1, 2, 2)
plt.title("CUDA")
plt.imshow(mask_cpu.reshape(50, 50).detach().cpu(), cmap="gray")
plt.show()

Expected behavior

The masks should be the same on CPU and GPU.