pytorch - 💡(How to fix) Fix torch.distributed._functional_collectives.reduce_scatter_tensor gets a wrong result [1 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
pytorch/pytorch#178766Fetched 2026-04-08 01:52:17
View on GitHub
Comments
1
Participants
2
Timeline
28
Reactions
0
Timeline (top)
mentioned ×12subscribed ×12labeled ×3commented ×1

Code Example

gpu0 and gpu1 have same input tensor, which is
[ -58,  -62,  -54,  -36,  -49,  -55,  -35,  -36,  -60,  -45,  -34,  -56,  -54,  -55,  -50,  -78,  -46,  -70,  -60,  -74],
        [ -69,  -38,  -49,  -58,  -62,  -51,  -48,  -51,  -60,  -69,  -78,  -59,  -28,  -63,  -54,  -55,  -49,  -50,  -63,  -43],
        [ -52,  -31,  -40,  -39,  -58,  -45,  -31,  -43,  -42,  -60,  -53,  -66,  -42,  -48,  -48,  -52,  -39,  -42,  -75,  -40],
        [ -63,  -61,  -76,  -57,  -53,  -58,  -60,  -56,  -73,  -72,  -36,  -46,  -66,  -52,  -45,  -64,  -70,  -59,  -57,  -42],
        [ -61,  -42,  -51,  -49,  -44,  -66,  -37,  -52,  -43,  -43,  -50,  -53,  -46,  -81,  -33,  -55,  -57,  -20,  -33,  -49],
        [ -71,  -63,  -61,  -60,  -29,  -60,  -58,  -58,  -46,  -32,  -25,  -51,  -40,  -69,  -46,  -60,  -19,  -55,  -46,  -61],
        [ -61,  -52,  -59,  -54,  -39,  -44,  -39,  -59,  -52,  -62,  -56,  -61,  -52,  -32,  -63,  -58,  -60,  -66,  -52,  -36],
        [ -68,  -41,  -61,  -56,  -67,  -50,  -76,  -64,  -74,  -38,  -39,  -64,  -61,  -42,  -64,  -50,  -44,  -65,  -66,  -58],
        [ -44,  -44,  -51,  -55,  -67,  -28,  -61,  -60,  -45,  -68,  -32,  -24,  -56,  -44,  -50,  -38,  -72,  -53,  -59,  -51],
        [ -58,  -49,  -52,  -55,  -71,  -27,  -57,  -82,  -51,  -75,  -54,  -50,  -41,  -43,  -48,  -42,  -57,  -50,  -59,  -47],
        [ -48,  -50,  -54,  -57,  -46,  -46,  -38,  -65,  -57,  -47,  -50,  -58,  -54,  -44,  -43,  -43,  -50,  -46,  -48,  -59],
        [ -43,  -69,  -46,  -65,  -55,  -65,  -62,  -55,  -57,  -58,  -42,  -59,  -49,  -48,  -51,  -55,  -56,  -77,  -44,  -50],
        [ -51,  -57,  -50,  -59,  -39,  -51,  -59,  -76,  -63,  -63,  -67,  -56,  -52,  -72,  -36,  -80,  -45,  -39,  -55,  -49],
        [ -53,  -48,  -52,  -61,  -71,  -44,  -56,  -48,  -49,  -74,  -63,  -74,  -84,  -54,  -53,  -50,  -29,  -18,  -42,  -49],
        [ -56,  -46,  -63,  -51,  -56,  -69,  -67,  -57,  -38,  -62,  -36,  -53,  -61,  -51,  -52,  -58,  -69,  -62,  -51,  -60],
        [ -42,  -60,  -59,  -63,  -68,  -32,  -59,  -59,  -72,  -37,  -55,  -61,  -61,  -66,  -47,  -58,  -59,  -46,  -45,  -36],
        [ -63,  -60,  -58,  -32,  -47,  -64,  -45,  -66,  -51,  -48,  -61,  -52,  -56,  -55,  -46,  -56,  -45,  -44,  -60,  -65],
        [ -64,  -58,  -50,  -55,  -57,  -67,  -44,  -34,  -34,  -59,  -55,  -58,  -62,  -42,  -38,  -51,  -62,  -78,  -61,  -56],
        [ -50,  -44,  -67,  -46,  -53,  -52,  -53,  -61,  -58,  -45,  -37,  -51,  -26,  -75,  -56,  -63,  -59,  -49,  -37,  -62],
        [ -55,  -50,  -38,  -65,  -48,  -60,  -20,  -67,  -58,  -72,  -63,  -49,  -50,  -64,  -52,  -45,  -61,  -19,  -65,  -53],
        [ -29,  -28,  -62,  -28,  -59,  -40,  -44,  -59,  -57,  -79,  -79,  -24,  -46,  -64,  -35,  -71,  -56,  -67,  -48,  -52],
        [ -42,  -72,  -57,  -49,  -48,  -46,  -47,  -40,  -60,  -27,  -56,  -45,  -58,  -34,  -62,  -59,  -88,  -37,  -51,  -72],
        [ -57,  -59,  -53,  -51,  -62,  -48,  -47,  -62,  -52,  -46,  -23,  -55,  -56,  -51,  -49,  -71,  -49,  -40,  -66,  -50],
        [ -56,  -48,  -50,  -52,  -54,  -51,  -44,  -49,  -60,  -61,  -63,  -67,  -48,  -56,  -63,  -44,  -64,  -45,  -25,  -67],
        [ -19,  -46,  -44,  -59,  -70,  -53,  -63,  -50,  -26,  -72,  -59,  -61,  -59,  -38,  -43,  -56,  -55,  -33,  -44,  -86],
        [ -57,  -56,  -59,  -41,  -70,  -60,  -38,  -25,  -65,  -53,  -30,  -52,  -69,  -47,  -71,  -57,  -62,  -57,  -59,  -42],
        [ -42,  -35,  -39,  -64,  -55,  -61,  -65,  -54,  -56,  -44,  -49,  -62,  -50,  -39,  -52,  -70,  -49,  -60,  -38,  -68],
        [ -66,  -51,  -53,  -55,  -34,  -56,  -63,  -46,  -50,  -48,  -82,  -62,  -69,  -48,  -70,  -69,  -35,  -52,  -67,  -57],
        [ -61,  -60,  -56,  -52,  -59,  -59,  -78,  -56,  -55,  -59,  -52,  -56,  -50,  -64,  -54,  -44,  -43,  -43,  -46,  -50],
        [ -54,  -62,  -53,  -39,  -62,  -61,  -69,  -24,  -54,  -56,  -64,  -46,  -62,  -66,  -43,  -60,  -54,  -71,  -52,  -64],
        [ -38,  -67,  -54,  -60,  -59,  -57,  -51,  -53,  -45,  -68,  -46,  -54,  -54,  -51,  -61,  -56,  -46,  -60,  -52,  -31],
        [ -54,  -49,  -68,  -45,  -51,  -41,  -34,  -47,  -58,  -35,  -53,  -54,  -62,  -53,  -66,  -37,  -61,  -73,  -56,  -42],
        [ -64,  -67,  -74,  -44,  -50,  -55,  -40,  -32,  -50,  -65,  -39,  -53,  -63,  -50,  -47,  -55,  -54,  -51,  -40,  -56],
        [ -29,  -32,  -58,  -80,  -46,  -39,  -48,  -55,  -64,  -50,  -71,  -63,  -68,  -49,  -68,  -71,  -50,  -59,  -52,  -55],
        [ -61,  -49,  -67,  -49,  -49,  -66,  -49,  -51,  -58,  -63,  -38,  -64,  -53,  -56,  -64,  -64,  -45,  -43,  -50,  -59],
        [ -79,  -58,  -38,  -57,  -45,  -43,  -50,  -47,  -64,  -70,  -45,  -68,  -75,  -60,  -61,  -47,  -57,  -43,  -51,  -62],
        [ -70,  -54,  -35,  -70,  -72,  -56,  -69,  -64,  -53,  -65,  -42,  -61,  -69,  -39,  -41,  -55,  -51,  -70,  -49,  -50],
        [ -59,  -66,  -76,  -52,  -50,  -61,  -58,  -59,  -56,  -37,  -67,  -47,  -72,  -54,  -44,  -47,  -69,  -41,  -56,  -34],
        [ -42,  -62,  -43,  -48,  -54,  -50,  -49,  -64,  -39,  -49,  -68,  -51,  -47,  -56,  -45,  -52,  -62,  -28,  -67,  -42],
        [ -91,  -56,  -51,  -34,  -62,  -62,  -59,  -50,  -51,  -62,  -58,  -49,  -65,  -49,  -59,  -61,  -39,  -46,  -55,  -49],
        [ -49,  -54,  -56,  -75,  -47,  -24,  -62,  -35,  -50,  -66,  -49,  -45,  -46,  -47,  -43,  -47,  -37,  -48,  -58,  -64],
        [ -50,  -60,  -49,  -50,  -54,  -63,  -57,  -60,  -45,  -60,  -54,  -50,  -83,  -52,  -63,  -53,  -57,  -77,  -51,  -60],
        [ -57,  -40,  -29,  -39,  -42,  -55,  -38,  -58,  -63,  -64,  -62,  -39,  -57,  -29,  -50,  -32,  -49,  -64,  -70,  -49],
        [ -33,  -54,  -70,  -76,  -54,  -59,  -50,  -62,  -41,  -48,  -59,  -61,  -51,  -52,  -66,  -43,  -36,  -77,  -42,  -51],
        [ -57,  -57,  -49,  -57,  -51,  -66,  -50,  -59,  -70,  -60,  -52,  -41,  -44,  -42,  -65,  -59,  -59,  -53,  -69,  -58],
        [ -42,  -54,  -59,  -75,  -31,  -54,  -65,  -72,  -70,  -53,  -37,  -77,  -59,  -63,  -43,  -56,  -69,  -63,  -47,  -61],
        [ -45,  -59,  -47,  -57,  -47,  -42,  -45,  -46,  -60,  -61,  -53,  -47,  -46,  -63,  -59,  -45,  -36,  -45,  -66,  -64],
        [ -43,  -71,  -50,  -27,  -71,  -53,  -41,  -45,  -64,  -67,  -61,  -62,  -54,  -47,  -46,  -52,  -41,  -42,  -59,  -44],
        [ -60,  -68,  -57,  -39,  -61,  -42,  -38,  -43,  -31,  -45,  -57,  -67,  -37,  -41,  -59,  -67,  -46,  -72,  -46,  -59],
        [ -40,  -74,  -48,  -78,  -45,  -60,  -49,  -71,  -46,  -51,  -56,  -34,  -40,  -42,  -72,  -49,  -49,  -51,  -53,  -82],
        [ -52,  -39,  -48,  -62,  -44,  -54,  -61,  -45,  -52,  -37,  -40,  -67,  -72,  -44,  -66,  -63,  -61,  -52,  -57,  -49],
        [ -60,  -56,  -54,  -46,  -75,  -62,  -46,  -75,  -59,  -55,  -41,  -47,  -57,  -52,  -44,  -65,  -73,  -48,  -57,  -47],
        [ -40,  -54,  -39,  -47,  -26,  -43,  -43,  -75,  -32,  -69,  -33,  -48,  -65,  -68,  -50,  -68,  -43,  -60,  -67,  -65],
        [ -60,  -51,  -51,  -44,  -44,  -51,  -69,  -58,  -59,  -61,  -49,  -54,  -69,  -39,  -61,  -57,  -45,  -65,  -49,  -61],
        [ -57,  -50,  -58,  -60,  -42,  -46,  -34,  -40,  -73,  -63,  -52,  -83,  -56,  -45,  -52,  -46,  -64,  -61,  -65,  -53],
        [ -77,  -50,  -49,  -33,  -69,  -56,  -49,  -64,  -29,  -42,  -43,  -54,  -55,  -63,  -62,  -40,  -76,  -69,  -50,  -64],
        [ -47,  -54,  -59,  -54,  -51,  -45,  -33,  -54,  -53,  -44,  -58,  -51,  -35,  -52,  -58,  -51,  -67,  -60,  -46,  -63],
        [ -56,  -32,  -41,  -47,  -61,  -36,  -49,  -55,  -36,  -67,  -63,  -56,  -32,  -45,  -43,  -39,  -70,  -55,  -37,  -48],
        [ -47,  -68,  -45,  -61,  -52,  -46,  -64,  -57,  -64,  -56,  -38,  -70,  -47,  -65,  -39,  -54,  -56,  -60,  -23,  -47],
        [ -44,  -44,  -75,  -41,  -49,  -54,  -55,  -41,  -81,  -57,  -60,  -47,  -54,  -46,  -68,  -60,  -50,  -45,  -50,  -47],
        [ -60,  -61,  -74,  -39,  -42,  -63,  -51,  -59,  -74,  -62,  -56,  -61,  -68,  -60,  -62,  -51,  -50,  -55,  -54,  -76],
        [ -59,  -67,  -40,  -62,  -55,  -47,  -40,  -53,  -62,  -51,  -30,  -52,  -50,  -51,  -62,  -49,  -37,  -71,  -41,  -48],
        [ -56,  -45,  -68,  -64,  -60,  -55,  -68,  -87,  -49,  -63,  -47,  -54,  -37,  -64,  -59,  -44,  -51,  -46,  -32,  -48],
        [ -54,  -42,  -58,  -38,  -48,  -24,  -40,  -50,  -70,  -33,  -56,  -90,  -64,  -73,  -65,  -59,  -41,  -22,  -56,  -61],
        [ -60,  -67,  -28,  -59,  -46,  -48,  -62,  -63,  -81,  -41,  -91,  -53,  -49,  -49,  -60,  -46,  -71,  -45,  -45,  -66],
        [ -59,  -45,  -61,  -67,  -32,  -39,  -51,  -33,  -74,  -60,  -71,  -62,  -39,  -36,  -37,  -64,  -45,  -62,  -84,  -51],
        [ -65,  -54,  -60,  -63,  -45,  -56,  -55,  -61,  -62,  -62,  -54,  -70,  -71,  -33,  -29,  -43,  -70,  -54,  -70,  -54],
        [ -59,  -71,  -68,  -54,  -57,  -62,  -81,  -36,  -45,  -48,  -57,  -43,  -32,  -45,  -42,  -65,  -52,  -71,  -51,  -59],
        [ -26,  -56,  -70,  -53,  -43,  -36,  -66,  -48,  -36,  -62,  -38,  -43,  -64,  -59,  -38,  -43,  -54,  -52,  -42,  -48],
        [ -56,  -59,  -55,  -52,  -49,  -51,  -44,  -80,  -78,  -50,  -42,  -85,  -59,  -66,  -57,  -71,  -48,  -47,  -53,  -56],
        [ -47,  -58,  -58,  -42,  -67,  -51,  -80,  -47,  -29,  -63,  -61,  -54,  -55,  -47,  -52,  -48,  -48,  -51,  -62,  -61],
        [ -62,  -63,  -91,  -57,  -58,  -46,  -61,  -51,  -65,  -70,  -47,  -61,  -60,  -49,  -48,  -50,  -56,  -63,  -57,  -38],
        [ -42,  -26,  -51,  -41,  -62,  -50,  -39,  -44,  -73,  -59,  -38,  -73,  -34,  -36,  -61,  -54,  -54,  -42,  -47,  -52],
        [ -38,  -51,  -60,  -34,  -34,  -45,  -71,  -67,  -60,  -55,  -40,  -57,  -49,  -49,  -67,  -49,  -55,  -45,  -81,  -52],
        [ -67,  -63,  -64,  -56,  -49,  -50,  -69,  -53,  -48,  -59,  -73,  -63,  -54,  -28,  -63,  -61,  -56,  -66,  -72,  -64],
        [ -54,  -67,  -48,  -44,  -55,  -68,  -46,  -34,  -51,  -64,  -63,  -58,  -54,  -45,  -70,  -62,  -62,  -37,  -38,  -83],
        [ -50,  -46,  -60,  -80,  -46,  -52,  -54,  -55,  -51,  -54,  -47,  -59,  -38,  -54,  -54,  -66,  -63,  -38,  -56,  -43],
        [ -60,  -45,  -69,  -59,  -61,  -51,  -45,  -58,  -40,  -62,  -49,  -38,  -47,  -70,  -50,  -45,  -69,  -76,  -46,  -26],
        [ -64,  -60,  -33,  -30,  -64,  -54,  -45,  -57,  -47,  -45,  -54,  -45,  -40,  -62,  -39,  -69,  -49,  -63,  -40,  -62],
        [ -39,  -46,  -60,  -67,  -20,  -36,  -44,  -65,  -49,  -38,  -58,  -52,  -57,  -44,  -27,  -40,  -50,  -37,  -57,  -44],
        [ -72,  -65,  -35,  -58,  -47,  -61,  -63,  -62,  -45,  -64,  -77,  -38,  -70,  -57,  -44,  -34,  -68,  -51,  -52,  -36],
        [ -52,  -49,  -53,  -43,  -65,  -42,  -50,  -49,  -61,  -51,  -66,  -58,  -47,  -63,  -56,  -66,  -47,  -57,  -45,  -53],
        [ -57,  -70,  -42,  -60,  -63,  -58,  -54,  -47,  -49,  -48,  -48,  -62,  -27,  -55,  -47,  -51,  -51,  -53,  -53,  -55],
        [ -69,  -46,  -41,  -45,  -49,  -47,  -64,  -79,  -52,  -51,  -40,  -62,  -59,  -50,  -52,  -78,  -54,  -60,  -61,  -50],
        [ -62,  -53,  -61,  -70,  -37,  -46,  -59,  -45,  -69,  -57,  -52,  -45,  -46,  -51,  -42,  -71,  -43,  -60,  -47,  -68],
        [ -35,  -78,  -50,  -62,  -48,  -71,  -59,  -75,  -69,  -43,  -55,  -33,  -40,  -50,  -40,  -39,  -49,  -61,  -61,  -60],
        [ -45,  -75,  -40,  -58,  -69,  -38,  -57,  -58,  -47,  -44,  -61,  -73,  -56,  -61,  -44,  -49,  -39,  -46,  -70,  -42],
        [ -60,  -69,  -54,  -55,  -40,  -45,  -44,  -36,  -65,  -58,  -64,  -48,  -45,  -66,  -41,  -55,  -65,  -54,  -62,  -30],
        [ -54,  -72,  -75,  -56,  -38,  -63,  -46,  -50,  -60,  -34,  -58,  -47,  -55,  -28,  -60,  -58,  -50,  -55,  -57,  -43],
        [ -65,  -53,  -59,  -37,  -44,  -59,  -76,  -48,  -68,  -38,  -79,  -48,  -62,  -72,  -35,  -47,  -42,  -61,  -62,  -62],
        [ -58,  -47,  -57,  -40,  -62,  -38,  -48,  -46,  -55,  -69,  -60,  -67,  -44,  -45,  -55,  -52,  -64,  -32,  -48,  -62],
        [ -59,  -46,  -54,  -67,  -55,  -55,  -56,  -54,  -66,  -59,  -64,  -55,  -53,  -64,  -53,  -55,  -55,  -56,  -44,  -59],
        [ -48,  -56,  -63,  -43,  -49,  -38,  -46,  -47,  -60,  -42,  -60,  -58,  -57,  -31,  -51,  -59,  -67,  -54,  -44,  -63],
        [ -47,  -27,  -50,  -68,  -29,  -76,  -58,  -55,  -54,  -50,  -51,  -44,  -53,  -64,  -62,  -59,  -46,  -50,  -68,  -24],
        [ -52,  -91,  -71,  -64,  -57,  -66,  -71,  -60,  -65,  -54,  -58,  -53,  -68,  -63,  -52,  -51,  -49,  -34,  -43,  -52],
        [ -42,  -66,  -45,  -43,  -39,  -49,  -46,  -53,  -63,  -54,  -65,  -70,  -49,  -53,  -44,  -46,  -44,  -51,  -43,  -65],
        [ -53,  -59,  -54,  -52,  -64,  -45,  -52,  -57,  -51,  -72,  -52,  -62,  -39,  -39,  -72,  -54,  -47,  -65,  -63,  -58],
        [ -59,  -38,  -34,  -66,  -41,  -57,  -80,  -57,  -56,  -56,  -48,  -53,  -42,  -55,  -47,  -52,  -57,  -79,  -39,  -64],
        [ -53,  -59,  -37,  -55,  -53,  -40,  -46,  -40,  -41,  -56,  -32,  -56,  -50,  -67,  -36,  -54,  -58,  -72,  -29,  -61],
        [ -51,  -56,  -48,  -65,  -73,  -36,  -59,  -47,  -71,  -56,  -80,  -47,  -52,  -57,  -53,  -79,  -41,  -65,  -54,  -53],
        [ -53,  -66,  -57,  -51,  -81,  -54,  -35,  -52,  -47,  -43,  -43,  -50,  -38,  -51,  -62,  -32,  -71,  -64,  -50,  -53],
        [ -51,  -54,  -52,  -60,  -42,  -52,  -48,  -24,  -50,  -56,  -72,  -52,  -59,  -26,  -79,  -45,  -72,  -44,  -52,  -48],
        [ -72,  -67,  -45,  -45,  -61,  -32,  -52,  -57,  -55,  -59,  -45,  -29,  -44,  -53,  -45,  -58,  -53,  -41,  -57,  -46],
        [ -43,  -56,  -54,  -60,  -41,  -20,  -48,  -48,  -47,  -55,  -61,  -33,  -64,  -44,  -41,  -47,  -73,  -63,  -44,  -46],
        [ -75,  -57,  -58,  -52,  -45,  -50,  -65,  -63,  -67,  -48,  -32,  -62,  -58,  -54,  -49,  -30,  -50,  -54,  -48,  -52],
        [ -42,  -67,  -60,  -45,  -46,  -36,  -42,  -46,  -59,  -56,  -60,  -50,  -44,  -73,  -41,  -44,  -45,  -37,  -39,  -39],
        [ -50,  -57,  -43,  -50,  -75,  -23,  -62,  -52,  -53,  -57,  -66,  -29,  -61,  -59,  -49,  -58,  -58,  -93,  -62,  -59],
        [ -53,  -53,  -51,  -56,  -51,  -38,  -66,  -46,  -78,  -43,  -52,  -67,  -70,  -50,  -40,  -65,  -53,  -49,  -50,  -42],
        [ -56,  -52,  -44,  -77,  -58,  -40,  -38,  -52,  -51,  -44,  -75,  -45,  -44,  -58,  -62,  -59,  -61,  -66,  -56,  -51],
        [ -63,  -55,  -55,  -46,  -54,  -48,  -47,  -25,  -69,  -68,  -81,  -55,  -39,  -52,  -79,  -54,  -90,  -33,  -51,  -44],
        [ -54,  -66,  -50,  -60,  -51,  -49,  -64,  -46,  -61,  -47,  -40,  -61,  -43,  -39,  -64,  -77,  -39,  -54,  -51,  -68],
        [ -52,  -52,  -58,  -67,  -77,  -73,  -72,  -48,  -61,  -88,  -47,  -42,  -35,  -41,  -54,  -66,  -46,  -20,  -46,  -44],
        [ -54,  -34,  -57,  -65,  -60,  -47,  -75,  -55,  -76,  -46,  -44,  -35,  -53,  -63,  -57,  -29,  -50,  -46,  -41,  -60],
        [ -51,  -57,  -55,  -55,  -54,  -56,  -37,  -61,  -40,  -40,  -62,  -49,  -40,  -56,  -40,  -57,  -51,  -53,  -58,  -60],
        [ -35,  -34,  -58,  -38,  -44,  -51,  -51,  -61,  -46,  -51,  -44,  -57,  -64,  -65,  -41,  -67,  -72,  -39,  -62,  -43],
        [ -54,  -46,  -59,  -42,  -62,  -59,  -53,  -39,  -46,  -75,  -55,  -52,  -35,  -53,  -63,  -39,  -54,  -41,  -73,  -59],
        [ -47,  -50,  -35,  -62,  -55,  -56,  -36,  -66,  -56,  -52,  -41,  -79,  -70,  -47,  -67,  -63,  -25,  -47,  -43,  -15],
        [ -43,  -44,  -83,  -46,  -43,  -26,  -43,  -53,  -43,  -64,  -75,  -46,  -39,  -43,  -50,  -53,  -48,  -70,  -54,  -46],
        [ -53,  -61,  -46,  -51,  -62,  -55,  -62,  -54,  -41,  -55,  -66,  -35,  -71,  -53,  -38,  -62,  -49,  -79,  -35,  -44],
        [ -35,  -41,  -44,  -48,  -45,  -58,  -59,  -49,  -53,  -59,  -53,  -42,  -64,  -57,  -60,  -65,  -62,  -79,  -52,  -66],
        [ -45,  -57,  -58,  -73,  -65,  -61,  -55,  -56,  -78,  -53,  -75,  -63,  -62,  -47,  -54,  -48,  -54,  -52,  -31,  -61],
        [ -70,  -80,  -45,  -60,  -72,  -53,  -74,  -45,  -59,  -45,  -52,  -48,  -37,  -38,  -58,  -57,  -67,  -61,  -59,  -45],
        [ -32,  -51,  -65,  -47,  -58,  -57,  -36,  -44,  -58,  -46,  -50,  -44,  -62,  -39,  -45,  -57,  -51,  -61,  -44,  -56],
        [ -58,  -23,  -45,  -46,  -67,  -59,  -39,  -59,  -53,  -63,  -51,  -38,  -54,  -82,  -65,  -58,  -37,  -45,  -63,  -50],
        [ -51,  -57,  -61,  -45,  -55,  -55,  -37,  -51,  -75,  -55,  -53,  -50,  -18,  -55,  -41,  -18,  -63,  -66,  -43,  -36],
        [ -57,  -64,  -44,  -61,  -61,  -56,  -52,  -63,  -50,  -51,  -39,  -71,  -60,  -57,  -55,  -46,  -52,  -32,  -54,  -45],
        [ -44,  -27,  -47,  -32,  -51,  -43,  -52,  -43,  -57,  -53,  -51,  -62,  -43,  -60,  -55,  -71,  -66,  -56,  -52,  -62],
        [ -53,  -53,  -38,  -79,  -43,  -66,  -55,  -66,  -44,  -52,  -29,  -62,  -76,  -46,  -56,  -55,  -56,  -44,  -64,  -47],
        [ -44,  -67,  -66,  -49,  -38,  -60,  -28,  -75,  -49,  -49,  -59,  -60,  -65,  -52,  -40,  -48,  -47,  -70,  -59,  -37],
        [ -67,  -43,  -46,  -66,  -40,  -56,  -59,  -68,  -66,  -48,  -64,  -26,  -50,  -68,  -52,  -57,  -58,  -60,  -41,  -65],
        [ -48,  -55,  -62,  -29,  -56,  -55,  -67,  -50,  -51,  -53,  -34,  -69,  -54,  -52,  -47,  -55,  -61,  -38,  -51,  -28],
        [ -47,  -52,  -54,  -43,  -55,  -48,  -46,  -65,  -76,  -51,  -41,  -52,  -46,  -14,  -44,  -65,  -72,  -55,  -64,  -41],
        [ -73,  -72,  -46,  -53,  -62,  -62,  -50,  -39,  -56,  -40,  -30,  -58,  -70,  -51,  -53,  -52,  -74,  -48,  -53,  -24],
        [ -62,  -64,  -46,  -27,  -60,  -61,  -53,  -69,  -52,  -61,  -48,  -56,  -15,  -60,  -62,  -47,  -49,  -85,  -49,  -43],
        [ -53,  -67,  -58,  -49,  -59,  -36,  -45,  -64,  -41,  -29,  -50,  -53,  -49,  -60,  -64,  -39,  -69,  -42,  -61,  -35],
        [ -58,  -66,  -58,  -63,  -62,  -67,  -77,  -39,  -40,  -64,  -59,  -52,  -69,  -46,  -52,  -49,  -40,  -64,  -59,  -53],
        [ -53,  -51,  -65,  -87,  -46,  -55,  -36,  -42,  -78,  -41,  -45,  -44,  -62,  -51,  -56,  -64,  -37,  -37,  -65,  -59],
        [ -52,  -66,  -80,  -37,  -69,  -54,  -61,  -69,  -44,  -38,  -63,  -45,  -36,  -66,  -52,  -52,  -71,  -57,  -55,  -53],
        [ -58,  -48,  -74,  -64,  -56,  -27,  -60,  -34,  -77,  -63,  -33,  -55,  -66,  -56,  -50,  -65,  -69,  -49,  -67,  -67],
        [ -77,  -54,  -47,  -52,  -40,  -51,  -42,  -68,  -69,  -35,  -63,  -63,  -70,  -56,  -47,  -46,  -59,  -44,  -63,  -70],
        [ -50,  -57,  -57,  -56,  -47,  -78,  -42,  -57,  -37,  -47,  -72,  -27,  -48,  -58,  -40,  -22,  -58,  -49,  -39,  -48],
        [ -54,  -51,  -38,  -78,  -73,  -56,  -45,  -69,  -81,  -59,  -58,  -60,  -55,  -58,  -44,  -47,  -61,  -35,  -56,  -46],
        [ -57,  -53,  -66,  -56,  -39,  -59,  -49,  -54,  -41,  -58,  -57,  -29,  -54,  -74,  -48,  -44,  -46,  -72,  -61,  -52],
        [ -39,  -55,  -35,  -64,  -56,  -52,  -47,  -49,  -77,  -51,  -60,  -64,  -42,  -48,  -51,  -64,  -62,  -51,  -73,  -48],
        [ -55,  -51,  -60,  -28,  -67,  -61,  -50,  -57,  -49,  -61,  -55,  -74,  -50,  -38,  -56,  -55,  -42,  -63,  -48,  -59],
        [ -53,  -53,  -61,  -54,  -47,  -54,  -66,  -43,  -62,  -64,  -48,  -71,  -70,  -56,  -41,  -64,  -45,  -49,  -51,  -52],
        [ -73,  -52,  -37,  -53,  -59,  -54,  -52,  -40,  -53,  -56,  -54,  -55,  -44,  -37,  -57,  -49,  -39,  -63,  -62,  -41],
        [ -49,  -40,  -49,  -65,  -54,  -50,  -54,  -60,  -55,  -58,  -53,  -63,  -47,  -35,  -70,  -52,  -53,  -36,  -48,  -71],
        [ -65,  -34,  -47,  -66,  -56,  -43,  -85,  -44,  -19,  -58,  -62,  -28,  -32,  -76,  -24,  -71,  -61,  -45,  -52,  -59],
        [ -50,  -54,  -51,  -48,  -56,  -75,  -55,  -43,  -37,  -45,  -70,  -69,  -54,  -37,  -51,  -62,  -50,  -43,  -46,  -33],
        [ -54,  -37,  -54,  -33,  -37,  -63,  -62,  -42,  -24,  -43,  -51,  -31,  -50,  -48,  -41,  -42,  -46,  -65,  -67,  -37],
        [ -78,  -73,  -43,  -55,  -57,  -52,  -51,  -68,  -37,  -65,  -58,  -51,  -31,  -60,  -21,  -32,  -55,  -55,  -65,  -69],
        [ -52,  -66,  -51,  -47,  -57,  -46,  -54,  -60,  -55,  -52,  -48,  -69,  -74,  -55,  -46,  -62,  -61,  -33,  -53,  -56],
        [ -49,  -61,  -46,  -51,  -52,  -67,  -38,  -67,  -55,  -47,  -50,  -73,  -25,  -71,  -44,  -73,  -54,  -67,  -82,  -68],
        [ -39,  -51,  -55,  -56,  -41,  -35,  -37,  -35,  -53,  -46,  -62,  -46,  -39,  -53,  -64,  -49,  -45,  -58,  -68,  -67],
        [ -59,  -51,  -63,  -76,  -67,  -84,  -40,  -46,  -51,  -57,  -40,  -53,  -49,  -51,  -47,  -50,  -49,  -37,  -59,  -48],
        [ -65,  -47,  -71,  -54,  -30,  -68,  -46,  -66,  -47,  -59,  -59,  -48,  -48,  -67,  -51,  -56,  -59,  -60,  -42,  -57],
        [ -55,  -71,  -43,  -59,  -42,  -48,  -40,  -40,  -47,  -51,  -54,  -64,  -61,  -80,  -52,  -48,  -40,  -68,  -42,  -54],
        [ -47,  -34,  -50,  -59,  -66,  -73,  -77,  -71,  -33,  -59,  -53,  -55,  -58,  -30,  -53,  -47,  -52,  -42,  -56,  -59],
        [ -84,  -50,  -59,  -53,  -75,  -48,  -71,  -44,  -56,  -49,  -50,  -59,  -45,  -20,  -66,  -37,  -52,  -65,  -58,  -39],
        [ -37,  -45,  -59,  -56,  -56,  -65,  -55,  -32,  -67,  -62,  -62,  -68,  -45,  -78,  -53,  -42,  -64,  -54,  -57,  -14],
        [ -39,  -70,  -71,  -59,  -39,  -74,  -53,  -61,  -51,  -62,  -63,  -44,  -54,  -55,  -45,  -52,  -61,  -65,  -62,  -56],
        [ -52,  -59,  -70,  -53,  -56,  -63,  -61,  -54,  -55,  -64,  -51,  -47,  -36,  -66,  -67,  -58,  -76,  -55,  -64,  -44],
        [ -55,  -53,  -58,  -60,  -43,  -43,  -58,  -42,  -70,  -49,  -49,  -55,  -60,  -57,  -47,  -44,  -37,  -79,  -39,  -53],
        [ -48,  -35,  -65,  -44,  -48,  -55,  -66,  -71,  -61,  -44,  -49,  -53,  -58,  -56,  -78,  -61,  -51,  -58,  -22,  -56],
        [ -42,  -66,  -47,  -66,  -70,  -64,  -49,  -56,  -51,  -36,  -41,  -64,  -73,  -52,  -48,  -61,  -26,  -54,  -59,  -57],
        [ -56,  -61,  -43,  -56,  -43,  -44,  -68,  -46,  -43,  -42,  -39,  -31,  -61,  -36,  -73,  -28,  -45,  -59,  -37,  -42],
        [ -55,  -56,  -58,  -51,  -40,  -65,  -69,  -63,  -53,  -50,  -65,  -76,  -48,  -54,  -68,  -55,  -48,  -53,  -39,  -59],
        [ -41,  -64,  -53,  -73,  -65,  -54,  -55,  -42,  -46,  -58,  -71,  -61,  -27,  -50,  -55,  -53,  -72,  -43,  -45,  -57],
        [ -58,  -48,  -39,  -61,  -62,  -57,  -38,  -52,  -64,  -69,  -44,  -64,  -86,  -34,  -64,  -54,  -55,  -59,  -51,  -33],
        [ -55,  -37,  -55,  -46,  -39,  -37,  -42,  -59,  -53,  -39,  -74,  -31,  -70,  -39,  -50,  -55,  -52,  -50,  -61,  -46],
        [ -38,  -53,  -53,  -48,  -52,  -41,  -55,  -56,  -50,  -60,  -55,  -67,  -57,  -36,  -49,  -60,  -48,  -60,  -58,  -62],
        [ -50,  -57,  -77,  -59,  -60,  -39,  -71,  -62,  -56,  -47,  -52,  -65,  -40,  -56,  -63,  -63,  -51,  -33,  -51,  -46],
        [ -42,  -62,  -62,  -56,  -42,  -74,  -73,  -42,  -54,  -55,  -52,  -50,  -56,  -50,  -55,  -69,  -52,  -32,  -46,  -65],
        [ -70,  -56,  -35,  -53,  -76,  -49,  -63,  -60,  -70,  -53,  -42,  -75,  -43,  -47,  -51,  -41,  -32,  -50,  -49,  -52],
        [ -68,  -63,  -57,  -58,  -62,  -60,  -56,  -71,  -40,  -50,  -35,  -50,  -46,  -48,  -58,  -63,  -57,  -60,  -60,  -47],
        [ -47,  -56,  -40,  -42,  -61,  -56,  -64,  -61,  -63,  -46,  -82,  -43,  -54,  -73,  -63,  -50,  -67,  -51,  -50,  -45],
        [ -41,  -72,  -46,  -74,  -64,  -47,  -34,  -69,  -39,  -38,  -51,  -50,  -53,  -53,  -62,  -74,  -39,  -37,  -45,  -49],
        [ -74,  -52,  -69,  -53,  -57,  -67,  -39,  -55,  -66,  -59,  -54,  -38,  -53,  -48,  -69,  -45,  -62,  -69,  -72,  -54],
        [ -61,  -62,  -45,  -56,  -64,  -55,  -48,  -43,  -55,  -55,  -38,  -82,  -54,  -47,  -55,  -56,  -59,  -56,  -44,  -52],
        [ -50,  -35,  -44,  -51,  -38,  -53,  -65,  -74,  -62,  -45,  -56,  -53,  -42,  -47,  -46,  -49,  -50,  -43,  -46,  -41],
        [ -38,  -65,  -65,  -63,  -43,  -43,  -44,  -57,  -63,  -50,  -61,  -57,  -53,  -51,  -46,  -47,  -45,  -52,  -58,  -42],
        [ -65,  -58,  -56,  -55,  -53,  -44,  -30,  -66,  -48,  -67,  -63,  -45,  -64,  -67,  -40,  -49,  -29,  -52,  -33,  -36],
        [ -45,  -66,  -71,  -56,  -71,  -49,  -66,  -36,  -55,  -48,  -62,  -69,  -55,  -59,  -38,  -58,  -60,  -55,  -67,  -55],
        [ -49,  -56,  -47,  -47,  -38,  -70,  -60,  -45,  -46,  -51,  -41,  -48,  -54,  -41,  -72,  -66,  -74,  -44,  -39,  -72],
        [ -60,  -63,  -40,  -44,  -49,  -70,  -62,  -66,  -50,  -47,  -62,  -60,  -50,  -41,  -62,  -51,  -62,  -72,  -49,  -62],
        [ -77,  -49,  -45,  -47,  -52,  -41,  -58,  -38,  -50,  -55,  -62,  -35,  -67,  -43,  -58,  -45,  -57,  -36,  -41,  -48],
        [ -60,  -54,  -61,  -53,  -37,  -34,  -56,  -47,  -61,  -55,  -49,  -31,  -72,  -41,  -53,  -52,  -66,  -82,  -53,  -44],
        [ -44,  -65,  -58,  -46,  -54,  -30,  -57,  -63,  -71,  -76,  -59,  -58,  -47,  -50,  -43,  -41,  -65,  -53,  -49,  -65],
        [ -36,  -56,  -53,  -46,  -55,  -50,  -59,  -68,  -52,  -57,  -46,  -68,  -29,  -44,  -71,  -43,  -83,  -52,  -43,  -42],
        [ -73,  -57,  -39,  -56,  -49,  -36,  -45,  -57,  -59,  -50,  -55,  -37,  -70,  -34,  -41,  -55,  -49,  -67,  -44,  -52],
        [ -50,  -46,  -32,  -47,  -64,  -36,  -61,  -73,  -54,  -52,  -47,  -40,  -60,  -40,  -68,  -57,  -57,  -49,  -54,  -57],
        [ -62,  -62,  -56,  -66,  -60,  -67,  -67,  -71,  -58,  -39,  -29,  -59,  -52,  -62,  -62,  -31,  -29,  -42,  -51,  -43],
        [ -44,  -43,  -49,  -28,  -48,  -73,  -63,  -66,  -58,  -69,  -56,  -71,  -65,  -54,  -44,  -45,  -29,  -46,  -54,  -57],
        [ -51,  -36,  -55,  -50,  -61,  -63,  -53,  -62,  -60,  -49,  -58,  -55,  -63,  -52,  -39,  -56,  -60,  -80,  -56,  -29],
        [ -49,  -49,  -63,  -63,  -44,  -45,  -43,  -65,  -76,  -51,  -51,  -56,  -61,  -45,  -43,  -65,  -41,  -64,  -61,  -55],
        [ -55,  -42,  -58,  -26,  -58,  -60,  -54,  -74,  -56,  -50,  -65,  -64,  -48,  -55,  -60,  -59,  -61,  -55,  -62,  -49],
        [ -48,  -39,  -65,  -74,  -55,  -46,  -43,  -43,  -58,  -50,  -43,  -54,  -37,  -68,  -54,  -63,  -63,  -60,  -37,  -57],
        [ -59,  -43,  -51,  -61,  -60,  -39,  -37,  -51,  -45,  -64,  -80,  -49,  -49,  -38,  -58,  -71,  -61,  -56,  -46,  -48],
        [ -50,  -66,  -50,  -51,  -46,  -73,  -51,  -63,  -50,  -40,  -57,  -46,  -72,  -41,  -60,  -67,  -56,  -42,  -45,  -71],
        [ -38,  -44,  -34,  -49,  -66,  -38,  -54,  -46,  -51,  -60,  -53,  -56,  -54,  -43,  -60,  -55,  -69,  -52,  -50,  -48],
        [ -77,  -56,  -33,  -49,  -73,  -33,  -51,  -60,  -40,  -55,  -51,  -30,  -46,  -55,  -63,  -50,  -77,  -50,  -45,  -60],
        [ -69,  -55,  -61,  -61,  -57,  -79,  -60,  -56,  -51,  -53,  -68,  -43,  -47,  -50,  -48,  -46,  -53,  -48,  -45,  -68],
        [ -59,  -54,  -67,  -56,  -26,  -64,  -58,  -35,  -63,  -61,  -52,  -44,  -68,  -70,  -50,  -62,  -49,  -41,  -63,  -52],
        [ -63,  -60,  -48,  -56,  -48,  -36,  -54,  -27,  -46,  -65,  -49,  -59,  -37,  -47,  -67,  -64,  -47,  -53,  -53,  -50],
        [ -61,  -61,  -52,  -83,  -41,  -44,  -46,  -62,  -48,  -54,  -73,  -46,  -50,  -35,  -33,  -46,  -54,  -47,  -37,  -51],
        [ -64,  -54,  -50,  -65,  -69,  -59,  -78,  -44,  -56,  -23,  -61,  -56,  -55,  -67,  -27,  -63,  -61,  -37,  -52,  -51],
        [ -21,  -31,  -62,  -15,  -75,  -62,  -44,  -36,  -73,  -39,  -65,  -60,  -55,  -52,  -42,  -56,  -41,  -41,  -63,  -50],
        [ -70,  -53,  -49,  -80,  -65,  -63,  -31,  -50,  -48,  -66,  -57,  -29,  -52,  -45,  -64,  -44,  -48,  -44,  -89,  -49],
        [ -57,  -66,  -54,  -57,  -62,  -34,  -74,  -38,  -44,  -42,  -60,  -73,  -71,  -43,  -47,  -38,  -58,  -59,  -55,  -23],
        [ -35,  -75,  -53,  -60,  -50,  -54,  -52,  -71,  -37,  -64,  -54,  -59,  -40,  -25,  -61,  -58,  -60,  -54,  -49,  -41],
        [ -53,  -56,  -61,  -52,  -75,  -43,  -43,  -60,  -49,  -39,  -57,  -51,  -43,  -56,  -43,  -35,  -55,  -38,  -74,  -58],
        [ -27,  -57,  -30,  -63,  -47,  -48,  -60,  -64,  -38,  -51,  -54,  -37,  -65,  -47,  -52,  -49,  -48,  -57,  -41,  -63],
        [ -54,  -60,  -64,  -47,  -52,  -42,  -40,  -54,  -67,  -79,  -55,  -62,  -60,  -58,  -45,  -40,  -27,  -48,  -54,  -51],
        [ -60,  -64,  -44,  -54,  -78,  -45,  -56,  -62,  -58,  -57,  -52,  -66,  -34,  -81,  -34,  -41,  -40,  -41,  -39,  -55],
        [ -33,  -54,  -67,  -60,  -46,  -43,  -51,  -45,  -44,  -53,  -43,  -62,  -55,  -47,  -49,  -64,  -57,  -54,  -78,  -41],
        [ -31,  -53,  -56,  -63,  -47,  -45,  -51,  -57,  -61,  -60,  -37,  -62,  -66,  -47,  -63,  -75,  -51,  -44,  -61,  -44],
        [ -66,  -55,  -48,  -67,  -58,  -45,  -50,  -65,  -52,  -48,  -29,  -61,  -51,  -64,  -47,  -34,  -38,  -31,  -63,  -64],
        [ -40,  -77,  -55,  -51,  -46,  -68,  -59,  -56,  -31,  -61,  -58,  -41,  -56,  -68,  -49,  -68,  -71,  -49,  -61,  -44],
        [ -49,  -60,  -41,  -53,  -67,  -52,  -49,  -56,  -47,  -34,  -68,  -62,  -34,  -58,  -85,  -52,  -55,  -29,  -47,  -45],
        [ -72,  -70,  -41,  -50,  -65,  -59,  -41,  -55,  -50,  -54,  -49,  -65,  -62,  -51,  -53,  -47,  -37,  -64,  -49,  -49],
        [ -49,  -39,  -58,  -76,  -41,  -67,  -51,  -34,  -65,  -57,  -49,  -84,  -38,  -66,  -47,  -54,  -56,  -48,  -56,  -64],
        [ -50,  -57,  -51,  -58,  -68,  -44,  -48,  -49,  -72,  -38,  -53,  -49,  -66,  -56,  -40,  -32,  -35,  -56,  -55,  -41],
        [ -36,  -24,  -31,  -69,  -48,  -53,  -59,  -43,  -55,  -65,  -37,  -46,  -76,  -62,  -62,  -67,  -76,  -75,  -40,  -55],
        [ -62,  -57,  -65,  -36,  -27,  -53,  -49,  -39,  -68,  -43,  -77,  -61,  -71,  -34,  -49,  -71,  -44,  -88,  -58,  -54],
        [ -57,  -80,  -71,  -44,  -45,  -45,  -74,  -73,  -58,  -55,  -35,  -39,  -48,  -60,  -63,  -54,  -42,  -42,  -33,  -32],
        [ -74,  -38,  -73,  -79,  -60,  -33,  -47,  -70,  -44,  -51,  -40,  -43,  -60,  -67,  -50,  -59,  -79,  -34,  -61,  -42],
        [ -57,  -46,  -60,  -41,  -40,  -72,  -67,  -65,  -34,  -72,  -68,  -49,  -49,  -45,  -60,  -35,  -47,  -57,  -37,  -50],
        [ -42,  -77,  -50,  -62,  -66,  -43,  -51,  -46,  -49,  -28,  -60,  -44,  -63,  -53,  -60,  -68,  -53,  -33,  -43,  -57],
        [ -46,  -37,  -53,  -42,  -46,  -52,  -64,  -39,  -38,  -57,  -48,  -50,  -46,  -41,  -59,  -49,  -42,  -40,  -55,  -53],
        [ -52,  -52,  -48,  -60,  -53,  -50,  -43,  -35,  -42,  -40,  -51,  -65,  -53,  -58,  -53,  -81,  -52,  -52,  -56,  -75],
        [ -68,  -66,  -59,  -34,  -36,  -41,  -41,  -45,  -58,  -65,  -54,  -45,  -46,  -34,  -48,  -47,  -50,  -36,  -45,  -59],
        [ -60,  -56,  -45,  -35,  -46,  -54,  -61,  -37,  -59,  -59,  -44,  -53,  -50,  -52,  -58,  -87,  -69,  -42,  -60,  -68],
        [ -50,  -20,  -38,  -48,  -70,  -52,  -48,  -38,  -47,  -48,  -46,  -45,  -30,  -69,  -48,  -62,  -38,  -68,  -57,  -49],
        [ -59,  -23,  -46,  -45,  -49,  -67,  -65,  -50,  -49,  -46,  -51,  -60,  -46,  -60,  -41,  -58,  -56,  -48,  -63,  -42],
        [ -65,  -37,  -43,  -43,  -66,  -41,  -42,  -48,  -55,  -33,  -57,  -80,  -56,  -79,  -71,  -73,  -44,  -37,  -53,  -57],
        [ -62,  -51,  -74,  -56,  -65,  -70,  -74,  -66,  -60,  -74,  -63,  -49,  -57,  -54,  -64,  -27,  -41,  -29,  -51,  -69],
        [ -57,  -59,  -61,  -60,  -40,  -62,  -48,  -75,  -45,  -61,  -38,  -66,  -47,  -65,  -62,  -47,  -62,  -63,  -34,  -59],
        [ -74,  -77,  -46,  -38,  -87,  -35,  -63,  -59,  -45,  -60,  -71,  -53,  -53,  -58,  -28,  -59,  -63,  -40,  -42,  -69],
        [ -52,  -41,  -49,  -40,  -64,  -35,  -58,  -51,  -48,  -45,  -56,  -45,  -52,  -47,  -55,  -44,  -46,  -43,  -28,  -47],
        [ -56,  -62,  -70,  -63,  -47,  -50,  -45,  -59,  -37,  -46,  -36,  -43,  -62,  -69,  -66,  -47,  -51,  -71,  -27,  -55],
        [ -48,  -42,  -54,  -44,  -71,  -48,  -45,  -49,  -77,  -64,  -47,  -62,  -55,  -27,  -23,  -67,  -22,  -60,  -70,  -45],
        [ -46,  -35,  -58,  -58,  -39,  -48,  -42,  -70,  -40,  -45,  -48,  -39,  -74,  -45,  -50,  -74,  -54,  -49,  -60,  -40],
        [ -56,  -49,  -60,  -51,  -57,  -38,  -52,  -38,  -74,  -67,  -49,  -49,  -53,  -59,  -43,  -57,  -52,  -40,  -49,  -50],
        [ -75,  -47,  -70,  -49,  -66,  -65,  -41,  -60,  -52,  -43,  -55,  -61,  -50,  -63,  -64,  -78,  -30,  -49,  -45,  -67],
        [ -44,  -38,  -60,  -49,  -55,  -34,  -37,  -48,  -42,  -65,  -48,  -39,  -59,  -58,  -70,  -70,  -61,  -37,  -65,  -50],
        [ -65,  -40,  -56,  -31,  -48,  -42,  -45,  -63,  -59,  -51,  -51,  -48,  -38,  -69,  -44,  -47,  -50,  -56,  -33,  -50],
        [ -61,  -44,  -63,  -52,  -67,  -26,  -82,  -53,  -37,  -45,  -58,  -42,  -45,  -70,  -55,  -43,  -64,  -50,  -62,  -50],
        [ -35,  -36,  -53,  -51,  -52,  -68,  -50,  -76,  -53,  -63,  -65,  -68,  -43,  -61,  -42,  -42,  -50,  -66,  -70,  -51],
        [ -48,  -59,  -46,  -33,  -55,  -70,  -43,  -58,  -52,  -32,  -55,  -69,  -48,  -69,  -59,  -28,  -52,  -50,  -60,  -58],
        [ -37,  -63,  -48,  -72,  -43,  -62,  -48,  -70,  -64,  -43,  -57,  -63,  -46,  -39,  -53,  -68,  -44,  -64,  -54,  -31],
        [ -55,  -27,  -43,  -49,  -57,  -65,  -59,  -60,  -69,  -57,  -43,  -52,  -64,  -68,  -37,  -44,  -45,  -41,  -59,  -43],
        [ -47,  -56,  -52,  -32,  -32,  -45,  -43,  -72,  -42,  -48,  -64,  -43,  -51,  -79,  -44,  -60,  -57,  -70,  -52,  -66],
        [ -64,  -48,  -53,  -42,  -47,  -52,  -57,  -52,  -60,  -74,  -58,  -66,  -38,  -43,  -62,  -38,  -51,  -41,  -68,  -38],
        [ -48,  -28,  -59,  -58,  -77,  -56,  -47,  -47,  -30,  -53,  -53,  -42,  -37,  -73,  -40,  -38,  -59,  -64,  -48,  -57],
        [ -50,  -42,  -41,  -81,  -68,  -71,  -56,  -63,  -49,  -37,  -51,  -64,  -69,  -53,  -74,  -63,  -49,  -68,  -62, -101]
output = torch.distributed._functional_collectives.reduce_scatter_tensor(input_tensor, "max", 0)
gpu0[122,10] is -51, which is different from input_tensor[122,10](-50).
RAW_BUFFERClick to expand / collapse

🐛 Describe the bug

gpu0 and gpu1 have same input tensor, which is
[ -58,  -62,  -54,  -36,  -49,  -55,  -35,  -36,  -60,  -45,  -34,  -56,  -54,  -55,  -50,  -78,  -46,  -70,  -60,  -74],
        [ -69,  -38,  -49,  -58,  -62,  -51,  -48,  -51,  -60,  -69,  -78,  -59,  -28,  -63,  -54,  -55,  -49,  -50,  -63,  -43],
        [ -52,  -31,  -40,  -39,  -58,  -45,  -31,  -43,  -42,  -60,  -53,  -66,  -42,  -48,  -48,  -52,  -39,  -42,  -75,  -40],
        [ -63,  -61,  -76,  -57,  -53,  -58,  -60,  -56,  -73,  -72,  -36,  -46,  -66,  -52,  -45,  -64,  -70,  -59,  -57,  -42],
        [ -61,  -42,  -51,  -49,  -44,  -66,  -37,  -52,  -43,  -43,  -50,  -53,  -46,  -81,  -33,  -55,  -57,  -20,  -33,  -49],
        [ -71,  -63,  -61,  -60,  -29,  -60,  -58,  -58,  -46,  -32,  -25,  -51,  -40,  -69,  -46,  -60,  -19,  -55,  -46,  -61],
        [ -61,  -52,  -59,  -54,  -39,  -44,  -39,  -59,  -52,  -62,  -56,  -61,  -52,  -32,  -63,  -58,  -60,  -66,  -52,  -36],
        [ -68,  -41,  -61,  -56,  -67,  -50,  -76,  -64,  -74,  -38,  -39,  -64,  -61,  -42,  -64,  -50,  -44,  -65,  -66,  -58],
        [ -44,  -44,  -51,  -55,  -67,  -28,  -61,  -60,  -45,  -68,  -32,  -24,  -56,  -44,  -50,  -38,  -72,  -53,  -59,  -51],
        [ -58,  -49,  -52,  -55,  -71,  -27,  -57,  -82,  -51,  -75,  -54,  -50,  -41,  -43,  -48,  -42,  -57,  -50,  -59,  -47],
        [ -48,  -50,  -54,  -57,  -46,  -46,  -38,  -65,  -57,  -47,  -50,  -58,  -54,  -44,  -43,  -43,  -50,  -46,  -48,  -59],
        [ -43,  -69,  -46,  -65,  -55,  -65,  -62,  -55,  -57,  -58,  -42,  -59,  -49,  -48,  -51,  -55,  -56,  -77,  -44,  -50],
        [ -51,  -57,  -50,  -59,  -39,  -51,  -59,  -76,  -63,  -63,  -67,  -56,  -52,  -72,  -36,  -80,  -45,  -39,  -55,  -49],
        [ -53,  -48,  -52,  -61,  -71,  -44,  -56,  -48,  -49,  -74,  -63,  -74,  -84,  -54,  -53,  -50,  -29,  -18,  -42,  -49],
        [ -56,  -46,  -63,  -51,  -56,  -69,  -67,  -57,  -38,  -62,  -36,  -53,  -61,  -51,  -52,  -58,  -69,  -62,  -51,  -60],
        [ -42,  -60,  -59,  -63,  -68,  -32,  -59,  -59,  -72,  -37,  -55,  -61,  -61,  -66,  -47,  -58,  -59,  -46,  -45,  -36],
        [ -63,  -60,  -58,  -32,  -47,  -64,  -45,  -66,  -51,  -48,  -61,  -52,  -56,  -55,  -46,  -56,  -45,  -44,  -60,  -65],
        [ -64,  -58,  -50,  -55,  -57,  -67,  -44,  -34,  -34,  -59,  -55,  -58,  -62,  -42,  -38,  -51,  -62,  -78,  -61,  -56],
        [ -50,  -44,  -67,  -46,  -53,  -52,  -53,  -61,  -58,  -45,  -37,  -51,  -26,  -75,  -56,  -63,  -59,  -49,  -37,  -62],
        [ -55,  -50,  -38,  -65,  -48,  -60,  -20,  -67,  -58,  -72,  -63,  -49,  -50,  -64,  -52,  -45,  -61,  -19,  -65,  -53],
        [ -29,  -28,  -62,  -28,  -59,  -40,  -44,  -59,  -57,  -79,  -79,  -24,  -46,  -64,  -35,  -71,  -56,  -67,  -48,  -52],
        [ -42,  -72,  -57,  -49,  -48,  -46,  -47,  -40,  -60,  -27,  -56,  -45,  -58,  -34,  -62,  -59,  -88,  -37,  -51,  -72],
        [ -57,  -59,  -53,  -51,  -62,  -48,  -47,  -62,  -52,  -46,  -23,  -55,  -56,  -51,  -49,  -71,  -49,  -40,  -66,  -50],
        [ -56,  -48,  -50,  -52,  -54,  -51,  -44,  -49,  -60,  -61,  -63,  -67,  -48,  -56,  -63,  -44,  -64,  -45,  -25,  -67],
        [ -19,  -46,  -44,  -59,  -70,  -53,  -63,  -50,  -26,  -72,  -59,  -61,  -59,  -38,  -43,  -56,  -55,  -33,  -44,  -86],
        [ -57,  -56,  -59,  -41,  -70,  -60,  -38,  -25,  -65,  -53,  -30,  -52,  -69,  -47,  -71,  -57,  -62,  -57,  -59,  -42],
        [ -42,  -35,  -39,  -64,  -55,  -61,  -65,  -54,  -56,  -44,  -49,  -62,  -50,  -39,  -52,  -70,  -49,  -60,  -38,  -68],
        [ -66,  -51,  -53,  -55,  -34,  -56,  -63,  -46,  -50,  -48,  -82,  -62,  -69,  -48,  -70,  -69,  -35,  -52,  -67,  -57],
        [ -61,  -60,  -56,  -52,  -59,  -59,  -78,  -56,  -55,  -59,  -52,  -56,  -50,  -64,  -54,  -44,  -43,  -43,  -46,  -50],
        [ -54,  -62,  -53,  -39,  -62,  -61,  -69,  -24,  -54,  -56,  -64,  -46,  -62,  -66,  -43,  -60,  -54,  -71,  -52,  -64],
        [ -38,  -67,  -54,  -60,  -59,  -57,  -51,  -53,  -45,  -68,  -46,  -54,  -54,  -51,  -61,  -56,  -46,  -60,  -52,  -31],
        [ -54,  -49,  -68,  -45,  -51,  -41,  -34,  -47,  -58,  -35,  -53,  -54,  -62,  -53,  -66,  -37,  -61,  -73,  -56,  -42],
        [ -64,  -67,  -74,  -44,  -50,  -55,  -40,  -32,  -50,  -65,  -39,  -53,  -63,  -50,  -47,  -55,  -54,  -51,  -40,  -56],
        [ -29,  -32,  -58,  -80,  -46,  -39,  -48,  -55,  -64,  -50,  -71,  -63,  -68,  -49,  -68,  -71,  -50,  -59,  -52,  -55],
        [ -61,  -49,  -67,  -49,  -49,  -66,  -49,  -51,  -58,  -63,  -38,  -64,  -53,  -56,  -64,  -64,  -45,  -43,  -50,  -59],
        [ -79,  -58,  -38,  -57,  -45,  -43,  -50,  -47,  -64,  -70,  -45,  -68,  -75,  -60,  -61,  -47,  -57,  -43,  -51,  -62],
        [ -70,  -54,  -35,  -70,  -72,  -56,  -69,  -64,  -53,  -65,  -42,  -61,  -69,  -39,  -41,  -55,  -51,  -70,  -49,  -50],
        [ -59,  -66,  -76,  -52,  -50,  -61,  -58,  -59,  -56,  -37,  -67,  -47,  -72,  -54,  -44,  -47,  -69,  -41,  -56,  -34],
        [ -42,  -62,  -43,  -48,  -54,  -50,  -49,  -64,  -39,  -49,  -68,  -51,  -47,  -56,  -45,  -52,  -62,  -28,  -67,  -42],
        [ -91,  -56,  -51,  -34,  -62,  -62,  -59,  -50,  -51,  -62,  -58,  -49,  -65,  -49,  -59,  -61,  -39,  -46,  -55,  -49],
        [ -49,  -54,  -56,  -75,  -47,  -24,  -62,  -35,  -50,  -66,  -49,  -45,  -46,  -47,  -43,  -47,  -37,  -48,  -58,  -64],
        [ -50,  -60,  -49,  -50,  -54,  -63,  -57,  -60,  -45,  -60,  -54,  -50,  -83,  -52,  -63,  -53,  -57,  -77,  -51,  -60],
        [ -57,  -40,  -29,  -39,  -42,  -55,  -38,  -58,  -63,  -64,  -62,  -39,  -57,  -29,  -50,  -32,  -49,  -64,  -70,  -49],
        [ -33,  -54,  -70,  -76,  -54,  -59,  -50,  -62,  -41,  -48,  -59,  -61,  -51,  -52,  -66,  -43,  -36,  -77,  -42,  -51],
        [ -57,  -57,  -49,  -57,  -51,  -66,  -50,  -59,  -70,  -60,  -52,  -41,  -44,  -42,  -65,  -59,  -59,  -53,  -69,  -58],
        [ -42,  -54,  -59,  -75,  -31,  -54,  -65,  -72,  -70,  -53,  -37,  -77,  -59,  -63,  -43,  -56,  -69,  -63,  -47,  -61],
        [ -45,  -59,  -47,  -57,  -47,  -42,  -45,  -46,  -60,  -61,  -53,  -47,  -46,  -63,  -59,  -45,  -36,  -45,  -66,  -64],
        [ -43,  -71,  -50,  -27,  -71,  -53,  -41,  -45,  -64,  -67,  -61,  -62,  -54,  -47,  -46,  -52,  -41,  -42,  -59,  -44],
        [ -60,  -68,  -57,  -39,  -61,  -42,  -38,  -43,  -31,  -45,  -57,  -67,  -37,  -41,  -59,  -67,  -46,  -72,  -46,  -59],
        [ -40,  -74,  -48,  -78,  -45,  -60,  -49,  -71,  -46,  -51,  -56,  -34,  -40,  -42,  -72,  -49,  -49,  -51,  -53,  -82],
        [ -52,  -39,  -48,  -62,  -44,  -54,  -61,  -45,  -52,  -37,  -40,  -67,  -72,  -44,  -66,  -63,  -61,  -52,  -57,  -49],
        [ -60,  -56,  -54,  -46,  -75,  -62,  -46,  -75,  -59,  -55,  -41,  -47,  -57,  -52,  -44,  -65,  -73,  -48,  -57,  -47],
        [ -40,  -54,  -39,  -47,  -26,  -43,  -43,  -75,  -32,  -69,  -33,  -48,  -65,  -68,  -50,  -68,  -43,  -60,  -67,  -65],
        [ -60,  -51,  -51,  -44,  -44,  -51,  -69,  -58,  -59,  -61,  -49,  -54,  -69,  -39,  -61,  -57,  -45,  -65,  -49,  -61],
        [ -57,  -50,  -58,  -60,  -42,  -46,  -34,  -40,  -73,  -63,  -52,  -83,  -56,  -45,  -52,  -46,  -64,  -61,  -65,  -53],
        [ -77,  -50,  -49,  -33,  -69,  -56,  -49,  -64,  -29,  -42,  -43,  -54,  -55,  -63,  -62,  -40,  -76,  -69,  -50,  -64],
        [ -47,  -54,  -59,  -54,  -51,  -45,  -33,  -54,  -53,  -44,  -58,  -51,  -35,  -52,  -58,  -51,  -67,  -60,  -46,  -63],
        [ -56,  -32,  -41,  -47,  -61,  -36,  -49,  -55,  -36,  -67,  -63,  -56,  -32,  -45,  -43,  -39,  -70,  -55,  -37,  -48],
        [ -47,  -68,  -45,  -61,  -52,  -46,  -64,  -57,  -64,  -56,  -38,  -70,  -47,  -65,  -39,  -54,  -56,  -60,  -23,  -47],
        [ -44,  -44,  -75,  -41,  -49,  -54,  -55,  -41,  -81,  -57,  -60,  -47,  -54,  -46,  -68,  -60,  -50,  -45,  -50,  -47],
        [ -60,  -61,  -74,  -39,  -42,  -63,  -51,  -59,  -74,  -62,  -56,  -61,  -68,  -60,  -62,  -51,  -50,  -55,  -54,  -76],
        [ -59,  -67,  -40,  -62,  -55,  -47,  -40,  -53,  -62,  -51,  -30,  -52,  -50,  -51,  -62,  -49,  -37,  -71,  -41,  -48],
        [ -56,  -45,  -68,  -64,  -60,  -55,  -68,  -87,  -49,  -63,  -47,  -54,  -37,  -64,  -59,  -44,  -51,  -46,  -32,  -48],
        [ -54,  -42,  -58,  -38,  -48,  -24,  -40,  -50,  -70,  -33,  -56,  -90,  -64,  -73,  -65,  -59,  -41,  -22,  -56,  -61],
        [ -60,  -67,  -28,  -59,  -46,  -48,  -62,  -63,  -81,  -41,  -91,  -53,  -49,  -49,  -60,  -46,  -71,  -45,  -45,  -66],
        [ -59,  -45,  -61,  -67,  -32,  -39,  -51,  -33,  -74,  -60,  -71,  -62,  -39,  -36,  -37,  -64,  -45,  -62,  -84,  -51],
        [ -65,  -54,  -60,  -63,  -45,  -56,  -55,  -61,  -62,  -62,  -54,  -70,  -71,  -33,  -29,  -43,  -70,  -54,  -70,  -54],
        [ -59,  -71,  -68,  -54,  -57,  -62,  -81,  -36,  -45,  -48,  -57,  -43,  -32,  -45,  -42,  -65,  -52,  -71,  -51,  -59],
        [ -26,  -56,  -70,  -53,  -43,  -36,  -66,  -48,  -36,  -62,  -38,  -43,  -64,  -59,  -38,  -43,  -54,  -52,  -42,  -48],
        [ -56,  -59,  -55,  -52,  -49,  -51,  -44,  -80,  -78,  -50,  -42,  -85,  -59,  -66,  -57,  -71,  -48,  -47,  -53,  -56],
        [ -47,  -58,  -58,  -42,  -67,  -51,  -80,  -47,  -29,  -63,  -61,  -54,  -55,  -47,  -52,  -48,  -48,  -51,  -62,  -61],
        [ -62,  -63,  -91,  -57,  -58,  -46,  -61,  -51,  -65,  -70,  -47,  -61,  -60,  -49,  -48,  -50,  -56,  -63,  -57,  -38],
        [ -42,  -26,  -51,  -41,  -62,  -50,  -39,  -44,  -73,  -59,  -38,  -73,  -34,  -36,  -61,  -54,  -54,  -42,  -47,  -52],
        [ -38,  -51,  -60,  -34,  -34,  -45,  -71,  -67,  -60,  -55,  -40,  -57,  -49,  -49,  -67,  -49,  -55,  -45,  -81,  -52],
        [ -67,  -63,  -64,  -56,  -49,  -50,  -69,  -53,  -48,  -59,  -73,  -63,  -54,  -28,  -63,  -61,  -56,  -66,  -72,  -64],
        [ -54,  -67,  -48,  -44,  -55,  -68,  -46,  -34,  -51,  -64,  -63,  -58,  -54,  -45,  -70,  -62,  -62,  -37,  -38,  -83],
        [ -50,  -46,  -60,  -80,  -46,  -52,  -54,  -55,  -51,  -54,  -47,  -59,  -38,  -54,  -54,  -66,  -63,  -38,  -56,  -43],
        [ -60,  -45,  -69,  -59,  -61,  -51,  -45,  -58,  -40,  -62,  -49,  -38,  -47,  -70,  -50,  -45,  -69,  -76,  -46,  -26],
        [ -64,  -60,  -33,  -30,  -64,  -54,  -45,  -57,  -47,  -45,  -54,  -45,  -40,  -62,  -39,  -69,  -49,  -63,  -40,  -62],
        [ -39,  -46,  -60,  -67,  -20,  -36,  -44,  -65,  -49,  -38,  -58,  -52,  -57,  -44,  -27,  -40,  -50,  -37,  -57,  -44],
        [ -72,  -65,  -35,  -58,  -47,  -61,  -63,  -62,  -45,  -64,  -77,  -38,  -70,  -57,  -44,  -34,  -68,  -51,  -52,  -36],
        [ -52,  -49,  -53,  -43,  -65,  -42,  -50,  -49,  -61,  -51,  -66,  -58,  -47,  -63,  -56,  -66,  -47,  -57,  -45,  -53],
        [ -57,  -70,  -42,  -60,  -63,  -58,  -54,  -47,  -49,  -48,  -48,  -62,  -27,  -55,  -47,  -51,  -51,  -53,  -53,  -55],
        [ -69,  -46,  -41,  -45,  -49,  -47,  -64,  -79,  -52,  -51,  -40,  -62,  -59,  -50,  -52,  -78,  -54,  -60,  -61,  -50],
        [ -62,  -53,  -61,  -70,  -37,  -46,  -59,  -45,  -69,  -57,  -52,  -45,  -46,  -51,  -42,  -71,  -43,  -60,  -47,  -68],
        [ -35,  -78,  -50,  -62,  -48,  -71,  -59,  -75,  -69,  -43,  -55,  -33,  -40,  -50,  -40,  -39,  -49,  -61,  -61,  -60],
        [ -45,  -75,  -40,  -58,  -69,  -38,  -57,  -58,  -47,  -44,  -61,  -73,  -56,  -61,  -44,  -49,  -39,  -46,  -70,  -42],
        [ -60,  -69,  -54,  -55,  -40,  -45,  -44,  -36,  -65,  -58,  -64,  -48,  -45,  -66,  -41,  -55,  -65,  -54,  -62,  -30],
        [ -54,  -72,  -75,  -56,  -38,  -63,  -46,  -50,  -60,  -34,  -58,  -47,  -55,  -28,  -60,  -58,  -50,  -55,  -57,  -43],
        [ -65,  -53,  -59,  -37,  -44,  -59,  -76,  -48,  -68,  -38,  -79,  -48,  -62,  -72,  -35,  -47,  -42,  -61,  -62,  -62],
        [ -58,  -47,  -57,  -40,  -62,  -38,  -48,  -46,  -55,  -69,  -60,  -67,  -44,  -45,  -55,  -52,  -64,  -32,  -48,  -62],
        [ -59,  -46,  -54,  -67,  -55,  -55,  -56,  -54,  -66,  -59,  -64,  -55,  -53,  -64,  -53,  -55,  -55,  -56,  -44,  -59],
        [ -48,  -56,  -63,  -43,  -49,  -38,  -46,  -47,  -60,  -42,  -60,  -58,  -57,  -31,  -51,  -59,  -67,  -54,  -44,  -63],
        [ -47,  -27,  -50,  -68,  -29,  -76,  -58,  -55,  -54,  -50,  -51,  -44,  -53,  -64,  -62,  -59,  -46,  -50,  -68,  -24],
        [ -52,  -91,  -71,  -64,  -57,  -66,  -71,  -60,  -65,  -54,  -58,  -53,  -68,  -63,  -52,  -51,  -49,  -34,  -43,  -52],
        [ -42,  -66,  -45,  -43,  -39,  -49,  -46,  -53,  -63,  -54,  -65,  -70,  -49,  -53,  -44,  -46,  -44,  -51,  -43,  -65],
        [ -53,  -59,  -54,  -52,  -64,  -45,  -52,  -57,  -51,  -72,  -52,  -62,  -39,  -39,  -72,  -54,  -47,  -65,  -63,  -58],
        [ -59,  -38,  -34,  -66,  -41,  -57,  -80,  -57,  -56,  -56,  -48,  -53,  -42,  -55,  -47,  -52,  -57,  -79,  -39,  -64],
        [ -53,  -59,  -37,  -55,  -53,  -40,  -46,  -40,  -41,  -56,  -32,  -56,  -50,  -67,  -36,  -54,  -58,  -72,  -29,  -61],
        [ -51,  -56,  -48,  -65,  -73,  -36,  -59,  -47,  -71,  -56,  -80,  -47,  -52,  -57,  -53,  -79,  -41,  -65,  -54,  -53],
        [ -53,  -66,  -57,  -51,  -81,  -54,  -35,  -52,  -47,  -43,  -43,  -50,  -38,  -51,  -62,  -32,  -71,  -64,  -50,  -53],
        [ -51,  -54,  -52,  -60,  -42,  -52,  -48,  -24,  -50,  -56,  -72,  -52,  -59,  -26,  -79,  -45,  -72,  -44,  -52,  -48],
        [ -72,  -67,  -45,  -45,  -61,  -32,  -52,  -57,  -55,  -59,  -45,  -29,  -44,  -53,  -45,  -58,  -53,  -41,  -57,  -46],
        [ -43,  -56,  -54,  -60,  -41,  -20,  -48,  -48,  -47,  -55,  -61,  -33,  -64,  -44,  -41,  -47,  -73,  -63,  -44,  -46],
        [ -75,  -57,  -58,  -52,  -45,  -50,  -65,  -63,  -67,  -48,  -32,  -62,  -58,  -54,  -49,  -30,  -50,  -54,  -48,  -52],
        [ -42,  -67,  -60,  -45,  -46,  -36,  -42,  -46,  -59,  -56,  -60,  -50,  -44,  -73,  -41,  -44,  -45,  -37,  -39,  -39],
        [ -50,  -57,  -43,  -50,  -75,  -23,  -62,  -52,  -53,  -57,  -66,  -29,  -61,  -59,  -49,  -58,  -58,  -93,  -62,  -59],
        [ -53,  -53,  -51,  -56,  -51,  -38,  -66,  -46,  -78,  -43,  -52,  -67,  -70,  -50,  -40,  -65,  -53,  -49,  -50,  -42],
        [ -56,  -52,  -44,  -77,  -58,  -40,  -38,  -52,  -51,  -44,  -75,  -45,  -44,  -58,  -62,  -59,  -61,  -66,  -56,  -51],
        [ -63,  -55,  -55,  -46,  -54,  -48,  -47,  -25,  -69,  -68,  -81,  -55,  -39,  -52,  -79,  -54,  -90,  -33,  -51,  -44],
        [ -54,  -66,  -50,  -60,  -51,  -49,  -64,  -46,  -61,  -47,  -40,  -61,  -43,  -39,  -64,  -77,  -39,  -54,  -51,  -68],
        [ -52,  -52,  -58,  -67,  -77,  -73,  -72,  -48,  -61,  -88,  -47,  -42,  -35,  -41,  -54,  -66,  -46,  -20,  -46,  -44],
        [ -54,  -34,  -57,  -65,  -60,  -47,  -75,  -55,  -76,  -46,  -44,  -35,  -53,  -63,  -57,  -29,  -50,  -46,  -41,  -60],
        [ -51,  -57,  -55,  -55,  -54,  -56,  -37,  -61,  -40,  -40,  -62,  -49,  -40,  -56,  -40,  -57,  -51,  -53,  -58,  -60],
        [ -35,  -34,  -58,  -38,  -44,  -51,  -51,  -61,  -46,  -51,  -44,  -57,  -64,  -65,  -41,  -67,  -72,  -39,  -62,  -43],
        [ -54,  -46,  -59,  -42,  -62,  -59,  -53,  -39,  -46,  -75,  -55,  -52,  -35,  -53,  -63,  -39,  -54,  -41,  -73,  -59],
        [ -47,  -50,  -35,  -62,  -55,  -56,  -36,  -66,  -56,  -52,  -41,  -79,  -70,  -47,  -67,  -63,  -25,  -47,  -43,  -15],
        [ -43,  -44,  -83,  -46,  -43,  -26,  -43,  -53,  -43,  -64,  -75,  -46,  -39,  -43,  -50,  -53,  -48,  -70,  -54,  -46],
        [ -53,  -61,  -46,  -51,  -62,  -55,  -62,  -54,  -41,  -55,  -66,  -35,  -71,  -53,  -38,  -62,  -49,  -79,  -35,  -44],
        [ -35,  -41,  -44,  -48,  -45,  -58,  -59,  -49,  -53,  -59,  -53,  -42,  -64,  -57,  -60,  -65,  -62,  -79,  -52,  -66],
        [ -45,  -57,  -58,  -73,  -65,  -61,  -55,  -56,  -78,  -53,  -75,  -63,  -62,  -47,  -54,  -48,  -54,  -52,  -31,  -61],
        [ -70,  -80,  -45,  -60,  -72,  -53,  -74,  -45,  -59,  -45,  -52,  -48,  -37,  -38,  -58,  -57,  -67,  -61,  -59,  -45],
        [ -32,  -51,  -65,  -47,  -58,  -57,  -36,  -44,  -58,  -46,  -50,  -44,  -62,  -39,  -45,  -57,  -51,  -61,  -44,  -56],
        [ -58,  -23,  -45,  -46,  -67,  -59,  -39,  -59,  -53,  -63,  -51,  -38,  -54,  -82,  -65,  -58,  -37,  -45,  -63,  -50],
        [ -51,  -57,  -61,  -45,  -55,  -55,  -37,  -51,  -75,  -55,  -53,  -50,  -18,  -55,  -41,  -18,  -63,  -66,  -43,  -36],
        [ -57,  -64,  -44,  -61,  -61,  -56,  -52,  -63,  -50,  -51,  -39,  -71,  -60,  -57,  -55,  -46,  -52,  -32,  -54,  -45],
        [ -44,  -27,  -47,  -32,  -51,  -43,  -52,  -43,  -57,  -53,  -51,  -62,  -43,  -60,  -55,  -71,  -66,  -56,  -52,  -62],
        [ -53,  -53,  -38,  -79,  -43,  -66,  -55,  -66,  -44,  -52,  -29,  -62,  -76,  -46,  -56,  -55,  -56,  -44,  -64,  -47],
        [ -44,  -67,  -66,  -49,  -38,  -60,  -28,  -75,  -49,  -49,  -59,  -60,  -65,  -52,  -40,  -48,  -47,  -70,  -59,  -37],
        [ -67,  -43,  -46,  -66,  -40,  -56,  -59,  -68,  -66,  -48,  -64,  -26,  -50,  -68,  -52,  -57,  -58,  -60,  -41,  -65],
        [ -48,  -55,  -62,  -29,  -56,  -55,  -67,  -50,  -51,  -53,  -34,  -69,  -54,  -52,  -47,  -55,  -61,  -38,  -51,  -28],
        [ -47,  -52,  -54,  -43,  -55,  -48,  -46,  -65,  -76,  -51,  -41,  -52,  -46,  -14,  -44,  -65,  -72,  -55,  -64,  -41],
        [ -73,  -72,  -46,  -53,  -62,  -62,  -50,  -39,  -56,  -40,  -30,  -58,  -70,  -51,  -53,  -52,  -74,  -48,  -53,  -24],
        [ -62,  -64,  -46,  -27,  -60,  -61,  -53,  -69,  -52,  -61,  -48,  -56,  -15,  -60,  -62,  -47,  -49,  -85,  -49,  -43],
        [ -53,  -67,  -58,  -49,  -59,  -36,  -45,  -64,  -41,  -29,  -50,  -53,  -49,  -60,  -64,  -39,  -69,  -42,  -61,  -35],
        [ -58,  -66,  -58,  -63,  -62,  -67,  -77,  -39,  -40,  -64,  -59,  -52,  -69,  -46,  -52,  -49,  -40,  -64,  -59,  -53],
        [ -53,  -51,  -65,  -87,  -46,  -55,  -36,  -42,  -78,  -41,  -45,  -44,  -62,  -51,  -56,  -64,  -37,  -37,  -65,  -59],
        [ -52,  -66,  -80,  -37,  -69,  -54,  -61,  -69,  -44,  -38,  -63,  -45,  -36,  -66,  -52,  -52,  -71,  -57,  -55,  -53],
        [ -58,  -48,  -74,  -64,  -56,  -27,  -60,  -34,  -77,  -63,  -33,  -55,  -66,  -56,  -50,  -65,  -69,  -49,  -67,  -67],
        [ -77,  -54,  -47,  -52,  -40,  -51,  -42,  -68,  -69,  -35,  -63,  -63,  -70,  -56,  -47,  -46,  -59,  -44,  -63,  -70],
        [ -50,  -57,  -57,  -56,  -47,  -78,  -42,  -57,  -37,  -47,  -72,  -27,  -48,  -58,  -40,  -22,  -58,  -49,  -39,  -48],
        [ -54,  -51,  -38,  -78,  -73,  -56,  -45,  -69,  -81,  -59,  -58,  -60,  -55,  -58,  -44,  -47,  -61,  -35,  -56,  -46],
        [ -57,  -53,  -66,  -56,  -39,  -59,  -49,  -54,  -41,  -58,  -57,  -29,  -54,  -74,  -48,  -44,  -46,  -72,  -61,  -52],
        [ -39,  -55,  -35,  -64,  -56,  -52,  -47,  -49,  -77,  -51,  -60,  -64,  -42,  -48,  -51,  -64,  -62,  -51,  -73,  -48],
        [ -55,  -51,  -60,  -28,  -67,  -61,  -50,  -57,  -49,  -61,  -55,  -74,  -50,  -38,  -56,  -55,  -42,  -63,  -48,  -59],
        [ -53,  -53,  -61,  -54,  -47,  -54,  -66,  -43,  -62,  -64,  -48,  -71,  -70,  -56,  -41,  -64,  -45,  -49,  -51,  -52],
        [ -73,  -52,  -37,  -53,  -59,  -54,  -52,  -40,  -53,  -56,  -54,  -55,  -44,  -37,  -57,  -49,  -39,  -63,  -62,  -41],
        [ -49,  -40,  -49,  -65,  -54,  -50,  -54,  -60,  -55,  -58,  -53,  -63,  -47,  -35,  -70,  -52,  -53,  -36,  -48,  -71],
        [ -65,  -34,  -47,  -66,  -56,  -43,  -85,  -44,  -19,  -58,  -62,  -28,  -32,  -76,  -24,  -71,  -61,  -45,  -52,  -59],
        [ -50,  -54,  -51,  -48,  -56,  -75,  -55,  -43,  -37,  -45,  -70,  -69,  -54,  -37,  -51,  -62,  -50,  -43,  -46,  -33],
        [ -54,  -37,  -54,  -33,  -37,  -63,  -62,  -42,  -24,  -43,  -51,  -31,  -50,  -48,  -41,  -42,  -46,  -65,  -67,  -37],
        [ -78,  -73,  -43,  -55,  -57,  -52,  -51,  -68,  -37,  -65,  -58,  -51,  -31,  -60,  -21,  -32,  -55,  -55,  -65,  -69],
        [ -52,  -66,  -51,  -47,  -57,  -46,  -54,  -60,  -55,  -52,  -48,  -69,  -74,  -55,  -46,  -62,  -61,  -33,  -53,  -56],
        [ -49,  -61,  -46,  -51,  -52,  -67,  -38,  -67,  -55,  -47,  -50,  -73,  -25,  -71,  -44,  -73,  -54,  -67,  -82,  -68],
        [ -39,  -51,  -55,  -56,  -41,  -35,  -37,  -35,  -53,  -46,  -62,  -46,  -39,  -53,  -64,  -49,  -45,  -58,  -68,  -67],
        [ -59,  -51,  -63,  -76,  -67,  -84,  -40,  -46,  -51,  -57,  -40,  -53,  -49,  -51,  -47,  -50,  -49,  -37,  -59,  -48],
        [ -65,  -47,  -71,  -54,  -30,  -68,  -46,  -66,  -47,  -59,  -59,  -48,  -48,  -67,  -51,  -56,  -59,  -60,  -42,  -57],
        [ -55,  -71,  -43,  -59,  -42,  -48,  -40,  -40,  -47,  -51,  -54,  -64,  -61,  -80,  -52,  -48,  -40,  -68,  -42,  -54],
        [ -47,  -34,  -50,  -59,  -66,  -73,  -77,  -71,  -33,  -59,  -53,  -55,  -58,  -30,  -53,  -47,  -52,  -42,  -56,  -59],
        [ -84,  -50,  -59,  -53,  -75,  -48,  -71,  -44,  -56,  -49,  -50,  -59,  -45,  -20,  -66,  -37,  -52,  -65,  -58,  -39],
        [ -37,  -45,  -59,  -56,  -56,  -65,  -55,  -32,  -67,  -62,  -62,  -68,  -45,  -78,  -53,  -42,  -64,  -54,  -57,  -14],
        [ -39,  -70,  -71,  -59,  -39,  -74,  -53,  -61,  -51,  -62,  -63,  -44,  -54,  -55,  -45,  -52,  -61,  -65,  -62,  -56],
        [ -52,  -59,  -70,  -53,  -56,  -63,  -61,  -54,  -55,  -64,  -51,  -47,  -36,  -66,  -67,  -58,  -76,  -55,  -64,  -44],
        [ -55,  -53,  -58,  -60,  -43,  -43,  -58,  -42,  -70,  -49,  -49,  -55,  -60,  -57,  -47,  -44,  -37,  -79,  -39,  -53],
        [ -48,  -35,  -65,  -44,  -48,  -55,  -66,  -71,  -61,  -44,  -49,  -53,  -58,  -56,  -78,  -61,  -51,  -58,  -22,  -56],
        [ -42,  -66,  -47,  -66,  -70,  -64,  -49,  -56,  -51,  -36,  -41,  -64,  -73,  -52,  -48,  -61,  -26,  -54,  -59,  -57],
        [ -56,  -61,  -43,  -56,  -43,  -44,  -68,  -46,  -43,  -42,  -39,  -31,  -61,  -36,  -73,  -28,  -45,  -59,  -37,  -42],
        [ -55,  -56,  -58,  -51,  -40,  -65,  -69,  -63,  -53,  -50,  -65,  -76,  -48,  -54,  -68,  -55,  -48,  -53,  -39,  -59],
        [ -41,  -64,  -53,  -73,  -65,  -54,  -55,  -42,  -46,  -58,  -71,  -61,  -27,  -50,  -55,  -53,  -72,  -43,  -45,  -57],
        [ -58,  -48,  -39,  -61,  -62,  -57,  -38,  -52,  -64,  -69,  -44,  -64,  -86,  -34,  -64,  -54,  -55,  -59,  -51,  -33],
        [ -55,  -37,  -55,  -46,  -39,  -37,  -42,  -59,  -53,  -39,  -74,  -31,  -70,  -39,  -50,  -55,  -52,  -50,  -61,  -46],
        [ -38,  -53,  -53,  -48,  -52,  -41,  -55,  -56,  -50,  -60,  -55,  -67,  -57,  -36,  -49,  -60,  -48,  -60,  -58,  -62],
        [ -50,  -57,  -77,  -59,  -60,  -39,  -71,  -62,  -56,  -47,  -52,  -65,  -40,  -56,  -63,  -63,  -51,  -33,  -51,  -46],
        [ -42,  -62,  -62,  -56,  -42,  -74,  -73,  -42,  -54,  -55,  -52,  -50,  -56,  -50,  -55,  -69,  -52,  -32,  -46,  -65],
        [ -70,  -56,  -35,  -53,  -76,  -49,  -63,  -60,  -70,  -53,  -42,  -75,  -43,  -47,  -51,  -41,  -32,  -50,  -49,  -52],
        [ -68,  -63,  -57,  -58,  -62,  -60,  -56,  -71,  -40,  -50,  -35,  -50,  -46,  -48,  -58,  -63,  -57,  -60,  -60,  -47],
        [ -47,  -56,  -40,  -42,  -61,  -56,  -64,  -61,  -63,  -46,  -82,  -43,  -54,  -73,  -63,  -50,  -67,  -51,  -50,  -45],
        [ -41,  -72,  -46,  -74,  -64,  -47,  -34,  -69,  -39,  -38,  -51,  -50,  -53,  -53,  -62,  -74,  -39,  -37,  -45,  -49],
        [ -74,  -52,  -69,  -53,  -57,  -67,  -39,  -55,  -66,  -59,  -54,  -38,  -53,  -48,  -69,  -45,  -62,  -69,  -72,  -54],
        [ -61,  -62,  -45,  -56,  -64,  -55,  -48,  -43,  -55,  -55,  -38,  -82,  -54,  -47,  -55,  -56,  -59,  -56,  -44,  -52],
        [ -50,  -35,  -44,  -51,  -38,  -53,  -65,  -74,  -62,  -45,  -56,  -53,  -42,  -47,  -46,  -49,  -50,  -43,  -46,  -41],
        [ -38,  -65,  -65,  -63,  -43,  -43,  -44,  -57,  -63,  -50,  -61,  -57,  -53,  -51,  -46,  -47,  -45,  -52,  -58,  -42],
        [ -65,  -58,  -56,  -55,  -53,  -44,  -30,  -66,  -48,  -67,  -63,  -45,  -64,  -67,  -40,  -49,  -29,  -52,  -33,  -36],
        [ -45,  -66,  -71,  -56,  -71,  -49,  -66,  -36,  -55,  -48,  -62,  -69,  -55,  -59,  -38,  -58,  -60,  -55,  -67,  -55],
        [ -49,  -56,  -47,  -47,  -38,  -70,  -60,  -45,  -46,  -51,  -41,  -48,  -54,  -41,  -72,  -66,  -74,  -44,  -39,  -72],
        [ -60,  -63,  -40,  -44,  -49,  -70,  -62,  -66,  -50,  -47,  -62,  -60,  -50,  -41,  -62,  -51,  -62,  -72,  -49,  -62],
        [ -77,  -49,  -45,  -47,  -52,  -41,  -58,  -38,  -50,  -55,  -62,  -35,  -67,  -43,  -58,  -45,  -57,  -36,  -41,  -48],
        [ -60,  -54,  -61,  -53,  -37,  -34,  -56,  -47,  -61,  -55,  -49,  -31,  -72,  -41,  -53,  -52,  -66,  -82,  -53,  -44],
        [ -44,  -65,  -58,  -46,  -54,  -30,  -57,  -63,  -71,  -76,  -59,  -58,  -47,  -50,  -43,  -41,  -65,  -53,  -49,  -65],
        [ -36,  -56,  -53,  -46,  -55,  -50,  -59,  -68,  -52,  -57,  -46,  -68,  -29,  -44,  -71,  -43,  -83,  -52,  -43,  -42],
        [ -73,  -57,  -39,  -56,  -49,  -36,  -45,  -57,  -59,  -50,  -55,  -37,  -70,  -34,  -41,  -55,  -49,  -67,  -44,  -52],
        [ -50,  -46,  -32,  -47,  -64,  -36,  -61,  -73,  -54,  -52,  -47,  -40,  -60,  -40,  -68,  -57,  -57,  -49,  -54,  -57],
        [ -62,  -62,  -56,  -66,  -60,  -67,  -67,  -71,  -58,  -39,  -29,  -59,  -52,  -62,  -62,  -31,  -29,  -42,  -51,  -43],
        [ -44,  -43,  -49,  -28,  -48,  -73,  -63,  -66,  -58,  -69,  -56,  -71,  -65,  -54,  -44,  -45,  -29,  -46,  -54,  -57],
        [ -51,  -36,  -55,  -50,  -61,  -63,  -53,  -62,  -60,  -49,  -58,  -55,  -63,  -52,  -39,  -56,  -60,  -80,  -56,  -29],
        [ -49,  -49,  -63,  -63,  -44,  -45,  -43,  -65,  -76,  -51,  -51,  -56,  -61,  -45,  -43,  -65,  -41,  -64,  -61,  -55],
        [ -55,  -42,  -58,  -26,  -58,  -60,  -54,  -74,  -56,  -50,  -65,  -64,  -48,  -55,  -60,  -59,  -61,  -55,  -62,  -49],
        [ -48,  -39,  -65,  -74,  -55,  -46,  -43,  -43,  -58,  -50,  -43,  -54,  -37,  -68,  -54,  -63,  -63,  -60,  -37,  -57],
        [ -59,  -43,  -51,  -61,  -60,  -39,  -37,  -51,  -45,  -64,  -80,  -49,  -49,  -38,  -58,  -71,  -61,  -56,  -46,  -48],
        [ -50,  -66,  -50,  -51,  -46,  -73,  -51,  -63,  -50,  -40,  -57,  -46,  -72,  -41,  -60,  -67,  -56,  -42,  -45,  -71],
        [ -38,  -44,  -34,  -49,  -66,  -38,  -54,  -46,  -51,  -60,  -53,  -56,  -54,  -43,  -60,  -55,  -69,  -52,  -50,  -48],
        [ -77,  -56,  -33,  -49,  -73,  -33,  -51,  -60,  -40,  -55,  -51,  -30,  -46,  -55,  -63,  -50,  -77,  -50,  -45,  -60],
        [ -69,  -55,  -61,  -61,  -57,  -79,  -60,  -56,  -51,  -53,  -68,  -43,  -47,  -50,  -48,  -46,  -53,  -48,  -45,  -68],
        [ -59,  -54,  -67,  -56,  -26,  -64,  -58,  -35,  -63,  -61,  -52,  -44,  -68,  -70,  -50,  -62,  -49,  -41,  -63,  -52],
        [ -63,  -60,  -48,  -56,  -48,  -36,  -54,  -27,  -46,  -65,  -49,  -59,  -37,  -47,  -67,  -64,  -47,  -53,  -53,  -50],
        [ -61,  -61,  -52,  -83,  -41,  -44,  -46,  -62,  -48,  -54,  -73,  -46,  -50,  -35,  -33,  -46,  -54,  -47,  -37,  -51],
        [ -64,  -54,  -50,  -65,  -69,  -59,  -78,  -44,  -56,  -23,  -61,  -56,  -55,  -67,  -27,  -63,  -61,  -37,  -52,  -51],
        [ -21,  -31,  -62,  -15,  -75,  -62,  -44,  -36,  -73,  -39,  -65,  -60,  -55,  -52,  -42,  -56,  -41,  -41,  -63,  -50],
        [ -70,  -53,  -49,  -80,  -65,  -63,  -31,  -50,  -48,  -66,  -57,  -29,  -52,  -45,  -64,  -44,  -48,  -44,  -89,  -49],
        [ -57,  -66,  -54,  -57,  -62,  -34,  -74,  -38,  -44,  -42,  -60,  -73,  -71,  -43,  -47,  -38,  -58,  -59,  -55,  -23],
        [ -35,  -75,  -53,  -60,  -50,  -54,  -52,  -71,  -37,  -64,  -54,  -59,  -40,  -25,  -61,  -58,  -60,  -54,  -49,  -41],
        [ -53,  -56,  -61,  -52,  -75,  -43,  -43,  -60,  -49,  -39,  -57,  -51,  -43,  -56,  -43,  -35,  -55,  -38,  -74,  -58],
        [ -27,  -57,  -30,  -63,  -47,  -48,  -60,  -64,  -38,  -51,  -54,  -37,  -65,  -47,  -52,  -49,  -48,  -57,  -41,  -63],
        [ -54,  -60,  -64,  -47,  -52,  -42,  -40,  -54,  -67,  -79,  -55,  -62,  -60,  -58,  -45,  -40,  -27,  -48,  -54,  -51],
        [ -60,  -64,  -44,  -54,  -78,  -45,  -56,  -62,  -58,  -57,  -52,  -66,  -34,  -81,  -34,  -41,  -40,  -41,  -39,  -55],
        [ -33,  -54,  -67,  -60,  -46,  -43,  -51,  -45,  -44,  -53,  -43,  -62,  -55,  -47,  -49,  -64,  -57,  -54,  -78,  -41],
        [ -31,  -53,  -56,  -63,  -47,  -45,  -51,  -57,  -61,  -60,  -37,  -62,  -66,  -47,  -63,  -75,  -51,  -44,  -61,  -44],
        [ -66,  -55,  -48,  -67,  -58,  -45,  -50,  -65,  -52,  -48,  -29,  -61,  -51,  -64,  -47,  -34,  -38,  -31,  -63,  -64],
        [ -40,  -77,  -55,  -51,  -46,  -68,  -59,  -56,  -31,  -61,  -58,  -41,  -56,  -68,  -49,  -68,  -71,  -49,  -61,  -44],
        [ -49,  -60,  -41,  -53,  -67,  -52,  -49,  -56,  -47,  -34,  -68,  -62,  -34,  -58,  -85,  -52,  -55,  -29,  -47,  -45],
        [ -72,  -70,  -41,  -50,  -65,  -59,  -41,  -55,  -50,  -54,  -49,  -65,  -62,  -51,  -53,  -47,  -37,  -64,  -49,  -49],
        [ -49,  -39,  -58,  -76,  -41,  -67,  -51,  -34,  -65,  -57,  -49,  -84,  -38,  -66,  -47,  -54,  -56,  -48,  -56,  -64],
        [ -50,  -57,  -51,  -58,  -68,  -44,  -48,  -49,  -72,  -38,  -53,  -49,  -66,  -56,  -40,  -32,  -35,  -56,  -55,  -41],
        [ -36,  -24,  -31,  -69,  -48,  -53,  -59,  -43,  -55,  -65,  -37,  -46,  -76,  -62,  -62,  -67,  -76,  -75,  -40,  -55],
        [ -62,  -57,  -65,  -36,  -27,  -53,  -49,  -39,  -68,  -43,  -77,  -61,  -71,  -34,  -49,  -71,  -44,  -88,  -58,  -54],
        [ -57,  -80,  -71,  -44,  -45,  -45,  -74,  -73,  -58,  -55,  -35,  -39,  -48,  -60,  -63,  -54,  -42,  -42,  -33,  -32],
        [ -74,  -38,  -73,  -79,  -60,  -33,  -47,  -70,  -44,  -51,  -40,  -43,  -60,  -67,  -50,  -59,  -79,  -34,  -61,  -42],
        [ -57,  -46,  -60,  -41,  -40,  -72,  -67,  -65,  -34,  -72,  -68,  -49,  -49,  -45,  -60,  -35,  -47,  -57,  -37,  -50],
        [ -42,  -77,  -50,  -62,  -66,  -43,  -51,  -46,  -49,  -28,  -60,  -44,  -63,  -53,  -60,  -68,  -53,  -33,  -43,  -57],
        [ -46,  -37,  -53,  -42,  -46,  -52,  -64,  -39,  -38,  -57,  -48,  -50,  -46,  -41,  -59,  -49,  -42,  -40,  -55,  -53],
        [ -52,  -52,  -48,  -60,  -53,  -50,  -43,  -35,  -42,  -40,  -51,  -65,  -53,  -58,  -53,  -81,  -52,  -52,  -56,  -75],
        [ -68,  -66,  -59,  -34,  -36,  -41,  -41,  -45,  -58,  -65,  -54,  -45,  -46,  -34,  -48,  -47,  -50,  -36,  -45,  -59],
        [ -60,  -56,  -45,  -35,  -46,  -54,  -61,  -37,  -59,  -59,  -44,  -53,  -50,  -52,  -58,  -87,  -69,  -42,  -60,  -68],
        [ -50,  -20,  -38,  -48,  -70,  -52,  -48,  -38,  -47,  -48,  -46,  -45,  -30,  -69,  -48,  -62,  -38,  -68,  -57,  -49],
        [ -59,  -23,  -46,  -45,  -49,  -67,  -65,  -50,  -49,  -46,  -51,  -60,  -46,  -60,  -41,  -58,  -56,  -48,  -63,  -42],
        [ -65,  -37,  -43,  -43,  -66,  -41,  -42,  -48,  -55,  -33,  -57,  -80,  -56,  -79,  -71,  -73,  -44,  -37,  -53,  -57],
        [ -62,  -51,  -74,  -56,  -65,  -70,  -74,  -66,  -60,  -74,  -63,  -49,  -57,  -54,  -64,  -27,  -41,  -29,  -51,  -69],
        [ -57,  -59,  -61,  -60,  -40,  -62,  -48,  -75,  -45,  -61,  -38,  -66,  -47,  -65,  -62,  -47,  -62,  -63,  -34,  -59],
        [ -74,  -77,  -46,  -38,  -87,  -35,  -63,  -59,  -45,  -60,  -71,  -53,  -53,  -58,  -28,  -59,  -63,  -40,  -42,  -69],
        [ -52,  -41,  -49,  -40,  -64,  -35,  -58,  -51,  -48,  -45,  -56,  -45,  -52,  -47,  -55,  -44,  -46,  -43,  -28,  -47],
        [ -56,  -62,  -70,  -63,  -47,  -50,  -45,  -59,  -37,  -46,  -36,  -43,  -62,  -69,  -66,  -47,  -51,  -71,  -27,  -55],
        [ -48,  -42,  -54,  -44,  -71,  -48,  -45,  -49,  -77,  -64,  -47,  -62,  -55,  -27,  -23,  -67,  -22,  -60,  -70,  -45],
        [ -46,  -35,  -58,  -58,  -39,  -48,  -42,  -70,  -40,  -45,  -48,  -39,  -74,  -45,  -50,  -74,  -54,  -49,  -60,  -40],
        [ -56,  -49,  -60,  -51,  -57,  -38,  -52,  -38,  -74,  -67,  -49,  -49,  -53,  -59,  -43,  -57,  -52,  -40,  -49,  -50],
        [ -75,  -47,  -70,  -49,  -66,  -65,  -41,  -60,  -52,  -43,  -55,  -61,  -50,  -63,  -64,  -78,  -30,  -49,  -45,  -67],
        [ -44,  -38,  -60,  -49,  -55,  -34,  -37,  -48,  -42,  -65,  -48,  -39,  -59,  -58,  -70,  -70,  -61,  -37,  -65,  -50],
        [ -65,  -40,  -56,  -31,  -48,  -42,  -45,  -63,  -59,  -51,  -51,  -48,  -38,  -69,  -44,  -47,  -50,  -56,  -33,  -50],
        [ -61,  -44,  -63,  -52,  -67,  -26,  -82,  -53,  -37,  -45,  -58,  -42,  -45,  -70,  -55,  -43,  -64,  -50,  -62,  -50],
        [ -35,  -36,  -53,  -51,  -52,  -68,  -50,  -76,  -53,  -63,  -65,  -68,  -43,  -61,  -42,  -42,  -50,  -66,  -70,  -51],
        [ -48,  -59,  -46,  -33,  -55,  -70,  -43,  -58,  -52,  -32,  -55,  -69,  -48,  -69,  -59,  -28,  -52,  -50,  -60,  -58],
        [ -37,  -63,  -48,  -72,  -43,  -62,  -48,  -70,  -64,  -43,  -57,  -63,  -46,  -39,  -53,  -68,  -44,  -64,  -54,  -31],
        [ -55,  -27,  -43,  -49,  -57,  -65,  -59,  -60,  -69,  -57,  -43,  -52,  -64,  -68,  -37,  -44,  -45,  -41,  -59,  -43],
        [ -47,  -56,  -52,  -32,  -32,  -45,  -43,  -72,  -42,  -48,  -64,  -43,  -51,  -79,  -44,  -60,  -57,  -70,  -52,  -66],
        [ -64,  -48,  -53,  -42,  -47,  -52,  -57,  -52,  -60,  -74,  -58,  -66,  -38,  -43,  -62,  -38,  -51,  -41,  -68,  -38],
        [ -48,  -28,  -59,  -58,  -77,  -56,  -47,  -47,  -30,  -53,  -53,  -42,  -37,  -73,  -40,  -38,  -59,  -64,  -48,  -57],
        [ -50,  -42,  -41,  -81,  -68,  -71,  -56,  -63,  -49,  -37,  -51,  -64,  -69,  -53,  -74,  -63,  -49,  -68,  -62, -101]
output = torch.distributed._functional_collectives.reduce_scatter_tensor(input_tensor, "max", 0)
gpu0[122,10] is -51, which is different from input_tensor[122,10](-50).

Versions

Is this a bug?

cc @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @pragupta @msaroufim @dcci @aditvenk @xmfan

extent analysis

Fix Plan

The issue seems to be related to the torch.distributed._functional_collectives.reduce_scatter_tensor function. To fix this, we need to ensure that the input tensor is properly synchronized across all GPUs before performing the reduction operation.

Here are the steps to fix the issue:

  • Verify the input tensor: Before calling reduce_scatter_tensor, verify that the input tensor is identical across all GPUs. You can do this by using torch.distributed.all_gather to gather the input tensors from all GPUs and check if they are identical.
  • Use torch.distributed.barrier(): Add a torch.distributed.barrier() call before reduce_scatter_tensor to ensure that all GPUs have finished their previous operations and are synchronized.
  • Check the reduction operation: Verify that the reduction operation ("max" in this case) is correct and suitable for the input tensor.

Here's an example code snippet that demonstrates the fix:

import torch
import torch.distributed as dist

#...

# Verify the input tensor
input_tensor =...  # your input tensor
gathered_tensors = [torch.zeros_like(input_tensor) for _ in range(dist.get_world_size())]
dist.all_gather(gathered_tensors, input_tensor)
for i, tensor in enumerate(gathered_tensors):
    if not torch.equal(tensor, input_tensor):
        print(f"Input tensor mismatch on GPU {i}")

# Add a barrier to ensure synchronization
dist.barrier()

# Perform the reduction operation
output = dist._functional_collectives.reduce_scatter_tensor(input_tensor, "max", 0)

Verification

To verify that the fix worked, you can add some debug prints or use a debugger to check the values of the input tensor and the output tensor. Specifically, you can check the value of gpu0[122,10] and input_tensor[122,10] to ensure they are identical.

Extra Tips

  • Make sure to use the correct reduction operation for your use case. In this example, we're using "max", but you may need to use a different operation depending on your specific requirements.
  • If you're using a custom reduction operation, ensure that it is correctly implemented and tested.
  • When working with distributed tensors, it's essential to ensure that all GPUs are properly synchronized to avoid inconsistencies and incorrect results.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING