pytorch - 💡(How to fix) Fix Audit license-files glob in pyproject.toml for over-collection [1 pull requests]

🚀 The feature, motivation and pitch

#180237 migrates PyTorch's bundled-license shipping mechanism from the legacy setup.py-concatenated LICENSE to the PEP 639 license-files field. As part of that PR's review, the question came up of whether the (old and new) license-files glob accurately represents the distribution's licensing surface.

The PR uses a minimal recursive glob that is designed to be identical to the old collection logic:

license-files = [
    "LICENSE",
    "third_party/**/LICENSE",
    "third_party/**/LICENSE.txt",
    "third_party/**/LICENSE.rst",
    "third_party/**/COPYING.BSD",
]

This matches ~108 files per built wheel. Inspection shows it over-collects in several categories:

After conservative exclusion, ~61 of the 108 files represent code that actually ships in the wheel; the remainder are over-collection artifacts of the recursive glob.

This is pre-existing behaviour. The old setup.py concat_license_files mechanism that #180237 replaces walked third_party/ with the same broad criteria and produced an equivalently over-collected LICENSE blob in every release wheel. The PEP 639 layout just makes it visible per-file rather than buried in concatenated text.

The technical layout migration (#180237) is in scope to land first; this issue is the follow-up to discuss and implement the audit of which files belong in the set.

cc @malfet @atalman @tinglvv @nWEIdia @rgommers @seemethere

Alternatives

In rough order of investment:

1. Tighten the recursive globs to a finite depth. Five explicit-depth patterns (third_party/*/LICENSE* through third_party/*/*/*/*/*/LICENSE*) drop all files at path depth >= 8, which removes the GPL-3.0 cpr/test/LICENSE and most dynolog-transitive over-collection. Doesn't address shallower over-collection (googletest at depth 3, cutlass/python at depth 4, etc.).

2. Enumerate license-files explicitly per shipped submodule. Matches NumPy's pattern. ~61 paths, sorted, each line auditable. Most precise; adds maintenance burden whenever a vendored submodule moves or a new one is added.

3. Compute license-files dynamically. Use dynamic = ["license-files"] with a build-backend hook that walks third_party/ and applies exclude patterns. Setuptools 77+ may support this; scikit-build-core may need a custom metadata provider. Most flexible, biggest infra investment.

4. Pair the chosen approach with the SPDX license expression update. @rgommers' review on #180237 also asked for the SPDX license field to reflect the actual distribution licenses; that's coupled to whichever subset of files we decide ships, so should be tackled in the same pass.

Additional context

Why cpr/test deserves a specific call-out

third_party/kineto/libkineto/third_party/dynolog/third_party/cpr/test/LICENSE is GPL-3.0. Including it in the distribution's license-files set would, on a strict reading of PEP 639, imply the distribution contains GPL-3.0 code.

It does not. The full chain of gates:

1. PyTorch only invokes libkineto's CMakeLists, not dynolog's top-level CMakeLists. So dynolog_lib (the full dynolog with cpr) is never built.
2. libkineto only links the dynolog_ipcfabric_lib target, which is add_library(... INTERFACE) -- an empty INTERFACE library with no compiled sources.
3. Even if dynolog_lib were built, add_subdirectory(third_party/cpr) is gated by if(USE_ODS_GRAPH_API), a Facebook-internal flag that PyTorch never enables.
4. Even if cpr were built, CPR_BUILD_TESTS defaults to OFF, so cpr/test/CMakeLists.txt is never added.

So no GPL-3.0 code from cpr/test is ever compiled, linked, or executed in any PyTorch build -- but the LICENSE file is structurally present in the source tree because of recursive submodule vendoring, and the recursive license-files glob picks it up.

Scope of this issue

Discussion of which approach to take and on which timeline. Implementation is a separate PR (or set of PRs), ideally after #180237 has landed so the PEP 639 layout is the stable baseline.

Refs

• #180237 (the PEP 639 layout migration)
• #158104 (earlier PEP 639 attempt, reverted twice for an unrelated torchvision-on-macOS issue)
• rgommers' inline review on #180237 (pyproject.toml:55) re: SPDX expression accuracy