claude-code - 💡(How to fix) Fix WebFetch/WebSearch: support content negotiation (Accept header), default to Accept: text/markdown

Summary

WebFetch (and WebSearch where applicable) currently has no way to send custom request headers. Please add support for content negotiation — at minimum an Accept header — and have WebFetch send Accept: text/markdown by default before falling back to HTML.

Why

A growing number of documentation sites serve clean markdown when asked. The most notable example is GitHub's own docs:

$ curl -sS -D - -H 'Accept: text/markdown' \
    https://docs.github.com/en/actions/how-tos/reuse-automations/reuse-workflows -o /dev/null \
    | grep -i '^content-type\|^vary'
content-type: text/markdown; charset=utf-8
vary: accept

The response is identical in content to what https://docs.github.com/api/article/body?pathname=... returns — just reached via standard content negotiation on the canonical URL. The vary: accept header confirms this is a first-class, CDN-cached representation.

Markdown is dramatically better input for an LLM than HTML-converted-to-markdown:

• No nav chrome, cookie banners, footers, or script tags to strip
• Tokens spent on content, not boilerplate
• Code blocks, tables, and headings preserved exactly as authored
• No lossy HTML→markdown conversion step in the middle

Today, the only way to get this in Claude Code is to bypass WebFetch entirely and shell out to curl via Bash. That works, but it means losing WebFetch's caching, summarization, and URL-redirect handling — and it requires per-user instructions in CLAUDE.md to remember to do it.

Proposal

1. Default behavior: WebFetch sends Accept: text/markdown, text/html;q=0.9, */*;q=0.5 (or similar). If the server returns text/markdown, use it verbatim and skip the HTML→markdown conversion step. If it returns HTML, fall back to the current behavior.
2. Optional override: Allow callers to pass headers explicitly (e.g. headers: { Accept: '...' }) for cases where the model wants JSON, plain text, or a specific representation.
3. Consider the same for any other web-fetching tools (WebSearch result fetches, MCP web tools that wrap fetch).

Benefits

• Free quality win for any site that honors content negotiation — GitHub Docs today, likely more tomorrow as the pattern spreads.
• Reduces token usage on docs-heavy tasks (which is most of what Claude Code does).
• Removes the need for users to write workaround instructions or hooks.
• Backward compatible — sites that don't recognize text/markdown just return HTML as they do today.

Workaround in the meantime

Shelling out:

curl -sS -H 'Accept: text/markdown' '<url>'

…with a CLAUDE.md note telling Claude to try this first. Works, but loses the ergonomics of WebFetch.