dify - 💡(How to fix) Fix Summary helper duplicates each line when merging text for summarization

Self-checks: searched existing issues, reproduced on latest main.

Summary

BuiltinTool.summary() in api/core/tools/builtin_tool/tool.py (and the identical copy in api/core/plugin/backwards_invocation/model.py) splits long content into lines and then merges them into token-bounded chunks before summarizing each chunk with the LLM. The merge loop uses two adjacent, non-mutually-exclusive if blocks:

for j in new_lines:
    if len(messages) == 0:
        messages.append(j)
    else:
        if len(messages[-1]) + len(j) < max_tokens * 0.5:
            messages[-1] += j          # (A) line placed into current chunk
        if get_prompt_tokens(messages[-1] + j) > max_tokens * 0.7:   # not elif
            messages.append(j)         # (B) line placed AGAIN
        else:
            messages[-1] += j          # (C) line placed AGAIN

When branch (A) fires, the line j is concatenated onto the current chunk, and then the separate if/else runs unconditionally and places j a second time (either as a new chunk via (B) or concatenated again via (C)). The token check at (B) is also evaluated on messages[-1] + j after j was already merged in (A), so it double-counts the line.

Impact

Every line that fits the character budget is duplicated in the text sent to the summarization model. This is on a live path — the builtin web scraper tool calls summary() to condense scraped pages (api/core/tools/builtin_tool/providers/webscraper/tools/webscraper.py). The result is wasted tokens and degraded/garbled summaries.

Expected

Each line should be placed into exactly one chunk — merged into the current chunk if it fits, otherwise starting a new chunk.

Fix

Make the two branches mutually exclusive (if → elif). A PR with a regression test follows.