dify - ✅(Solved) Fix qdrant delete items is so large lead server error [1 pull requests, 1 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
langgenius/dify#35041Fetched 2026-04-16 06:34:33
View on GitHub
Comments
1
Participants
2
Timeline
6
Reactions
1
Author
Assignees
Timeline (top)
assigned ×1closed ×1commented ×1cross-referenced ×1

Error Message

  1. error occur no error {"ts":"2026-04-11T06:42:38.569Z","severity":"ERROR","service":"langgenius/dify","caller":"batch_clean_document_task.py:85","message":"Failed to clean vector index for dataset_id: 0561270c-6024-4c07-bf8e-bc68f6972622, document_ids: ['243e9451-e370-4017-9a32-b31110e21115', '2a2c2961-5f67-4034-afb3-173f2fc87a60', '2f8571ec-36b2-4715-b4fc-ad1925e598e3', '42a9d049-8194-4dd1-b9b7-39f434b8076b', '4e3cd7bc-57df-4d63-b35d-ac35a05ce9a9', '76372b36-e4a0-4813-8d47-142318d7f066', '7e1628b6-30d0-463c-a17e-f4ad3eab69ea', '86720383-c978-4513-bc7a-b3e75c096400', '874f1f6e-7dc1-4257-873d-fd191d396e19', '914d920a-5dc6-418c-a398-397abd89fbf9', '94bfba04-4918-4b17-a6dd-1ecf3319f846', 'aaf539f1-f979-4859-b2d7-92b96e1d7409', 'c7962cd2-1138-4ca2-8b76-5ff055108b27'], index_node_ids count: 8546","trace_id":"4e1072e9ecb15407840e468ff2d57130","stack_trace":"Traceback (most recent call last):\n File "/app/api/tasks/batch_clean_document_task.py", line 81, in batch_clean_document_task\n index_processor.clean(\n File "/app/api/core/rag/index_processor/processor/parent_child_index_processor.py", line 194, in clean\n vector.delete_by_ids(child_node_ids)\n File "/app/api/core/rag/datasource/vdb/vector_factory.py", line 270, in delete_by_ids\n self._vector_processor.delete_by_ids(ids)\n File "/app/api/core/rag/datasource/vdb/tidb_on_qdrant/tidb_on_qdrant_vector.py", line 310, in delete_by_ids\n raise e\n File "/app/api/core/rag/datasource/vdb/tidb_on_qdrant/tidb_on_qdrant_vector.py", line 300, in delete_by_ids\n self._client.delete(\n File "/app/api/.venv/lib/python3.12/site-packages/qdrant_client/qdrant_client.py", line 1201, in delete\n return self._client.delete(\n ^^^^^^^^^^^^^^^^^^^^\n File "/app/api/.venv/lib/python3.12/site-packages/qdrant_client/qdrant_remote.py", line 1700, in delete\n result: Optional[types.UpdateResult] = self.openapi_client.points_api.delete_points(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/app/api/.venv/lib/python3.12/site-packages/qdrant_client/http/api/points_api.py", line 1173, in delete_points\n return self._build_for_delete_points(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/app/api/.venv/lib/python3.12/site-packages/qdrant_client/http/api/points_api.py", line 206, in build_for_delete_points\n return self.api_client.request(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File "/app/api/.venv/lib/python3.12/site-packages/qdrant_client/http/api_client.py", line 76, in request\n return self.send(request, type)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/app/api/.venv/lib/python3.12/site-packages/qdrant_client/http/api_client.py", line 99, in send\n raise UnexpectedResponse.for_response(response)\nqdrant_client.http.exceptions.UnexpectedResponse: Unexpected Response: 500 (Internal Server Error)\nRaw response content:\nb'{"time":5e-8,"status":{"error":"Internal server error: [trace-id:662e6390-660c-4242-8071-63a02c681538] database error"}}'\n"}

Fix Action

Fixed

PR fix notes

PR #35042: fix: fix qdrant delete size is too large

Description (problem / solution / changelog)

[!IMPORTANT]

  1. Make sure you have read our contribution guidelines
  2. Ensure there is an associated issue and you have been assigned to it
  3. Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

fix #35041

limit qdrant delete item max batch size

<!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> <!-- If this PR was created by an automated agent, add `From <Tool Name>` as the final line of the description. Example: `From Codex`. -->

Screenshots

BeforeAfter
......

Checklist

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran make lint && make type-check (backend) and cd web && pnpm exec vp staged (frontend) to appease the lint gods

Changed files

  • api/core/rag/datasource/vdb/tidb_on_qdrant/tidb_on_qdrant_vector.py (modified, +21/-20)
RAW_BUFFERClick to expand / collapse

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.13.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

  1. delete multi indexes
  2. select 8600 document
  3. error occur

✔️ Expected Behavior

no error

❌ Actual Behavior

{"ts":"2026-04-11T06:42:38.569Z","severity":"ERROR","service":"langgenius/dify","caller":"batch_clean_document_task.py:85","message":"Failed to clean vector index for dataset_id: 0561270c-6024-4c07-bf8e-bc68f6972622, document_ids: ['243e9451-e370-4017-9a32-b31110e21115', '2a2c2961-5f67-4034-afb3-173f2fc87a60', '2f8571ec-36b2-4715-b4fc-ad1925e598e3', '42a9d049-8194-4dd1-b9b7-39f434b8076b', '4e3cd7bc-57df-4d63-b35d-ac35a05ce9a9', '76372b36-e4a0-4813-8d47-142318d7f066', '7e1628b6-30d0-463c-a17e-f4ad3eab69ea', '86720383-c978-4513-bc7a-b3e75c096400', '874f1f6e-7dc1-4257-873d-fd191d396e19', '914d920a-5dc6-418c-a398-397abd89fbf9', '94bfba04-4918-4b17-a6dd-1ecf3319f846', 'aaf539f1-f979-4859-b2d7-92b96e1d7409', 'c7962cd2-1138-4ca2-8b76-5ff055108b27'], index_node_ids count: 8546","trace_id":"4e1072e9ecb15407840e468ff2d57130","stack_trace":"Traceback (most recent call last):\n File "/app/api/tasks/batch_clean_document_task.py", line 81, in batch_clean_document_task\n index_processor.clean(\n File "/app/api/core/rag/index_processor/processor/parent_child_index_processor.py", line 194, in clean\n vector.delete_by_ids(child_node_ids)\n File "/app/api/core/rag/datasource/vdb/vector_factory.py", line 270, in delete_by_ids\n self._vector_processor.delete_by_ids(ids)\n File "/app/api/core/rag/datasource/vdb/tidb_on_qdrant/tidb_on_qdrant_vector.py", line 310, in delete_by_ids\n raise e\n File "/app/api/core/rag/datasource/vdb/tidb_on_qdrant/tidb_on_qdrant_vector.py", line 300, in delete_by_ids\n self._client.delete(\n File "/app/api/.venv/lib/python3.12/site-packages/qdrant_client/qdrant_client.py", line 1201, in delete\n return self._client.delete(\n ^^^^^^^^^^^^^^^^^^^^\n File "/app/api/.venv/lib/python3.12/site-packages/qdrant_client/qdrant_remote.py", line 1700, in delete\n result: Optional[types.UpdateResult] = self.openapi_client.points_api.delete_points(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/app/api/.venv/lib/python3.12/site-packages/qdrant_client/http/api/points_api.py", line 1173, in delete_points\n return self._build_for_delete_points(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/app/api/.venv/lib/python3.12/site-packages/qdrant_client/http/api/points_api.py", line 206, in build_for_delete_points\n return self.api_client.request(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File "/app/api/.venv/lib/python3.12/site-packages/qdrant_client/http/api_client.py", line 76, in request\n return self.send(request, type)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/app/api/.venv/lib/python3.12/site-packages/qdrant_client/http/api_client.py", line 99, in send\n raise UnexpectedResponse.for_response(response)\nqdrant_client.http.exceptions.UnexpectedResponse: Unexpected Response: 500 (Internal Server Error)\nRaw response content:\nb'{"time":5e-8,"status":{"error":"Internal server error: [trace-id:662e6390-660c-4242-8071-63a02c681538] database error"}}'\n"}

extent analysis

TL;DR

The issue is likely caused by a database error when trying to delete vector indexes, resulting in an internal server error.

Guidance

  • The error message indicates a database error, specifically an internal server error, when trying to delete points from the Qdrant database.
  • The delete_by_ids method in tidb_on_qdrant_vector.py is raising an exception, which suggests that the issue may be related to the Qdrant client or the database connection.
  • To mitigate the issue, you can try to:
    • Check the Qdrant database logs for any errors or issues related to the delete operation.
    • Verify that the Qdrant client is properly configured and connected to the database.
    • Consider implementing retry logic or error handling for the delete_by_ids method to handle temporary database errors.

Example

No code example is provided as the issue is related to a specific database error and requires further investigation.

Notes

The issue may be related to a bug in the Qdrant client or the database itself, and further investigation is required to determine the root cause.

Recommendation

Apply workaround: Implement retry logic or error handling for the delete_by_ids method to handle temporary database errors, as this may help mitigate the issue until the root cause is determined.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

dify - ✅(Solved) Fix qdrant delete items is so large lead server error [1 pull requests, 1 comments, 2 participants]