litellm - 💡(How to fix) Fix [Bug]: Azure Document Intelligence `prebuilt-layout` via LiteLLM OCR does not return layout-aware Markdown [1 participants]

## Azure Document Intelligence
- model_name: azure-doc-intel-read
  litellm_params:
    model: azure_ai/doc-intelligence/prebuilt-read
    api_key: os.environ/AZURE_DOCUMENT_INTELLIGENCE_API_KEY
    api_base: os.environ/AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT
  model_info:
    mode: ocr

- model_name: azure-doc-intel-layout
  litellm_params:
    model: azure_ai/doc-intelligence/prebuilt-layout
    api_key: os.environ/AZURE_DOCUMENT_INTELLIGENCE_API_KEY
    api_base: os.environ/AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT
  model_info:
    mode: ocr

## VertexAI / Mistral
- model_name: mistral-ocr-2505
  litellm_params:
    model: vertex_ai/mistral-ocr-2505
    vertex_project: "project"
    vertex_location: "location"
    vertex_credentials: "/app/credentials/credentials.json"
  model_info:
    mode: ocr

curl --location 'http://localhost:4000/ocr' \
--header 'Authorization: Bearer <API_KEY>' \
--header 'Content-Type: application/json' \
--data '{
  "document": {
    "type": "document_url",
    "document_url": "https://github.com/Azure-Samples/document-intelligence-code-samples/blob/main/Data/layout/layout-pageobject.png?raw=true"
  },
  "model": "azure-doc-intel-read",
  "user": "test"
}'

curl --location 'http://localhost:4000/ocr' \
--header 'Authorization: Bearer <API_KEY>' \
--header 'Content-Type: application/json' \
--data '{
  "document": {
    "type": "document_url",
    "document_url": "https://github.com/Azure-Samples/document-intelligence-code-samples/blob/main/Data/layout/layout-pageobject.png?raw=true"
  },
  "model": "azure-doc-intel-layout",
  "user": "test"
}'

curl --location 'http://localhost:4000/ocr' \
--header 'Authorization: Bearer <API_KEY>' \
--header 'Content-Type: application/json' \
--data '{
  "document": {
    "type": "document_url",
    "document_url": "https://github.com/Azure-Samples/document-intelligence-code-samples/blob/main/Data/layout/layout-pageobject.png?raw=true"
  },
  "model": "mistral-ocr-2505",
  "user": "test"
}'

{
  "status": "succeeded",
  "createdDateTime": "2026-04-14T09:13:46Z",
  "lastUpdatedDateTime": "2026-04-14T09:13:46Z",
  "analyzeResult": {
    "apiVersion": "2024-11-30",
    "modelId": "prebuilt-layout",
    "stringIndexType": "utf16CodeUnit",
    "content": "\n\n\n# This is title\n\n\n## 1. Text\n\nLatin refers to an ancient Italic language\noriginating in the region of Latium in\nancient Rome.\n\n\n## 2. Page Objects\n\n\n### 2.1 Table\n\nHere's a sample table below, designed to\nbe simple for easy understand and quick\nreference.\n\n\n<table>\n<caption>Table 1: This is a dummy table</caption>\n<tr>\n<th>Name</th>\n<th>Corp</th>\n<th>Remark</th>\n</tr>\n<tr>\n<td>Foo</td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td>Bar</td>\n<td>Microsoft</td>\n<td>Dummy</td>\n</tr>\n</table>\n\n\n### 2.2. Figure\n\n\n<figure>\n<figcaption>Figure 1: Here is a figure with text</figcaption>\n\nValues\n\n500\n\n450\n\n400\n\n400\n\n350\n\n300\n\n300\n\n250\n\n200\n\n200\n\n100\n\n0\n\nJan\n\nFeb\n\nMar\n\nApr\n\nMay\n\nJun\n\nMonths\n\n</figure>\n\n\n## 3. Others\n\nAI Document Intelligence is an AI service\nthat applies advanced machine learning\nto extract text, key-value pairs, tables,\nand structures from documents\nautomatically and accurately:\n\n☒\nclear\n\n☒\nprecise\n\n☐\nvague\n\n☒\ncoherent\n\n☐\nIncomprehensible\n\nTurn documents into usable data and\nshift your focus to acting on information\nrather than compiling it. Start with\nprebuilt models or create custom models\ntailored to your documents both on\npremises and in the cloud with the AI\nDocument Intelligence studio or SDK.\n\nLearn how to accelerate your business\nprocesses by automating text extraction\nwith AI Document Intelligence. This\nwebinar features hands-on demos for key\nuse cases such as document processing,\nknowledge mining, and industry-specific\nAI model customization.\n\n\n\n",
    "pages": [
      ...
    ],
    "tables": [
      ...
    ],
    "paragraphs": [
      ...
    ],
    "contentFormat": "markdown",
    "sections": [
      ...
    ],
    "figures": [
      ...
    ]
  }
}

### Relevant log output

```shell
Response excerpt for azure-doc-intel-read:

{
    "pages": [
        {
            "index": 0,
            "markdown": "This is the header of the document.\nThis is title\n1. Text\nLatin refers to an ancient Italic language\noriginating in the region of Latium in\nancient Rome.\n2. Page Objects\n2.1 Table\nHere's a sample table below, designed to\nbe simple for easy understand and quick\nreference.\nName\nCorp\nRemark\nFoo\nBar\nMicrosoft\nDummy\nTable 1: This is a dummy table\n2.2. Figure\nFigure 1: Here is a figure with text\n500\n450\n400\n400\n350\n250\n200\n200-\nFeb\nVisu\nMents\n3. Others\nAl Document Intelligence is an Al service\nthat applies advanced machine learning\nto extract text, key-value pairs, tables,\nand structures from documents\nautomatically and accurately:\nclear\nIprecise\nvague\ncoherent\nIncomprehensible\nTurn documents into usable data and\nshift your focus to acting on information\nrather than compiling it. Start with\nprebuilt models or create custom models\ntailored to your documents both on\npremises and in the cloud with the Al\nDocument Intelligence studio or SDK.\nLearn how to accelerate your business\nprocesses by automating text extraction\nwith Al Document Intelligence. This\nwebinar features hands-on demos for key\nuse cases such as document processing,\nknowledge mining, and industry-specific\nAl model customization.\nThis is the footer of the document.\n1 | Page",
            "images": null,
            "dimensions": {
                "dpi": 96,
                "height": 909,
                "width": 1200
            }
        }
    ],
    "model": "azure-doc-intel-read",
    "document_annotation": null,
    "usage_info": {
        "pages_processed": 1,
        "doc_size_bytes": null
    },
    "object": "ocr"
}



Response excerpt for azure-doc-intel-layout:

{
    "pages": [
        {
            "index": 0,
            "markdown": "This is the header of the document.\nThis is title\n1. Text\nLatin refers to an ancient Italic language\noriginating in the region of Latium in\nancient Rome.\n2. Page Objects\n2.1 Table\nHere's a sample table below, designed to\nbe simple for easy understand and quick\nreference.\nName\nCorp\nRemark\nFoo\nBar\nMicrosoft\nDummy\nTable 1: This is a dummy table\n2.2. Figure\nFigure 1: Here is a figure with text\nValues\n500\n450\n100\n400\n350\n300\n300\n250\n200\n200\n200-\nn\nJan\nFeb\nMar\nİçr\nMay\n2um\nMeness\n3. Others\nAl Document Intelligence is an Al service\nthat applies advanced machine learning\nto extract text, key-value pairs, tables,\nand structures from documents\nautomatically and accurately:\nclear\nprecise\nvague\ncoherent\nIncomprehensible\nTurn documents into usable data and\nshift your focus to acting on information\nrather than compiling it. Start with\nprebuilt models or create custom models\ntailored to your documents both on\npremises and in the cloud with the Al\nDocument Intelligence studio or SDK.\nLearn how to accelerate your business\nprocesses by automating text extraction\nwith Al Document Intelligence. This\nwebinar features hands-on demos for key\nuse cases such as document processing,\nknowledge mining, and industry-specific\nAl model customization.\nThis is the footer of the document.\n1 | Page",
            "images": null,
            "dimensions": {
                "dpi": 96,
                "height": 909,
                "width": 1200
            }
        }
    ],
    "model": "azure-doc-intel-layout",
    "document_annotation": null,
    "usage_info": {
        "pages_processed": 1,
        "doc_size_bytes": null
    },
    "object": "ocr"
}



Response excerpt for mistral-ocr-2505:

{
    "pages": [
        {
            "index": 0,
            "markdown": "This is the header of the document.\n\n# This is title\n\n## 1. Text\n\nLatin refers to an ancient Italic language originating in the region of Latium in ancient Rome.\n\n## 2. Page Objects\n\n### 2.1 Table\n\nHere's a sample table below, designed to be simple for easy understand and quick reference.\n\n|  Name | Corp | Remark  |\n| --- | --- | --- |\n|  Foo |  |   |\n|  Bar | Microsoft | Dummy  |\n\n*Table 1: This is a dummy table*\n\n### 2.2. Figure\n\n*Figure 1: Here is a figure with text*\n\n![img-0.jpeg](img-0.jpeg)\n\n## 3. Others\n\nAl Document Intelligence is an AI service that applies advanced machine learning to extract text, key-value pairs, tables, and structures from documents automatically and accurately:\n\n- ☑ clear\n- ☑ precise\n- ☐ vague\n- ☑ coherent\n- ☐ Incomprehensible\n\nTurn documents into usable data and shift your focus to acting on information rather than compiling it. Start with prebuilt models or create custom models tailored to your documents both on premises and in the cloud with the AI Document Intelligence studio or SDK.\n\nLearn how to accelerate your business processes by automating text extraction with AI Document Intelligence. This webinar features hands-on demos for key use cases such as document processing, knowledge mining, and industry-specific AI model customization.",
            "images": [
                {
                    "image_base64": null,
                    "bbox": null,
                    "id": "img-0.jpeg",
                    "top_left_x": 330,
                    "top_left_y": 586,
                    "bottom_right_x": 594,
                    "bottom_right_y": 758,
                    "image_annotation": null
                }
            ],
            "dimensions": {
                "dpi": 200,
                "height": 909,
                "width": 1200
            },
            "tables": [],
            "hyperlinks": [],
            "header": null,
            "footer": null
        }
    ],
    "model": "mistral-ocr-2505",
    "document_annotation": null,
    "usage_info": {
        "pages_processed": 1,
        "doc_size_bytes": 151213
    },
    "object": "ocr"
}