Introduction

When working with Elasticsearch for data analysis, you often need to query millions or even tens of millions of records. Traditional pagination with from/size suffers from severe performance degradation at deeper pages. That’s where the Scroll API comes to the rescue.

This article covers the Scroll API with real-world examples and best practices.

What is the Scroll API?

The Scroll API is a batch retrieval mechanism in Elasticsearch. Unlike traditional pagination, it creates a snapshot of the query context in memory, allowing you to paginate through all matching documents efficiently — without the deep pagination overhead.

Why Use Scroll?

• Performance: Avoids deep pagination penalties
• Memory-friendly: Processes data in batches rather than loading everything at once
• Ideal for: Data exports, batch processing, ETL pipelines, analytics

Step-by-Step Guide

Step 1: Initialize a Scroll Query

Start by sending a search request with a scroll parameter to set the context lifetime:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

GET /yk_callinfo_v2/_search?scroll=5m
{
  "_source": false,
  "fields": ["_id","userId","planCustomerId"],
  "size": 50000,               
  "query": {
    "bool": {
      "must": [
        { 
          "range": { 
            "createdTime": { 
              "gte": "2026-04-01T00:00:00.000Z", 
              "lt": "2026-04-02T00:00:00.000Z" 
            } 
          } 
        },
        { "term": { "ai": 1 } },
        { "exists": { "field": "planCustomerId" } }
      ]
    }
  }
}

Parameter Breakdown:

Note: The initial response includes a _scroll_id. Some clusters return a new _scroll_id on every subsequent call — always use the latest one.

Step 2: Fetch Subsequent Batches

Use the _scroll_id from the previous response to get the next batch:

1
2
3
4
5

POST /_search/scroll
{
  "scroll": "5m",
  "scroll_id": "FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoBRZTSm9hS0lWa1JMMjlVZWdnQTVDamt3AAAAAACboVkWQ0NIOF9CQk9UclNERHVSLTduNk1MdxY3cDRpY1lDeFFVcVZXR2VQYVJ6NFRRAAAAAACfpmIWNFVqVVAxUGtTS2VkWDVUcGRfWXRDQRZTSm9hS0lWa1JMMjlVZWdnQTVDamt3AAAAAACboVgWQ0NIOF9CQk9UclNERHVSLTduNk1MdxZrZU9xREtpWFQ0YUdJaHd3a3Q0WFFRAAAAAACnZz4WRmdvem1WS0RUdTZ6eDA1M1NMdGh2dxZrZU9xREtpWFQ0YUdJaHd3a3Q0WFFRAAAAAACnZz0WRmdvem1WS0RUdTZ6eDA1M1NMdGh2dw=="
}

Key Points:

• Each response may return a new _scroll_id — always use the most recent one
• When you receive an empty hits array, you’ve reached the end of the dataset
• Re-set scroll on every request to reset the TTL (prevents premature expiry)

Step 3: Clean Up Resources

Always clear the scroll context when done to free server memory:

1
2
3
4

DELETE /_search/scroll
{
  "scroll_id": "FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoBRZrZU9xREtpWFQ0YUdJaHd3a3Q0WFFRAAAAAACqwqwWRmdvem1WS0RUdTZ6eDA1M1NMdGh2dxZTSm9hS0lWa1JMMjlVZWdnQTVDamt3AAAAAACe0BQWQ0NIOF9CQk9UclNERHVSLTduNk1MdxZTSm9hS0lWa1JMMjlVZWdnQTVDamt3AAAAAACe0BMWQ0NIOF9CQk9UclNERHVSLTduNk1MdxY3cDRpY1lDeFFVcVZXR2VQYVJ6NFRRAAAAAACi23wWNFVqVVAxUGtTS2VkWDVUcGRfWXRDQRZrZU9xREtpWFQ0YUdJaHd3a0RFUlRfWFFRAAAAAACqwq0WRmdvem1WS0RUdTZ6eDA1M1NMdGh2dw=="
}

You can also clear multiple scroll IDs at once:

1
2
3
4
5
6
7
8

DELETE /_search/scroll
{
  "scroll_id": [
    "scroll_id_1",
    "scroll_id_2",
    "scroll_id_3"
  ]
}

Practical Use Cases

1. Data Export (Python)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

from elasticsearch import Elasticsearch

es = Elasticsearch("http://localhost:9200")

# Initialize scroll
response = es.search(
    index="yk_callinfo_v2",
    scroll="5m",
    size=50000,
    body={
        "query": {
            "range": {
                "createdTime": {
                    "gte": "2026-04-01T00:00:00.000Z",
                    "lt": "2026-04-02T00:00:00.000Z"
                }
            }
        }
    }
)

scroll_id = response['_scroll_id']
total = 0

while True:
    hits = response['hits']['hits']
    if not hits:
        break
    
    for hit in hits:
        process_record(hit)
    
    total += len(hits)
    response = es.scroll(scroll_id=scroll_id, scroll="5m")

es.clear_scroll(scroll_id=scroll_id)
print(f"Processed {total} records")

2. Batch Data Analysis

Process large datasets in chunks to avoid out-of-memory errors in your analysis pipeline.

3. Cross-Cluster Migration

Scroll API is the standard approach for reading data from a source cluster during index migration.

Best Practices

1. Set Appropriate TTL

• Keep scroll context TTL between 1–5 minutes
• Reset the TTL on each request by re-sending scroll
• Long TTLs consume more memory — avoid excessive durations

2. Control Batch Size

• Recommended size: 5,000 — 50,000
• Too large → slow queries and higher memory pressure
• Too small → excessive network round-trips

3. Always Clean Up

• Manual cleanup is more reliable than relying on TTL expiry
• Orphaned scrolls consume cluster resources

4. Consider search_after for Real-Time

For user-facing search features, use search_after instead of scroll — it doesn’t create a snapshot and has lower overhead.

Important Caveats

1. Not real-time: Scroll queries work on a snapshot. Updates made after the initial query won’t appear.
2. Memory cost: Each scroll context uses memory. Too many concurrent scrolls can degrade cluster performance.
3. Not for user requests: Scroll is designed for background batch jobs, not real-time search.
4. Sorting: For best performance with scroll, use _doc sorting.

Summary

The Elasticsearch Scroll API is an essential tool for batch data processing at scale. Key takeaways:

• Set a reasonable scroll TTL
• Choose an appropriate batch size
• Always clean up after yourself
• Use search_after for real-time search instead

This guide should help you handle large-scale data exports, migrations, and analysis with confidence!

References:

• Elasticsearch Scroll API Documentation
• Elasticsearch Performance Tuning