Elasticsearch Patterns

Quick Guide: Use @elastic/elasticsearch (v8.x/v9.x) as the TypeScript client. Elasticsearch is near real-time -- documents are NOT searchable immediately after indexing; they become visible after a refresh (default: every 1 second on active indices). You MUST define explicit mappings before indexing -- dynamic mapping infers types from the first document, and mismatched types in later documents cause hard failures you cannot fix without reindexing. Use search_after + Point in Time (PIT) for deep pagination -- NOT from/size beyond 10,000 hits and NOT the scroll API (deprecated for search). Use the bulk API or client.helpers.bulk() for any batch operation -- never loop individual index calls.

---

<critical_requirements>

CRITICAL: Before Using This Skill

All code must follow project conventions in CLAUDE.md (kebab-case, named exports, import ordering, import type, named constants)

(You MUST define explicit index mappings BEFORE indexing documents -- dynamic mapping infers types from the first document, and if a later document sends a different type for the same field, indexing fails with a mapper_parsing_exception that CANNOT be fixed without reindexing into a new index)

(You MUST use the bulk API for batch operations -- looping individual client.index() calls is orders of magnitude slower and can overwhelm the cluster with HTTP connections)

(You MUST NOT use from/size pagination beyond 10,000 results -- Elasticsearch throws Result window is too large by default; use search_after + PIT instead)

(You MUST NOT use refresh: true or refresh: "wait_for" in production request handlers -- forcing a refresh on every write degrades cluster performance; let the default 1-second refresh interval handle it)

</critical_requirements>

---

Examples

• Core Patterns -- Client setup, index management, document CRUD, search basics, TypeScript integration
• Aggregations -- Terms, range, date_histogram, nested, pipeline aggregations
• Vector Search -- Dense vector fields, kNN queries, hybrid search, similarity metrics
• Pagination -- from/size, search_after, Point in Time, scroll helpers
• Bulk Operations -- Bulk API, bulk helper, reindexing patterns

Additional resources:

• reference.md -- Search DSL cheat sheet, mapping types, aggregation reference, decision frameworks

---

Auto-detection: Elasticsearch, elasticsearch, @elastic/elasticsearch, client.search, client.index, client.bulk, client.indices.create, client.indices.putMapping, dense_vector, knn, search_after, point in time, openPIT, aggregations, aggs, bool query, match query, term query, multi_match, nested query, range query, client.helpers.bulk, client.helpers.scrollSearch, BulkResponse, SearchResponse, MappingProperty

When to use:

• Full-text search with advanced relevance tuning (BM25, custom analyzers, boosting)
• Aggregations and analytics (terms, histograms, pipeline aggregations)
• Vector/semantic search with kNN on dense_vector fields
• Log and event data search with time-based queries
• Complex structured queries combining bool, nested, range, and geo filters
• Search across large datasets requiring deep pagination (search_after + PIT)

Key patterns covered:

• Client initialization and connection management
• Index management with explicit mappings and settings
• Document CRUD (index, get, update, delete)
• Search DSL (match, term, bool, range, nested, multi_match)
• Aggregations (terms, range, date_histogram, nested, pipeline)
• Full-text analysis (custom analyzers, tokenizers, filters)
• Vector search (dense_vector, kNN, hybrid text+vector)
• Bulk operations and reindexing
• Deep pagination (search_after + PIT)

When NOT to use:

• Simple keyword search on small datasets (client-side filtering or database LIKE queries are simpler)
• Primary data store (Elasticsearch is a search engine, not a database -- always have a source of truth elsewhere)
• Strong consistency requirements (Elasticsearch is eventually consistent by design)
• Simple autocomplete on a small list (a prefix trie or client-side filter is simpler)

---

<philosophy>

Philosophy

Elasticsearch is a distributed search and analytics engine built on Apache Lucene. It excels at full-text search, structured queries, aggregations, and vector search at scale. Core principles:

1. Near real-time, not real-time -- Documents are indexed into segments. A refresh (default: every 1 second on active indices) makes new segments searchable. Do not expect immediate consistency after writes.
2. Mappings are immutable -- Once a field type is set (text, keyword, integer, etc.), it cannot be changed. Wrong types require reindexing into a new index. Always define mappings explicitly before first document.
3. Search engine, not database -- Elasticsearch should not be your source of truth. Always have a primary database and sync to Elasticsearch for search.
4. Bulk everything -- The bulk API amortizes HTTP overhead across thousands of operations. Never loop individual index/update/delete calls.
5. Pagination has limits -- from/size is capped at 10,000 hits by default (index.max_result_window). Deep pagination requires search_after + Point in Time (PIT). The scroll API is deprecated for search use cases.
6. Text vs keyword matters -- text fields are analyzed (tokenized, lowercased) for full-text search. keyword fields are exact-match only. Getting this wrong means either broken search or broken aggregations/filters.

</philosophy>

---

<patterns>

Core Patterns

Pattern 1: Client Setup

Initialize the client with node URL and authentication. The client supports Elastic Cloud, API keys, basic auth, and bearer tokens.

// Good Example -- Typed client setup with environment validation
import { Client } from "@elastic/elasticsearch";

function createElasticsearchClient(): Client {
  const node = process.env.ELASTICSEARCH_URL;
  if (!node) {
    throw new Error("ELASTICSEARCH_URL environment variable is required");
  }

  return new Client({
    node,
    auth: {
      apiKey: process.env.ELASTICSEARCH_API_KEY ?? "",
    },
  });
}

export { createElasticsearchClient };

Why good: Environment variable validation, named export, API key auth (preferred over basic auth in production)

// Bad Example -- Hardcoded credentials
import { Client } from "@elastic/elasticsearch";
const client = new Client({
  node: "http://localhost:9200",
  auth: { username: "elastic", password: "changeme" },
});

Why bad: Hardcoded node URL and credentials leak in version control, basic auth with default password

See examples/core.md for Elastic Cloud setup, health checks, and child clients.

---

Pattern 2: Index with Explicit Mappings

Always define mappings before indexing. Dynamic mapping infers types from the first document -- if wrong, you must reindex.

// Good Example -- Explicit mappings with text + keyword multi-field
const INDEX_NAME = "products";

await client.indices.create({
  index: INDEX_NAME,
  settings: {
    number_of_replicas: 1,
    refresh_interval: "1s",
  },
  mappings: {
    properties: {
      name: {
        type: "text",
        fields: { keyword: { type: "keyword" } },
      },
      description: { type: "text", analyzer: "standard" },
      price: { type: "float" },
      categories: { type: "keyword" },
      inStock: { type: "boolean" },
      createdAt: { type: "date" },
    },
  },
});

Why good: Explicit types prevent mapping conflicts, text + keyword multi-field allows both full-text search and exact-match filtering/aggregation on name

// Bad Example -- No mappings, relying on dynamic mapping
await client.indices.create({ index: "products" });
await client.index({
  index: "products",
  document: { price: "29.99" }, // Oops -- "29.99" is a string, mapped as text
});
// All future numeric price documents will fail with mapper_parsing_exception

Why bad: Dynamic mapping infers price as text from the string "29.99", and this mapping is immutable -- all future documents with numeric price will fail

See examples/core.md for analysis settings, custom analyzers, and mapping migration.

---

Pattern 3: Search with Bool Query

The bool query is the workhorse of Elasticsearch. It combines must, should, must_not, and filter clauses.

// Good Example -- Bool query with filter context for exact matches
const MIN_PRICE = 10;
const MAX_PRICE = 100;

const result = await client.search<Product>({
  index: INDEX_NAME,
  query: {
    bool: {
      must: [{ match: { description: "wireless headphones" } }],
      filter: [
        { range: { price: { gte: MIN_PRICE, lte: MAX_PRICE } } },
        { term: { inStock: true } },
      ],
    },
  },
  size: 20,
});
// result.hits.hits[0]._source is typed as Product | undefined

Why good: filter context for exact matches (no scoring overhead, cacheable), must for full-text relevance scoring, named constants for range values, typed search with generic

// Bad Example -- Everything in must (no filter context)
const result = await client.search({
  index: "products",
  query: {
    bool: {
      must: [
        { match: { description: "wireless headphones" } },
        { range: { price: { gte: 10, lte: 100 } } }, // Wasteful scoring
        { term: { inStock: true } }, // Wasteful scoring
      ],
    },
  },
});

Why bad: Range and term queries in must waste CPU on relevance scoring for yes/no conditions; filter context skips scoring and enables Elasticsearch's filter cache

See examples/core.md for multi_match, nested queries, and function_score.

---

Pattern 4: Aggregations

Aggregations compute analytics over search results. terms for category counts, range for bucketing, date_histogram for time series.

// Good Example -- Terms aggregation with sub-aggregation
const AGGREGATION_SIZE = 50;

const result = await client.search({
  index: INDEX_NAME,
  size: 0, // No hits needed, only aggregations
  aggs: {
    categories: {
      terms: { field: "categories", size: AGGREGATION_SIZE },
      aggs: {
        avgPrice: { avg: { field: "price" } },
      },
    },
  },
});
// result.aggregations?.categories.buckets -> [{ key: "electronics", doc_count: 42, avgPrice: { value: 89.5 } }]

Why good: size: 0 skips hits when only aggregations are needed (faster), nested sub-aggregation for metrics per bucket, named constant for aggregation size

See examples/aggregations.md for date_histogram, range, nested, and pipeline aggregations.

---

Pattern 5: Bulk Operations

The bulk API batches multiple index/update/delete operations in a single request. Use client.helpers.bulk() for the best developer experience.

// Good Example -- Bulk helper with async generator
const result = await client.helpers.bulk<Product>({
  datasource: products,
  onDocument(doc) {
    return { index: { _index: INDEX_NAME, _id: doc.productId } };
  },
  refreshOnCompletion: INDEX_NAME,
});
// result.total, result.successful, result.failed

Why good: Bulk helper handles batching, concurrency, retries, and back-pressure automatically; refreshOnCompletion triggers one refresh at the end instead of per-document

// Bad Example -- Looping individual index calls
for (const product of products) {
  await client.index({
    index: "products",
    document: product,
    refresh: true, // Refresh after EVERY document!
  });
}

Why bad: N individual HTTP requests instead of 1 bulk request, refresh: true on every document causes N segment refreshes (devastating to cluster performance)

See examples/bulk-operations.md for error handling, update operations, and reindexing.

---

Pattern 6: Deep Pagination with search_after + PIT

from/size is limited to 10,000 hits. For deep pagination, use search_after with a Point in Time (PIT) for consistent results.

// Good Example -- search_after with PIT
const PIT_KEEP_ALIVE = "1m";

const pit = await client.openPointInTime({
  index: INDEX_NAME,
  keep_alive: PIT_KEEP_ALIVE,
});

let searchAfter: Array<string | number> | undefined;
let allHits: Product[] = [];

while (true) {
  const result = await client.search<Product>({
    pit: { id: pit.id, keep_alive: PIT_KEEP_ALIVE },
    sort: [{ createdAt: "desc" }, { _id: "asc" }], // Tiebreaker!
    size: 100,
    ...(searchAfter ? { search_after: searchAfter } : {}),
  });

  const hits = result.hits.hits;
  if (hits.length === 0) break;

  allHits = allHits.concat(
    hits.filter((h) => h._source !== undefined).map((h) => h._source!),
  );
  searchAfter = hits[hits.length - 1].sort as Array<string | number>;
}

await client.closePointInTime({ id: pit.id });

Why good: PIT ensures consistent snapshot across pages, tiebreaker _id prevents missing/duplicate documents, keep_alive refreshed on each request

See examples/pagination.md for from/size limits, scroll helpers, and pagination decision framework.

</patterns>

---

<decision_framework>

Decision Framework

Which Query Type?

What kind of search do I need?
-- Full-text relevance search? -> match / multi_match in must
-- Exact value filtering? -> term / terms / range in filter
-- Combining text + filters? -> bool query (must for text, filter for exact)
-- Fuzzy matching? -> match with fuzziness: "AUTO"
-- Phrase matching? -> match_phrase
-- Complex nested objects? -> nested query with path
-- Vector similarity? -> knn with dense_vector field
-- Text + vector hybrid? -> query + knn in same request

Pagination Strategy?

How deep do results go?
-- Under 10,000 total? -> from/size (simplest)
-- Over 10,000 hits? -> search_after + PIT (recommended)
-- Bulk data export? -> client.helpers.scrollSearch() or scrollDocuments()
-- Real-time infinite scroll? -> search_after (no PIT needed for forward-only)

text vs keyword?

What will I do with this field?
-- Full-text search (tokenized, relevance)? -> text
-- Exact match, filtering, aggregations, sorting? -> keyword
-- Both? -> Multi-field: { type: "text", fields: { keyword: { type: "keyword" } } }
-- Neither (just stored, never queried)? -> { type: "keyword", index: false }

Filter vs Must?

Does relevance scoring matter for this clause?
-- YES (affects result order) -> must
-- NO (binary yes/no filter) -> filter (cached, no scoring overhead)
-- Exclude documents -> must_not (in filter context)
-- Boost if present (optional) -> should with minimum_should_match: 0

</decision_framework>

---

<red_flags>

RED FLAGS

High Priority Issues:

• Relying on dynamic mapping without explicit mappings -- wrong type inference causes mapper_parsing_exception that requires reindexing to fix
• Using from/size beyond 10,000 results -- Elasticsearch throws Result window is too large; use search_after + PIT
• Looping individual client.index() calls instead of client.bulk() or client.helpers.bulk() -- orders of magnitude slower, can overwhelm the cluster
• Using refresh: true or refresh: "wait_for" in production request handlers -- forces a segment refresh on every write, degrades cluster performance under load

Medium Priority Issues:

• Putting exact-match conditions (term, range) in must instead of filter -- wastes CPU on scoring, misses filter cache
• Using text type for fields that need exact matching or aggregation -- text fields are analyzed (tokenized), making aggregations return individual tokens instead of full values
• Not including a tiebreaker field in sort when using search_after -- documents with identical sort values may be skipped or duplicated across pages
• Missing _source check -- hit._source can be undefined if _source is disabled or fields are excluded; always handle this

Gotchas & Edge Cases:

• Mapping types are immutable -- once a field is mapped as text, you cannot change it to keyword. The only fix is to create a new index with correct mappings and reindex all documents
• text vs keyword confusion -- text fields are tokenized ("New York" becomes ["new", "york"]). Aggregating on a text field gives you individual tokens, not full values. Use keyword or a .keyword sub-field for aggregations
• match vs term on text fields -- term on a text field often returns no results because term does NOT analyze the query but the field value IS analyzed (e.g., term "New York" won't match the analyzed tokens "new" and "york")
• Near real-time delay -- after indexing, documents are NOT searchable until the next refresh (default: 1 second). Tests that index then immediately search must use refresh: "wait_for" or explicit client.indices.refresh()
• Default index.max_result_window is 10,000 -- increasing this is possible but NOT recommended; deep pagination with from/size holds all skipped results in memory
• Nested objects require nested mapping type -- arrays of objects are flattened by default, losing the association between fields within each object. If you need to query "color: red AND size: large" on the same object in an array, use nested
• Aggregation on _id is disabled by default (8.x+) -- use a separate id field if you need to aggregate by document ID
• _score is null in filter context -- clauses in filter do not contribute to scoring; if you need scoring, use must
• Bulk API partial failures -- a bulk request can succeed overall but have individual failures. Always check result.errors and iterate result.items to find failed operations
• Scroll API is deprecated for search -- use search_after + PIT for deep pagination. Scroll is still valid for one-time data export but consumes cluster resources (open search contexts)
• PIT must be closed -- failing to close Point in Time contexts leaks resources on the cluster; always close in a finally block

</red_flags>

---

<critical_reminders>

CRITICAL REMINDERS

All code must follow project conventions in CLAUDE.md (kebab-case, named exports, import ordering, import type, named constants)

(You MUST define explicit index mappings BEFORE indexing documents -- dynamic mapping infers types from the first document, and if a later document sends a different type for the same field, indexing fails with a mapper_parsing_exception that CANNOT be fixed without reindexing into a new index)

(You MUST use the bulk API for batch operations -- looping individual client.index() calls is orders of magnitude slower and can overwhelm the cluster with HTTP connections)

(You MUST NOT use from/size pagination beyond 10,000 results -- Elasticsearch throws Result window is too large by default; use search_after + PIT instead)

(You MUST NOT use refresh: true or refresh: "wait_for" in production request handlers -- forcing a refresh on every write degrades cluster performance; let the default 1-second refresh interval handle it)

Failure to follow these rules will cause mapping conflicts, pagination failures, cluster performance degradation, and silent data loss.

</critical_reminders>