> ## Documentation Index > Fetch the complete documentation index at: https://docs.lancedb.com/llms.txt > Use this file to discover all available pages before exploring further. # Scalar Indexes > Learn how to use scalar indexes in LanceDB for efficient metadata filtering and query optimization. Scalar indexes organize data by scalar attributes (e.g., numbers, categories) and enable fast filtering of vector data. They accelerate retrieval of scalar data associated with vectors, thus enhancing query performance. LanceDB supports four types of scalar indexes: * `BTREE`: Stores column data in sorted order for binary search. Best for columns with many unique values. * `BITMAP`: Uses bitmaps to track value presence. Ideal for columns with few unique values (e.g., categories, tags). * `LABEL_LIST`: Special index for `List` and `LargeList` columns of primitive values supporting `array_contains_all` and `array_contains_any` queries. * `FM`: FM-Index over string or binary columns that accelerates substring search via `contains(col, 'needle')`. > [!TIP] > > Scalar indices serve as a foundational optimization layer, accelerating filtering across diverse search workloads. They can be combined with: > > * Vector search (prefilter results using metadata) > * Full-text search (combining keyword matching with structured filters) > * SQL scans (optimizing WHERE clauses on scalar columns) > * Key-value lookups (enabling rapid primary key-based retrievals) ## Choosing the Right Index Type | Data Type | Filter | Index Type | | :- | :- | :- | | Numeric, String, Temporal | `<`, `=`, `>`, `in`, `between`, `is null` | `BTREE` | | Boolean, numbers or strings with fewer than 1,000 unique values | `<`, `=`, `>`, `in`, `between`, `is null` | `BITMAP` | | List of low cardinality of numbers or strings | `array_has_any`, `array_has_all` | `LABEL_LIST` | | String or binary (`Utf8`, `LargeUtf8`, `Binary`, `LargeBinary`) | `contains(col, 'needle')` | `FM` | ## Scalar Index Operations ### 1. Build the Index You can create multiple scalar indexes within a table. By default, the index will be `BTREE`, but you can always configure another type like `BITMAP` tbl = db.open\_table("scalar\_index\_build") tbl.create\_scalar\_index("book\_id") tbl.create\_scalar\_index("publisher", index\_type="BITMAP") > [!NOTE] > > If you are using LanceDB Enterprise, the `create_scalar_index` API returns immediately, but the building of the scalar index is asynchronous. To wait until all data is fully indexed, you can specify the `wait_timeout` parameter on `create_scalar_index()` or call `wait_for_index()` on the table. ### 2. Check Index Status index\_name = "label\_idx" table.wait\_for\_index(\[index\_name]) `wait_for_index(...)` waits until the named scalar indexes exist and `index_stats(...)` reports `num_unindexed_rows == 0`. If a table is receiving steady writes, that fully indexed state may not stabilize before the timeout. ### 3. Update the Index Updating the table data (adding, deleting, or modifying records) requires that you also update the scalar index. This can be done by calling `optimize`, which will trigger an update to the existing scalar index. table.add(\[\{"vector": \[7, 8], "book\_id": 4}]) table.optimize() > [!NOTE] > > New data added after creating the scalar index will still appear in search results if optimize is not used, but with increased latency due to a flat search on the unindexed portion. LanceDB Enterprise automates the optimize process, minimizing the impact on search speed. ### 4. Run Indexed Searches The following scan will be faster if the column `book_id` has a scalar index: table = db.open\_table("books") result = table.search().where("book\_id = 2").limit(10).to\_pandas() Scalar indexes can also speed up scans containing a vector search or full text search, and a prefilter: table = db.open\_table("book\_with\_embeddings") table.search(\[1.2] \* 2).where("book\_id != 3").limit(10).to\_pandas() ## Indexing nested fields Scalar indexes can target a scalar field inside a struct by passing its full dotted path. The path is preserved end to end: it's the value you pass to `create_scalar_index`, it's what `list_indices()` reports under `columns`, and it's the column reference you use in filter predicates. ```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} # Schema: pa.struct([pa.field("user_id", pa.int32())]) stored under the `metadata` column. table.create_scalar_index("metadata.user_id", name="metadata_user_id_idx") # The same dotted path works in WHERE clauses. table.search().where("metadata.user_id = 42").limit(1).to_list() ``` > [!NOTE] > > Nested paths follow Lance field-path semantics: dot-separate each struct field from root to leaf (for example, `metadata.author.name`). The same convention applies to FTS and vector indexes. ## FM-Index for substring search The `FM` index is a scalar index built over string or binary columns that accelerates substring lookups expressed as `contains(col, 'needle')`. Unlike the tokenized [FTS index](https://docs.lancedb.com/indexing/fts-index.md), which matches whole words after tokenization, the FM-Index matches arbitrary substrings of the raw bytes — so it works well for URLs, file paths, identifiers, log lines, or any column where you search for a fragment rather than a word. Use the FM-Index when: * Filters use `contains(col, 'needle')` (substring), not equality or word search. * The column is `Utf8`, `LargeUtf8`, `Binary`, or `LargeBinary`. * You want substring matches without paying for tokenization, language analysis, or BM25 scoring. Pick `FTS` instead when you need word-level relevance ranking, phrase queries, or language-aware tokenization. ### Create an FM-Index Build an FM-Index with the async `create_index` API by passing the `Fm` config in Python or `Index.fm()` in TypeScript. In Rust, use `Index::Fm(FmIndexBuilder::default())`. ```python Python icon="python" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} from lancedb.index import Fm await tbl.create_index("text", config=Fm()) ``` ```typescript TypeScript icon="square-js" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} import * as lancedb from "@lancedb/lancedb"; import { Index } from "@lancedb/lancedb"; await tbl.createIndex("text", { config: Index.fm() }); ``` ```rust Rust icon="rust" theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} use lancedb::index::Index; use lancedb::index::scalar::FmIndexBuilder; tbl.create_index(&["text"], Index::Fm(FmIndexBuilder::default())) .execute() .await?; ``` After the index is built, substring filters use it automatically: ```python theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}} table.search().where("contains(text, 'needle')").limit(10).to_pandas() ``` `list_indices()` reports the index type as `"Fm"`. ## Index UUID Columns LanceDB supports scalar indexes on UUID columns (stored as `FixedSizeBinary(16)`), enabling efficient lookups and filtering on UUID-based primary keys. > [!NOTE] > > **To use `FixedSizeBinary`, ensure you have:** > > * Python SDK version `0.22.0` or later > * TypeScript SDK version `0.19.0` or later ### 1. Define UUID Type import pyarrow as pa ### 2. Generate UUID Data def generate\_random\_names(): base\_names = \["Alice", "Bob", "Carla", "David", "Eve", "Frank", "Grace"] letter = random.choice(string.ascii\_uppercase) return f"\{random.choice(base\_names)} \{letter}." def generate\_uuids(num\_items): return \[uuid.uuid4().bytes for \_ in range(num\_items)] \# Generate some UUIDs and random names n = 7 uuids = generate\_uuids(n) names = \[generate\_random\_names() for \_ in range(n)] ### 3. Create Table with UUID Column table\_name = "index-on-uuid" uuid\_array = pa.array(uuids, pa.uuid()) name\_array = pa.array(names, pa.string()) schema = pa.schema( \[ pa.field("id", pa.uuid()), pa.field("name", pa.string()), ] ) data\_table = pa.Table.from\_arrays(\[uuid\_array, name\_array], schema=schema) table = db.create\_table(table\_name, data=data\_table, mode="overwrite") ### 4. Create and Wait for the Index index\_name = "id\_idx" table.create\_scalar\_index("id") table.wait\_for\_index(\[index\_name]) ### 5. Perform Operations with the UUID Index new\_users = \[ \{"id": uuid.uuid4().bytes, "name": "Hannah D."}, \{"id": uuid.uuid4().bytes, "name": "Ian B."}, ] \# Insert or update using the UUID index table.merge\_insert( "id" ).when\_matched\_update\_all().when\_not\_matched\_insert\_all().execute(new\_users) ## Index nested fields You can build a scalar index on a field inside a struct column by passing the canonical dot-separated path to `create_index`. This is useful when filters target attributes nested under a `metadata`-style column, for example `metadata.user_id` or `metadata.event.type`. If a literal segment of the path itself contains a dot (for example a column named `user.id` nested inside `metadata`), wrap that segment in backticks so LanceDB can tell the dot apart from the path separator: `` metadata.`user.id` ``. `list_indices()` echoes the same canonical path back, so the column you pass in round-trips through index metadata regardless of nesting depth or escaping. import pyarrow as pa from lancedb.index import BTree metadata\_type = pa.struct( \[ pa.field("user\_id", pa.int32()), pa.field("user.id", pa.int32()), ] ) data = pa.Table.from\_arrays( \[ pa.array(\[1, 2, 3], type=pa.int32()), pa.array( \[ \{"user\_id": 10, "user.id": 100}, \{"user\_id": 20, "user.id": 200}, \{"user\_id": 30, "user.id": 300}, ], type=metadata\_type, ), ], names=\["user\_id", "metadata"], ) table = await db.create\_table("nested\_scalar\_index", data) \# Index a nested struct field. await table.create\_index( "metadata.user\_id", config=BTree(), name="nested\_user\_id\_idx" ) \# Escape literal dots inside a segment with backticks. await table.create\_index( "metadata.\`user.id\`", config=BTree(), name="escaped\_user\_id\_idx" ) \# \`columns\` is returned as the canonical path you passed in. for index in await table.list\_indices(): print(index.name, index.columns) \# nested\_user\_id\_idx \['metadata.user\_id'] \# escaped\_user\_id\_idx \['metadata.\`user.id\`'] > [!NOTE] > > Composite indexes that cover multiple columns aren't supported yet. Each > `create_index` call must target a single (possibly nested) field path. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.