> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lancedb.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Ingesting Data

> Learn about different methods to ingest data into tables in LanceDB, including from various data sources and empty tables.

In LanceDB, tables store records with a defined schema that specifies column names and types. Across the SDKs, you can create tables from row-oriented data and Apache Arrow data structures. The Python SDK additionally supports:

* PyArrow schemas for explicit schema control
* `LanceModel` for Pydantic-based validation

## Create a table with data

Initialize a LanceDB connection and create a table

import lancedb

uri = "data/sample-lancedb"
db = lancedb.connect(uri)


import \* as lancedb from "@lancedb/lancedb";

async function connectExample(uri: string) \{
&#x20; const db = await lancedb.connect(uri);
&#x20; return db;
}


async fn connect\_example(uri: \&str) \{
&#x20;   let db = connect(uri).execute().await.unwrap();
&#x20;   let \_ = db;
}


Depending on the SDK, LanceDB can ingest arrays of records, Arrow tables or record batches, and Arrow batch iterators or readers. Let's take a look at some of the common patterns.

### From list of objects

You can provide a list of objects to create a table. The Python and TypeScript SDKs
support lists/arrays of dictionaries, while the Rust SDK supports lists of structs.
In Python, pass a list or other batch-like object; a single bare `dict` or single
`LanceModel` is rejected.

data = \[
&#x20;   \{"vector": \[1.1, 1.2], "lat": 45.5, "long": -122.7},
&#x20;   \{"vector": \[0.2, 1.8], "lat": 40.1, "long": -74.1},
]
db = tmp\_db
db.create\_table("test\_table", data, mode="overwrite")
tbl = db\["test\_table"]
tbl.head()


type Location = \{
&#x20; vector: number\[];
&#x20; lat: number;
&#x20; long: number;
};

const data: Location\[] = \[
&#x20; \{ vector: \[1.1, 1.2], lat: 45.5, long: -122.7 },
&#x20; \{ vector: \[0.2, 1.8], lat: 40.1, long: -74.1 },
];
const table = await db.createTable("test\_table", data, \{
&#x20; mode: "overwrite",
});


struct Location \{
&#x20;   vector: \[f32; 2],
&#x20;   lat: f32,
&#x20;   long: f32,
}

let data = vec!\[
&#x20;   Location \{
&#x20;       vector: \[1.1, 1.2],
&#x20;       lat: 45.5,
&#x20;       long: -122.7,
&#x20;   },
&#x20;   Location \{
&#x20;       vector: \[0.2, 1.8],
&#x20;       lat: 40.1,
&#x20;       long: -74.1,
&#x20;   },
];

let schema = Arc::new(Schema::new(vec!\[
&#x20;   Field::new(
&#x20;       "vector",
&#x20;       DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), 2),
&#x20;       false,
&#x20;   ),
&#x20;   Field::new("lat", DataType::Float32, false),
&#x20;   Field::new("long", DataType::Float32, false),
]));

let batch = RecordBatch::try\_new(
&#x20;   schema.clone(),
&#x20;   vec!\[
&#x20;       Arc::new(
&#x20;           FixedSizeListArray::from\_iter\_primitive::\<Float32Type, \_, \_>(
&#x20;               data.iter()
&#x20;                   .map(|row| Some(row\.vector.iter().copied().map(Some).collect::\<Vec\<\_>>())),
&#x20;               2,
&#x20;           ),
&#x20;       ),
&#x20;       Arc::new(Float32Array::from\_iter\_values(
&#x20;           data.iter().map(|row| row\.lat),
&#x20;       )),
&#x20;       Arc::new(Float32Array::from\_iter\_values(
&#x20;           data.iter().map(|row| row\.long),
&#x20;       )),
&#x20;   ],
)
.unwrap();
let reader: Box\<dyn RecordBatchReader + Send> =
&#x20;   Box::new(RecordBatchIterator::new(vec!\[Ok(batch)].into\_iter(), schema.clone()));
let table = db
&#x20;   .create\_table("test\_table", reader)
&#x20;   .mode(CreateTableMode::Overwrite)
&#x20;   .execute()
&#x20;   .await
&#x20;   .unwrap();


### Handle existing tables

By default, `create_table` raises an error if a table with the same name already exists.
You can change this behavior with two parameters that resolve the conflict in different ways:

* **Idempotent open**: return the existing table without modifying it. Use when your
  code may run more than once (notebooks, retries, init scripts) and you want to reuse
  the table on subsequent runs. The provided data is ignored, but the schema is
  validated against the existing table and a mismatch raises an error.
* **Overwrite**: drop the existing table and create a new one with the provided data.
  Use this for test fixtures or when you intentionally want to replace prior contents.
  This permanently discards the old table's data.

\# Idempotent open: reuse the existing table if it exists.
\# The provided data is ignored; the schema is validated against the
\# existing table and a mismatch raises an error.
tbl = db.create\_table("conflict\_table", data, exist\_ok=True)

\# Overwrite: drop the existing table and create a new one with the
\# provided data. This permanently discards the old table's data.
tbl = db.create\_table("conflict\_table", data, mode="overwrite")


// Idempotent open: reuse the existing table if it exists.
// The provided data is ignored; the schema is validated against the
// existing table and a mismatch raises an error.
let conflictTable = await db.createTable("conflict\_table", data, \{
&#x20; existOk: true,
});

// Overwrite: drop the existing table and create a new one with the
// provided data. This permanently discards the old table's data.
conflictTable = await db.createTable("conflict\_table", data, \{
&#x20; mode: "overwrite",
});


// Idempotent open: reuse the existing table if it exists.
// The provided data is ignored; the schema is validated against the
// existing table and a mismatch raises an error.
let \_conflict\_table = db
&#x20;   .create\_table("conflict\_table", exist\_ok\_reader)
&#x20;   .mode(CreateTableMode::exist\_ok(|req| req))
&#x20;   .execute()
&#x20;   .await
&#x20;   .unwrap();

// Overwrite: drop the existing table and create a new one with the
// provided data. This permanently discards the old table's data.
let conflict\_table = db
&#x20;   .create\_table("conflict\_table", overwrite\_reader)
&#x20;   .mode(CreateTableMode::Overwrite)
&#x20;   .execute()
&#x20;   .await
&#x20;   .unwrap();


> [!NOTE]
>
> `exist_ok` / `existOk` does not append the provided data to an existing table. Use
> [`table.add()`](https://docs.lancedb.com/tables/update) for that. If you need to ensure a table exists *and*
> contains specific rows, prefer the [empty-table-then-add pattern](#create-empty-table).

### From a custom schema

You can define a custom Arrow schema for the table. This is useful when you want to have more control over the column types and metadata.

import pyarrow as pa

custom\_schema = pa.schema(
&#x20;   \[
&#x20;       pa.field("vector", pa.list\_(pa.float32(), 4)),
&#x20;       pa.field("lat", pa.float32()),
&#x20;       pa.field("long", pa.float32()),
&#x20;   ]
)

data = \[
&#x20;   \{"vector": \[1.1, 1.2, 1.3, 1.4], "lat": 45.5, "long": -122.7},
&#x20;   \{"vector": \[0.2, 1.8, 0.4, 3.6], "lat": 40.1, "long": -74.1},
]
db = tmp\_db
tbl = db.create\_table(
&#x20;   "my\_table\_custom\_schema", data, schema=custom\_schema, mode="overwrite"
)


const customSchema = new arrow\.Schema(\[
&#x20; new arrow\.Field(
&#x20;   "vector",
&#x20;   new arrow\.FixedSizeList(
&#x20;     4,
&#x20;     new arrow\.Field("item", new arrow\.Float32(), true),
&#x20;   ),
&#x20; ),
&#x20; new arrow\.Field("lat", new arrow\.Float32()),
&#x20; new arrow\.Field("long", new arrow\.Float32()),
]);

const customSchemaData = lancedb.makeArrowTable(
&#x20; \[
&#x20;   \{ vector: \[1.1, 1.2, 1.3, 1.4], lat: 45.5, long: -122.7 },
&#x20;   \{ vector: \[0.2, 1.8, 0.4, 3.6], lat: 40.1, long: -74.1 },
&#x20; ],
&#x20; \{ schema: customSchema },
);
const customSchemaTable = await db.createTable(
&#x20; "my\_table\_custom\_schema",
&#x20; customSchemaData,
&#x20; \{ mode: "overwrite" },
);


let custom\_schema = Arc::new(Schema::new(vec!\[
&#x20;   Field::new(
&#x20;       "vector",
&#x20;       DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), 4),
&#x20;       false,
&#x20;   ),
&#x20;   Field::new("lat", DataType::Float32, false),
&#x20;   Field::new("long", DataType::Float32, false),
]));

let custom\_batch = RecordBatch::try\_new(
&#x20;   custom\_schema.clone(),
&#x20;   vec!\[
&#x20;       Arc::new(
&#x20;           FixedSizeListArray::from\_iter\_primitive::\<Float32Type, \_, \_>(
&#x20;               vec!\[
&#x20;                   Some(vec!\[Some(1.1), Some(1.2), Some(1.3), Some(1.4)]),
&#x20;                   Some(vec!\[Some(0.2), Some(1.8), Some(0.4), Some(3.6)]),
&#x20;               ],
&#x20;               4,
&#x20;           ),
&#x20;       ),
&#x20;       Arc::new(Float32Array::from(vec!\[45.5, 40.1])),
&#x20;       Arc::new(Float32Array::from(vec!\[-122.7, -74.1])),
&#x20;   ],
)
.unwrap();
let custom\_reader: Box\<dyn RecordBatchReader + Send> =
&#x20;   Box::new(RecordBatchIterator::new(vec!\[Ok(custom\_batch)].into\_iter(), custom\_schema.clone()));
let custom\_table = db
&#x20;   .create\_table("my\_table\_custom\_schema", custom\_reader)
&#x20;   .mode(CreateTableMode::Overwrite)
&#x20;   .execute()
&#x20;   .await
&#x20;   .unwrap();


An explicit schema is also where you control nullability. If later writes omit a
non-nullable column, or provide actual nulls for it, ingestion fails; nullable columns can be
omitted or written with null values. Without an explicit schema, Python infers list-like vector
values as fixed-size `float32` vector fields from the observed dimension.

For Python ingest, malformed vector values fail by default. If you expect occasional wrong-length,
null, or NaN vectors, choose an `on_bad_vectors` policy: `"drop"` removes those rows, `"fill"` writes
`fill_value`, and `"null"` writes nulls.

### From an Arrow Table

You can also create LanceDB tables directly from Arrow tables.
Rust uses an Arrow `RecordBatchReader` for the same Arrow-native ingest flow.

import numpy as np
import pyarrow as pa

dim = 16
total = 2
schema = pa.schema(
&#x20;   \[pa.field("vector", pa.list\_(pa.float16(), dim)), pa.field("text", pa.string())]
)
data = pa.Table.from\_arrays(
&#x20;   \[
&#x20;       pa.array(
&#x20;           \[np.random.randn(dim).astype(np.float16) for \_ in range(total)],
&#x20;           pa.list\_(pa.float16(), dim),
&#x20;       ),
&#x20;       pa.array(\["foo", "bar"]),
&#x20;   ],
&#x20;   \["vector", "text"],
)
db = tmp\_db
tbl = db.create\_table("f16\_tbl", data, schema=schema, mode="overwrite")


const arrowSchema = new arrow\.Schema(\[
&#x20; new arrow\.Field(
&#x20;   "vector",
&#x20;   new arrow\.FixedSizeList(
&#x20;     16,
&#x20;     new arrow\.Field("item", new arrow\.Float32(), true),
&#x20;   ),
&#x20; ),
&#x20; new arrow\.Field("text", new arrow\.Utf8()),
]);
const arrowData = lancedb.makeArrowTable(
&#x20; \[
&#x20;   \{ vector: Array(16).fill(0.1), text: "foo" },
&#x20;   \{ vector: Array(16).fill(0.2), text: "bar" },
&#x20; ],
&#x20; \{ schema: arrowSchema },
);
const arrowTable = await db.createTable("f32\_tbl", arrowData, \{
&#x20; mode: "overwrite",
});


let arrow\_schema = Arc::new(Schema::new(vec!\[
&#x20;   Field::new(
&#x20;       "vector",
&#x20;       DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), 16),
&#x20;       false,
&#x20;   ),
&#x20;   Field::new("text", DataType::Utf8, false),
]));

let arrow\_batch = RecordBatch::try\_new(
&#x20;   arrow\_schema.clone(),
&#x20;   vec!\[
&#x20;       Arc::new(
&#x20;           FixedSizeListArray::from\_iter\_primitive::\<Float32Type, \_, \_>(
&#x20;               vec!\[Some(vec!\[Some(0.1); 16]), Some(vec!\[Some(0.2); 16])],
&#x20;               16,
&#x20;           ),
&#x20;       ),
&#x20;       Arc::new(StringArray::from(vec!\["foo", "bar"])),
&#x20;   ],
)
.unwrap();
let arrow\_reader: Box\<dyn RecordBatchReader + Send> =
&#x20;   Box::new(RecordBatchIterator::new(vec!\[Ok(arrow\_batch)].into\_iter(), arrow\_schema.clone()));
let arrow\_table = db
&#x20;   .create\_table("arrow\_table\_example", arrow\_reader)
&#x20;   .mode(CreateTableMode::Overwrite)
&#x20;   .execute()
&#x20;   .await
&#x20;   .unwrap();


### From a Pandas DataFrame

Python Only

import pandas as pd

data = pd.DataFrame(
&#x20;   \{
&#x20;       "vector": \[\[1.1, 1.2, 1.3, 1.4], \[0.2, 1.8, 0.4, 3.6]],
&#x20;       "lat": \[45.5, 40.1],
&#x20;       "long": \[-122.7, -74.1],
&#x20;   }
)
db = tmp\_db
db.create\_table("my\_table\_pandas", data, mode="overwrite")
db\["my\_table\_pandas"].head()


> [!NOTE]
>
> Data is converted to Arrow before being written to disk. For maximum control over how data is saved, either provide the PyArrow schema to convert to or else provide a PyArrow Table directly.

> [!NOTE]
>
> The **`vector`** column needs to be a [Vector](https://docs.lancedb.com/integrations/data/pydantic.md#vector-field) (defined as [pyarrow.FixedSizeList](https://arrow.apache.org/docs/python/generated/pyarrow.list_.html)) type.

### From a Polars DataFrame

Python Only

LanceDB supports [Polars](https://pola.rs/), a modern, fast DataFrame library
written in Rust. Just like in Pandas, the Polars integration is enabled by PyArrow
under the hood. A deeper integration between LanceDB Tables and Polars DataFrames
is on the way.

import polars as pl

data = pl.DataFrame(
&#x20;   \{
&#x20;       "vector": \[\[3.1, 4.1], \[5.9, 26.5]],
&#x20;       "item": \["foo", "bar"],
&#x20;       "price": \[10.0, 20.0],
&#x20;   }
)
db = tmp\_db
tbl = db.create\_table("my\_table\_pl", data, mode="overwrite")


### From Pydantic Models

Python Only

When you create an empty table without data, you must specify the table schema.
LanceDB supports creating tables by specifying a PyArrow schema or a specialized
Pydantic model called `LanceModel`.

For example, the following Content model specifies a table with 5 columns:
`movie_id`, `vector`, `genres`, `title`, and `imdb_id`. When you create a table, you can
pass the class as the value of the `schema` parameter to `create_table`.
The `vector` column is a `Vector` type, which is a specialized Pydantic type that
can be configured with the vector dimensions. It is also important to note that
LanceDB only understands subclasses of `lancedb.pydantic.LanceModel`
(which itself derives from `pydantic.BaseModel`).

from lancedb.pydantic import LanceModel, Vector

class Content(LanceModel):
&#x20;   movie\_id: int
&#x20;   vector: Vector(128)
&#x20;   genres: str
&#x20;   title: str
&#x20;   imdb\_id: int

&#x20;   @property
&#x20;   def imdb\_url(self) -> str:
&#x20;       return f"https\://www\.imdb.com/title/tt\{self.imdb\_id}"

db = tmp\_db
tbl = db.create\_table("movielens\_small", schema=Content, mode="overwrite")


#### Nested schemas

Sometimes your data model may contain nested objects. For example, you may want to store the document string and the document source name as a nested Document object:

from pydantic import BaseModel

class Document(BaseModel):
&#x20;   content: str
&#x20;   source: str


This can be used as the type of a LanceDB table column:

from lancedb.pydantic import LanceModel, Vector

\# --8\<-- \[start:tables\_document\_model]
from pydantic import BaseModel

class Document(BaseModel):
&#x20;   content: str
&#x20;   source: str

\# --8\<-- \[end:tables\_document\_model]

class NestedSchema(LanceModel):
&#x20;   id: str
&#x20;   vector: Vector(1536)
&#x20;   document: Document

db = tmp\_db
tbl = db.create\_table("nested\_table", schema=NestedSchema, mode="overwrite")


This creates a struct column called "document" that has two subfields
called "content" and "source":

```bash theme={"theme":{"light":"vitesse-light","dark":"catppuccin-mocha"}}
In [28]: tbl.schema
Out[28]:
id: string not null
vector: fixed_size_list<item: float>[1536] not null
    child 0, item: float
document: struct<content: string not null, source: string not null> not null
    child 0, content: string not null
    child 1, source: string not null
```

#### Validators

Because `LanceModel` inherits from Pydantic's `BaseModel`, you can combine them with Pydantic's
[field validators](https://docs.pydantic.dev/latest/concepts/validators). The example
below shows how to add a validator to ensure that only valid timezone-aware datetime objects are used
for a `created_at` field.

from datetime import datetime
from zoneinfo import ZoneInfo

from lancedb.pydantic import LanceModel
from pydantic import Field, ValidationError, ValidationInfo, field\_validator

tzname = "America/New\_York"
tz = ZoneInfo(tzname)

class TestModel(LanceModel):
&#x20;   dt\_with\_tz: datetime = Field(json\_schema\_extra=\{"tz": tzname})

&#x20;   @field\_validator("dt\_with\_tz")
&#x20;   @classmethod
&#x20;   def tz\_must\_match(cls, dt: datetime) -> datetime:
&#x20;       assert dt.tzinfo == tz
&#x20;       return dt

ok = TestModel(dt\_with\_tz=datetime.now(tz))

try:
&#x20;   TestModel(dt\_with\_tz=datetime.now(ZoneInfo("Asia/Shanghai")))
&#x20;   assert 0 == 1, "this should raise ValidationError"
except ValidationError:
&#x20;   print("A ValidationError was raised.")
&#x20;   pass


When you run this code it, should raise the `ValidationError`.

### Loading Large Datasets

When ingesting large datasets, use `table.add()` on an existing table rather than
passing all data to `create_table()`. The `add()` method auto-parallelizes large
writes, while `create_table(name, data)` does not.

> [!TIP]
>
> For best performance with large datasets, create an empty table first and then call
> `table.add()`. This enables automatic write parallelism for materialized data sources.

#### From files (Parquet, CSV, etc.)

Python Only

For file-based data, pass a `pyarrow.dataset.Dataset` to `table.add()`. This streams
data from disk without loading the entire dataset into memory.

import pyarrow\.dataset as ds

dataset = ds.dataset(data\_path, format="parquet")
db = tmp\_db
table = db.create\_table("my\_table", schema=dataset.schema, mode="overwrite")
table.add(dataset)


> [!NOTE]
>
> `pa.dataset()` input is currently Python-only. TypeScript and Rust support for
> file-based dataset ingestion is tracked in
> [lancedb#3173](https://github.com/lancedb/lancedb/issues/3173).

#### From iterators (custom batch generation)

When you need custom batch logic — generating embeddings on the fly, transforming
rows from an external source, etc. — use an iterator of `RecordBatch` objects.

import pyarrow as pa

schema = pa.schema(
&#x20;   \[
&#x20;       pa.field("vector", pa.list\_(pa.float32(), 4)),
&#x20;       pa.field("item", pa.utf8()),
&#x20;       pa.field("price", pa.float32()),
&#x20;   ]
)

def make\_batches():
&#x20;   for i in range(5):
&#x20;       yield pa.RecordBatch.from\_arrays(
&#x20;           \[
&#x20;               pa.array(
&#x20;                   \[\[3.1, 4.1, 5.1, 6.1], \[5.9, 26.5, 4.7, 32.8]],
&#x20;                   pa.list\_(pa.float32(), 4),
&#x20;               ),
&#x20;               pa.array(\["foo", "bar"]),
&#x20;               pa.array(\[10.0, 20.0]),
&#x20;           ],
&#x20;           \["vector", "item", "price"],
&#x20;       )

db = tmp\_db
db.create\_table("batched\_table", make\_batches(), schema=schema, mode="overwrite")


const batchSchema = new arrow\.Schema(\[
&#x20; new arrow\.Field(
&#x20;   "vector",
&#x20;   new arrow\.FixedSizeList(
&#x20;     4,
&#x20;     new arrow\.Field("item", new arrow\.Float32(), true),
&#x20;   ),
&#x20; ),
&#x20; new arrow\.Field("item", new arrow\.Utf8()),
&#x20; new arrow\.Field("price", new arrow\.Float32()),
]);

const tableForBatches = await db.createEmptyTable(
&#x20; "batched\_table",
&#x20; batchSchema,
&#x20; \{
&#x20;   mode: "overwrite",
&#x20; },
);

const rows = Array.from(\{ length: 10 }, (\_, i) => (\{
&#x20; vector: \[i + 0.1, i + 0.2, i + 0.3, i + 0.4],
&#x20; item: \`item-\$\{i + 1}\`,
&#x20; price: (i + 1) \* 10,
}));

const chunkSize = 2;
for (let i = 0; i \< rows.length; i += chunkSize) \{
&#x20; const batch = lancedb.makeArrowTable(rows.slice(i, i + chunkSize), \{
&#x20;   schema: batchSchema,
&#x20; });
&#x20; await tableForBatches.add(batch);
}


let batch\_schema = Arc::new(Schema::new(vec!\[
&#x20;   Field::new(
&#x20;       "vector",
&#x20;       DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), 4),
&#x20;       false,
&#x20;   ),
&#x20;   Field::new("item", DataType::Utf8, false),
&#x20;   Field::new("price", DataType::Float32, false),
]));

let batches = (0..5)
&#x20;   .map(|i| \{
&#x20;       RecordBatch::try\_new(
&#x20;           batch\_schema.clone(),
&#x20;           vec!\[
&#x20;               Arc::new(
&#x20;                   FixedSizeListArray::from\_iter\_primitive::\<Float32Type, \_, \_>(
&#x20;                       vec!\[
&#x20;                           Some(vec!\[Some(3.1 + i as f32), Some(4.1), Some(5.1), Some(6.1)]),
&#x20;                           Some(vec!\[
&#x20;                               Some(5.9),
&#x20;                               Some(26.5 + i as f32),
&#x20;                               Some(4.7),
&#x20;                               Some(32.8),
&#x20;                           ]),
&#x20;                       ],
&#x20;                       4,
&#x20;                   ),
&#x20;               ),
&#x20;               Arc::new(StringArray::from(vec!\[
&#x20;                   format!("item\{}", i \* 2 + 1),
&#x20;                   format!("item\{}", i \* 2 + 2),
&#x20;               ])),
&#x20;               Arc::new(Float32Array::from(vec!\[
&#x20;                   ((i \* 2 + 1) \* 10) as f32,
&#x20;                   ((i \* 2 + 2) \* 10) as f32,
&#x20;               ])),
&#x20;           ],
&#x20;       )
&#x20;       .unwrap()
&#x20;   })
&#x20;   .collect::\<Vec\<\_>>();

let batch\_reader: Box\<dyn RecordBatchReader + Send> =
&#x20;   Box::new(RecordBatchIterator::new(batches.into\_iter().map(Ok), batch\_schema.clone()));
let batch\_table = db
&#x20;   .create\_table("batched\_table", batch\_reader)
&#x20;   .mode(CreateTableMode::Overwrite)
&#x20;   .execute()
&#x20;   .await
&#x20;   .unwrap();


Use this pattern when:

* Your source data already arrives in Arrow batches, readers, datasets, or streams.
* Materializing the entire ingest as one giant in-memory list or array would be too expensive.
* You want to control chunk size explicitly during ingestion.

Python can also consume iterators of other supported types like Pandas DataFrames or Python lists.

#### Write parallelism

> [!NOTE]
>
> For materialized data (`pa.Table`, `pd.DataFrame`, `pa.dataset()`), LanceDB
> automatically parallelizes large writes — no configuration needed. Auto-parallelism
> targets approximately 1M rows or 2GB per write partition.
>
> For streaming sources (iterators, `RecordBatchReader`), LanceDB cannot determine
> total size upfront. A `parallelism` parameter to control this manually is planned
> but not yet exposed in Python or TypeScript
> ([tracking issue](https://github.com/lancedb/lancedb/issues/3173)).

#### Tracking ingestion progress

TypeScript Only

For long-running writes, pass a `progress` callback to `table.add()` to surface
per-batch progress in your UI, logs, or metrics pipeline. The callback fires
once per batch written and once more with `done: true` when the write completes.

Each invocation receives a `WriteProgress` object:

| Field | Description |
| :- | :- |
| `outputRows` | Rows written so far. |
| `outputBytes` | Bytes written so far. |
| `totalRows` | Expected total rows when the input source reports one. Always set on the final callback. |
| `elapsedSeconds` | Wall-clock seconds since the write started. |
| `activeTasks` | Parallel write tasks currently in flight. |
| `totalTasks` | Total parallel write tasks (the write parallelism). |
| `done` | `true` only for the final callback. |

{TsAddProgress}

A few things to know before you wire this up:

* Back-pressures the writer: callback invocations are serialized and run inline with each batch, so a slow callback will slow the write rather than drop updates. Every batch update is delivered, and the final `done: true` callback always fires (even on error or cancellation). Keep the callback cheap — offload heavy work to a queue you drain elsewhere.
* Errors swallowed: anything your callback throws is logged with `console.warn` and won't abort the write, so keep the callback side-effect-only and don't rely on it for control flow.
* Row totals: `totalRows` is only populated when the input source can report it up front (for example, a materialized `arrow.Table`). For streaming sources it stays `undefined` until the final callback, where it falls back to the actual rows written.

## Create empty table

You can create an empty table for scenarios where you want to add data to the table later.
An example would be when you want to collect data from a stream/external file and then add it to a table in
batches.

An empty table can be initialized via an Arrow schema.

import pyarrow as pa

schema = pa.schema(
&#x20;   \[
&#x20;       pa.field("vector", pa.list\_(pa.float32(), 2)),
&#x20;       pa.field("item", pa.string()),
&#x20;       pa.field("price", pa.float32()),
&#x20;   ]
)
db = tmp\_db
tbl = db.create\_table("test\_empty\_table", schema=schema, mode="overwrite")


const emptySchema = new arrow\.Schema(\[
&#x20; new arrow\.Field(
&#x20;   "vector",
&#x20;   new arrow\.FixedSizeList(
&#x20;     2,
&#x20;     new arrow\.Field("item", new arrow\.Float32(), true),
&#x20;   ),
&#x20; ),
&#x20; new arrow\.Field("item", new arrow\.Utf8()),
&#x20; new arrow\.Field("price", new arrow\.Float32()),
]);
const emptyTable = await db.createEmptyTable(
&#x20; "test\_empty\_table",
&#x20; emptySchema,
&#x20; \{
&#x20;   mode: "overwrite",
&#x20; },
);


let empty\_schema = Arc::new(Schema::new(vec!\[
&#x20;   Field::new(
&#x20;       "vector",
&#x20;       DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), 2),
&#x20;       false,
&#x20;   ),
&#x20;   Field::new("item", DataType::Utf8, false),
&#x20;   Field::new("price", DataType::Float32, false),
]));
let empty\_table = db
&#x20;   .create\_empty\_table("test\_empty\_table", empty\_schema)
&#x20;   .mode(CreateTableMode::Overwrite)
&#x20;   .execute()
&#x20;   .await
&#x20;   .unwrap();


Alternatively, you can also use Pydantic to specify the schema for the empty table. Note that we do not
directly import `pydantic` but instead use `lancedb.pydantic` which is a subclass of `pydantic.BaseModel`
that has been extended to support LanceDB specific types like `Vector`.

from lancedb.pydantic import LanceModel, Vector

class Item(LanceModel):
&#x20;   vector: Vector(2)
&#x20;   item: str
&#x20;   price: float

db = tmp\_db
tbl = db.create\_table(
&#x20;   "test\_empty\_table\_new", schema=Item.to\_arrow\_schema(), mode="overwrite"
)


Once the empty table has been created, you can append to it or modify its contents,
as explained in the [updating and modifying tables](https://docs.lancedb.com/tables/update.md) section.

## Open an existing table

You can open an existing table by specifying the name of the table to the `open_table` / `openTable` method.
If you forget the name of your table, you can always get a listing of all table names.

db = tmp\_db
\# Create a table first
data = \[\{"vector": \[1.1, 1.2], "lat": 45.5, "long": -122.7}]
db.create\_table("test\_table", data, mode="overwrite")

\# List table names
print(db.list\_tables().tables)

\# Open existing table
tbl = db.open\_table("test\_table")


const openTableData = \[\{ vector: \[1.1, 1.2], lat: 45.5, long: -122.7 }];
await db.createTable("test\_table\_open", openTableData, \{
&#x20; mode: "overwrite",
});

console.log(await db.tableNames());

const openedTable = await db.openTable("test\_table\_open");


let open\_schema = Arc::new(Schema::new(vec!\[
&#x20;   Field::new(
&#x20;       "vector",
&#x20;       DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), 2),
&#x20;       false,
&#x20;   ),
&#x20;   Field::new("lat", DataType::Float32, false),
&#x20;   Field::new("long", DataType::Float32, false),
]));
let open\_batch = RecordBatch::try\_new(
&#x20;   open\_schema.clone(),
&#x20;   vec!\[
&#x20;       Arc::new(
&#x20;           FixedSizeListArray::from\_iter\_primitive::\<Float32Type, \_, \_>(
&#x20;               vec!\[Some(vec!\[Some(1.1), Some(1.2)])],
&#x20;               2,
&#x20;           ),
&#x20;       ),
&#x20;       Arc::new(Float32Array::from(vec!\[45.5])),
&#x20;       Arc::new(Float32Array::from(vec!\[-122.7])),
&#x20;   ],
)
.unwrap();
let open\_reader: Box\<dyn RecordBatchReader + Send> =
&#x20;   Box::new(RecordBatchIterator::new(vec!\[Ok(open\_batch)].into\_iter(), open\_schema.clone()));
db.create\_table("test\_table", open\_reader)
&#x20;   .mode(CreateTableMode::Overwrite)
&#x20;   .execute()
&#x20;   .await
&#x20;   .unwrap();

println!("\{:?}", db.table\_names().execute().await.unwrap());

let opened\_table = db.open\_table("test\_table").execute().await.unwrap();


## Drop a table

Use the `drop_table()` method on the database to remove a table.

db = tmp\_db
\# Create a table first
data = \[\{"vector": \[1.1, 1.2], "lat": 45.5}]
db.create\_table("my\_table", data, mode="overwrite")

\# Drop the table
db.drop\_table("my\_table")


await db.createTable("my\_table", \[\{ vector: \[1.1, 1.2], lat: 45.5 }], \{
&#x20; mode: "overwrite",
});

await db.dropTable("my\_table");


let drop\_schema = Arc::new(Schema::new(vec!\[
&#x20;   Field::new(
&#x20;       "vector",
&#x20;       DataType::FixedSizeList(Arc::new(Field::new("item", DataType::Float32, true)), 2),
&#x20;       false,
&#x20;   ),
&#x20;   Field::new("lat", DataType::Float32, false),
]));
let drop\_batch = RecordBatch::try\_new(
&#x20;   drop\_schema.clone(),
&#x20;   vec!\[
&#x20;       Arc::new(
&#x20;           FixedSizeListArray::from\_iter\_primitive::\<Float32Type, \_, \_>(
&#x20;               vec!\[Some(vec!\[Some(1.1), Some(1.2)])],
&#x20;               2,
&#x20;           ),
&#x20;       ),
&#x20;       Arc::new(Float32Array::from(vec!\[45.5])),
&#x20;   ],
)
.unwrap();
let drop\_reader: Box\<dyn RecordBatchReader + Send> =
&#x20;   Box::new(RecordBatchIterator::new(vec!\[Ok(drop\_batch)].into\_iter(), drop\_schema.clone()));
db.create\_table("my\_table", drop\_reader)
&#x20;   .mode(CreateTableMode::Overwrite)
&#x20;   .execute()
&#x20;   .await
&#x20;   .unwrap();

db.drop\_table("my\_table", &\[]).await.unwrap();


This permanently removes the table and is not recoverable, unlike deleting rows.
By default, if the table does not exist an exception is raised. To suppress this,
you can pass in `ignore_missing=True`.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.