> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pixeltable.com/llms.txt
> Use this file to discover all available pages before exploring further.
# io
> 
# module pixeltable.io
Functions for importing and exporting Pixeltable data.
## func export\_csv()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
export_csv(
table_or_query: pxt.Table | pxt.Query,
file_path: str | Path,
*,
delimiter: str = ',',
quoting: int = 0
) -> None
```
Exports a query result or table to a CSV file.
Pixeltable column types are mapped to CSV values as follows:
* String, Int, Float, Bool: native CSV representation
* Timestamp, Date: ISO 8601 string representation
* UUID: string representation
* Json: JSON-encoded string
* Array: JSON-encoded string (via `tolist()`)
* Binary: excluded from export (not representable in CSV)
* Image, Video, Audio, Document: file path or URL string
**Parameters:**
* **`table_or_query`** (`pxt.Table | pxt.Query`): Table or Query to export.
* **`file_path`** (`str | Path`): Path to the output CSV file.
* **`delimiter`** (`str`, default: `','`): Field delimiter character. Default `','`.
* **`quoting`** (`int`, default: `0`): CSV quoting style (a `csv.QUOTE_*` constant). Default `csv.QUOTE_MINIMAL`.
## func export\_iceberg()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
export_iceberg(
table_or_query: pxt.Table | pxt.Query,
catalog: Catalog,
table_name: str,
*,
batch_size_bytes: int = 134217728,
if_exists: Literal['error', 'replace', 'append'] = 'error',
schema_overrides: Mapping[str, pa.DataType] | None = None
) -> None
```
Exports a query result or table to an Apache Iceberg table.
Data is streamed into the Iceberg table via pyarrow
[`RecordBatches`](https://arrow.apache.org/docs/python/generated/pyarrow.RecordBatch.html), the size of
which can be controlled with the `batch_size_bytes` parameter.
**Requirements:**
* `pip install pyiceberg`
**Parameters:**
* **`table_or_query`** (`pxt.Table | pxt.Query`): Pixeltable `Table` or `Query` to export.
* **`catalog`** (`Catalog`): An Iceberg `Catalog` instance to write the table into.
* **`table_name`** (`str`): Fully-qualified Iceberg table identifier (e.g. `'my_namespace.my_table'`). If the namespace
does not exist, it will be created.
* **`batch_size_bytes`** (`int`, default: `134217728`): Maximum size in bytes for each in-memory pyarrow batch.
* **`if_exists`** (`Literal['error', 'replace', 'append']`, default: `'error'`): Determines the behavior if the table already exists. Must be one of the following:
* `'error'`: raise an error
* `'replace'`: drop the existing table and create a new one
* `'append'`: append to the existing table (source schema must be compatible)
* **`schema_overrides`** (`Mapping[str, pa.DataType] | None`): If specified, then for each (name, type) pair in `schema_overrides`, the column with
name `name` will be given type `type`, instead of being inferred from the Pixeltable schema. The keys in
`schema_overrides` should be the column names of the Pixeltable schema.
## func export\_images\_as\_fo\_dataset()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
export_images_as_fo_dataset(
tbl: pxt.Table,
images: exprs.Expr,
image_format: str = 'webp',
classifications: exprs.Expr | list[exprs.Expr] | dict[str, exprs.Expr] | None = None,
detections: exprs.Expr | list[exprs.Expr] | dict[str, exprs.Expr] | None = None
) -> fo.Dataset
```
Export images from a Pixeltable table as a Voxel51 dataset. The data must consist of a single column
(or expression) containing image data, along with optional additional columns containing labels. Currently, only
classification and detection labels are supported.
The [Working with Voxel51 in Pixeltable](https://docs.pixeltable.com/howto/working-with-fiftyone)
tutorial contains a fully worked example showing how to export data from a Pixeltable table and
load it into Voxel51.
Images in the dataset that already exist on disk will be exported directly, in whatever format they
are stored in. Images that are not already on disk (such as frames extracted using a
[`frame_iterator`](./video#iterator-frame_iterator)) will first be written to disk in the specified
`image_format`.
The label parameters accept one or more sets of labels of each type. If a single `Expr` is provided, then it will
be exported as a single set of labels with a default name such as `classifications`.
(The single set of labels may still containing multiple individual labels; see below.)
If a list of `Expr`s is provided, then each one will be exported as a separate set of labels with a default name
such as `classifications`, `classifications_1`, etc. If a dictionary of `Expr`s is provided, then each entry will
be exported as a set of labels with the specified name.
**Requirements:**
* `pip install fiftyone`
**Parameters:**
* **`tbl`** (`pxt.Table`): The table from which to export data.
* **`images`** (`exprs.Expr`): A column or expression that contains the images to export.
* **`image_format`** (`str`, default: `'webp'`): The format to use when writing out images for export.
* **`classifications`** (`exprs.Expr | list[exprs.Expr] | dict[str, exprs.Expr] | None`): Optional image classification labels. If a single `Expr` is provided, it must be a table
column or an expression that evaluates to a list of dictionaries. Each dictionary in the list corresponds
to an image class and must have the following structure:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
{'label': 'zebra', 'confidence': 0.325}
```
If multiple `Expr`s are provided, each one must evaluate to a list of such dictionaries.
* **`detections`** (`exprs.Expr | list[exprs.Expr] | dict[str, exprs.Expr] | None`): Optional image detection labels. If a single `Expr` is provided, it must be a table column or an
expression that evaluates to a list of dictionaries. Each dictionary in the list corresponds to an image
detection, and must have the following structure:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
{
'label': 'giraffe',
'confidence': 0.99,
# [x, y, w, h], fractional coordinates
'bounding_box': [0.081, 0.836, 0.202, 0.136],
}
```
If multiple `Expr`s are provided, each one must evaluate to a list of such dictionaries.
**Returns:**
* `'fo.Dataset'`: A Voxel51 dataset.
**Examples:**
Export the images in the `image` column of the table `tbl` as a Voxel51 dataset, using classification
labels from `tbl.classifications`:
```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
export_images_as_fo_dataset(
tbl, tbl.image, classifications=tbl.classifications
)
```
See the [Working with Voxel51 in Pixeltable](https://docs.pixeltable.com/howto/working-with-fiftyone)
tutorial for a fully worked example.
## func export\_json()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
export_json(
table_or_query: pxt.Table | pxt.Query,
file_path: str | Path
) -> None
```
Exports a query result or table to a JSONL file.
Pixeltable column types are mapped to JSON values as follows:
* String: string
* Int: number
* Float: number
* Bool: boolean
* Timestamp: ISO 8601 string
* Date: ISO 8601 string
* UUID: string
* Json: native JSON value (object, array, etc.)
* Array: nested JSON array (via `tolist()`)
* Binary: excluded from export (not representable in JSON)
* Image, Video, Audio, Document: file path or URL string
**Parameters:**
* **`table_or_query`** (`pxt.Table | pxt.Query`): Table or Query to export.
* **`file_path`** (`str | Path`): Path to the output JSONL file.
## func export\_lancedb()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
export_lancedb(
table_or_query: pxt.Table | pxt.Query,
db_uri: Path,
table_name: str,
batch_size_bytes: int = 134217728,
if_exists: Literal['error', 'overwrite', 'append'] = 'error'
) -> None
```
Exports a Query's data to a LanceDB table.
This utilizes LanceDB's streaming interface for efficient table creation, via a sequence of in-memory pyarrow
`RecordBatches`, the size of which can be controlled with the `batch_size_bytes` parameter.
**Requirements:**
* `pip install lancedb pylance`
**Parameters:**
* **`table_or_query `** (`Any`): Table or Query to export.
* **`db_uri`** (`Path`): Local Path to the LanceDB database.
* **`table_name `** (`Any`): Name of the table in the LanceDB database.
* **`batch_size_bytes `** (`Any`): Maximum size in bytes for each batch.
* **`if_exists`** (`Literal['error', 'overwrite', 'append']`, default: `'error'`): Determines the behavior if the table already exists. Must be one of the following:
* `'error'`: raise an error
* `'overwrite'`: overwrite the existing table
* `'append'`: append to the existing table
## func export\_parquet()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
export_parquet(
table_or_query: pxt.Table | pxt.Query,
parquet_path: Path,
partition_size_bytes: int = 100000000,
inline_images: bool = False,
_write_md: bool = False
) -> None
```
Exports a query result or table to one or more Parquet files. Requires pyarrow to be installed.
Pixeltable column types are mapped to Parquet types as follows:
* String: string
* Int: int64
* Float: float32
* Bool: bool
* Timestamp: timestamp\[us, tz=UTC]
* Date: date32
* UUID: uuid
* Binary: binary
* Image: binary (when `inline_images=True`)
* Audio, Video, Document: string (file paths)
* Array (requires shape to be known):
* fixed\_shape\_tensor for fixed-shape arrays
* list for ragged arrays (one or more dimensions are None)
* Json: struct
* Schema is inferred from data via `pyarrow.infer_type()`
* Fields that contain empty dicts cannot be mapped to a Parquet type and will result in an exception
**Parameters:**
* **`table_or_query `** (`Any`): Table or Query to export.
* **`parquet_path `** (`Any`): Path to directory to write the parquet files to.
* **`partition_size_bytes `** (`Any`): The maximum target size for each chunk. Default 100\_000\_000 bytes.
* **`inline_images `** (`Any`): If True, images are stored inline in the parquet file. This is useful
for small images, to be imported as pytorch dataset. But can be inefficient
for large images, and cannot be imported into pixeltable.
If False, will raise an error if the Query has any image column.
Default False.
## func export\_sql()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
export_sql(
table_or_query: pixeltable.catalog.table.Table | pixeltable._query.Query,
target_table_name: str,
*,
db_connect_str: str,
target_schema_name: str | None = None,
if_exists: Literal['error', 'replace', 'insert'] = 'error',
if_not_exists: Literal['error', 'create'] = 'create'
) -> None
```
Exports a query result or table to an RDBMS table.
**Parameters:**
* **`table_or_query `** (`Any`): Table or Query to export.
* **`target_table_name `** (`Any`): Name of the target table.
* **`db_connect_str `** (`Any`): Connection string to the target database.
* **`target_schema_name `** (`Any`): Optional name of the target schema.
* **`if_exists `** (`Any`): What to do if the target table already exists.
* 'error': raise an error
* 'replace': drop the existing table and create a new one
* 'insert': insert new rows into the existing table
* **`if_not_exists `** (`Any`): What to do if the target table does not exist.
* 'error': raise an error
* 'create': create the table from the source schema
## func import\_csv()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import_csv(
tbl_name: str,
filepath_or_buffer: str | os.PathLike,
schema_overrides: dict[str, Any] | None = None,
primary_key: str | list[str] | None = None,
comment: str | None = None,
**kwargs: Any
) -> pxt.Table
```
Creates a new base table from a csv file. This is a convenience method and is equivalent
to calling `import_pandas(table_path, pd.read_csv(filepath_or_buffer, **kwargs), schema=schema)`.
See the Pandas documentation for [`read_csv`](https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html)
for more details.
**Returns:**
* `pxt.Table`: A handle to the newly created [`Table`](./table).
## func import\_excel()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import_excel(
tbl_name: str,
io: str | os.PathLike,
*,
schema_overrides: dict[str, Any] | None = None,
primary_key: str | list[str] | None = None,
comment: str = '',
**kwargs: Any
) -> pxt.Table
```
Creates a new base table from an Excel (.xlsx) file. This is a convenience method and is
equivalent to calling `import_pandas(table_path, pd.read_excel(io, *args, **kwargs), schema=schema)`.
See the Pandas documentation for [`read_excel`](https://pandas.pydata.org/docs/reference/api/pandas.read_excel.html)
for more details.
**Returns:**
* `pxt.Table`: A handle to the newly created [`Table`](./table).
## func import\_huggingface\_dataset()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import_huggingface_dataset(
table_path: str,
dataset: datasets.Dataset | datasets.DatasetDict | datasets.IterableDataset | datasets.IterableDatasetDict,
*,
schema_overrides: dict[str, Any] | None = None,
primary_key: str | list[str] | None = None,
**kwargs: Any
) -> pxt.Table
```
Create a new base table from a Huggingface dataset, or dataset dict with multiple splits.
Requires `datasets` library to be installed.
HuggingFace feature types are mapped to Pixeltable column types as follows:
* `Value(bool)`: `Bool`
`Value(int*/uint*)`: `Int`
`Value(float*)`: `Float`
`Value(string/large_string)`: `String`
`Value(timestamp*)`: `Timestamp`
`Value(date*)`: `Date`
* `ClassLabel`: `String` (converted to label names)
* `Sequence`/`LargeList` of numeric types: `Array`
* `Sequence`/`LargeList` of string: `Json`
* `Sequence`/`LargeList` of dicts: `Json`
* `Array2D`-`Array5D`: `Array` (preserves shape)
* `Image`: `Image`
* `Audio`: `Audio`
* `Video`: `Video`
* `Translation`/`TranslationVariableLanguages`: `Json`
**Parameters:**
* **`table_path`** (`str`): Path to the table.
* **`dataset`** (`datasets.Dataset | datasets.DatasetDict | datasets.IterableDataset | datasets.IterableDatasetDict`): An instance of any of the Huggingface dataset classes:
[`datasets.Dataset`](https://huggingface.co/docs/datasets/en/package_reference/main_classes#datasets.Dataset),
[`datasets.DatasetDict`](https://huggingface.co/docs/datasets/en/package_reference/main_classes#datasets.DatasetDict),
[`datasets.IterableDataset`](https://huggingface.co/docs/datasets/en/package_reference/main_classes#datasets.IterableDataset),
[`datasets.IterableDatasetDict`](https://huggingface.co/docs/datasets/en/package_reference/main_classes#datasets.IterableDatasetDict)
* **`schema_overrides`** (`dict[str, Any] | None`): If specified, then for each (name, type) pair in `schema_overrides`, the column with
name `name` will be given type `type`, instead of being inferred from the `Dataset` or `DatasetDict`.
The keys in `schema_overrides` should be the column names of the `Dataset` or `DatasetDict` (whether or not
they are valid Pixeltable identifiers).
* **`primary_key`** (`str | list[str] | None`): The primary key of the table (see [`create_table()`](./pixeltable#func-create_table)).
* **`kwargs`** (`Any`): Additional arguments to pass to `create_table`.
An argument of `column_name_for_split` must be provided if the source is a DatasetDict.
This column name will contain the split information. If None, no split information will be stored.
**Returns:**
* `pxt.Table`: A handle to the newly created [`Table`](./table).
## func import\_json()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import_json(
tbl_path: str,
filepath_or_url: str,
*,
schema_overrides: dict[str, Any] | None = None,
primary_key: str | list[str] | None = None,
comment: str | None = None,
**kwargs: Any
) -> pxt.Table
```
Creates a new base table from a JSON file. This is a convenience method and is
equivalent to calling [`create_table()`](./pixeltable#func-create_table) with
`pxt.create_table(tbl_path, source=filepath_or_url, extra_args=kwargs, ...)`.
The contents of `filepath_or_url` are read and parsed as JSON internally (using `json.loads(**kwargs)`).
**Parameters:**
* **`tbl_path`** (`str`): The name of the table to create.
* **`filepath_or_url`** (`str`): The path or URL of the JSON file.
* **`schema_overrides`** (`dict[str, Any] | None`): If specified, then columns in `schema_overrides` will be given the specified types
(see [`import_rows()`](./io#func-import_rows)).
* **`primary_key`** (`str | list[str] | None`): The primary key of the table (see [`create_table()`](./pixeltable#func-create_table)).
* **`comment`** (`str | None`): A comment to attach to the table (see [`create_table()`](./pixeltable#func-create_table)).
* **`kwargs`** (`Any`): Additional keyword arguments to pass to `json.loads`.
**Returns:**
* `pxt.Table`: A handle to the newly created [`Table`](./table).
## func import\_pandas()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import_pandas(
tbl_name: str,
df: pd.DataFrame,
*,
schema_overrides: dict[str, Any] | None = None,
primary_key: str | list[str] | None = None,
comment: str = ''
) -> pxt.Table
```
Creates a new base table from a Pandas
[`DataFrame`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html), with the
specified name. The schema of the table will be inferred from the DataFrame.
The column names of the new table will be identical to those in the DataFrame, as long as they are valid
Pixeltable identifiers. If a column name is not a valid Pixeltable identifier, it will be normalized according to
the following procedure:
* first replace any non-alphanumeric characters with underscores;
* then, preface the result with the letter 'c' if it begins with a number or an underscore;
* then, if there are any duplicate column names, suffix the duplicates with '\_2', '\_3', etc., in column order.
**Parameters:**
* **`tbl_name`** (`str`): The name of the table to create.
* **`df`** (`pd.DataFrame`): The Pandas `DataFrame`.
* **`schema_overrides`** (`dict[str, Any] | None`): If specified, then for each (name, type) pair in `schema_overrides`, the column with
name `name` will be given type `type`, instead of being inferred from the `DataFrame`. The keys in
`schema_overrides` should be the column names of the `DataFrame` (whether or not they are valid
Pixeltable identifiers).
**Returns:**
* `pxt.Table`: A handle to the newly created [`Table`](./table).
## func import\_parquet()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import_parquet(
table: str,
*,
parquet_path: str,
schema_overrides: dict[str, Any] | None = None,
primary_key: str | list[str] | None = None,
**kwargs: Any
) -> pxt.Table
```
Creates a new base table from a Parquet file or set of files. Requires pyarrow to be installed.
**Parameters:**
* **`table`** (`str`): Fully qualified name of the table to import the data into.
* **`parquet_path`** (`str`): Path to an individual Parquet file or directory of Parquet files.
* **`schema_overrides`** (`dict[str, Any] | None`): If specified, then for each (name, type) pair in `schema_overrides`, the column with
name `name` will be given type `type`, instead of being inferred from the Parquet dataset. The keys in
`schema_overrides` should be the column names of the Parquet dataset (whether or not they are valid
Pixeltable identifiers).
* **`primary_key`** (`str | list[str] | None`): The primary key of the table (see [`create_table()`](./pixeltable#func-create_table)).
* **`kwargs`** (`Any`): Additional arguments to pass to `create_table`.
**Returns:**
* `pxt.Table`: A handle to the newly created table.
## func import\_rows()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import_rows(
tbl_path: str,
rows: list[dict[str, Any]],
*,
schema_overrides: dict[str, Any] | None = None,
primary_key: str | list[str] | None = None,
comment: str = ''
) -> pxt.Table
```
Creates a new base table from a list of dictionaries. The dictionaries must be of the
form `{column_name: value, ...}`. Pixeltable will attempt to infer the schema of the table from the
supplied data, using the most specific type that can represent all the values in a column.
If `schema_overrides` is specified, then for each entry `(column_name, type)` in `schema_overrides`,
Pixeltable will force the specified column to the specified type (and will not attempt any type inference
for that column).
All column types of the new table will be nullable unless explicitly specified as non-nullable in
`schema_overrides`.
**Parameters:**
* **`tbl_path`** (`str`): The qualified name of the table to create.
* **`rows`** (`list[dict[str, Any]]`): The list of dictionaries to import.
* **`schema_overrides`** (`dict[str, Any] | None`): If specified, then columns in `schema_overrides` will be given the specified types
as described above.
* **`primary_key`** (`str | list[str] | None`): The primary key of the table (see [`create_table()`](./pixeltable#func-create_table)).
* **`comment`** (`str`, default: `''`): A comment to attach to the table (see [`create_table()`](./pixeltable#func-create_table)).
**Returns:**
* `pxt.Table`: A handle to the newly created [`Table`](./table).
## func import\_sql()
```python Signature theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
import_sql(
selectable: sqlalchemy.sql.selectable.Selectable,
conn: sqlalchemy.engine.base.Engine | sqlalchemy.engine.base.Connection,
tbl_name: str,
*,
schema_overrides: dict[str, typing.Any] | None = None,
primary_key: str | list[str] | None = None,
comment: str | None = None,
custom_metadata: Any = None,
if_exists: Literal['error', 'append'] = 'error',
on_error: Literal['abort', 'ignore'] = 'abort',
send_connect_url: bool = False
) -> pixeltable.catalog.table.Table
```
Import a SQL source into a Pixeltable table.
Rows are streamed from the source via a server-side cursor and inserted in batches.
**Parameters:**
* **`selectable`** (`sqlalchemy.sql.selectable.Selectable`): A SQLAlchemy `Selectable` (a `Table`, a `select()` statement, or `text(...).columns(...)`)
describing the source rows.
* **`conn`** (`sqlalchemy.engine.base.Engine | sqlalchemy.engine.base.Connection`): A SQLAlchemy `Engine` or `Connection` to execute `selectable` against. If a `Connection` is
passed, it must remain open and untouched (no commits, rollbacks, or other statements) for the
duration of the import; rows are streamed directly from the server-side cursor.
* **`tbl_name`** (`str`): Pixeltable path of the destination table.
* **`schema_overrides`** (`dict[str, typing.Any] | None`): Optional per-column overrides applied on top of the inferred schema. Keys are column
names; values accept any Pixeltable type spec recognized by `pxt.create_table` (eg, `pxt.Image`,
`pxt.Required[pxt.String]`).
* **`primary_key`** (`str | list[str] | None`): An optional column name or list of column names to use as the primary key(s) of the
table. Only applies when a new table is created; ignored when appending to an existing table.
* **`comment`** (`str | None`): An optional comment; its meaning is user-defined. Only applies when a new table is created;
ignored when appending to an existing table.
* **`custom_metadata`** (`typing.Any`): Optional user-defined metadata to associate with the table. Must be a valid
JSON-serializable object \[str, int, float, bool, dict, list]. Only applies when a new table is
created; ignored when appending to an existing table.
* **`if_exists`** (`typing.Literal['error', 'append']`, default: `'error'`): How to handle the destination table.
* `'error'`: create the table; fail if it already exists.
* `'append'`: append into the table if it already exists (verifying the source schema is
compatible); otherwise create it.
* **`on_error`** (`typing.Literal['abort', 'ignore']`, default: `'abort'`): Determines the behavior if an error occurs while evaluating a computed column or detecting an
invalid media file (such as a corrupt image) for one of the inserted rows.
* If `on_error='abort'`, then an exception will be raised and the rows will not be inserted.
* If `on_error='ignore'`, then execution will continue and the rows will be inserted. Any cells
with errors will have a `None` value for that cell, with information about the error stored in the
corresponding `tbl.col_name.errortype` and `tbl.col_name.errormsg` fields.
* **`send_connect_url`** (`bool`, default: `False`): Determines where the query is run, and is only relevant when the destination table is hosted.
It has no effect on a local table.
* If `False`, the query is run locally and the result is sent to the server.
* If `True`, the server connects to the source database itself (using the source connection URL, including
credentials); this requires the server's host to be able to reach the database and is faster for large
imports.
**Returns:**
* `pixeltable.catalog.table.Table`: The destination `Table`.