> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pixeltable.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Ecosystem

> Explore Pixeltable ecosystem of built-in integrations for AI/ML workflows

From language models to computer vision frameworks, Pixeltable integrates with the entire ecosystem. All integrations are available out-of-the-box with Pixeltable installation. No additional setup required unless specified.

<Note>
  Missing an integration? Build your own with Pixeltable [UDFs](/platform/udfs-in-pixeltable), or suggest one on [GitHub Discussions](https://github.com/pixeltable/pixeltable/discussions).
</Note>

## Cloud LLM providers

<CardGroup cols={3}>
  <Card title="Anthropic Claude" icon="brain" href="/howto/providers/working-with-anthropic">
    Integrate Claude models for advanced language understanding and generation with multimodal capabilities
  </Card>

  <Card title="Google Gemini" icon="sparkles" href="/howto/providers/working-with-gemini">
    Access Google's Gemini models via Google AI Studio or Vertex AI for state-of-the-art multimodal AI capabilities
  </Card>

  <Card title="OpenAI" icon="square-code" href="/howto/providers/working-with-openai">
    Leverage GPT models for text generation, embeddings, and image analysis
  </Card>

  <Card title="Azure OpenAI" icon="microsoft" href="/howto/providers/working-with-openai">
    Use OpenAI models via Azure with enterprise security and compliance
  </Card>

  <Card title="Mistral AI" icon="wind" href="/howto/providers/working-with-mistralai">
    Use Mistral's efficient language models for various NLP tasks
  </Card>

  <Card title="DeepSeek" icon="robot" href="/howto/providers/working-with-deepseek">
    Leverage DeepSeek's powerful language and code models for text and code generation
  </Card>

  <Card title="Groq" icon="microchip" href="/howto/providers/working-with-groq">
    Access Groq's models for text generation
  </Card>
</CardGroup>

## Model hubs

Platforms that host or route many models from a shared catalog.

<CardGroup cols={3}>
  <Card title="Hugging Face Hub" icon="face-smile" href="/howto/providers/working-with-hugging-face">
    Access thousands of pre-trained models across vision, text, and audio domains
  </Card>

  <Card title="Replicate" icon="clone" href="/howto/providers/working-with-replicate">
    Deploy and run ML models through Replicate's cloud infrastructure
  </Card>

  <Card title="Together AI" icon="users" href="/howto/providers/working-with-together">
    Access a variety of open-source models through Together AI's platform
  </Card>

  <Card title="Fireworks" icon="rocket" href="/howto/providers/working-with-fireworks">
    Use Fireworks.ai's optimized model inference infrastructure
  </Card>

  <Card title="OpenRouter" icon="route" href="/howto/providers/working-with-openrouter">
    Unified access to 100+ LLMs from various providers through a single API
  </Card>

  <Card title="AWS Bedrock" icon="aws" href="/howto/providers/working-with-bedrock">
    Access a variety of AI models through AWS Bedrock's unified API
  </Card>

  <Card title="Nebius" icon="cloud" href="/sdk/latest/nebius">
    Nebius Token Factory language and embedding models via an OpenAI-compatible API
  </Card>
</CardGroup>

## Embeddings & Reranking

<CardGroup cols={3}>
  <Card title="Voyage AI" icon="compass" href="/howto/providers/working-with-voyageai">
    High-quality embeddings and reranking for text, images, and video
  </Card>

  <Card title="Jina AI" icon="magnifying-glass" href="/howto/providers/working-with-jina">
    Embeddings and reranking optimized for search and RAG pipelines
  </Card>

  <Card title="Twelve Labs" icon="clapperboard" href="/howto/providers/working-with-twelvelabs">
    Multimodal embeddings for text, image, audio, and video via the TwelveLabs Embed API
  </Card>
</CardGroup>

## Media Generation

<CardGroup cols={3}>
  <Card title="BFL (FLUX)" icon="image" href="/howto/providers/working-with-bfl">
    Image generation, editing, fill, and expansion with FLUX models from Black Forest Labs
  </Card>

  <Card title="fal.ai" icon="wand-magic-sparkles" href="/howto/providers/working-with-fal">
    Fast image generation with Flux, Stable Diffusion, and other models
  </Card>

  <Card title="RunwayML" icon="film" href="/howto/providers/working-with-runwayml">
    AI video generation with Gen-4 and other Runway models
  </Card>
</CardGroup>

## Local LLM runtimes

<CardGroup cols={3}>
  <Card title="Llama.cpp" icon="microchip" href="/howto/providers/working-with-llama-cpp">
    High-performance C++ implementation for running LLMs on CPU and GPU
  </Card>

  <Card title="Ollama" icon="box" href="/howto/providers/working-with-ollama">
    Easy-to-use toolkit for running and managing open-source models locally
  </Card>

  <Card title="vLLM" icon="microchip" href="/howto/providers/working-with-vllm">
    High-throughput local LLM inference with PagedAttention and continuous batching
  </Card>
</CardGroup>

## Hugging Face models

Pixeltable provides seamless integration with Hugging Face's transformers library through built-in UDFs. These functions allow you to use state-of-the-art models directly in your data workflows.

<Note>
  Requirements: Install required dependencies with `pip install transformers`. Some models may require additional packages like `sentence-transformers` or `torch`.
</Note>

### CLIP models

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions.huggingface import clip

# For text embedding
t.add_computed_column(
    text_embedding=clip(
        t.text_column,
        model_id='openai/clip-vit-base-patch32'
    )
)

# For image embedding
t.add_computed_column(
    image_embedding=clip(
        t.image_column,
        model_id='openai/clip-vit-base-patch32'
    )
)
```

Perfect for multimodal applications combining text and image understanding.

### Cross-encoders

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions.huggingface import cross_encoder

t.add_computed_column(
    similarity_score=cross_encoder(
        t.sentence1,
        t.sentence2,
        model_id='cross-encoder/ms-marco-MiniLM-L-4-v2'
    )
)
```

Ideal for semantic similarity tasks and sentence pair classification.

### DETR object detection

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions.huggingface import detr_for_object_detection

t.add_computed_column(
    detections=detr_for_object_detection(
        t.image,
        model_id='facebook/detr-resnet-50',
        threshold=0.8
    )
)

# Convert to COCO format if needed
t.add_computed_column(
    coco_format=detr_to_coco(t.image, t.detections)
)
```

Powerful object detection with end-to-end transformer architecture.

### Sentence transformers

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions.huggingface import sentence_transformer

t.add_computed_column(
    embeddings=sentence_transformer(
        t.text,
        model_id='sentence-transformers/all-mpnet-base-v2',
        normalize_embeddings=True
    )
)
```

State-of-the-art sentence and document embeddings for semantic search and similarity.

### Speech2Text models

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions.huggingface import speech2text_for_conditional_generation

# Basic transcription
t.add_computed_column(
    transcript=speech2text_for_conditional_generation(
        t.audio,
        model_id='facebook/s2t-small-librispeech-asr'
    )
)

# Multilingual translation
t.add_computed_column(
    translation=speech2text_for_conditional_generation(
        t.audio,
        model_id='facebook/s2t-medium-mustc-multilingual-st',
        language='fr'
    )
)
```

Support for both transcription and translation of audio content.

### Vision Transformer (ViT)

```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
from pixeltable.functions.huggingface import vit_for_image_classification

t.add_computed_column(
    classifications=vit_for_image_classification(
        t.image,
        model_id='google/vit-base-patch16-224',
        top_k=5
    )
)
```

Modern image classification using transformer architecture.

## Model selection guide

<Steps>
  <Step title="Choose Task">
    Select the appropriate model family based on your task:

    * Text/Image Similarity → CLIP
    * Object Detection → DETR
    * Text Embeddings → Sentence Transformers
    * Speech Processing → Speech2Text
    * Image Classification → ViT
  </Step>

  <Step title="Check Requirements">
    Install necessary dependencies:

    ```bash theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
    pip install transformers torch sentence-transformers
    ```
  </Step>

  <Step title="Setup Integration">
    Import and use the model in your Pixeltable workflow:

    ```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
    from pixeltable.functions.huggingface import clip, sentence_transformer
    ```
  </Step>
</Steps>

## Computer vision

<CardGroup cols={2}>
  <Card title="YOLOX" icon="camera" href="/howto/use-cases/object-detection-in-videos">
    State-of-the-art object detection with YOLOX models
  </Card>

  <Card title="Voxel51" icon="cube" href="/howto/working-with-fiftyone">
    Advanced video and image dataset management with Voxel51
  </Card>
</CardGroup>

## Audio processing

<Card title="Whisper/WhisperX" icon="waveform" href="/howto/use-cases/audio-transcriptions">
  High-quality speech recognition and transcription using OpenAI's Whisper models
</Card>

## Enterprise Platforms

<Card title="Microsoft Fabric" icon="microsoft" href="/howto/providers/working-with-fabric">
  Azure OpenAI integration through Microsoft Fabric for enterprise AI workloads
</Card>

## Data Wrangling

<Card title="Pandas" icon="table" href="/tutorials/tables-and-data-operations">
  Import and export from and to Pandas DataFrames
</Card>

## Usage examples

<AccordionGroup>
  <Accordion title="LLM Integration">
    ```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
    import pixeltable as pxt
    from pixeltable.functions import openai

    # Create a table with computed column for OpenAI completion
    table = pxt.create_table('responses', {'prompt': pxt.String})

    table.add_computed_column(
        response=openai.chat_completions(
            messages=[{'role': 'user', 'content': table.prompt}],
            model='gpt-4'
        )
    )
    ```
  </Accordion>

  <Accordion title="Computer Vision">
    ```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
    from pixeltable.functions.yolox import yolox

    # Add object detection to video frames
    frames_view.add_computed_column(
        detections=yolox(
            frames_view.frame,
            model_id='yolox_l'
        )
    )
    ```
  </Accordion>

  <Accordion title="Audio Processing">
    ```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
    from pixeltable.functions import openai

    # Transcribe audio files
    audio_table.add_computed_column(
        transcription=openai.transcriptions(
            audio=audio_table.file,
            model='whisper-1'
        )
    )
    ```
  </Accordion>

  <Accordion title="Model Output Format">
    ```python theme={"theme":{"light":"light-plus","dark":"dark-plus"}}
    # Object Detection Output
    {
        'scores': [0.99, 0.98],  # confidence scores
        'labels': [25, 30],      # class labels
        'label_text': ['cat', 'dog'], # human-readable labels
        'boxes': [[x1, y1, x2, y2], ...] # bounding boxes
    }

    # Image Classification Output
    {
        'scores': [0.8, 0.15],   # class probabilities
        'labels': [340, 353],    # class IDs
        'label_text': ['zebra', 'gazelle'] # class names
    }
    ```
  </Accordion>
</AccordionGroup>

## Integration features

<Steps>
  <Step title="Easy Setup">
    Most integrations work out-of-the-box with simple API configuration
  </Step>

  <Step title="Computed Columns">
    Use integrations directly in computed columns for automated processing
  </Step>

  <Step title="Batch Processing">
    Efficient handling of batch operations with automatic optimization
  </Step>
</Steps>

<Tip>
  Check the [provider notebooks](https://github.com/pixeltable/pixeltable/tree/main/docs/release/howto/providers) for detailed usage instructions for each integration.
</Tip>