Skip to main content

LLM Support

CocoIndex provides builtin functions integrating with various LLM APIs, for various inference tasks:

LLM API Types​

We support integrating with LLM with different types of APIs. Each LLM API type is specified by a cocoindex.LlmApiType enum.

We support the following types of LLM APIs:

API NameLlmApiType enumText GenerationText Embedding
OpenAILlmApiType.OPENAI✅✅
OllamaLlmApiType.OLLAMA✅❌
Google GeminiLlmApiType.GEMINI✅✅
AnthropicLlmApiType.ANTHROPIC✅❌
VoyageLlmApiType.VOYAGE❌✅
LiteLLMLlmApiType.LITE_LLM✅❌
OpenRouterLlmApiType.OPEN_ROUTER✅❌

LLM Tasks​

Text Generation​

Generation is used as a building block for certain CocoIndex functions that process data using LLM generation.

We have one builtin functions using LLM generation for now:

LLM Spec​

When calling a CocoIndex function that uses LLM generation, you need to provide a cocoindex.LlmSpec dataclass, to configure the LLM you want to use in these functions. It has the following fields:

  • api_type (type: cocoindex.LlmApiType, required): The type of integrated LLM API to use, e.g. cocoindex.LlmApiType.OPENAI or cocoindex.LlmApiType.OLLAMA. See supported LLM APIs in the LLM API integrations section below.
  • model (type: str, required): The name of the LLM model to use.
  • address (type: str, optional): The address of the LLM API.

Text Embedding​

Embedding means converting text into a vector space, usually for similarity matching.

We provide a builtin function EmbedText that converts a given text into a vector space. The spec takes the following fields:

  • api_type (type: cocoindex.LlmApiType, required)
  • model (type: str, required)
  • address (type: str, optional)
  • output_dimension (type: int, optional)
  • task_type (type: str, optional)

See documentation for EmbedText for more details about these fields.

LLM API Integrations​

CocoIndex integrates with various LLM APIs for these functions.

OpenAI​

To use the OpenAI LLM API, you need to set the environment variable OPENAI_API_KEY. You can generate the API key from OpenAI Dashboard.

Currently we don't support custom address for OpenAI API.

You can find the full list of models supported by OpenAI here.

For text generation, a spec for OpenAI looks like this:

cocoindex.LlmSpec(
    api_type=cocoindex.LlmApiType.OPENAI,
    model="gpt-4o",
)

For text embedding, a spec for OpenAI looks like this:

cocoindex.functions.EmbedText(
    api_type=cocoindex.LlmApiType.OPENAI,
    model="text-embedding-3-small",
)

Ollama​

Ollama allows you to run LLM models on your local machine easily. To get started:

  • Download and install Ollama.
  • Pull your favorite LLM models by the ollama pull command, e.g. 
    ollama pull llama3.2

You can find the list of models supported by Ollama.

A spec for Ollama looks like this:

cocoindex.LlmSpec(
    api_type=cocoindex.LlmApiType.OLLAMA,
    model="llama3.2:latest",
    # Optional, use Ollama's default port (11434) on localhost if not specified
    address="http://localhost:11434",
)

Google Gemini​

To use the Gemini LLM API, you need to set the environment variable GEMINI_API_KEY. You can generate the API key from Google AI Studio.

You can find the full list of models supported by Gemini here.

For text generation, a spec looks like this:

cocoindex.LlmSpec(
    api_type=cocoindex.LlmApiType.GEMINI,
    model="gemini-2.0-flash",
)

For text embedding, a spec looks like this:

cocoindex.functions.EmbedText(
    api_type=cocoindex.LlmApiType.GEMINI,
    model="text-embedding-004",
    task_type="SEMANTICS_SIMILARITY",
)

All supported embedding models can be found here. Gemini supports task type (optional), which can be found here.

Anthropic​

To use the Anthropic LLM API, you need to set the environment variable ANTHROPIC_API_KEY. You can generate the API key from Anthropic API.

A text generation spec for Anthropic looks like this:

cocoindex.LlmSpec(
    api_type=cocoindex.LlmApiType.ANTHROPIC,
    model="claude-3-5-sonnet-latest",
)

You can find the full list of models supported by Anthropic here.

Voyage​

To use the Voyage LLM API, you need to set the environment variable VOYAGE_API_KEY. You can generate the API key from Voyage dashboard.

A text embedding spec for Voyage looks like this:

cocoindex.functions.EmbedText(
    api_type=cocoindex.LlmApiType.VOYAGE,
    model="voyage-code-3",
    task_type="document",
)

Voyage API supports document and query as task types (optional, a.k.a. input_type in Voyage API, see Voyage API documentation for details).

LiteLLM​

To use the LiteLLM API, you need to set the environment variable LITELLM_API_KEY.

1. Install LiteLLM Proxy​

pip install 'litellm[proxy]'

2. Create a config.yml for LiteLLM​

Example for DeepSeek:

Use this in your config.yml:

model_list:
  - model_name: deepseek-chat
    litellm_params:
        model: deepseek/deepseek-chat
        api_key: os.environ/DEEPSEEK_API_KEY

You need to set the environment variable DEEPSEEK_API_KEY to your DeepSeek API key.

Example for Groq:

Use this in your config.yml:

model_list:
  - model_name: groq-llama-3.3-70b-versatile
    litellm_params:
      model: groq/llama-3.3-70b-versatile
      api_key: "os.environ/GROQ_API_KEY"

You need to set the environment variable GROQ_API_KEY to your Groq API key.

3. Run LiteLLM Proxy​

litellm --config config.yml

4. A Spec for LiteLLM will look like this:​

cocoindex.LlmSpec(
    api_type=cocoindex.LlmApiType.LITE_LLM,
    model="deepseek-chat",
    address="http://127.0.0.1:4000", # default url of LiteLLM
)

You can find the full list of models supported by LiteLLM here.

OpenRouter​

To use the OpenRouter API, you need to set the environment variable OPENROUTER_API_KEY. You can generate the API key from here.

A spec for OpenRouter looks like this:

cocoindex.LlmSpec(
    api_type=cocoindex.LlmApiType.OPEN_ROUTER,
    model="deepseek/deepseek-r1:free",
)

You can find the full list of models supported by OpenRouter here.