Questions

Docs · Questions

LOPsUsage

How do I use local AI models (like `llama.cpp` or Ollama) with LOPs?

Problem

Users frequently encounter errors or limitations when trying to use LOPs with their own local AI models, such as those served with llama.cpp or Ollama.

1. Connecting to a llama.cpp Server

  • Symptom: You receive errors when trying to connect LOPs to a local llama.cpp server, even if it exposes an OpenAI-compatible endpoint (e.g., via llama-swap).
  • Solution: This is a known issue that will be addressed in a future update. In the meantime, you can try the following:
    • Verify Endpoint URL: Double-check that the URL and port are correct in the ChatTD API configuration.
    • Check Firewall Settings: Ensure that your firewall is not blocking the connection between TouchDesigner and the local server.
    • Consult litellm Docs: For advanced configurations, refer to the litellm documentation, as it is the underlying library that LOPs uses to connect to models.

2. Running Multiple Local Models in Parallel

  • Symptom: You experience errors or instability when trying to use more than one local model at the same time (e.g., a vision model and a text model).
  • Solution: This is a known limitation that will be fixed in a future update. As a temporary workaround, you can:
    • Consolidate Models: If possible, serve all your models from a single server endpoint.
    • Use Separate Machines: Run each local model on a separate computer to avoid port conflicts and resource competition.

3. RAGIndex Fails with Local Ollama Models

  • Symptom: When using the RAGIndex LOP with a local Ollama model, you get a “Could not load OpenAI embedding model” error.
  • Solution: This is a missing dependency issue. You can fix it by running the following command in the LOPs Python Manager:
    pip install llama-index-embeddings-ollama