NVIDIA-NeMo/Switchyard

title: NVIDIA-NeMo/Switchyard
source: https://gemini.google.com/app/75dd752b47a0aae2?utm_source=deepmind.google&utm_medium=referral&utm_campaign=gdm&utm_content=
author:
published:
created: 2026-09-04
description: Gemini conversation with 38 messages
tags:
  - LLM

NVIDIA-NeMo/Switchyard

Can this work with local ollama: https://github.com/NVIDIA-NeMo/Switchyard

Gemini

Yes, absolutely. NVIDIA NeMo Switchyard can work with local Ollama.

Switchyard is designed to be provider-agnostic. Its standalone proxy server (switchyard-server) and library components communicate with backends using standard API formats like OpenAI Chat Completions. Because Ollama exposes an OpenAI-compatible API endpoint (http://localhost:11434/v1), you can easily configure Switchyard to target local models running in Ollama.

How to configure it

In your Switchyard routes.toml configuration file, define an llm_clients block pointing to your local Ollama instance. Because Ollama does not require an API key, you can omit the api_key_env setting.

Here is an example setup demonstrating how to route requests between a local Ollama model (e.g., llama3 or nemotron) and an external cloud model:

schema_version = 1

# Define your local Ollama backend
[llm_clients.ollama_local]
format = "openai_chat"
base_url = "http://localhost:11434/v1"

# Define your cloud provider (optional, for hybrid routing)
[llm_clients.openrouter]
format = "openai_chat"
base_url = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"

# Map targets to local/cloud models
[targets.local_fast]
id = "llama3"
llm_client = "ollama_local"

[targets.cloud_smart]
id = "openai/gpt-4o"
llm_client = "openrouter"

# Create a smart/stage/random routing policy over them
[routes.my_agent_router]
id = "my-smart-router"
type = "llm_classifier"
classifier_target = "local_fast" # Can use a local model as the judge/classifier tier
weak_target = "local_fast"
strong_target = "cloud_smart"

Key things to keep in mind:

Ollama URL: Ensure Ollama is running locally and accessible at http://localhost:11434/v1.
No API Key Required: Switchyard allows you to drop the api_key_env property for local endpoints like Ollama since authentication isn't needed.
Model Names: The id under [targets ...] must match the exact model tag you have pulled locally into Ollama (e.g., ollama run llama3).

You

Can I put all my local ollama models in the Yaml?: [targets.local_fast]id = "llama3"llm_client = "ollama_local"

Gemini

Yes, you can register as many local Ollama models in Switchyard as you like— but note that configuration files in Switchyard are written in TOML, not YAML (as shown in the routes.toml structure).

To add multiple Ollama models, you define a single [llm_clients.ollama_local] block for the client connection, and then define a separate [targets.<name>] block for every local model you have pulled in Ollama.

Here is how to set up multiple local models in your routes.toml:

schema_version = 1

# Single client pointing to your local Ollama instance
[llm_clients.ollama_local]
format = "openai_chat"
base_url = "http://localhost:11434/v1"

# Target 1: Llama 3
[targets.local_llama3]
id = "llama3"
llm_client = "ollama_local"

# Target 2: DeepSeek R1
[targets.local_deepseek]
id = "deepseek-r1:8b"
llm_client = "ollama_local"

# Target 3: Qwen 2.5
[targets.local_qwen]
id = "qwen2.5:7b"
llm_client = "ollama_local"

# Example Policy: Dynamic routing between local models
[routes.local_router]
id = "local-only-router"
type = "llm_classifier"
classifier_target = "local_llama3" # Uses Llama 3 as the router judge
weak_target = "local_llama3"       # Sends simple queries to Llama 3
strong_target = "local_deepseek"   # Sends complex reasoning queries to DeepSeek

Important details to remember:

Match exact Ollama tags: The id under each [targets.<name>] block must match the exact string you see when you run ollama list in your terminal (e.g., llama3:latest, deepseek-r1:8b, mistral).
VRAM management: Ollama handles swapping models in and out of GPU memory automatically when Switchyard routes incoming requests between your different local targets.

You

Then Switchyard would choose the best model for the job automatically?

Gemini

Yes, exactly. Switchyard evaluates your incoming request and automatically routes it to the most appropriate target based on the routing strategy you configure.

However, Switchyard doesn't possess inherent magic to determine model capability—it relies on the specific routing algorithm defined in your TOML config to decide when to use a fast/lightweight model versus a heavy, reasoning model.

Common Routing Strategies Available

LLM Classifier (llm_classifier):
- How it works: Switchyard uses a lightweight "judge" model (e.g., your fast local model like qwen2.5:7b) to evaluate the user's prompt first.
  LangChain
  - Behavior: If the judge determines the query is simple, it routes to local_llama3. If it detects high complexity, math, or difficult reasoning, it routes the query to local_deepseek.
Stage Router (stage_router):
- How it works: Designed specifically for multi-turn AI agents. It monitors execution context (such as recent tool output, token counts, or repeated script errors).
  Wavect+ 1
  - Behavior: It keeps routine tasks on a fast local model, but automatically "escalates" to a stronger local or cloud model when it detects tool errors or stalled loops.
Escalation Router:
- How it works: Starts every task on your smaller, faster model by default to save compute and lower latency.
  LangChain
  - Behavior: If the smaller model fails or flags uncertainty, Switchyard transparently reroutes the task to your heavier model.
    LangChain
Sub-Agent or Passthrough Routing (passthrough):
- How it works: Directs requests explicitly based on the model ID requested by your agent or client, acting as a unified proxy gateway.

Example: Setting Up an Automated Local Classifier

Here is how you configure Switchyard so it uses a lightweight local model to evaluate intent and select between your local models automatically:

schema_version = 1

[llm_clients.ollama_local]
format = "openai_chat"
base_url = "http://localhost:11434/v1"

# Fast, small model for general chat & classification
[targets.fast_model]
id = "qwen2.5:7b"
llm_client = "ollama_local"

# Heavy reasoning model
[targets.reasoning_model]
id = "deepseek-r1:8b"
llm_client = "ollama_local"

# Auto-routing policy exposed under model name "auto-select"
[routes.auto_select]
id = "auto-select"
type = "llm_classifier"
classifier_target = "fast_model"  # Small model analyzes the intent
weak_target = "fast_model"        # Handled locally by 7B if simple
strong_target = "reasoning_model" # Handled locally by DeepSeek if complex

When your frontend or application sends requests to Switchyard using "model": "auto-select", Switchyard automatically runs the decision logic and forwards the request to the optimal local model.

This setup guide provides visual context for configuring local targets and defining classifier rules in Switchyard:

Switchyard GitHub Setup Guide: Route Agent Traffic

You

Does Switchyard have a gui?

Gemini

No, standalone NeMo Switchyard does not have a dedicated built-in Graphical User Interface (GUI).

By default, Switchyard is designed as a lightweight, headless proxy service and Python library. Interaction, configuration, and monitoring are handled entirely via CLI, code, and TOML configuration files.

However, depending on how you deploy it, there are a few ways to get a graphical interface:

1. External Web UIs (Open WebUI, AnythingLLM, Chatbox, etc.)

Because switchyard-server exposes a standard OpenAI-compatible REST API (typically running on http://localhost:8000/v1), you can connect any frontend Web UI to it.

Point your favorite frontend (like Open WebUI) to Switchyard's local server port.
Set the target model name in the UI to match the route ID defined in your routes.toml (e.g., auto-select).
The Web UI acts as the visual chat interface, while Switchyard transparently handles the background model routing behind the scenes.

2. NVIDIA NeMo Studio (Platform Ecosystem)

If you run Switchyard as part of the broader NVIDIA NeMo Platform, NVIDIA includes NeMo Studio (an alpha browser-based UI).

It provides a web dashboard to monitor agent execution, test models, and inspect routing job metrics managed under the NeMo suite.
NVIDIA Documentation

3. API Gateway Dashboards

If you integrate Switchyard into broader API gateway tools (such as Kong AI Gateway or LiteLLM), you can view request logs, latency, and routing stats inside those platforms' management dashboards.

You

What about page assist?

Gemini

Yes, Page Assist works seamlessly with Switchyard.

Page Assist (the browser extension that provides a sidebar and Web UI for local LLMs) includes built-in support for OpenAI-compatible API endpoints. Because Switchyard exposes an OpenAI-compliant API endpoint when running, you can point Page Assist directly to Switchyard instead of pointing it straight to Ollama.

This setup allows you to use Page Assist's web-browsing sidebar while letting Switchyard dynamically route your prompts to the best local model behind the scenes.

How to Connect Page Assist to Switchyard

1. Start Switchyard

Ensure your switchyard-server is running locally (by default on port 8000):

switchyard-server --config routes.toml --port 8000

2. Configure Page Assist

Click the Page Assist icon in your browser toolbar to open the settings.
Page Assist
Navigate to Settings OpenAI Compatible API.
Page Assist
Click Add Provider and select Custom.
Page Assist
Fill in the endpoint settings:
- Base URL: http://localhost:8000/v1 (or your Switchyard address)
  - API Key: Leave empty (or enter none if a key is required by the UI)
Save the settings.
Page Assist

3. Select Your Switchyard Route

In Page Assist's model selector dropdown:

Type or select the Route ID defined in your Switchyard routes.toml (e.g., auto-select or my-smart-router).

When you type a prompt into Page Assist's sidebar or web UI, Page Assist sends the request to Switchyard. Switchyard evaluates the query, picks the best local model running in Ollama, and streams the answer back to your browser sidebar.

You

Would that combination allow me to use OpenRouter?

Gemini

Yes, absolutely. That combination (Page Assist Switchyard Ollama + OpenRouter) creates a powerful, hybrid setup.

With this setup, Page Assist acts as your front-end browser interface, Switchyard acts as the intelligent traffic controller running in the background, and OpenRouter provides access to cloud models alongside your local Ollama models.

How the Full Stack Works Together

Page Assist (Front-End): You highlight text on a webpage or open the extension sidebar and type a prompt. Page Assist sends the request to http://localhost:8000/v1 (Switchyard's local server).
Switchyard (The Router): Switchyard inspects the query using your configured strategy.
LangChain
- Simple task / summary? Route to local Ollama.
  - Complex logic / huge context / difficult reasoning? Route to OpenRouter.
Backends (Ollama & OpenRouter):
- Ollama handles local inference at $0 cost.
  - OpenRouter handles cloud calls (e.g., DeepSeek, Claude, GPT-4, or Qwen cloud models) using your OpenRouter API key.

Example Configuration (routes.toml)

Here is how you set up routes.toml to blend local Ollama and OpenRouter together seamlessly:

schema_version = 1

# 1. Local Ollama Client
[llm_clients.ollama_local]
format = "openai_chat"
base_url = "http://localhost:11434/v1"

# 2. OpenRouter Cloud Client
[llm_clients.openrouter]
format = "openai_chat"
base_url = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"  # Export this in your shell environment

# Local Target (e.g., Qwen 2.5 or Llama 3)
[targets.local_fast]
id = "qwen2.5:7b"
llm_client = "ollama_local"

# OpenRouter Target (e.g., DeepSeek R1, Claude, or GPT-4o)
[targets.cloud_smart]
id = "deepseek/deepseek-r1" # Or "anthropic/claude-3.5-sonnet"
llm_client = "openrouter"

# Hybrid Auto-Router Route
[routes.hybrid_browser_router]
id = "hybrid-assistant"
type = "llm_classifier"
classifier_target = "local_fast" # Small local model acts as the judge
weak_target = "local_fast"       # Simple page tasks run locally for free
strong_target = "cloud_smart"    # Complex queries get forwarded to OpenRouter

Launching the Stack

Before launching Switchyard, export your OpenRouter key into your shell environment:

export OPENROUTER_API_KEY="sk-or-v1-your-key-here"
switchyard-server --config routes.toml --port 8000

Then in Page Assist:

Point custom OpenAI provider to http://localhost:8000/v1.
Set your model to hybrid-assistant.

Page Assist will give you a single interface, while Switchyard automatically handles whether your query stays local or calls OpenRouter.

You

Can all free models be used in OpenRouter?

Gemini

Yes, all free models available on OpenRouter can be used in this setup.

OpenRouter regularly hosts models with a $0.00 cost per million tokens (often identified by the :free suffix in their model IDs, such as meta-llama/llama-3.3-70b-instruct:free, google/gemma-2-9b-it:free, or qwen/qwen-2.5-coder-32b-instruct:free).

Because Switchyard interacts with OpenRouter using standard OpenAI-compatible API calls, any free model accessible through your OpenRouter API key works identically to paid models.

How to use OpenRouter Free Models in Switchyard

To add OpenRouter's free models to your routes.toml, simply set the model id to the specific free model slug from OpenRouter:

schema_version = 1

# OpenRouter Client Setup
[llm_clients.openrouter]
format = "openai_chat"
base_url = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"

# Target 1: Free Llama 3.3 70B on OpenRouter
[targets.openrouter_free_llama]
id = "meta-llama/llama-3.3-70b-instruct:free"
llm_client = "openrouter"

# Target 2: Free Qwen 2.5 Coder 32B on OpenRouter
[targets.openrouter_free_qwen]
id = "qwen/qwen-2.5-coder-32b-instruct:free"
llm_client = "openrouter"

# Local Ollama Target
[targets.local_fast]
id = "qwen2.5:7b"
llm_client = "ollama_local"

# Example Policy: Routing between local models and OpenRouter free tiers
[routes.free_hybrid_router]
id = "all-free-router"
type = "llm_classifier"
classifier_target = "local_fast"
weak_target = "local_fast"                   # Runs locally
strong_target = "openrouter_free_llama"     # Offloads complex prompts to free 70B cloud model

Important Considerations for OpenRouter Free Models

Model ID Suffix (:free): Make sure to include the exact model tag (including :free at the end) in your id field. If you omit :free, OpenRouter will attempt to route to the paid variant of that model.
Rate Limits & Availability: Free models on OpenRouter are subject to global request rate limits and higher traffic concurrency constraints compared to paid tiers. If a free model hits a provider rate limit or downtime, Switchyard will receive a rate-limit error (429).
API Key Requirement: Even though the models are $0 cost, OpenRouter still requires you to pass a valid OPENROUTER_API_KEY in the request header to manage rate limits per account.

You

I only need one key?

Gemini

Yes, exactly. Just one OpenRouter API key handles everything.

You do not need separate accounts or API keys for different models. A single key generated from your OpenRouter account gives you unified access to:

All free models (like meta-llama/llama-3.3-70b-instruct:free or qwen/qwen-2.5-coder-32b-instruct:free)
Paid models (like GPT-4o, Claude 3.5 Sonnet, or DeepSeek R1)
Multimodal capabilities (vision, audio, image generation)
OpenRouter

How it stays organized in your local stack

┌──► Local Ollama (No API Key) ──► Runs local models (Qwen, Llama, etc.)
               │
Switchyard ────┤
               │
               └──► OpenRouter (1 API Key) ────► Accesses ALL cloud models (Free & Paid)

In your environment, you export that single key:

export OPENROUTER_API_KEY="sk-or-v1-your-key-here"

Then in your Switchyard routes.toml, every cloud target points to llm_client = "openrouter", regardless of which underlying provider (Meta, Anthropic, Google, Qwen, DeepSeek) or model tier you choose. Switchyard passes that single key to OpenRouter, and OpenRouter routes the request upstream automatically.

You

If I get a rate limit will Switchyard use another model (like a local ollama model)?

Gemini

Yes, Switchyard includes built-in fallback and failover mechanisms.

If an upstream provider (like OpenRouter) returns a rate-limit error (HTTP 429), hits a context limit, times out, or encounters a server error, Switchyard automatically catches the exception and routes the prompt to your configured fallback target (like your local Ollama model).

How to Configure Fallbacks in routes.toml

Switchyard uses a cascade / fallback target chain. You can explicitly tell it where to send traffic if the primary route fails or gets rate-limited.

Here is how you set up OpenRouter as your primary target, with local Ollama as the automatic fallback:

schema_version = 1

# Local Ollama client
[llm_clients.ollama_local]
format = "openai_chat"
base_url = "http://localhost:11434/v1"

# OpenRouter client
[llm_clients.openrouter]
format = "openai_chat"
base_url = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"

# Primary Target: Free OpenRouter model
[targets.openrouter_free]
id = "meta-llama/llama-3.3-70b-instruct:free"
llm_client = "openrouter"

# Fallback Target: Local Ollama model (always available, no rate limits)
[targets.local_fallback]
id = "qwen2.5:7b"
llm_client = "ollama_local"

# Router strategy with built-in fallback
[routes.rate_limit_safe_router]
id = "smart-fallback-router"
type = "llm_classifier"
classifier_target = "local_fallback"   # Use local model as judge to avoid wasting cloud API limits
weak_target = "local_fallback"
strong_target = "openrouter_free"

# Explicit fallback rule when strong_target hits errors or 429 rate limits
fallback_target = "local_fallback"

Why this setup is ideal for OpenRouter Free Models:

Seamless Continuity: When OpenRouter throws a 429 Too Many Requests response, Switchyard immediately re-executes the prompt against local_fallback without failing your chat session or throwing an error to Page Assist.
Zero Downtime: Your browser sidebar stays active even if OpenRouter's free tier is experiencing high traffic or temporary downtime.
No wasted credits/limits: You can use local Ollama for the initial classifier/routing evaluation so you don't burn through rate limits just deciding which model to pick.

You

You can use local Ollama for the initial classifier/routing evaluation.: Show me how.

Gemini

Here is how to set up local Ollama as your classifier (the "router judge").

By offloading the evaluation step to a fast local model, Switchyard checks prompt complexity locally for $0 cost without touching your OpenRouter API rate limits.

Step 1: Complete routes.toml Setup

In this configuration, your local qwen2.5:7b acts as the judge to evaluate incoming prompts. If the prompt is simple, it runs locally on qwen2.5:7b. If it requires heavy reasoning, Switchyard routes it to OpenRouter's free Llama 3.3 70B.

schema_version = 1

# ==========================================
# 1. CLIENT ENDPOINTS
# ==========================================

# Local Ollama Backend
[llm_clients.ollama_local]
format = "openai_chat"
base_url = "http://localhost:11434/v1"

# OpenRouter Cloud Backend
[llm_clients.openrouter]
format = "openai_chat"
base_url = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"

# ==========================================
# 2. TARGET MODELS
# ==========================================

# Local Model (Fast & Free)
[targets.local_fast]
id = "qwen2.5:7b"
llm_client = "ollama_local"

# Cloud Model (Free OpenRouter Tier)
[targets.openrouter_70b]
id = "meta-llama/llama-3.3-70b-instruct:free"
llm_client = "openrouter"

# ==========================================
# 3. ROUTE CONFIGURATION
# ==========================================

[routes.smart_hybrid_route]
id = "smart-hybrid"                   # Target ID exposed to Page Assist
type = "llm_classifier"               # Tells Switchyard to judge prompt content

# Local Ollama evaluates the query first
classifier_target = "local_fast"     

# Targets selected based on local evaluation:
weak_target = "local_fast"           # Simple queries stay local
strong_target = "openrouter_70b"     # Complex queries go to OpenRouter

# Fallback in case OpenRouter hits rate limits (429) or errors
fallback_target = "local_fast"

Step 2: How the Request Flow Operates

Page Assist Prompt
        │
        ▼
Switchyard Proxy (port 8000)
        │
        ├─ Step 1: Evaluates prompt locally using [targets.local_fast] (qwen2.5:7b via Ollama)
        │
        ├─ Query is Simple ────────► Runs on local_fast (Ollama)
        │
        └─ Query is Complex ───────► Routes to openrouter_70b (OpenRouter Cloud)
                                            │
                                            └─ If Rate-Limited (429) ──► Falls back to local_fast

Step 3: Running the Stack

Export your OpenRouter key in your terminal:

export OPENROUTER_API_KEY="sk-or-v1-your-key-here"

Start Switchyard pointing to your TOML config:

switchyard-server --config routes.toml --port 8000

In Page Assist Settings:
- Set Custom OpenAI Base URL to http://localhost:8000/v1
  - Select model ID: smart-hybrid

Now, every prompt sent through Page Assist gets analyzed by your local Ollama instance before Switchyard decides whether to answer it locally or call OpenRouter.

Gemini

Yes, that port conflict will prevent Switchyard from binding if Portainer is using port 8000.

Portainer defaults to binding host port 8000 (for its HTTP tunnel/edge agent port) alongside port 9000 or 9443.

By default, switchyard-server listens on port 4000, but if you specify --port 8000, it will crash with an EADDRINUSE (address already in use) error because Portainer is actively occupying it.

How to Fix It

You have two simple options:

Option 1: Run Switchyard on its default port or a custom port (Recommended)

Simply launch Switchyard on port 4000 (its native default) or any unassigned port like 8080 or 8001:

switchyard-server --config routes.toml --port 4000

Then, inside Page Assist settings, update the Custom Base URL to match: http://localhost:4000/v1

Option 2: Change Portainer's host mapping

If you specifically need Switchyard on port 8000, you can re-create your Portainer container to remap or drop host port 8000:

# Stop and remove existing Portainer container
docker stop portainer
docker rm portainer

# Run Portainer without mapping host port 8000 (only keeping web UI on 9000/9443)
docker run -d -p 9000:9000 -p 9443:9443 --name portainer \
    --restart=always \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v portainer_data:/data \
    portainer/portainer-ce:latest

Gemini

Looking at your Portainer container list, you actually have two active port conflicts on host port 8000:

portainer has mapped 8000:8000 (Portainer's Edge Tunnel port).
talkwithme has mapped 8082:8000 (another container binding host port 8082 internally to 8000).

How to Fix It in Portainer

You have two ways to resolve this without using the command line:

Method 1: Change Switchyard's Port (Easiest)

Avoid host port 8000 entirely. Run switchyard-server on its standard default port 4000 or another unused port (like 8085 or 4001):

switchyard-server --config routes.toml --port 4000

Then, inside Page Assist, update the Base URL to: http://localhost:4000/v1

Method 2: Modify Portainer via the UI to Free Up Host Port 8000

If you want to free up port 8000 on your host machine using your Portainer web dashboard:

Click on the portainer container in the list.
Click Duplicate/Edit at the top.
Scroll down to Network ports configuration.
Locate the binding for host port 8000 and click Remove (or change host port 8000 to 8001).
Scroll down and click Deploy the container (select Replace when prompted).

You

Can't Duplicate/Edit portainer.

Gemini

Portainer intentionally disables and grays out the Duplicate/Edit button for its own container. A running instance of Portainer cannot modify, recreate, or replace itself from inside its own Web UI.

To remove host port 8000 from Portainer so Switchyard can use it, you must modify Portainer from your host terminal using the Docker CLI.

Step-by-Step Fix via Terminal

Execute these commands on your host system:

Stop and remove the current Portainer container:
```
docker stop portainer
docker rm portainer
```
(Note: Your Portainer configuration and stacks will stay completely intact because they are saved in the portainer_data volume).

Re-launch Portainer without mapping host port 8000:

docker run -d \
  -p 9000:9000 \
  -p 9443:9443 \
  --name portainer \
  --restart=always \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v portainer_data:/data \
  portainer/portainer-ce:lts

Verify host port 8000 is free:
```
sudo lsof -i :8000
```
(If talkwithme is also using port 8000, note that its published port is 8082:8000, which means it binds internal container port 8000 to host port 8082 —so it will not conflict with host port 8000).

Once finished, host port 8000 will be wide open for switchyard-server, and your Portainer Web UI will remain accessible on `

You

Will my containers be OK?

Gemini

Yes, absolutely. Re-launching Portainer will not affect your other containers or your data.

Portainer is just a viewer: Portainer is simply a visual management wrapper around the Docker daemon. Stopping or removing the Portainer container does not stop, remove, or modify any of your other active application containers (like devdocs, kokoro-tts, perplexica, talkwithme, whisper, etc.). They will continue running without interruption.
Portainer's settings are saved in a volume: The deployment command uses -v portainer_data:/data. All your user accounts, stacks, custom settings, and configurations are stored safely in that persistent portainer_data volume on your system. When the new Portainer container attaches to portainer_data, everything will load exactly as you left it.

You

garyrust@MX25-GPU-win:~

$ docker stop portainer

docker rm portainer

portainer

garyrust@MX25-GPU-win:~

$ docker run -d \

-p 9000:9000 \

-p 9443:9443 \

--name portainer \

--restart=always \

-v /var/run/docker.sock:/var/run/docker.sock \

-v portainer_data:/data \

portainer/portainer-ce:lts

f5deeb28d753f82a63f4e4eb354c8f1c2b13f86d2b065eacc5facf8c9f083234

garyrust@MX25-GPU-win:~

$ sudo lsof -i:8000

[sudo] password for garyrust:

garyrust@MX25-GPU-win:~

Gemini

Your command executed cleanly, and host port 8000 is now completely free on MX25-GPU-win.

Notice that running sudo lsof -i :8000 returned an empty prompt—meaning no process on your system is occupying host port 8000. Your Portainer instance is safely running on ports 9000 and 9443 without interfering with 8000.

Next Step: Launch Switchyard

Now you can run switchyard-server directly on port 8000:

export OPENROUTER_API_KEY="sk-or-v1-your-key-here"
switchyard-server --config routes.toml --port 8000

Once running, set Page Assist 's Custom API URL to http://localhost:8000/v1, and it will connect directly to Switchyard.

You

Whisper is on port 9001. Is that going to be a problem?

Gemini

No, that won't be a problem at all.

Your whisper container has a internal-to-external port mapping of 9001:8000 (or 9001:9001). Because Docker maps whisper to host port 9001, it is completely separate from host port 8000.

Ports only conflict if two services try to bind to the exact same host port at the same time. Since 8000 is now completely open, running Switchyard on port 8000 will run smoothly without affecting whisper on 9001 or portainer on 9000 / 9443.

You

I don't think my mic is working on page assist

Gemini

Because Page Assist runs as a browser extension side panel / popup, microphone issues usually come down to browser permissions, Whisper/Speech-To-Text API settings, or HTTP security restrictions.

Here is how to troubleshoot and fix voice input in Page Assist:

Step 1: Grant WebCam/Microphone Permission to the Extension

Browser side panels don't always trigger the automatic microphone permission prompt like normal websites do.

In Vivaldi / Chrome, open the Page Assist Options / Settings page directly in a full tab rather than inside the side panel.
Click the microphone icon in Page Assist.
When the browser prompts "Page Assist wants to use your microphone", click Allow.
Alternatively, go to your browser's site settings:
- URL: chrome://settings/content/microphone (or vivaldi://settings/privacy)
  - Make sure Page Assist's extension ID is listed under Allowed.

Step 2: Check Page Assist's Speech-to-Text Settings

Page Assist supports two main voice modes:

Open Page Assist Settings.
Look for Speech-to-Text (STT) / Voice Input Settings.
Option A (Web Speech API): Uses the browser's built-in speech recognition (requires an active web connection in Chrome/Vivaldi).
Option B (Whisper / Custom API): If you set Page Assist to use a custom Whisper server for local transcription, check the URL:
- Since your Docker local whisper container is running on port 9001, ensure the Whisper API endpoint in Page Assist points to http://localhost:9001/v1/audio/transcriptions (or [http://127.0.0.1:9001](http://127.0.0.1:9001)).

Step 3: Test Audio Input on the System Level

If the browser permissions are allowed but no sound is captured:

Open your system audio settings in XFCE / MX Linux (e.g., pavucontrol / Volume Control).
Go to the Input Devices tab.
Speak into your microphone and verify that the audio level meter jumps.
Go to the Recording tab while clicking the mic button in Page Assist to ensure Vivaldi/Chrome is actively attached to your input device.

Step 4: Full Page Extension Tab Trick

If the mic button in the side-panel sidebar fails to register clicks:

Right-click the Page Assist icon Open in New Tab (or click "Expand" to full page).
Click the microphone button inside the dedicated tab. Chrome extensions are far more reliable at requesting media device stream permissions when running in a full browser tab than in a compressed side-panel iframe.

You

Gemini

Awesome! Everything is clean, free of port conflicts, and configured for Switchyard.

When you're ready to spin up switchyard-server, drop in your routes.toml, or test out the setup with Page Assist, let me know if you run into any hiccups!

You

Show me a complete Toml for Switchyard using the following ollama list: garyrust@MX25-GPU-win:~

$ ollama list

NAME ID SIZE MODIFIED

llama4:scout bf31604e25c2 67 GB 11 minutes ago

glm-4.7-flash:q4_K_M 4475827791a2 19 GB 4 days ago

granite4.2:8b f586c02fdecd 5.3 GB 5 days ago

qwen3.8:27b 22130167c4c2 17 GB 2 weeks ago

nemotron-3.5-lightning:30b e7a64ff15fb1 25 GB 3 weeks ago

muse-glimmer:30b de878ce33ad8 18 GB 3 weeks ago

MichelRosselli/ternary-bonsai:8b-f16 812e80f7cf27 16 GB 7 weeks ago

gemma4:31b-it-qat e0812a55773b 18 GB 2 months ago

gemma4:26b-a4b-it-qat 2dd70431afed 15 GB 2 months ago

nemotron-3-super:120b-a12b-q4_K_M 95acc78b3ffd 86 GB 2 months ago

fredrezones55/Gemma-4-Uncensored-HauhauCS-Aggressive:e4b 0810d70aa9aa 6.3 GB 2 months ago

qwen3-coder:30b-a3b-q8_0 7b438a19895a 32 GB 3 months ago

granite-code:34b 4ce00960ca84 19 GB 3 months ago

huihui_ai/granite4.1-abliterated:30b f823d08bc5a4 17 GB 3 months ago

nemotron-3-super:120b 95acc78b3ffd 86 GB 4 months ago

ministral-3:latest 1922accd5827 6.0 GB 4 months ago

glm-ocr:bf16 6effedd0dc8a 2.2 GB 4 months ago

nemotron-3-nano:30b b725f1117407 24 GB 4 months ago

deepseek-r1:70b d37b54d01a76 42 GB 4 months ago

translategemma:4b c49d986b0764 3.3 GB 4 months ago

lfm2:24b d6c816d74887 14 GB 4 months ago

gpt-oss:120b a951a23b46a1 65 GB 7 months ago

huihui_ai/huihui-moe-abliterated:60b b8a02689b1bc 36 GB 7 months ago

aiasistentworld/ERNIE-4.5-21B-A3B-Thinking-LLM:latest b116c0c3324a 13 GB 9 months ago

cogito:14b d0cac86a2347 9.0 GB 9 months ago

hermes3:latest 4f6b83f30b62 4.7 GB 9 months ago

marco-o1:latest 4752e62baa0a 4.7 GB 9 months ago

hir0rameel/qwen-claude:latest 87381b6b55a9 5.2 GB 9 months ago

snowflake-arctic-embed2:latest 5de93a84837d 1.2 GB 9 months ago

nomic-embed-text:latest 0a109f422b47 274 MB 9 months ago

gpt-oss:latest 17052f91a42e 13 GB 9 months ago

phi4:latest ac896e5b8b34 9.1 GB 9 months ago

jina/jina-embeddings-v2-base-en:latest 377193292084 274 MB 9 months ago

mistral-small3.2:latest 5a408ab55df5 15 GB 9 months ago

mistral-nemo:latest e7e06d107c6c 7.1 GB 9 months ago

garyrust@MX25-GPU-win:~

Gemini

Here is a complete, production-ready routes.toml configured specifically for your local Ollama instance on MX25-GPU-win.

Embedding models (like nomic-embed-text and jina-embeddings) are excluded from general text chat routing as they are meant strictly for vector embeddings.

schema_version = 1

# ==============================================================================
# LOCAL OLLAMA CLIENT
# ==============================================================================
[llm_clients.ollama_local]
format = "openai_chat"
base_url = "http://localhost:11434/v1"

# ==============================================================================
# TARGET DEFINITIONS (ALL LOCAL OLLAMA MODELS)
# ==============================================================================

# --- Fast & Lightweight Models ---
[targets.granite_8b]
id = "granite4.2:8b"
llm_client = "ollama_local"

[targets.ternary_bonsai_8b]
id = "MichelRosselli/ternary-bonsai:8b-f16"
llm_client = "ollama_local"

[targets.gemma4_uncensored]
id = "fredrezones55/Gemma-4-Uncensored-HauhauCS-Aggressive:e4b"
llm_client = "ollama_local"

[targets.ministral_3]
id = "ministral-3:latest"
llm_client = "ollama_local"

[targets.translategemma_4b]
id = "translategemma:4b"
llm_client = "ollama_local"

[targets.cogito_14b]
id = "cogito:14b"
llm_client = "ollama_local"

[targets.hermes3]
id = "hermes3:latest"
llm_client = "ollama_local"

[targets.marco_o1]
id = "marco-o1:latest"
llm_client = "ollama_local"

[targets.qwen_claude]
id = "hir0rameel/qwen-claude:latest"
llm_client = "ollama_local"

[targets.gpt_oss_13b]
id = "gpt-oss:latest"
llm_client = "ollama_local"

[targets.phi4]
id = "phi4:latest"
llm_client = "ollama_local"

[targets.mistral_small_32]
id = "mistral-small3.2:latest"
llm_client = "ollama_local"

[targets.mistral_nemo]
id = "mistral-nemo:latest"
llm_client = "ollama_local"

# --- Mid-Tier / Reasoning Models ---
[targets.glm_47_flash]
id = "glm-4.7-flash:q4_K_M"
llm_client = "ollama_local"

[targets.qwen38_27b]
id = "qwen3.8:27b"
llm_client = "ollama_local"

[targets.nemotron_35_lightning]
id = "nemotron-3.5-lightning:30b"
llm_client = "ollama_local"

[targets.muse_glimmer_30b]
id = "muse-glimmer:30b"
llm_client = "ollama_local"

[targets.gemma4_31b]
id = "gemma4:31b-it-qat"
llm_client = "ollama_local"

[targets.gemma4_26b]
id = "gemma4:26b-a4b-it-qat"
llm_client = "ollama_local"

[targets.granite_abliterated_30b]
id = "huihui_ai/granite4.1-abliterated:30b"
llm_client = "ollama_local"

[targets.nemotron_3_nano_30b]
id = "nemotron-3-nano:30b"
llm_client = "ollama_local"

[targets.lfm2_24b]
id = "lfm2:24b"
llm_client = "ollama_local"

[targets.ernie_thinking_21b]
id = "aiasistentworld/ERNIE-4.5-21B-A3B-Thinking-LLM:latest"
llm_client = "ollama_local"

# --- Heavy / Large Scale Models ---
[targets.llama4_scout]
id = "llama4:scout"
llm_client = "ollama_local"

[targets.nemotron_3_super_120b_a12b]
id = "nemotron-3-super:120b-a12b-q4_K_M"
llm_client = "ollama_local"

[targets.nemotron_3_super_120b]
id = "nemotron-3-super:120b"
llm_client = "ollama_local"

[targets.deepseek_r1_70b]
id = "deepseek-r1:70b"
llm_client = "ollama_local"

[targets.gpt_oss_120b]
id = "gpt-oss:120b"
llm_client = "ollama_local"

[targets.huihui_moe_60b]
id = "huihui_ai/huihui-moe-abliterated:60b"
llm_client = "ollama_local"

# --- Specialized Domain Models ---
[targets.qwen3_coder]
id = "qwen3-coder:30b-a3b-q8_0"
llm_client = "ollama_local"

[targets.granite_code]
id = "granite-code:34b"
llm_client = "ollama_local"

[targets.glm_ocr]
id = "glm-ocr:bf16"
llm_client = "ollama_local"

# ==============================================================================
# ROUTE DEFINITIONS (EXPOSED TO PAGE ASSIST / FRONTENDS)
# ==============================================================================

# 1. Automatic Hybrid Classifier Route
# Uses lightweight granite4.2:8b as judge, routes simple queries to Granite 8B and complex prompts to DeepSeek-R1 70B
[routes.auto_local_router]
id = "auto-local"
type = "llm_classifier"
classifier_target = "granite_8b"
weak_target = "granite_8b"
strong_target = "deepseek_r1_70b"
fallback_target = "granite_8b"

# 2. Coding Specialist Route
# Uses Granite 8B as judge, routes coding tasks between Qwen3 Coder and Granite Code
[routes.code_router]
id = "code-assistant"
type = "llm_classifier"
classifier_target = "granite_8b"
weak_target = "granite_code"
strong_target = "qwen3_coder"
fallback_target = "granite_code"

# 3. Heavy Reasoning Route (Direct Passthrough)
[routes.reasoning_passthrough]
id = "heavy-reasoning"
type = "passthrough"
target = "deepseek_r1_70b"

# 4. Fast Chat Route (Direct Passthrough)
[routes.fast_chat]
id = "fast-chat"
type = "passthrough"
target = "granite_8b"

Validate and Test

Validate the config syntax without starting the server:

switchyard-server --config routes.toml --dry-run

If valid, launch Switchyard on port 8000:

switchyard-server --config routes.toml --port 8000

In Page Assist, select auto-local, code-assistant, or heavy-reasoning as your model.

For a visual breakdown of establishing custom local targets and classifier policies inside routes.toml, watch this

Switchyard Setup Guide on YouTube

. This video provides helpful visual context on configuring target nodes, testing local proxy endpoints, and checking routing performance.

You

OK. Can this also have OpenRoute's Free Models Router and fallover?

Gemini

Yes, absolutely. You can incorporate OpenRouter’s native Free Models Router (openrouter/free) directly as a target inside Switchyard, and configure your local Ollama models as a failover / fallback chain.

By setting openrouter/free as your primary target, OpenRouter handles selecting an available free cloud model. If OpenRouter hits rate limits (429), encounters downtime, or fails, Switchyard automatically falls back to your local Ollama instance.

Updated routes.toml with OpenRouter Free Router & Local Fallbacks

Here is the complete configuration snippet to add to your existing routes.toml:

schema_version = 1

# ==============================================================================
# CLIENT ENDPOINTS
# ==============================================================================

[llm_clients.ollama_local]
format = "openai_chat"
base_url = "http://localhost:11434/v1"

[llm_clients.openrouter]
format = "openai_chat"
base_url = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"

# ==============================================================================
# TARGET DEFINITIONS
# ==============================================================================

# --- OpenRouter Native Free Models Router ---
# Automatically dynamically selects from available $0 cloud models on OpenRouter
[targets.openrouter_free_router]
id = "openrouter/free"
llm_client = "openrouter"

# --- Local Ollama Targets (Fallbacks & Local Execution) ---
[targets.local_fast]
id = "granite4.2:8b"
llm_client = "ollama_local"

[targets.local_heavy]
id = "llama4:scout"
llm_client = "ollama_local"

[targets.local_reasoning]
id = "deepseek-r1:70b"
llm_client = "ollama_local"

# ==============================================================================
# ROUTE CONFIGURATIONS
# ==============================================================================

# 1. Cloud-First Free Router with Local Fallback
# Directs queries to OpenRouter's Free Router. If rate-limited or offline, falls back to local Granite 8B.
[routes.openrouter_free_with_fallback]
id = "openrouter-free"
type = "passthrough"
target = "openrouter_free_router"
fallback_target = "local_fast"

# 2. Smart Classifier: Local Judge -> OpenRouter Free Router -> Local DeepSeek
# Evaluates prompt locally using Granite 8B. Simple prompts run locally; 
# complex prompts route to OpenRouter's Free Router, with local DeepSeek-R1 70B as secondary fallback.
[routes.smart_hybrid_openrouter]
id = "smart-hybrid-free"
type = "llm_classifier"
classifier_target = "local_fast"         # Evaluates intent locally for free
weak_target = "local_fast"               # Light prompts run on local Granite 8B
strong_target = "openrouter_free_router" # Complex prompts offloaded to OpenRouter Free Router
fallback_target = "local_reasoning"      # Falls back to local 70B if OpenRouter is rate-limited

How this routing behavior works:

openrouter-free Route: When selected in Page Assist, Switchyard sends the request to OpenRouter's openrouter/free endpoint. OpenRouter selects a available $0 cloud model. If OpenRouter throws a 429 Too Many Requests or connection error, Switchyard transparently re-runs the prompt on your local granite4.2:8b without interrupting Page Assist.
smart-hybrid-free Route: Uses your local granite4.2:8b model as a local judge to inspect incoming prompts. Light prompts stay local. Heavy prompts call OpenRouter's Free Router, and if rate-limited, fail over to your local heavy deepseek-r1:70b.