Troubleshooting

Common issues and solutions for the AI-Q blueprint.

Installation Issues

Issue	Cause	Fix
ModuleNotFoundError: aiq_agent	Package not installed in editable mode	uv pip install -e .
nat command not found	Using system nat instead of venv	Use .venv/bin/nat or activate the venv
NeMo Agent Toolkit plugins not found	Plugins not installed	uv pip install -e . to register entry points
Pre-commit hook failures	Missing pre-commit setup	pre-commit install && pre-commit run --all-files
ormsgpack attribute error	Version conflict with LangGraph	uv pip install "ormsgpack>=1.5.0"

API Key Issues

Issue	Cause	Fix
[404] Not found for account	Invalid or expired NVIDIA API key	Regenerate key at build.nvidia.com
Gateway timeout (504)	Model endpoint overloaded or unavailable	Retry, or switch to a different model in config
Tavily search returns empty	Invalid TAVILY_API_KEY	Verify key at tavily.com
You.com tools return an unavailable or 401 error	Missing or invalid YDC_API_KEY	Create or verify the key using the You.com quickstart and restart AI-Q
Exa search returns empty or 401	Invalid or missing EXA_API_KEY	Verify key at exa.ai
Nimble search returns empty or 401	Invalid or missing NIMBLE_API_KEY	Verify the key through Nimble
Nimble search returns 403 with "enterprise"	search_depth: fast requires an Enterprise plan	Switch to search_depth: lite (default) or deep, or upgrade your Nimble plan
Serper search fails	Missing SERPER_API_KEY	Set key or remove paper_search_tool from config

Runtime Issues

Issue	Cause	Fix
Agent hangs on deep research	LLM timeout or rate limit	Inspect Relay logs/traces and check LLM API availability and rate limits
HTTP 429 or 503 on deep research	Nemotron hosted endpoint availability	Retry after a short delay, reduce concurrency, or follow the self-hosting guidance for consistent throughput
Intermittent shallow-research failure with Nemotron 3.5 Lightning on NVIDIA API Catalog	The hosted serving profile can produce citation-incomplete or malformed final drafts	Use Nemotron Ultra for the shallow role, or use a validated self-hosted Lightning serving profile; see Nemotron 3.5 Lightning on NVIDIA API Catalog
Shallow research returns generic answers	Insufficient tool calls	Increase max_tool_iterations (default: 5)
Clarifier keeps asking questions	Too many clarification turns	Reduce max_turns, or set enable_clarifier: false in the workflow to disable clarification
SSE stream disconnects	Network timeout	Client auto-reconnects using last_event_id; refer to Data Flow
Job status stuck on RUNNING	Dask worker crashed	Check Dask logs; the ghost job reaper will eventually mark it FAILURE
OpenShell setup, attestation, readiness, or deletion fails	Gateway, version, policy/config, image, or service-owner mismatch	Follow the canonical OpenShell inspection and troubleshooting guide

Nemotron Hosted Endpoint Availability

Nemotron 3.5 Lightning (nvidia/nemotron-3.5-lightning-30b-a3b) and Nemotron 3 Ultra (nvidia/nemotron-3-ultra-550b-a55b) are compatible and tested with AIQ, but their NVIDIA-hosted endpoints can have limited availability during high demand. During peak periods you may observe:

• Elevated latency or timeouts on LLM inference calls
• HTTP 429 (rate-limited) or 503 (service unavailable) responses from the Build API
• Degraded agent workflow performance due to upstream model availability

Default Configuration: The default configs use Nemotron 3.5 Lightning for intent classification and shallow research, and Nemotron 3 Ultra for clarification and all deep-research roles. If a hosted endpoint is saturated, retry after a short delay, reduce concurrency, or self-host a downloadable model for consistent throughput.

Recommended Mitigation: Self-Host the Affected Model

For production and staging deployments that require consistent throughput and low-latency inference, self-host a downloadable NVIDIA NIM rather than relying on shared endpoints. Preview endpoint availability and downloadable NIM availability do not necessarily move in lockstep; verify the current model card before choosing an image.

• Self-host Nemotron 3.5 Lightning 30B A3B for the default intent and shallow-research roles
• Self-host Nemotron 3 Ultra 550B A55B for the default clarification and deep-research roles

Once your self-hosted endpoint is running, update the corresponding base_url in your config to point at it. AIQ's configuration validator currently requires NVIDIA_API_KEY for every _type: nim profile, even when a local NIM does not enforce client authentication. Set a non-secret placeholder for the local deployment before starting AIQ:

export NVIDIA_API_KEY=local-nim

Then reference that variable in the local profile:

llms:
  local_ultra_llm:
    _type: nim
    # Use the identifier returned by the local NIM's /v1/models endpoint.
    model_name: nvidia/nemotron-3-ultra-550b-a55b
    base_url: "https://<your-ultra-endpoint>/v1"
    api_key: ${NVIDIA_API_KEY}
    temperature: 0.2
    top_p: 0.7
    max_tokens: 16384
    num_retries: 5
    chat_template_kwargs:
      enable_thinking: false

Nemotron 3.5 Lightning on NVIDIA API Catalog

The default profiles retain Nemotron 3.5 Lightning for intent classification and shallow research. When the shallow role uses Lightning through the NVIDIA API Catalog endpoint (integrate.api.nvidia.com), the hosted serving profile can intermittently return citation-incomplete or malformed final drafts. AI-Q verifies the draft against the captured source registry and fails closed instead of publishing an unsupported answer, so an affected request ends with a failed workflow outcome even when its search completed successfully.

This behavior depends on the serving profile, not only the model weights. It did not reproduce at the same rate in validation with the tested self-hosted NVFP4 vLLM profile. For a deployment that prioritizes shallow-answer reliability, use one of these configurations:

• Assign Nemotron Ultra to shallow_research_agent.llm, while keeping Lightning for intent classification.
• Serve Lightning through a self-hosted profile that you validate end to end with AI-Q's citation and tool-calling workflow.

The Brev getting-started launchable uses the first option. This keeps the launchable reliable without changing the model assignment in the general-purpose shipped profiles.

Knowledge Layer Issues

Issue	Cause	Fix
Unknown backend	Adapter module not imported	Ensure backend package is installed: uv pip install -e "sources/knowledge_layer[llamaindex]"
Empty retrieval results	Ingestion and retrieval resolved different collections	Verify the upload-path collection and active conversation-id; without session context, verify the configured collection_name fallback
Foundational RAG connection refused	RAG Blueprint not running	Start the RAG Blueprint server; verify rag_url and ingest_url
milvus-lite required	Missing dependency	uv pip install "pymilvus[milvus_lite]"

Docker / Deployment Issues

Issue	Cause	Fix
Container fails to start	Missing environment variables	Check deploy/.env has all required keys
Port already in use	Another service on port 3000/8000	Set PORT=8100 or FRONTEND_PORT=3100 in .env
UI shows "Backend unavailable"	Backend not healthy	curl http://localhost:8000/health; check backend container logs

VM / Remote Development

If you are running the AI-Q blueprint on a remote VM (cloud instance, WSL, SSH server) and accessing it from your local browser, localhost:3000 and localhost:8000 will not resolve because the services are listening on the VM — not your local machine.

SSH Port Forwarding

Forward the required ports through your SSH connection:

# Forward both the frontend and backend ports
ssh -L 3000:localhost:3000 -L 8000:localhost:8000 user@your-vm-host

Then open http://localhost:3000 on your local machine as usual.

To forward ports to an already-active SSH session, you can also use ~C (SSH escape sequence) to open the SSH command line and type the following on a single line (press Enter at the end):

-L 3000:localhost:3000 -L 8000:localhost:8000

VS Code Remote SSH

If you are using VS Code Remote-SSH, ports are typically forwarded automatically when the server starts listening. If not, open the Ports panel (Ctrl+Shift+P → "Ports: Focus on Ports View") and add ports 3000 and 8000 manually.

Common Symptoms

Symptom	Cause	Fix
"This site can't be reached" on localhost:3000 or localhost:8000	Ports not forwarded from VM to local machine	Use SSH port forwarding (see above)
Connection refused after forwarding	Service not running on the VM	SSH into the VM and verify with curl http://localhost:8000/health
Port forwarding conflicts	Local port already in use	Use alternate local ports: ssh -L 3001:localhost:3000 -L 8001:localhost:8000 user@vm

Docker Compose deployments on the VM handle container-to-host port mapping automatically. The SSH forwarding described here is for making the VM's ports accessible on your local machine.

Debugging Tips

Inspect Relay Logging

# In your config YAML
workflow:
  _type: chat_deepresearcher_agent
  relay:
    logging: true

Phoenix Tracing Through Relay

For full setup and trace-reading instructions, see Observability with NeMo Relay.

Start a Phoenix server and enable tracing in config:

workflow:
  relay:
    observability:
      opentelemetry:
        enabled: true
        endpoints:
          - type: openinference
            endpoint: ${RELAY_OTEL_ENDPOINT:-http://localhost:6006/v1/traces}
            resource_attributes:
              openinference.project.name: aiq-relay

Then open http://localhost:6006 to inspect traces, token usage, and latency. Set RELAY_OTEL_ENDPOINT to use a remote Phoenix or collector endpoint; local Phoenix is the default. If the trace is missing, also inspect the project configured in ~/.config/nemo-relay/plugins.toml; Relay can discover an existing user-level Phoenix destination.

Check Registered Components

# List registered NeMo Agent Toolkit plugins
.venv/bin/nat info components