VirtualMCPServer Kubernetes Guide

July 3, 2026 · View on GitHub

This guide provides specialized content for migrating to Kubernetes and troubleshooting VirtualMCPServer deployments.

For general VirtualMCPServer documentation, see the ToolHive Documentation Website:

For API field definitions, see the VirtualMCPServer API Reference.

Table of Contents

Migration Guide: CLI to Kubernetes

Overview

Migrating from CLI (thv) to Kubernetes deployment provides several benefits:

  • Scalability: Run multiple instances, automatic restarts
  • Multi-tenancy: Isolate workloads by namespace
  • GitOps: Declarative configuration management
  • High availability: Kubernetes self-healing and scheduling

This guide covers migrating both individual MCPServers and VirtualMCPServers.

Migrating Individual MCP Servers

Step 1: Export from CLI

Export your existing workload configuration:

# Export as Kubernetes YAML (recommended)
thv export my-server ./my-server.yaml --format k8s

# Or export as RunConfig JSON for manual conversion
thv export my-server ./my-server-config.json --format json

The --format k8s option automatically converts to MCPServer CRD format.

Step 2: Review and Adjust

Review the exported YAML and make any necessary adjustments:

apiVersion: toolhive.stacklok.dev/v1beta1
kind: MCPServer
metadata:
  name: my-server
  namespace: default  # Adjust namespace if needed
spec:
  image: ghcr.io/example/my-server:latest
  transport: streamable-http
  proxyPort: 8080
  mcpPort: 8080
  # Review and adjust these fields:
  resources:
    requests:
      cpu: "100m"
      memory: "128Mi"
    limits:
      cpu: "200m"
      memory: "256Mi"

Key adjustments:

  • Namespace: Choose appropriate namespace
  • Resources: Set CPU/memory limits for Kubernetes
  • Service Type: Defaults to ClusterIP (change to LoadBalancer if needed)
  • Authentication: OIDC configs may need URLs updated for cluster context

Step 3: Deploy to Kubernetes

# Install operator if not already installed
helm install toolhive-operator-crds oci://ghcr.io/stacklok/toolhive/toolhive-operator-crds
helm install toolhive-operator oci://ghcr.io/stacklok/toolhive/toolhive-operator \
  -n toolhive-system --create-namespace

# Apply the MCPServer
kubectl apply -f my-server.yaml

# Verify deployment
kubectl get mcpserver my-server
kubectl get pods -l app.kubernetes.io/name=my-server

Step 4: Update Clients

Update MCP clients to use the new Kubernetes service endpoint:

Before (CLI):

http://localhost:8080

After (Kubernetes - in cluster):

http://my-server.default.svc.cluster.local:8080

After (Kubernetes - external):

# Option 1: Port-forward for testing
kubectl port-forward service/my-server 8080:8080

# Option 2: Use LoadBalancer
kubectl get service my-server
# Use EXTERNAL-IP from output

# Option 3: Use Ingress
https://my-server.example.com

Step 5: Decommission CLI Instance

Once verified in Kubernetes:

# Stop and remove CLI workload
thv stop my-server
thv rm my-server

Migrating VirtualMCPServers

Understanding the Migration

A VirtualMCPServer in Kubernetes aggregates multiple backend MCPServers. The CLI equivalent would be running multiple thv instances with a group.

CLI Setup Example:

# CLI: Running multiple servers
thv run github --image ghcr.io/example/github-mcp
thv run jira --image ghcr.io/example/jira-mcp
thv run slack --image ghcr.io/example/slack-mcp

# Note: CLI grouping works differently - backends reference groups via config

Kubernetes Equivalent: VirtualMCPServer + MCPGroup + MCPServers

Step 1: Export Backend Servers

Export each backend server individually:

thv export github ./github.yaml --format k8s
thv export jira ./jira.yaml --format k8s
thv export slack ./slack.yaml --format k8s

Step 2: Create MCPGroup

Create an MCPGroup to organize the backends:

# mcp-group.yaml
apiVersion: toolhive.stacklok.dev/v1beta1
kind: MCPGroup
metadata:
  name: my-services
  namespace: default
spec:
  description: Migrated from CLI group 'my-services'

Add groupRef to each exported MCPServer:

# github.yaml
apiVersion: toolhive.stacklok.dev/v1beta1
kind: MCPServer
metadata:
  name: github
  namespace: default
spec:
  groupRef:
    name: my-services  # Add this field
  image: ghcr.io/example/github-mcp
  transport: streamable-http
  proxyPort: 8080
  mcpPort: 8080

Repeat for jira.yaml and slack.yaml.

Step 4: Create VirtualMCPServer

Create a VirtualMCPServer to aggregate the backends:

# virtual-mcp-server.yaml
apiVersion: toolhive.stacklok.dev/v1beta1
kind: VirtualMCPServer
metadata:
  name: my-vmcp
  namespace: default
spec:
  groupRef:
    name: my-services
  config: {}

  # Configure authentication (adjust from CLI if using OIDC)
  # For OIDC, use oidcConfigRef with a shared MCPOIDCConfig resource:
  #   type: oidc
  #   oidcConfigRef:
  #     name: my-oidc-config
  #     audience: my-vmcp
  incomingAuth:
    type: anonymous  # Or configure OIDC (see above)
    authzConfig:
      type: inline
      inline:
        policies:
          - 'permit(principal, action, resource);'

  # Backend authentication discovery
  outgoingAuth:
    source: discovered

  # Tool aggregation strategy
  aggregation:
    conflictResolution: prefix
    conflictResolutionConfig:
      prefixFormat: "{workload}_"

Step 5: Deploy Everything

# Deploy in order: Group → Backends → VirtualMCP
kubectl apply -f mcp-group.yaml
kubectl apply -f github.yaml
kubectl apply -f jira.yaml
kubectl apply -f slack.yaml
kubectl apply -f virtual-mcp-server.yaml

# Verify deployment
kubectl get mcpgroup my-services
kubectl get mcpserver
kubectl get virtualmcpserver my-vmcp

Step 6: Verify and Test

Check that the VirtualMCPServer discovered all backends:

# Check discovered backends
kubectl get virtualmcpserver my-vmcp -o jsonpath='{.status.discoveredBackends}' | jq

# Test connectivity
kubectl port-forward service/my-vmcp 8080:8080
# Test with MCP client at http://localhost:8080

Step 7: Update Clients and Decommission CLI

Update clients to use the VirtualMCPServer endpoint and remove CLI instances:

# Stop CLI instances
thv stop github jira slack

# Remove CLI instances
thv rm github jira slack

# Remove CLI group
thv group rm my-services

Migration Checklist

Use this checklist to ensure complete migration:

Pre-Migration:

  • Document all running CLI workloads (thv list)
  • Export configurations for all workloads
  • Note any custom authentication or middleware configurations
  • Identify workload dependencies and groups
  • Plan namespace strategy for Kubernetes

During Migration:

  • Install ToolHive operator in Kubernetes
  • Create namespaces if needed
  • Deploy MCPGroups (if using VirtualMCPServers)
  • Deploy all backend MCPServers
  • Link MCPServers to MCPGroups
  • Deploy VirtualMCPServers
  • Verify all resources are Ready

Post-Migration:

  • Test all MCP server endpoints
  • Verify tool/resource/prompt availability
  • Update client configurations
  • Test authentication flows
  • Monitor for errors or issues
  • Decommission CLI instances
  • Update documentation with new endpoints

Common Migration Scenarios

Scenario 1: Simple MCP Server

CLI:

thv run weather --image ghcr.io/example/weather:latest

Kubernetes:

thv export weather ./weather.yaml --format k8s
kubectl apply -f weather.yaml

Scenario 2: MCP Server with OIDC

CLI (with local OIDC config):

thv run github \
  --image ghcr.io/example/github-mcp \
  --oidc-issuer https://auth.example.com \
  --oidc-client-id github-client

Kubernetes:

The preferred approach is to create a shared MCPOIDCConfig resource and reference it via oidcConfigRef. This lets you define OIDC provider settings once and reuse them across multiple servers.

See example configurations:

Inline OIDC and Kubernetes SA OIDC variants were deprecated and removed; use MCPOIDCConfig references instead.

Scenario 3: Grouped Servers (CLI) → VirtualMCPServer (K8s)

CLI:

thv run backend1 --image ghcr.io/example/backend1
thv run backend2 --image ghcr.io/example/backend2
thv group create services
# Note: In CLI, workloads are linked to groups via their configuration

Kubernetes:

# Export backends
thv export backend1 ./backend1.yaml --format k8s
thv export backend2 ./backend2.yaml --format k8s

# Create manifests (add groupRef to each backend YAML)
cat > resources.yaml <<EOF
apiVersion: toolhive.stacklok.dev/v1beta1
kind: MCPGroup
metadata:
  name: services
---
# Include backend1.yaml content with groupRef: {name: services}
# Include backend2.yaml content with groupRef: {name: services}
---
apiVersion: toolhive.stacklok.dev/v1beta1
kind: VirtualMCPServer
metadata:
  name: services-vmcp
spec:
  groupRef:
    name: services
  incomingAuth:
    type: anonymous
  outgoingAuth:
    source: discovered
  aggregation:
    conflictResolution: prefix
EOF

kubectl apply -f resources.yaml

Troubleshooting Migration Issues

Issue: Exported YAML fails validation

Solution: Check for CLI-specific fields that need adjustment:

  • Update URLs from localhost to cluster DNS names
  • Add namespace to metadata
  • Set appropriate resource limits
  • Remove CLI-specific configurations

Issue: OIDC authentication not working

Solution: Update OIDC URLs for Kubernetes context:

  • resourceUrl should use cluster service DNS
  • issuer should be accessible from pods
  • Verify secrets are in the same namespace
  • Check RBAC permissions for service accounts

Issue: Backend servers not discovered by VirtualMCPServer

Solution:

  • Verify all MCPServers have groupRef.name set
  • Ensure all resources are in the same namespace
  • Check MCPServer status: kubectl get mcpserver
  • Review VirtualMCPServer conditions: kubectl describe virtualmcpserver <name>

Issue: Performance degradation after migration

Solution:

  • Increase pod resources (CPU/memory)
  • Adjust timeout configurations
  • Check network policies aren't blocking traffic
  • Monitor pod metrics: kubectl top pod

Best Practices

  1. Test in Staging First: Migrate to a staging Kubernetes cluster before production
  2. Gradual Migration: Migrate one workload at a time, verify before proceeding
  3. Keep CLI Running: Run CLI and K8s in parallel during testing
  4. Document Endpoints: Maintain a mapping of old (CLI) to new (K8s) endpoints
  5. Monitor Closely: Watch logs and metrics after migration
  6. Plan Rollback: Keep CLI configurations as backup until migration is stable
  7. Use GitOps: Store Kubernetes manifests in Git for versioning and rollback

Using MCPServerEntry for Remote Backends

For remote MCP servers that don't need a dedicated proxy, use MCPServerEntry instead of MCPRemoteProxy. This avoids deploying unnecessary proxy pods.

Before (MCPRemoteProxy — deploys a proxy pod):

apiVersion: toolhive.stacklok.dev/v1beta1
kind: MCPRemoteProxy
metadata:
  name: context7
spec:
  remoteUrl: https://mcp.context7.com/mcp
  transport: streamable-http
  groupRef:
    name: engineering-team
  # Requires OIDC config, deploys proxy pod

After (MCPServerEntry — zero infrastructure):

apiVersion: toolhive.stacklok.dev/v1beta1
kind: MCPServerEntry
metadata:
  name: context7
spec:
  remoteUrl: https://mcp.context7.com/mcp
  transport: streamable-http
  groupRef:
    name: engineering-team
  # No pods deployed, VirtualMCPServer connects directly

MCPServerEntry supports the same auth mechanisms as other backends via externalAuthConfigRef, and can use caBundleRef for internal CA certificates. See the examples for complete configurations.

Troubleshooting

Deployment Issues

VirtualMCPServer Stuck in "Pending" Phase

Symptoms:

kubectl get virtualmcpserver my-vmcp
# NAME      PHASE     AGE
# my-vmcp   Pending   5m

Common Causes and Solutions:

1. MCPGroup Not Found

kubectl get virtualmcpserver my-vmcp -o yaml | grep -A 5 conditions
# Look for: GroupRefValidated: False

Solution: Verify the MCPGroup exists:

kubectl get mcpgroup <group-name>

Create if missing or fix spec.groupRef.name in VirtualMCPServer spec.

2. No Backend MCPServers in Group

kubectl get mcpserver -o custom-columns=NAME:.metadata.name,GROUP:.spec.groupRef.name

Solution: Create MCPServers and link them to the group:

spec:
  groupRef:
    name: <group-name>

3. Backend MCPServers Not Ready

kubectl get mcpserver
# Check STATUS column

Solution: Check backend server logs:

kubectl logs -l app.kubernetes.io/name=<mcpserver-name>
kubectl describe mcpserver <mcpserver-name>

VirtualMCPServer in "Degraded" Phase

Symptoms:

kubectl get virtualmcpserver my-vmcp -o jsonpath='{.status.phase}'
# Degraded

Common Causes and Solutions:

1. Some Backends Unhealthy

kubectl get virtualmcpserver my-vmcp -o jsonpath='{.status.discoveredBackends}' | jq
# Check "status" field for each backend

Solution: Investigate unhealthy backends:

kubectl get mcpserver <backend-name>
kubectl logs <backend-pod-name>
kubectl describe pod <backend-pod-name>

2. Partial Failure Mode Configuration

Check your configuration:

spec:
  operational:
    failureHandling:
      partialFailureMode: best_effort  # vs fail

Solution: If using best_effort mode, this is expected behavior when some backends are down. VirtualMCPServer continues serving healthy backends.

To require all backends to be healthy, use partialFailureMode: fail.

Authentication Failures

Symptoms:

  • Clients cannot connect to VirtualMCPServer
  • 401 Unauthorized errors
  • 403 Forbidden errors

Common Causes and Solutions:

1. Missing OIDC Client Secret

kubectl get secret oidc-client-secret

Solution: Create the secret:

apiVersion: v1
kind: Secret
metadata:
  name: oidc-client-secret
  namespace: default
type: Opaque
stringData:
  clientSecret: "YOUR_SECRET"

2. Incorrect OIDC Configuration

Check VirtualMCPServer events:

kubectl describe virtualmcpserver my-vmcp

Solution: Verify OIDC settings:

  • issuer: Must match your OIDC provider URL exactly
  • clientId: Must match the registered client in OIDC provider
  • audience: Must match the expected audience claim
  • resourceUrl: Must match the VirtualMCPServer's accessible URL

3. Authorization Policy Errors

Solution: Test with a permissive policy first:

authzConfig:
  type: inline
  inline:
    policies:
      - 'permit(principal, action, resource);'

Then gradually add restrictions. Common Cedar policy issues:

  • Check syntax is correct
  • Verify attribute names match token claims
  • Test policies with different user roles

Multiple upstream IDPs: when spec.authServerConfig declares more than one upstreamProviders entry, Cedar evaluates claims from the first one by default. Pin a specific provider explicitly via spec.authServerConfig.primaryUpstreamProvider:

spec:
  authServerConfig:
    issuer: https://vmcp.example.com
    primaryUpstreamProvider: okta   # must match one of the configured upstreams
    upstreamProviders:
      - name: okta
        type: oidc
        # ...
      - name: github
        type: oauth2
        # ...
  incomingAuth:
    authzConfig:
      type: inline
      inline:
        policies:
          - 'permit(principal, action, resource);'

Migration: primaryUpstreamProvider location

The field used to live under spec.incomingAuth.authzConfig.inline.primaryUpstreamProvider. It has moved to spec.authServerConfig.primaryUpstreamProvider to sit next to the upstreamProviders list it selects from. The old location is read for one release for backward compatibility; the controller emits a Warning event with reason AuthzPrimaryUpstreamProviderDeprecated whenever it consumes the deprecated location. Move the value to the new location to clear the warning. The deprecated field is planned for removal one release after the deprecation cycle.

Authorization policy errors: misconfigured authz surfaces on the AuthConfigured condition with one of:

  • AuthzConfigMapNotFound: the ConfigMap referenced by spec.incomingAuth.authzConfig.configMap does not exist in the namespace. Create it before reconciling, or fix the name.
  • AuthzConfigMapInvalid: the ConfigMap exists but the payload is missing the configured key, empty, malformed YAML/JSON, fails Cedar validation, or is a registered non-Cedar authorizer (vMCP supports Cedar only). Check the payload shape (see the Cedar v1 schema in the example above).

Enterprise Cedar policies that deny every request: when a policy walks a transitive hierarchy like Client → ClaimGroup → PlatformRole, both Cedar JWT-claim mapping settings and the static entity store must agree on the entity type. The configuration fields live on spec.incomingAuth.authzConfig:

  • groupClaimName / roleClaimName: JWT claim keys to extract.
  • groupEntityType: Cedar entity type used for principal parent UIDs. Must match the entity type used in entitiesJson (e.g. ClaimGroup rather than the default THVGroup).

For configMap-sourced authz, the same fields can be set in the ConfigMap payload (cedar.group_claim_name, cedar.role_claim_name, cedar.group_entity_type); spec-level values on authzConfig override the ConfigMap when set.

Backend Discovery Issues

Backends Not Discovered

Symptoms:

kubectl get virtualmcpserver my-vmcp -o jsonpath='{.status.discoveredBackends}' | jq
# Empty array or missing backends

Common Causes and Solutions:

1. Backend Not in MCPGroup

kubectl get mcpserver <backend-name> -o yaml | grep -A1 groupRef

Solution: Verify backend has correct groupRef:

kubectl patch mcpserver <backend-name> --type merge -p '{"spec":{"groupRef":{"name":"<group-name>"}}}'

2. Namespace Mismatch

Solution: Ensure VirtualMCPServer, MCPGroup, and all MCPServers are in the same namespace (security requirement):

kubectl get virtualmcpserver,mcpgroup,mcpserver -n <namespace>

All resources must be in the same namespace. Move resources if needed.

3. Backend Authentication Config Not Found

When using outgoingAuth.source: discovered:

kubectl get mcpserver <backend-name> -o yaml | grep externalAuthConfigRef

Solution: Either:

  • Create MCPExternalAuthConfig if backend requires auth
  • Remove externalAuthConfigRef from backend if no auth required
  • Use outgoingAuth.source: inline and configure explicitly

Tool Conflict Issues

Tool Name Conflicts Not Resolved

Symptoms:

  • Error messages about unresolved tool conflicts
  • Tools missing from aggregated capabilities
  • VirtualMCPServer status shows validation errors

Common Causes and Solutions:

1. Priority Strategy Missing Order

aggregation:
  conflictResolution: priority
  # Missing: conflictResolutionConfig.priorityOrder

Solution: Add priority order with all backend names:

aggregation:
  conflictResolution: priority
  conflictResolutionConfig:
    priorityOrder:
      - backend1
      - backend2
      - backend3

2. Manual Strategy Missing Tool Configuration

Solution: Add explicit tool configuration for all backends:

aggregation:
  conflictResolution: manual
  tools:
    - workload: backend1
      filter: ["tool1", "tool2"]
    - workload: backend2
      filter: ["tool3", "tool4"]

3. Invalid Tool Names in Filter

Solution: Verify actual tool names from backend:

# Port-forward to backend
kubectl port-forward service/<backend-name> 8080:8080

# Query tools endpoint (method depends on transport)
# Or check backend logs during startup
kubectl logs <backend-pod-name> | grep -i tool

Composite Workflow Issues

Workflow Validation Errors

Symptoms:

kubectl get virtualmcpcompositetooldefinition <name> -o jsonpath='{.status.validationStatus}'
# Invalid

Check validation errors:

kubectl get virtualmcpcompositetooldefinition <name> -o jsonpath='{.status.validationErrors}' | jq

Common Causes and Solutions:

1. Circular Dependencies

steps:
  - id: step1
    dependsOn: [step2]
  - id: step2
    dependsOn: [step1]  # Circular!

Solution: Remove circular dependencies. Draw dependency graph if needed.

2. Invalid Tool References

steps:
  - id: deploy
    tool: invalid-format  # Should be: workload.tool_name

Solution: Use correct format: <workload>.<tool_name>

Check available tools from the backend MCPServers directly or test the VirtualMCPServer endpoint.

3. Missing Step Dependencies

steps:
  - id: step2
    dependsOn: [step1]  # step1 doesn't exist

Solution: Ensure all referenced steps exist and are defined before they're referenced.

Performance Issues

Slow Tool Execution

Common Causes and Solutions:

1. Backend Timeouts Too Short

Solution: Increase timeouts:

spec:
  operational:
    timeouts:
      default: 60s
      perWorkload:
        slow-backend: 120s

2. Resource Constraints

Check pod resources:

kubectl top pod -l app.kubernetes.io/name=<vmcp-name>

Solution: Increase pod resources:

spec:
  podTemplateSpec:
    spec:
      containers:
        - name: vmcp
          resources:
            requests:
              cpu: "1000m"
              memory: "1Gi"
            limits:
              cpu: "2000m"
              memory: "2Gi"

3. Too Many Backends

Solution: Consider splitting into multiple VirtualMCPServers by function or team.

4. Network Latency

Check backend connectivity:

kubectl exec -it <vmcp-pod> -- sh
# Inside pod:
ping <backend-service-name>
curl http://<backend-service-name>:8080/health

Monitoring and Debugging

Viewing Logs

# VirtualMCPServer proxy logs
kubectl logs -l app.kubernetes.io/name=<vmcp-name> --tail=100 -f

# Backend server logs
kubectl logs -l app.kubernetes.io/name=<backend-name> --tail=100 -f

# Operator logs (for reconciliation issues)
kubectl logs -n toolhive-system -l app.kubernetes.io/name=toolhive-operator --tail=100 -f

Checking Events

# VirtualMCPServer events
kubectl describe virtualmcpserver <name>

# All events in namespace sorted by time
kubectl get events --sort-by='.lastTimestamp' | tail -20

Status Inspection

# Full status YAML
kubectl get virtualmcpserver <name> -o yaml

# Just conditions
kubectl get virtualmcpserver <name> -o jsonpath='{.status.conditions}' | jq

# Backend health
kubectl get virtualmcpserver <name> -o jsonpath='{.status.discoveredBackends}' | jq

Testing Connectivity

# Port-forward to VirtualMCPServer
kubectl port-forward service/<vmcp-name> 8080:8080

# Test health endpoint
curl http://localhost:8080/health

# Port-forward to backend
kubectl port-forward service/<backend-name> 8080:8080
curl http://localhost:8080/health

Enable Debug Logging

spec:
  podTemplateSpec:
    spec:
      containers:
        - name: vmcp
          env:
            - name: LOG_LEVEL
              value: "debug"

Apply changes and check logs for detailed information.

Getting Help

If you continue to experience issues:

  1. Check Examples: Review working examples in examples/operator/virtual-mcps/
  2. GitHub Issues: Search or create issues at ToolHive GitHub
  3. Operator Logs: Check operator logs for reconciliation errors
  4. Documentation: Review: