VMware AVI

VMware AVI

Author: Wei Zhou, VMware by Broadcom — wei-wz.zhou@broadcom.com This is a community-driven project by a VMware engineer, not an official VMware product. For official VMware developer tools see developer.broadcom.com.

English | 中文

AVI (NSX Advanced Load Balancer) management and AKO Kubernetes operations tool — 28 tools across 10 categories.

Dual mode: Traditional AVI Controller management + AKO K8s operations in one skill.

Companion skills handle everything else:

Skill	Scope	Install
vmware-aiops	VM lifecycle, deployment, guest ops, cluster	uv tool install vmware-aiops
vmware-monitor	Read-only: inventory, health, alarms, events	uv tool install vmware-monitor
vmware-storage	Datastores, iSCSI, vSAN management	uv tool install vmware-storage
vmware-vks	Tanzu Namespaces, TKC cluster lifecycle	uv tool install vmware-vks
vmware-nsx	NSX networking: segments, gateways, NAT	uv tool install vmware-nsx-mgmt
vmware-nsx-security	DFW firewall rules, security groups	uv tool install vmware-nsx-security
vmware-aria	Aria Ops: metrics, alerts, capacity	uv tool install vmware-aria

Upgrading to 2.0

2.0 renames all 28 MCP tools so every name starts with a verb (e.g. ako_amko_status → get_ako_amko_status, ako_clusters → list_ako_clusters, ako_config_diff → diff_ako_config). The full old → new table is in RELEASE_NOTES.md. Nothing else changes: same tools, parameters, safety gates and CLI commands. If an agent prompt, script or ~/.vmware/rules.yaml names a tool, update it — though deny rules written with the old names keep working with vmware-policy ≥ 1.18 (vmware-audit policy lists any rule the renames widened), and vmware-pilot ≥ 1.13 translates old names in saved workflows. If an install older than 2.0 shares that rules.yaml, give it vmware-policy ≥ 1.18.2 too: it matches each rule against both the old and the new name, so one file serves both.

Quick Install

# Via uv (recommended)
uv tool install vmware-avi

# Or via pip
pip install vmware-avi

# China mainland mirror
pip install vmware-avi -i https://pypi.tuna.tsinghua.edu.cn/simple

# Verify installation
vmware-avi doctor

Offline / Air-Gapped Install (from source)

This project uses the modern PEP 517 build system (hatchling), so there is no setup.py by design — that is expected, not a missing file. If you cloned the source and hit ERROR: File "setup.py" or "setup.cfg" not found ... editable mode currently requires a setuptools-based build, your pip is older than 21.3 and cannot do an editable (-e) install with a non-setuptools backend. Editable mode is a developer convenience, not needed to run the tool — do one of:

# From the source tree — a normal (non-editable) install builds a wheel:
pip install .              # NOT  pip install -e .

# ...or upgrade pip first, and editable works too:
pip install --upgrade pip && pip install -e .

For a truly air-gapped host, build the wheels on a connected machine and copy them over — the target then needs no network:

# On a connected machine, collect this package + its dependencies as wheels:
pip wheel . -w dist        # → dist/*.whl   (or: uv build, for just this package)

# Copy dist/ to the air-gapped host, then install offline:
pip install --no-index --find-links dist vmware-avi

Capabilities Overview

What This Skill Does

CLI vs MCP: Which Mode to Use

Rule of thumb: Use CLI for cost efficiency and small models. Use MCP for structured automation with large models.

Architecture

User (Natural Language)
  |
AI CLI Tool (Claude Code / Gemini / Codex / Cursor / Trae)
  | reads SKILL.md
  |
vmware-avi CLI
  |--- avisdk (AVI REST API) ---> AVI Controller ---> Virtual Services / Pools / SEs
  |--- kubectl / kubernetes ---> K8s Cluster ---> AKO Pods / Ingress / Services

Configuration

Step 1: Create Config Directory

mkdir -p ~/.vmware-avi
vmware-avi init          # generates config.yaml and .env templates
chmod 600 ~/.vmware-avi/.env

Step 2: Edit config.yaml

controllers:
  - name: prod-avi
    host: avi-controller.example.com
    username: admin
    api_version: "22.1.4"
    tenant: admin
    port: 443
    verify_ssl: true

default_controller: prod-avi

ako:
  kubeconfig: ~/.kube/config
  default_context: ""
  namespace: avi-system

Step 3: Set Passwords

Create ~/.vmware-avi/.env:

# AVI Controller passwords
# Format: {CONTROLLER_NAME_UPPER}_PASSWORD
PROD_AVI_PASSWORD=your-password-here

Password environment variable naming convention:

{CONTROLLER_NAME_UPPER}_PASSWORD
# Replace hyphens with underscores, UPPERCASE
# Example: controller "prod-avi" -> PROD_AVI_PASSWORD
# Example: controller "staging-alb" -> STAGING_ALB_PASSWORD

Step 4: Verify

vmware-avi doctor    # checks Controller connectivity + kubeconfig + avisdk

CLI Usage

Virtual Service Management

# List all virtual services
vmware-avi vs list [--controller prod-avi]

# Check status of a specific VS
vmware-avi vs status my-webapp-vs

# Enable / disable a VS (disable requires double confirmation)
vmware-avi vs enable my-webapp-vs
vmware-avi vs disable my-webapp-vs

Pool Member Drain / Restore

# List pool members and health status
vmware-avi pool members my-pool

# Graceful drain (disable) — double confirmation required
vmware-avi pool disable my-pool 10.1.1.5

# Restore traffic (enable)
vmware-avi pool enable my-pool 10.1.1.5

SSL Certificate Expiry Check

# List all certificates
vmware-avi ssl list

# Check certificates expiring within 30 days
vmware-avi ssl expiry --days 30

Analytics and Error Logs

# VS analytics: throughput, latency, error rates
vmware-avi analytics my-webapp-vs

# Request error logs
vmware-avi logs my-webapp-vs --since 1h

Service Engine Health

# Name, mgmt IP, operational status, SE group — status sourced from the
# serviceengine-inventory endpoint (config + runtime merged)
vmware-avi se list

# Per-SE operational status + connected-VS counts
vmware-avi se health

AKO Troubleshooting

# Check AKO pod status
vmware-avi ako status [--context my-k8s-context]

# View AKO logs
vmware-avi ako logs [--tail 100] [--since 30m]

# Restart AKO pod (double confirmation)
vmware-avi ako restart

# Show AKO version
vmware-avi ako version

AKO Helm Config Management

The AKO Helm release is discovered automatically (official installs use helm install --generate-name, so the release is not named ako). Upgrades pull the official Broadcom OCI chart oci://projects.packages.broadcom.com/ako/helm-charts/ako with --reuse-values.

# View current AKO Helm values (release auto-discovered)
vmware-avi ako config show

# Show pending changes (diff against the official OCI chart)
vmware-avi ako config diff

# Helm upgrade (double confirmation + --dry-run default)
vmware-avi ako config upgrade

Ingress Diagnostics

# Validate Ingress annotations
vmware-avi ako ingress check <namespace>

# Show Ingress-to-VS mapping
vmware-avi ako ingress map

# Diagnose why an Ingress has no VS
vmware-avi ako ingress diagnose <ingress-name>

Sync Diagnostics

# Check K8s-Controller sync status
vmware-avi ako sync status

# Show inconsistencies between K8s and Controller
vmware-avi ako sync diff

# Force AKO resync (double confirmation)
vmware-avi ako sync force

Multi-cluster AKO

# List clusters with AKO deployed
vmware-avi ako clusters

# Cross-cluster AKO status overview
vmware-avi ako cluster-overview

# AMKO GSLB status
vmware-avi ako amko status

MCP Server

The MCP server exposes all 28 tools via the Model Context Protocol. Works with any MCP-compatible client.

After uv tool install vmware-avi, start the MCP server with one command (v1.5.15+):

# Recommended — single command, no network re-resolve
vmware-avi mcp

# With custom config path
VMWARE_AVI_CONFIG=/path/to/config.yaml vmware-avi mcp

Claude Desktop Config

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "vmware-avi": {
      "command": "vmware-avi",
      "args": ["mcp"],
      "env": {
        "VMWARE_AVI_CONFIG": "~/.vmware-avi/config.yaml"
      }
    }
  }
}

# Run without installing (requires PyPI access each launch)
uvx --from vmware-avi vmware-avi mcp

# Legacy entry point (still works, kept for backward compatibility)
vmware-avi-mcp

MCP Tools (28)

Common Workflows

1. Maintenance Window -- Drain a Pool Member

When taking a backend server offline for patching:

1. List pool members and health status

   vmware-avi pool members my-pool
   

2. Disable the target server (graceful drain)

   vmware-avi pool disable my-pool 10.1.1.5
   

3. Monitor analytics to confirm active connections are draining

   vmware-avi analytics my-vs
   

4. Perform maintenance on the server
5. Re-enable the server

   vmware-avi pool enable my-pool 10.1.1.5
   

6. Verify health status is green

   vmware-avi pool members my-pool
   

2. AKO Ingress Not Creating VS

When a developer reports their Ingress is not producing a Virtual Service:

1. Verify AKO is running

   vmware-avi ako status
   

2. Validate Ingress annotations

   vmware-avi ako ingress check <namespace>
   

3. Check sync status between K8s and Controller

   vmware-avi ako sync status
   

4. If annotations are wrong, diagnose the specific Ingress

   vmware-avi ako ingress diagnose <ingress-name>
   

5. If sync drift is detected, review the diff and force resync if needed

   vmware-avi ako sync diff
   vmware-avi ako sync force
   

3. SSL Certificate Expiry Audit

Expired certificates cause outages. Run periodic checks:

1. Check all certificates expiring within 30 days

   vmware-avi ssl expiry --days 30
   

2. Review which VS uses each expiring certificate (output includes VS mapping)
3. Plan renewal with the certificate team
4. After renewal, verify the new certificate is in place

   vmware-avi ssl list
   

Troubleshooting

"Controller unreachable" error

1. Run vmware-avi doctor to verify connectivity
2. Check if the controller address and port are correct in ~/.vmware-avi/config.yaml
3. For self-signed certs: set verify_ssl: false in config.yaml (lab environments only)

AKO Pod in CrashLoopBackOff

1. Check logs: vmware-avi ako logs --tail 50
2. Common causes: wrong controller IP in values.yaml, network policy blocking AKO to Controller, expired credentials
3. Fix config: vmware-avi ako config show to inspect, then vmware-avi ako config upgrade with corrected values (release auto-discovered; pulls the official Broadcom OCI chart)

Ingress created but no VS on Controller

1. Validate annotations: vmware-avi ako ingress check <namespace>
2. Check AKO logs for rejection reason: vmware-avi ako logs --since 5m
3. Run sync diff: vmware-avi ako sync diff to see if the object is stuck

Pool member shows "down" after enable

Health monitor may still be failing. The member is enabled but unhealthy. Check the actual health status on the Controller side. Fix the backend service first, then the health status will auto-recover.

SSL expiry check shows 0 certificates

Verify the controller connection has tenant-level access. Certificates are tenant-scoped in AVI. The configured user may only see certs in their tenant.

AKO sync force has no effect

Force resync triggers AKO to re-reconcile all K8s objects. If the drift persists, the issue is likely in the K8s resource definition itself (bad annotation, missing secret). Use vmware-avi ako ingress diagnose to pinpoint the root cause.

Safety Features

Security Details

• Source Code: github.com/vmware-skills/VMware-AVI
• Config File Contents: config.yaml stores controller addresses, usernames, and AKO settings. No passwords or tokens. All secrets stored exclusively in .env
• Webhook Data Scope: Disabled by default. No third-party data transmission
• TLS Verification: Enabled by default. Disable only for self-signed certificate environments
• Prompt Injection Protection: _sanitize() truncation + control character cleanup on all AVI API responses
• Least Privilege: Use a dedicated AVI service account with minimal permissions. AKO operations require only namespace-scoped kubeconfig access

Companion Skills

Version Compatibility

Official Broadcom References

• SDKs: https://developer.broadcom.com/sdks — VCF Python SDK
• REST APIs: https://developer.broadcom.com/xapis — AVI Controller REST API
• CLI Tools: https://developer.broadcom.com/tools — VCF PowerCLI 9.1

Troubleshooting & Contributing

If you encounter any errors or issues, please send the error message, logs, or screenshots to zhouwei008@gmail.com. Contributions are welcome -- feel free to join us in maintaining and improving this project!

Remote access (HTTPS) and limiting tools

By default vmware-avi mcp speaks stdio to one local client and offers every tool. Two options change that (vmware-policy ≥ 1.21.0).

Serve over HTTPS so clients on other machines can connect. Every caller needs a bearer token:

vmware-audit token create alice          # prints the token once; give it to alice
vmware-audit token list                  # names only
vmware-audit token revoke alice          # refused from its next request, no restart
vmware-avi mcp --transport https --host 0.0.0.0 --port 8443 \
    --cert /etc/vmware-mcp/server.pem --key /etc/vmware-mcp/server-key.pem \
    --public-name mcp.example.com

Connect a client (Claude Code shown; any MCP client that supports streamable HTTP and custom headers works):

claude mcp add --transport http vmware-avi https://mcp.example.com:8443/mcp \
    --header "Authorization: Bearer $VMWARE_MCP_TOKEN"

{
  "mcpServers": {
    "vmware-avi": {
      "type": "http",
      "url": "https://mcp.example.com:8443/mcp",
      "headers": { "Authorization": "Bearer ${VMWARE_MCP_TOKEN}" }
    }
  }
}

• The server will not start without a token, refuses Host/Origin values outside --public-name, and checks before it binds that unauthenticated requests get 401. Use a certificate your clients trust for those names.
• Behind a TLS-terminating reverse proxy on the same machine use --transport http --host 127.0.0.1 --public-name mcp.example.com and pass the Authorization header through.
• Behind a load balancer or gateway on another machine, keep --transport https (a self-signed certificate is fine), list in --public-name the host name the balancer sends as Host, and have it forward Authorization. Its health probe can use GET /healthz, which needs no token and returns only {"status": "ok"} (vmware-policy ≥ 1.23.0).
• Send traceparent or X-Request-ID with each request to have it stored on the audit row; vmware-audit log --correlation-id ID then finds the calls behind one platform request.
• The audit log records each caller as token:<name>. Calls run side by side: each holds its own session to a Controller (or Kubernetes context), up to VMWARE_MCP_POOL_SIZE (default 2) sessions per target. Each Controller session is a login on the Controller; a call waits at most 300 s for a free one.
• Everyone who can reach the endpoint acts with this server's account. Tokens identify callers; they grant nothing on the platform. Run one endpoint per privilege level.
• Several people or roles on one endpoint: run vmware-audit roles enable once, then give each token a role (vmware-audit token create NAME --role reader, or a role in roles.yaml naming tools and targets; vmware-policy ≥ 1.24.0). A role limits what the token may call on this endpoint; it cannot grant more than the account has. See vmware-policy's README, "Roles for network callers".

Offer only read tools, plus the writes you choose:

vmware-avi mcp --expose read --allow-write disable_lb_pool_member
# or: VMWARE_AVI_MCP_EXPOSE=read VMWARE_AVI_MCP_ALLOW_WRITE=disable_lb_pool_member vmware-avi mcp

Tools not offered do not appear in the client's tool list. A bad value or an unknown write tool stops the server (exit 2). This limits what clients are offered; it is not authorization, so the account itself should be least-privilege.

Details: skills/vmware-avi/references/setup-guide.md ("Serving MCP over HTTPS", "Limiting which tools are offered").

License

MIT