
testdriver
TestDriver.ai MCP server, TestDriver expert agent, and skills for authoring computer-use end-to-end tests with Vitest.
- GitHub stars
- 357
Install
/plugin install testdriver@testdriverRun it inside Claude Code after adding the marketplace with /plugin marketplace add testdriverai/testdriverai.
Contents
Discovered from sourceSkills 55
Overview
The ai() method allows you to execute complex tasks using natural language descriptions. TestDriver's AI will figure out the steps needed to accomplish the task.
Syntax
await testdriver.ai(task, options)
Parameters
Natural language description of what to do Execution options Whether to validate completion and retry if incompleteReturns
Promise<string | void> - Final AI response if validateAndLoop is true
Examples
Basic Usage
// Simple task execution
await testdriver.ai('Click the submit button');
// Complex multi-step task
await testdriver.ai('Fill out the contact form and submit it');
// Navigation task
await testdriver.ai('Go to the settings page and enable notifications');
With Validation
// AI will verify the task completed successfully
const result = await testdriver.ai('Complete the checkout process', {
validateAndLoop: true
});
console.log('Task result:', result);
Multi-Step Workflows
// The AI will break down complex tasks
await testdriver.ai('Search for "laptop", add the first result to cart, and proceed to checkout');
// UI exploration
await testdriver.ai('Find and click all menu items to explore the application');
Use Cases
Use AI to explore unfamiliar applications:```javascript
await testdriver.ai('Explore the main navigation menu');
await testdriver.ai('Try to find the user profile settings');
```
Let AI handle multi-step processes:
```javascript
await testdriver.ai('Complete the multi-step registration form');
await testdriver.ai('Configure all the advanced settings to default values');
```
When exact element locations aren't critical:
```javascript
await testdriver.ai('Close any popup dialogs');
await testdriver.ai('Accept the cookie consent if it appears');
```
Best Practices
**Be specific but flexible**: Give enough context without over-constraining the AI// ✅ Good
await testdriver.ai('Add the first product to the shopping cart');
// ❌ Too vague
await testdriver.ai('do something');
// ❌ Too specific (defeats the purpose)
await testdriver.ai('click at coordinates 500, 300');
**Use for exploration, not precision**: For critical assertions, use explicit methods
// Use AI for setup
await testdriver.ai('Navigate to the login page');
// Use explicit methods for critical steps
const usernameField = await testdriver.find('username input');
await usernameField.click();
await testdriver.type('testuser');
await testdriver.assert('login page is displayed');
**AI tasks may be slower**: The AI needs to analyze the screen and plan actions. For performance-critical tests, use explicit methods.
When to Use AI vs Explicit Methods
Use ai() when:
- Exploring unfamiliar applications
- Handling optional UI elements (popups, cookies, etc.)
- Prototyping tests quickly
- Tasks where exact steps may vary
Use explicit methods when:
- Performance is critical
- You need precise control
- Making assertions
- Debugging test failures
- Repetitive, predictable actions
Complete Example
import { beforeAll, afterAll, describe, it } from 'vitest';
import TestDriver from 'testdriverai';
describe('E-commerce Flow with AI', () => {
let testdriver;
beforeAll(async () => {
client = new TestDriver(process.env.TD_API_KEY);
await testdriver.auth();
await testdriver.connect();
});
afterAll(async () => {
await testdriver.disconnect();
});
it('should complete shopping flow using AI assistance', async () => {
await testdriver.focusApplication('Google Chrome');
// Use AI for navigation and exploration
await testdriver.ai('Browse to the electronics section');
await testdriver.ai('Find and add a laptop to the cart');
// Use explicit methods for critical steps
const cartIcon = await testdriver.find('shopping cart icon');
await cartIcon.click();
const total = await testdriver.extract('the cart total amount');
console.log('Cart total:', total);
// Use AI for checkout flow
await testdriver.ai('Proceed to checkout and fill in shipping details', {
validateAndLoop: true
});
// Explicit assertion
await testdriver.assert('order confirmation page is displayed');
});
});
Related Methods
Overview
Make assertions about the screen state with natural language and AI. The AI examines the screen. It makes sure that your assertion is true.
Syntax
await testdriver.assert(assertion)
await testdriver.assert(assertion, options)
Parameters
Natural language description of what should be true Optional configuration AI sampling configuration for this assert call (overrides global `ai` config from constructor). <Expandable title="properties">
<ParamField path="temperature" type="number">
Controls randomness. `0` = deterministic, higher = more creative. Default: model default.
</ParamField>
<ParamField path="top" type="object">
Sampling parameters
<Expandable title="properties">
<ParamField path="p" type="number">
Top-P (nucleus sampling). Range: 0-1.
</ParamField>
<ParamField path="k" type="number">
Top-K sampling. `1` = most deterministic.
</ParamField>
</Expandable>
</ParamField>
</Expandable>
</ParamField>
Returns
Promise<boolean> - true if the assertion passes. It throws an error if the assertion fails.
Examples
Basic Assertions
// Verify page elements
await testdriver.assert('the login page is displayed');
await testdriver.assert('submit button is visible');
await testdriver.assert('error message is shown');
// Verify text content
await testdriver.assert('the page title is "Welcome"');
await testdriver.assert('username field contains "john.doe"');
await testdriver.assert('success message says "Account created"');
// Verify states
await testdriver.assert('the form is empty');
await testdriver.assert('the checkbox is checked');
await testdriver.assert('the dropdown shows "United States"');
// Verify visual appearance
await testdriver.assert('the button is blue');
await testdriver.assert('the loading spinner is displayed');
await testdriver.assert('the modal dialog is open');
Best Practices
**Be specific in assertions**More specific assertions are more reliable:
// ❌ Too vague
await testdriver.assert('button is visible');
// ✅ Specific
await testdriver.assert('blue submit button is visible below the form');
**Assert state changes**
Make sure of the state before and after actions:
// Before
await testdriver.assert('cart is empty');
// Action
const addBtn = await testdriver.find('add to cart');
await addBtn.click();
// After
await testdriver.assert('cart contains 1 item');
**Use with test framework assertions**
Combine AI assertions with traditional test assertions:
// AI assertion
const result = await testdriver.assert('success message is displayed');
// Framework assertion
expect(result).toBeTruthy();
// Extract for detailed comparison
const message = await testdriver.extract('the success message text');
expect(message).toContain('successfully');
Polling Assertions
For conditions that may take time to become true:
async function waitForAssertion(testdriver, assertion, timeout = 30000) {
const startTime = Date.now();
while (Date.now() - startTime < timeout) {
try {
await testdriver.assert(assertion);
return true; // Assertion passed
} catch (error) {
// Assertion failed, wait and retry
await new Promise(r => setTimeout(r, 1000));
}
}
throw new Error(`Assertion timeout: "${assertion}"`);
}
// Usage
await waitForAssertion(testdriver, 'page has finished loading', 30000);
await waitForAssertion(testdriver, 'results are displayed', 10000);
Use Cases
```javascript // Try to submit empty form const submitBtn = await testdriver.find('submit button'); await submitBtn.click();// Verify validation errors
await testdriver.assert('email field shows "required" error');
await testdriver.assert('password field shows "required" error');
// Verify form not submitted
await testdriver.assert('still on the form page');
```
```javascript
const loginBtn = await testdriver.find('login button');
await loginBtn.click();
// Verify navigation
await testdriver.assert('user dashboard is displayed');
await testdriver.assert('welcome message shows user name');
await testdriver.assert('logout button is visible');
```
```javascript
const loadBtn = await testdriver.find('load more button');
await loadBtn.click();
// Poll for content using helper
await waitForAssertion(testdriver, 'more than 10 items are shown', 10000);
await testdriver.assert('load more button is still visible');
```
```javascript
// Verify hover effect
const button = await testdriver.find('primary button');
await button.hover();
await testdriver.assert('button background is darker');
// Verify button is enabled
await testdriver.assert('submit button is enabled');
```
```javascript
// Step 1
await testdriver.assert('step 1 is active');
const nextBtn = await testdriver.find('next button');
await nextBtn.click();
// Step 2
await testdriver.assert('step 2 is active');
await testdriver.assert('step 1 is completed');
await nextBtn.click();
// Step 3
await testdriver.assert('step 3 is active');
await testdriver.assert('step 2 is completed');
```
Complete Example
import { beforeAll, afterAll, describe, it, expect } from 'vitest';
import TestDriver from 'testdriverai';
describe('Assertions', () => {
let testdriver;
beforeAll(async () => {
client = new TestDriver(process.env.TD_API_KEY);
await testdriver.auth();
await testdriver.connect();
});
afterAll(async () => {
await testdriver.disconnect();
});
it('should validate login flow', async () => {
await testdriver.focusApplication('Google Chrome');
// Verify initial state
await testdriver.assert('the login page is displayed');
await testdriver.assert('username field is empty');
await testdriver.assert('password field is empty');
// Fill form
const usernameField = await testdriver.find('username input');
await usernameField.click();
await testdriver.type('testuser');
await testdriver.pressKeys(['tab']);
await testdriver.type('password123');
// Verify fields filled
await testdriver.assert('username field contains "testuser"');
await testdriver.assert('password field is not empty');
// Submit
await testdriver.pressKeys(['enter']);
// Poll for success
await waitForAssertion(testdriver, 'user dashboard is displayed', 10000);
// Verify logged in state
await testdriver.assert('welcome message is shown');
await testdriver.assert('logout button is visible');
// Verify login page is gone
await testdriver.assert('login form is displayed', false, true);
});
it('should show validation errors', async () => {
// Try empty submission
const submitBtn = await testdriver.find('submit button');
await submitBtn.click();
// Verify multiple errors
const errors = [
'email field shows error',
'name field shows error',
'password field shows error'
];
for (const error of errors) {
const result = await testdriver.assert(error);
expect(result).toBeTruthy();
}
// Verify not submitted
await testdriver.assert('confirmation page is shown', false, true);
});
});
Related Methods
This guide helps you to set up self-hosted TestDriver instances on AWS. At the end, you have automated test infrastructure. It starts and stops instances when you need them.
graph LR
A[Vitest Test] --> B[setup-aws hook]
B --> C[Spawns EC2]
C --> D[Runs Test]
D --> E[Terminates EC2]
How It Works
TestDriver manages the AWS EC2 instances for your tests automatically:
- Deploy CloudFormation. Set up the infrastructure one time.
- Configure Vitest. Add one line to your config.
- Run Tests. The instances start automatically, run the tests, and stop.
That is all. You do not need to manage the instances by hand.
Quickstart
One-click AWS setup ```javascript vitest.config.mjs setupFiles: ['testdriverai/vitest/setup', 'testdriverai/vitest/setup-aws'] ``` ```bash TD_OS=windows AWS_REGION=us-east-2 \ AWS_LAUNCH_TEMPLATE_ID=lt-xxx AMI_ID=ami-xxx \ vitest run ```Overview
The setup process is simple:
- Deploy CloudFormation — Creates VPC, security groups, IAM roles, and launch templates
- Configure Vitest — Add
setup-awsto automatically manage instance lifecycle - Run Tests — Set
TD_OS=windowswith AWS credentials and instances spawn/terminate automatically
Prerequisites
Before you begin, ensure you have:
- AWS account with CloudFormation permissions
- AWS CLI installed and configured (
aws configure) - Access to the TestDriver AMI — Contact us with your AWS region
- A GitHub repository for your tests
Step 1: Deploy CloudFormation Stack
Our CloudFormation template creates all the AWS infrastructure you need:
- Dedicated VPC with public subnet
- Security group with required port access
- IAM roles and instance profiles
- EC2 launch template for instance creation
<Card
title="Launch Stack"
icon="aws"
href="https://console.aws.amazon.com/cloudformation/home#/stacks/create/review?templateURL=https://v7-cloudformation-template.s3.us-east-2.amazonaws.com/cloudformation.yaml"
horizontal
arrow
>
Deploy TestDriver infrastructure with one click
</Card>
Configure the stack parameters:
- **Stack name**: `testdriver-infrastructure` (or your preferred name)
- **ProjectTag**: `testdriver`
- **AllowedIngressCidr**: Your IP range (e.g., `203.0.113.0/24`)
- **InstanceType**: `c5.xlarge` (recommended)
- **CreateKeyPair**: `true`
<Warning>
**Security**: Replace `AllowedIngressCidr` with your specific IP ranges to restrict VPC access. Avoid using `0.0.0.0/0` in production.
</Warning>
### Get Your Launch Template ID
After the stack creation completes, navigate to the **Outputs** tab to find your `LaunchTemplateId`:

<Tip>
**Save this ID** — you'll need it for spawning instances and CI configuration.
</Tip>
Download the template from the [TestDriver CLI repository](https://github.com/testdriverai/testdriverai/blob/main/setup/aws/cloudformation.yaml), then deploy:
```bash
aws cloudformation deploy \
--template-file setup/aws/cloudformation.yaml \
--stack-name testdriver-infrastructure \
--parameter-overrides \
ProjectTag=testdriver \
AllowedIngressCidr=0.0.0.0/0 \
InstanceType=c5.xlarge \
CreateKeyPair=true \
--capabilities CAPABILITY_IAM
```
<Warning>
**Security**: Replace `AllowedIngressCidr=0.0.0.0/0` with your specific IP ranges to restrict VPC access.
</Warning>
### Get Your Launch Template ID
After deployment completes, retrieve the launch template ID:
```bash
aws cloudformation describe-stacks \
--stack-name testdriver-infrastructure \
--query 'Stacks[0].Outputs[?OutputKey==`LaunchTemplateId`].OutputValue' \
--output text
```
<Tip>
**Save this ID** — you'll need it for spawning instances and CI configuration.
</Tip>
Step 2: Configure Vitest
Add the AWS setup hook to your vitest.config.mjs:
import { defineConfig } from 'vitest/config';
import { config } from 'dotenv';
import TestDriver from 'testdriverai/vitest';
config(); // Load .env file
export default defineConfig({
test: {
testTimeout: 900000,
hookTimeout: 900000,
maxConcurrency: 3,
reporters: [
'default',
TestDriver(),
['junit', { outputFile: 'test-report.junit.xml' }]
],
setupFiles: ['testdriverai/vitest/setup', 'testdriverai/vitest/setup-aws'],
},
});
**That's it!** The `setup-aws` hook automatically spawns and terminates instances when `TD_OS=windows` is set. No manual instance management needed.
Step 3: Write Your Tests
Tests should use context.ip || process.env.TD_IP for the IP configuration:
import { describe, it } from "vitest";
import { TestDriver } from "testdriverai/vitest/hooks";
describe("My Test", () => {
it("should run on self-hosted instance", async (context) => {
const testdriver = TestDriver(context, {
ip: context.ip || process.env.TD_IP,
});
await testdriver.provision.chrome({ url: "https://example.com" });
// ... your test steps
});
});
**How it works**: When `TD_OS=windows` with AWS credentials, `context.ip` is automatically set by the setup hook. When running without AWS setup (cloud-hosted), both are undefined and TestDriver uses the cloud. When `TD_IP` is provided manually, it takes precedence.
Step 4: Run Tests
Locally
TD_OS=windows \
AWS_REGION=us-east-2 \
AWS_LAUNCH_TEMPLATE_ID=lt-xxx \
AMI_ID=ami-0504bf50fad62f312 \
vitest run
Each test gets its own fresh EC2 instance that's automatically terminated after completion.
In GitHub Actions
Automate testing with self-hosted instances in your CI/CD pipeline. TestDriver automatically spawns a fresh instance for each test, runs the test, and terminates the instance.
name: TestDriver Self-Hosted Windows Tests
on:
push:
branches: [main]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Run Windows tests with self-hosted instances
run: npx vitest run examples/*.test.mjs
env:
TD_API_KEY: ${{ secrets.TD_API_KEY }}
TD_OS: windows
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_REGION: us-east-2
AWS_LAUNCH_TEMPLATE_ID: ${{ secrets.AWS_LAUNCH_TEMPLATE_ID }}
AMI_ID: ${{ secrets.AMI_ID }}
- name: Upload test results
if: always()
uses: actions/upload-artifact@v4
with:
name: test-results
path: test-report.junit.xml
**Automatic Instance Management**: Setting `TD_OS=windows` with AWS credentials enables automatic instance spawning. Each test gets its own fresh instance that's terminated after the test completes.
Required GitHub Secrets
| Secret | Description | Example |
|---|---|---|
AWS_ACCESS_KEY_ID | AWS access key | AKIAIOSFODNN7EXAMPLE |
AWS_SECRET_ACCESS_KEY | AWS secret key | wJalrXUtnFEMI/K7MDENG... |
AWS_REGION | AWS region | us-east-2 |
AWS_LAUNCH_TEMPLATE_ID | From CloudFormation output | lt-07c53ce8349b958d1 |
AMI_ID | TestDriver AMI ID | ami-0504bf50fad62f312 |
TD_API_KEY | Your TestDriver API key | From console.testdriver.ai |
Example Workflows
For complete production examples, see:
- Scheduled Tests — Daily automated test runs
- Reusable Workflow — Shared configuration for PR and scheduled tests
Advanced Usage
Using an Existing Instance
If you already have a running instance, you can skip automatic spawning by providing TD_IP:
TD_OS=windows TD_IP=1.2.3.4 vitest run
The setup-aws hook will detect TD_IP is already set and skip spawning a new instance.
Manual Instance Management
For advanced use cases, you can manually spawn instances using the spawn-runner.sh script:
AWS_REGION=us-east-2 \
AMI_ID=ami-0504bf50fad62f312 \
AWS_LAUNCH_TEMPLATE_ID=lt-xxx \
bash setup/aws/spawn-runner.sh
Output:
PUBLIC_IP=1.2.3.4
INSTANCE_ID=i-1234567890abcdef0
AWS_REGION=us-east-2
Then manually terminate when done:
aws ec2 terminate-instances \
--instance-ids i-1234567890abcdef0 \
--region us-east-2
For complete production examples, see:
- Scheduled Tests — Daily automated test runs
- Reusable Workflow — Shared configuration for PR and scheduled tests
Connecting to an Instance
You can connect to running instances via:
- RDP — Use the public IP on port 3389
- VNC — Access via web browser at
http://<public-ip>:5900 - AWS Console — Use EC2 Instance Connect or Session Manager
AMI Customization
The TestDriver Golden Image comes pre-configured with:
- Windows Server with desktop environment
- VNC + web server for remote access
- Python, Node.js, Git
- TestDriver agent and dependencies
Creating a Custom AMI
You can customize the AMI to include additional software or configurations:
Use the default credentials: - **Username**: `testdriver` - **Password**: `wwv9uJ0sqlulbN3` **Critical**: Run the password rotation script immediately: ```powershell C:\testdriver\RotateLocalPasswords.ps1 ``` Save the new password securely. Install any additional dependencies, configure settings, or modify the environment as needed. Use the AWS console or CLI to create an AMI from your modified instance. Update your workflow to use the new AMI ID. **Security**: Never use the default password in production. Always rotate passwords before creating custom AMIs.Security Best Practices
Network Security
- Restrict CIDR blocks — Only allow access from known IP ranges
- Use VPC endpoints — For private AWS service communication
- Enable VPC Flow Logs — For network monitoring
AWS Authentication
Use OIDC instead of long-term credentials for GitHub Actions:
permissions:
id-token: write
contents: read
steps:
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789012:role/GitHubActionsRole
aws-region: us-east-2
See GitHub's OIDC documentation for setup instructions.
Instance Security
- Terminate immediately after tests complete
- Monitor costs with AWS billing alerts
- Use least-privilege IAM roles
- Enable CloudTrail for audit logging
Troubleshooting
Instance Not Responding
- Check security groups — Verify required ports are open (RDP 3389, VNC 5900, TestDriver ports)
- Verify status checks — Ensure instance has passed AWS status checks
- Check AMI compatibility — Some instance types don't support certain AMIs
Connection Timeouts
- Verify network connectivity from CI runner to instance
- Check VPC routing and internet gateway configuration
- Confirm instance is in the correct subnet
AWS CLI Errors
- Validate credentials and permissions
- Check service quotas and limits
- Verify region consistency across all resources
Overview
The cache system makes repeated test runs faster. It compares screenshots to cached results. When the screen did not change much, TestDriver uses the cached element positions again. It does not make an AI call.
The cache works at two levels:
- Screen cache: a pixel diff comparison between the present screenshot and the cached screenshot
- Element cache: OpenCV template matching to make sure that the cached element position is still correct
How It Works
- On
find(), the SDK sends the present screenshot and the cache metadata to the API. - The API compares the screenshot against the cached results for the same
cacheKey. - If the screen pixel diff is in the
screenthreshold AND the element template match is more than theelementthreshold, TestDriver returns the cached position. - If not, TestDriver makes a new AI call and caches the result.
flowchart LR
A[Screenshot + cacheKey] --> B{Screen Diff\n< threshold?}
B -- Yes --> C{Template\nMatch OK?}
C -- Yes --> D[Return cached position]
B -- No --> E[AI Call - fresh]
C -- No --> F[AI Call - fresh]
Configuration
Constructor Options
const testdriver = new TestDriver({
cache: {
enabled: true,
thresholds: {
find: {
screen: 0.05, // 5% pixel diff allowed (default)
element: 0.8, // 80% OpenCV correlation required (default)
},
assert: 0.05, // 5% pixel diff for assertions (default)
},
},
cacheKey: 'my-custom-key', // overrides auto-generated key
});
Cache configuration object, or `false` to disable entirely.
Enable or disable the cache system. Requires a valid `cacheKey` to actually activate.
<ParamField path="thresholds" type="CacheThresholds">
Threshold configuration for different command types.
<Expandable title="properties">
<ParamField path="find" type="FindCacheThresholds">
Thresholds for `find()` and `findAll()`.
<Expandable title="properties">
<ParamField path="screen" type="number" default={0.05}>
Maximum pixel diff percentage allowed between the current screenshot and the cached screenshot. Lower values require a closer match. Range: `0` to `1`.
</ParamField>
<ParamField path="element" type="number" default={0.8}>
Minimum OpenCV template matching correlation required for the cached element crop. Higher values require a closer match. Range: `0` to `1`. Only used for `find()`, not `findAll()`.
</ParamField>
</Expandable>
</ParamField>
<ParamField path="assert" type="number" default={0.05}>
Maximum pixel diff allowed for assertion cache hits.
</ParamField>
</Expandable>
</ParamField>
Unique key for cache lookups. If not provided, an auto-generated key is created from a SHA-256 hash of the calling test file (first 16 hex characters). The cache key changes automatically when your test file changes, providing automatic cache invalidation.
Disabling Cache
// Via constructor
const testdriver = new TestDriver({ cache: false });
// Via environment variable
// TD_NO_CACHE=true npx vitest run
When cache is disabled, all thresholds are set to -1 internally, causing the API to skip cache lookups.
Per-Command Overrides
Override cache thresholds for individual commands:
// Stricter screen matching for this specific find
const el = await testdriver.find('submit button', {
cache: {
thresholds: { screen: 0.01, element: 0.95 },
},
});
// Custom cache key for a specific assertion
await testdriver.assert('dashboard loaded', {
cache: { threshold: 0.01 },
cacheKey: 'dashboard-check',
});
Threshold Priority
Thresholds are resolved in priority order (highest wins):
| Priority | Source | Example |
|---|---|---|
| 1 (highest) | Per-command option | find(desc, { cache: { thresholds: { screen: 0.1 } } }) |
| 2 | Legacy number argument | find(desc, 0.1) |
| 3 | Global constructor config | new TestDriver({ cache: { thresholds: { find: { screen: 0.1 } } } }) |
| 4 (lowest) | Hard-coded defaults | screen: 0.05, element: 0.8, assert: 0.05 |
Auto-Generated Cache Key
When you don't specify a cacheKey, the SDK automatically generates one:
- Walks the call stack to find your test file
- Reads the file content
- Computes a SHA-256 hash of the content
- Uses the first 16 hex characters as the cache key
This means:
- Same test file → same cache key → cache hits
- Modified test file → different hash → automatic cache invalidation
- Different test files → different keys → isolated caches
// Auto-generated cache key from test file hash
const testdriver = new TestDriver();
// cacheKey = "a3f2b1c4d5e6f7a8" (auto)
// Manual override
const testdriver = new TestDriver({ cacheKey: 'login-test-v2' });
Template Matching (OpenCV)
Element cache validation uses OpenCV's normalized cross-correlation coefficient (TM_CCOEFF_NORMED) to verify that the cached element is still visible at the expected position.
Algorithm:
- Load the cached element crop (needle) and current screenshot (haystack)
- Run
cv.matchTemplate()withTM_CCOEFF_NORMED - Binary threshold at the configured element threshold
- Find contours to extract match positions
- Return matches with
{ x, y, width, height, centerX, centerY }
Scale factors tried: [1, 0.5, 2, 0.75, 1.25, 1.5]
Thresholds tried: [0.9, 0.8, 0.7] (picks highest matching threshold)
This accounts for minor scaling differences between screenshots taken at different times or resolutions.
Cache Partitioning
Cache entries are partitioned by:
cacheKey— identifies the test fileos— operating system (linux, windows, darwin)resolution— screen resolution
This means cache from a Linux run won't be used for a Windows run, even with the same cache key.
Debugging Cache
API responses include cache metadata:
| Field | Description |
|---|---|
cacheHit | true if cache was used |
similarity | Pixel diff percentage between screenshots |
cacheSimilarity | OpenCV template match score |
Use getDebugInfo() on an element to inspect cache results:
const el = await testdriver.find('submit button');
const debug = el.getDebugInfo();
console.log(debug);
// { cacheHit: true, similarity: 0.02, cacheSimilarity: 0.92, ... }
Types
interface CacheConfig {
enabled?: boolean; // Default: true
thresholds?: CacheThresholds;
}
interface CacheThresholds {
find?: FindCacheThresholds;
assert?: number; // Default: 0.05
}
interface FindCacheThresholds {
screen?: number; // Default: 0.05
element?: number; // Default: 0.8
}
interface CacheDebugInfo {
cacheHit: boolean;
similarity: number;
cacheSimilarity: number;
}
After the agent explored your app, TestDriver keeps what it found. TestDriver caches each element that the AI vision agent finds. It uses a vision fingerprint. This is a perceptual hash of the screen state at the element location. On the next run, TestDriver matches against that cache. It does not call the AI again. Tests that pass replay quickly, in the same way each time, and at a low cost.
This learning makes TestDriver fast. The cache makes the test run a maximum of 1.7x faster. It does not do the same AI vision analysis again. The agent thinks only when it sees something new.
// First run: builds cache
await testdriver.find('submit button');
// Second run: exact match
await testdriver.find('submit button');
Automatic Caching
TestDriver enables learning automatically. You do not need configuration. TestDriver computes the cache key from these. The cache key is the fingerprint that TestDriver uses to know what it learned:
- File hash: The SHA-256 hash of the test file contents
- Selector prompt: The exact text description that you give to
find() - Screenshot context: The perceptual hash of the present screen state
- Platform: The operating system and the browser version
When you change your test file, the hash changes automatically. This makes the old cache entries not valid. Then TestDriver does a new AI analysis with your new test logic.
import { test } from 'vitest';
import { chrome } from 'testdriverai/presets';
test('auto-cached test', async (context) => {
const { testdriver } = await chrome(context, {
url: 'https://example.com'
});
// First call: AI analyzes screen, saves to cache
await testdriver.find('More information link'); // 2.1s
// Second call: cache hit, instant response
await testdriver.find('More information link'); // 12ms ⚡
});
Managing the Cache
You can clear the cache in the TestDriver console. There, you also find previews of cached elements, the input prompts, and analytics on the cache hit rates.
Manage and clear your test cache from the TestDriver console.Debugging Cache Hits and Misses
You can track what TestDriver has learned by inspecting cache performance in your tests:
test('monitor cache performance', async (context) => {
const { testdriver } = await chrome(context, { url });
const element = await testdriver.find('submit button');
if (element.cacheHit) {
console.log('✅ Cache hit - instant response');
console.log('Strategy:', element.cacheStrategy); // 'exact', 'pixeldiff', or 'template'
console.log('Similarity:', `${(element.similarity * 100).toFixed(1)}%`);
console.log('Cache age:', element.cacheCreatedAt);
} else {
console.log('⏱️ Cache miss - AI analysis performed');
console.log('New cache entry created');
}
});
Configuring the Cache
You can configure how TestDriver learns globally when initializing TestDriver:
import { TestDriver } from 'testdriverai';
const testdriver = new TestDriver({
apiKey: process.env.TD_API_KEY,
cacheKey: 'my-test-suite', // cache-key for this instance
cacheDefaults: {
threshold: 0.05, // 95% similarity
}
});
It's also possible to override cache settings per find() call:
// Default: 95% similarity required
await testdriver.find('submit button');
// Explicit strict threshold
await testdriver.find('submit button', {
cacheThreshold: 0.01 // 99% similarity
});
Caching with Variables
Custom cache keys prevent cache pollution when using variables in prompts, dramatically improving cache hit rates—so TestDriver reuses what it learned even when your data changes.
// ❌ Without cache key - creates new cache for each variable value
const email = 'user@example.com';
await testdriver.find(`input for ${email}`); // Cache miss every time
// ✅ With cache key - reuses cache regardless of variable
const email = 'user@example.com';
await testdriver.find(`input for ${email}`, {
cacheKey: 'email-input'
});
// Also useful for dynamic IDs, names, or other changing data
const orderId = generateOrderId();
await testdriver.find(`order ${orderId} status`, {
cacheKey: 'order-status' // Same cache for all orders
});
Next
Now that TestDriver has learned your app, run your tests in CI and locally—replaying the cache for fast, deterministic results.Captcha Solving
TestDriver can solve captchas automatically during your tests. It uses the 2captcha service. This feature works on Linux sandboxes and Windows sandboxes.
Quick Start
const result = await testdriver.captcha({
apiKey: process.env.TWOCAPTCHA_API_KEY,
});
console.log(result.success); // true
console.log(result.token); // The solved captcha token
That is all. TestDriver does these automatically:
- Finds the captcha type on the page
- Gets the sitekey
- Solves the captcha with 2captcha
- Puts the token into the page
- Starts the callbacks
Supported Captcha Types
| Type | Auto-Detected | Notes |
|---|---|---|
| reCAPTCHA v2 | ✅ | Including invisible |
| reCAPTCHA v3 | ✅ | Action is auto-detected |
| hCaptcha | ✅ | |
| Cloudflare Turnstile | ✅ |
Getting a 2captcha API Key
- Sign up at 2captcha.com
- Add funds to your account
- Copy your API key from the dashboard
Configuration Options
const result = await testdriver.captcha({
// Required
apiKey: '2CAPTCHA_API_KEY',
// Optional - usually auto-detected
sitekey: '6Le...', // Override auto-detected sitekey
type: 'recaptcha_v3', // Override auto-detected type
action: 'submit', // reCAPTCHA v3 action
// Timing
timeout: 120000, // Max wait time (default: 120s)
pollInterval: 5000, // Poll interval (default: 5s)
});
Full Example
import { describe, expect, it } from "vitest";
import { TestDriver } from "testdriver";
describe("Checkout flow", () => {
it("should complete checkout with captcha", async (context) => {
const testdriver = TestDriver(context);
// Navigate to checkout page
await testdriver.provision.chrome({
url: 'https://example.com/checkout',
});
// Fill out form
await testdriver.type({ text: 'John Doe', selector: '#name' });
await testdriver.type({ text: 'john@example.com', selector: '#email' });
// Solve the captcha
const result = await testdriver.captcha({
apiKey: process.env.TWOCAPTCHA_API_KEY,
});
expect(result.success).toBe(true);
// Submit the form
await testdriver.click({ selector: '#submit' });
// Verify success
await testdriver.find({ text: 'Order confirmed' });
}, 180000);
});
Environment Variables
You can set your API key as an environment variable:
export TWOCAPTCHA_API_KEY=your_api_key_here
Then use it in your tests:
const result = await testdriver.captcha({
apiKey: process.env.TWOCAPTCHA_API_KEY,
});
GitHub Actions
Add the key to your repository secrets and expose it in your workflow:
- name: Run Tests
run: vitest run
env:
TWOCAPTCHA_API_KEY: ${{ secrets.TWOCAPTCHA_API_KEY }}
How It Works
- Detection: Scans the page for captcha elements (
data-sitekey, script tags, etc.) - Submit: Sends the captcha challenge to 2captcha's solving service
- Poll: Waits for human solvers to complete the captcha
- Inject: Injects the solved token into the page's hidden fields
- Callback: Triggers any JavaScript callbacks the page expects
Troubleshooting
"Could not auto-detect captcha"
The captcha element wasn't found on the page. Try:
- Waiting for the page to fully load before calling
captcha() - Providing the
sitekeyandtypemanually
Timeout errors
Captcha solving typically takes 10-30 seconds. If you're getting timeouts:
- Increase the
timeoutoption - Check your 2captcha balance
- Verify the captcha type is correct
Token not working
Some sites validate tokens immediately. Make sure:
- The token is injected before form submission
- The captcha type matches what the site expects
- For reCAPTCHA v3, the
actionparameter matches the site's expected action
Requirements
- Chrome must be launched with remote debugging enabled (automatic on all sandboxes)
- A valid 2captcha API key with sufficient balance
TestDriver integrates with the common CI providers. It lets you do automated end-to-end tests on each push and pull request.
Authentication
On GitHub Actions, use OIDC through the published testdriverai/action. Then
there is no TD_API_KEY secret to store, copy, or change. The action shows that the
workflow runs in your org. TestDriver changes that proof for your
team's key at run time. See the GitHub Actions tab below.
For other CI providers (or self-hosted runners without OIDC), use a
stored API key from console.testdriver.ai/settings.
Add it as a TD_API_KEY secret in the settings of your CI provider.
CI Provider Examples
### Authenticate with OIDC via `testdriverai/action` (recommended)Use the published [`testdriverai/action`](https://github.com/testdriverai/action). It makes the OIDC token, changes it for your team's API key, and exports `TD_API_KEY` for the steps that come after. **There is no `TD_API_KEY` secret to store or change.**
<Note>
Do this one time: authorize the [TestDriver GitHub App](https://console.testdriver.ai) for your org. Then the org → team binding exists. If your org authorized the App before OIDC support was available, authorize it again one time. If the App is not authorized, the action fails with a console link. (Or it uses the `api-key` secret if you give one.)
</Note>
```yaml .github/workflows/testdriver.yml
name: TestDriver Tests
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
test:
runs-on: ubuntu-latest
permissions:
id-token: write # REQUIRED to mint an OIDC token
contents: read
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
- run: npm ci
- name: Authenticate to TestDriver
uses: testdriverai/action@stable # pin @stable / @canary / @test to your SDK channel
with:
api-key: ${{ secrets.TD_API_KEY }} # optional fallback if OIDC isn't set up
- name: Run TestDriver tests
run: npx vitest run
```
### Stored-key fallback
Only if you can't use OIDC (e.g. self-hosted runners without an OIDC provider). Add the key as a secret and pass it via `env`:
1. Navigate to your GitHub repository
2. Go to **Settings** → **Secrets and variables** → **Actions**
3. Click **New repository secret**
4. Name: `TD_API_KEY`, Value: your API key
5. Click **Add secret**
### Basic Workflow
Create `.github/workflows/testdriver.yml`:
```yaml .github/workflows/testdriver.yml
name: TestDriver Tests
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
- run: npm ci
- name: Run TestDriver tests
env:
TD_API_KEY: ${{ secrets.TD_API_KEY }}
run: vitest --run
```
### Parallel Execution
Use matrix strategy to run tests in parallel:
```yaml .github/workflows/testdriver-parallel.yml
name: TestDriver Tests (Parallel)
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3, 4]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
- run: npm ci
- name: Run tests (shard ${{ matrix.shard }}/4)
env:
TD_API_KEY: ${{ secrets.TD_API_KEY }}
run: vitest --run --shard=${{ matrix.shard }}/4
```
### Multi-Platform Testing
```yaml .github/workflows/testdriver-multiplatform.yml
name: TestDriver Tests (Multi-Platform)
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
td-os: [linux, windows]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
- run: npm ci
- name: Run tests on ${{ matrix.td-os }}
env:
TD_API_KEY: ${{ secrets.TD_API_KEY }}
TD_OS: ${{ matrix.td-os }}
run: vitest --run
```
### Adding Secrets
1. Go to your GitLab project
2. Navigate to **Settings** → **CI/CD** → **Variables**
3. Click **Add variable**
4. Key: `TD_API_KEY`, Value: your API key
5. Check **Mask variable** and click **Add variable**
### Basic Pipeline
Create `.gitlab-ci.yml`:
```yaml .gitlab-ci.yml
stages:
- test
testdriver:
stage: test
image: node:20
cache:
paths:
- node_modules/
script:
- npm ci
- vitest --run
variables:
TD_API_KEY: $TD_API_KEY
```
### Parallel Execution
```yaml .gitlab-ci.yml
stages:
- test
.testdriver-base:
stage: test
image: node:20
cache:
paths:
- node_modules/
before_script:
- npm ci
variables:
TD_API_KEY: $TD_API_KEY
testdriver-shard-1:
extends: .testdriver-base
script:
- vitest --run --shard=1/4
testdriver-shard-2:
extends: .testdriver-base
script:
- vitest --run --shard=2/4
testdriver-shard-3:
extends: .testdriver-base
script:
- vitest --run --shard=3/4
testdriver-shard-4:
extends: .testdriver-base
script:
- vitest --run --shard=4/4
```
### Multi-Platform Testing
```yaml .gitlab-ci.yml
stages:
- test
.testdriver-base:
stage: test
image: node:20
cache:
paths:
- node_modules/
before_script:
- npm ci
variables:
TD_API_KEY: $TD_API_KEY
testdriver-linux:
extends: .testdriver-base
variables:
TD_OS: linux
script:
- vitest --run
testdriver-windows:
extends: .testdriver-base
variables:
TD_OS: windows
script:
- vitest --run
```
### Adding Secrets
1. Go to your CircleCI project
2. Click **Project Settings** → **Environment Variables**
3. Click **Add Environment Variable**
4. Name: `TD_API_KEY`, Value: your API key
### Basic Config
Create `.circleci/config.yml`:
```yaml .circleci/config.yml
version: 2.1
jobs:
test:
docker:
- image: cimg/node:20.0
steps:
- checkout
- restore_cache:
keys:
- npm-deps-{{ checksum "package-lock.json" }}
- run: npm ci
- save_cache:
key: npm-deps-{{ checksum "package-lock.json" }}
paths:
- node_modules
- run:
name: Run TestDriver tests
command: vitest --run
environment:
TD_API_KEY: ${TD_API_KEY}
workflows:
test:
jobs:
- test
```
### Parallel Execution
```yaml .circleci/config.yml
version: 2.1
jobs:
test:
docker:
- image: cimg/node:20.0
parallelism: 4
steps:
- checkout
- restore_cache:
keys:
- npm-deps-{{ checksum "package-lock.json" }}
- run: npm ci
- save_cache:
key: npm-deps-{{ checksum "package-lock.json" }}
paths:
- node_modules
- run:
name: Run TestDriver tests
command: |
vitest --run --shard=$((CIRCLE_NODE_INDEX + 1))/$CIRCLE_NODE_TOTAL
environment:
TD_API_KEY: ${TD_API_KEY}
workflows:
test:
jobs:
- test
```
### Multi-Platform Testing
```yaml .circleci/config.yml
version: 2.1
jobs:
test:
docker:
- image: cimg/node:20.0
parameters:
td-os:
type: string
steps:
- checkout
- run: npm ci
- run:
name: Run TestDriver tests on << parameters.td-os >>
command: vitest --run
environment:
TD_API_KEY: ${TD_API_KEY}
TD_OS: << parameters.td-os >>
workflows:
test:
jobs:
- test:
td-os: linux
- test:
td-os: windows
```
### Adding Secrets
1. Go to your Azure DevOps project
2. Navigate to **Pipelines** → **Library** → **Variable groups**
3. Create a new variable group or edit existing
4. Add variable: `TD_API_KEY` with your API key
5. Click the lock icon to make it secret
### Basic Pipeline
Create `azure-pipelines.yml`:
```yaml azure-pipelines.yml
trigger:
- main
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool@0
inputs:
versionSpec: '20.x'
displayName: 'Setup Node.js'
- script: npm ci
displayName: 'Install dependencies'
- script: vitest --run
displayName: 'Run TestDriver tests'
env:
TD_API_KEY: $(TD_API_KEY)
```
### Parallel Execution
```yaml azure-pipelines.yml
trigger:
- main
pool:
vmImage: 'ubuntu-latest'
strategy:
matrix:
shard1:
SHARD: '1/4'
shard2:
SHARD: '2/4'
shard3:
SHARD: '3/4'
shard4:
SHARD: '4/4'
steps:
- task: NodeTool@0
inputs:
versionSpec: '20.x'
- script: npm ci
displayName: 'Install dependencies'
- script: vitest --run --shard=$(SHARD)
displayName: 'Run TestDriver tests'
env:
TD_API_KEY: $(TD_API_KEY)
```
### Multi-Platform Testing
```yaml azure-pipelines.yml
trigger:
- main
pool:
vmImage: 'ubuntu-latest'
strategy:
matrix:
linux:
TD_OS: 'linux'
windows:
TD_OS: 'windows'
steps:
- task: NodeTool@0
inputs:
versionSpec: '20.x'
- script: npm ci
displayName: 'Install dependencies'
- script: vitest --run
displayName: 'Run TestDriver tests on $(TD_OS)'
env:
TD_API_KEY: $(TD_API_KEY)
TD_OS: $(TD_OS)
```
### Adding Secrets
1. Go to **Manage Jenkins** → **Credentials**
2. Select the appropriate domain
3. Click **Add Credentials**
4. Kind: **Secret text**
5. ID: `td-api-key`, Secret: your API key
### Basic Pipeline
Create `Jenkinsfile`:
```groovy Jenkinsfile
pipeline {
agent {
docker {
image 'node:20'
}
}
environment {
TD_API_KEY = credentials('td-api-key')
}
stages {
stage('Install') {
steps {
sh 'npm ci'
}
}
stage('Test') {
steps {
sh 'vitest --run'
}
}
}
}
```
### Parallel Execution
```groovy Jenkinsfile
pipeline {
agent none
environment {
TD_API_KEY = credentials('td-api-key')
}
stages {
stage('Test') {
parallel {
stage('Shard 1') {
agent { docker { image 'node:20' } }
steps {
sh 'npm ci'
sh 'vitest --run --shard=1/4'
}
}
stage('Shard 2') {
agent { docker { image 'node:20' } }
steps {
sh 'npm ci'
sh 'vitest --run --shard=2/4'
}
}
stage('Shard 3') {
agent { docker { image 'node:20' } }
steps {
sh 'npm ci'
sh 'vitest --run --shard=3/4'
}
}
stage('Shard 4') {
agent { docker { image 'node:20' } }
steps {
sh 'npm ci'
sh 'vitest --run --shard=4/4'
}
}
}
}
}
}
```
### Multi-Platform Testing
```groovy Jenkinsfile
pipeline {
agent none
environment {
TD_API_KEY = credentials('td-api-key')
}
stages {
stage('Test') {
parallel {
stage('Linux') {
agent { docker { image 'node:20' } }
environment {
TD_OS = 'linux'
}
steps {
sh 'npm ci'
sh 'vitest --run'
}
}
stage('Windows') {
agent { docker { image 'node:20' } }
environment {
TD_OS = 'windows'
}
steps {
sh 'npm ci'
sh 'vitest --run'
}
}
}
}
}
}
```
Reading Platform in Tests
When using multi-platform testing, read the TD_OS environment variable in your test:
import { describe, expect, it } from "vitest";
import { TestDriver } from "testdriverai/vitest/hooks";
describe("Cross-platform tests", () => {
it("should work on both Linux and Windows", async (context) => {
const os = process.env.TD_OS || 'linux';
const testdriver = TestDriver(context, {
os: os // 'linux' or 'windows'
});
await testdriver.provision.chrome({
url: 'https://example.com',
});
const result = await testdriver.assert("the page loaded successfully");
expect(result).toBeTruthy();
});
});
Concurrency limits
Your plan allows a fixed number of sandboxes running at once. When a test asks for a sandbox and you're already at that limit, the request is queued rather than failed immediately: the SDK waits for a slot to free up, retrying every 10 seconds, then proceeds automatically once one opens. This is what lets a parallel CI matrix (many jobs starting at once) work on a plan with fewer slots than jobs — the extra jobs simply wait their turn instead of erroring.
By default the SDK waits up to 60 seconds for a slot before giving up with a
concurrency-limit error. Control that ceiling with TD_CONCURRENCY_MAX_WAIT:
| Value | Behavior |
|---|---|
| unset | Wait up to 60 seconds (the default). |
TD_CONCURRENCY_MAX_WAIT=300 | Wait up to 300 seconds (5 minutes) before giving up. |
TD_CONCURRENCY_MAX_WAIT=0 | Don't queue — fail on the first denial. |
The value is in seconds (fractional values are allowed and rounded to the nearest millisecond). Any invalid or negative value falls back to the 60-second default. The wait applies per sandbox request, across both the initial allocation and the realtime slot-approval handshake.
# Example: a large parallel matrix that may queue for a while.
# Give each job up to 5 minutes to acquire a slot before failing.
- name: Run TestDriver tests
env:
TD_API_KEY: ${{ secrets.TD_API_KEY }}
TD_CONCURRENCY_MAX_WAIT: "300"
run: npx vitest run
Raise `TD_CONCURRENCY_MAX_WAIT` when you run more parallel jobs than your plan has
slots and would rather they queue than fail. Set it to `0` when you'd prefer a job
to **fail fast** on a busy account (e.g. a quick smoke test that shouldn't sit
waiting). When jobs routinely give up waiting, that's the signal to
[add more slots](https://console.testdriver.ai/checkout/pro).
Viewing Results
All test runs are automatically recorded and visible in your TestDriver dashboard at console.testdriver.ai:
- All test runs with pass/fail status
- Video replays of each test
- Error messages and screenshots on failure
- Git commit and branch information
- Duration trends over time
Element Click
When you call this on an Element object, it clicks on the found element.
Syntax
await element.click(action)
Parameters
Type of click action: `'click'`, `'double-click'`, `'right-click'`, `'hover'`, `'mouseDown'`, `'mouseUp'`Returns
Promise<void>
Examples
// Regular click
const button = await testdriver.find('submit button');
await button.click();
// Double-click
const file = await testdriver.find('README.txt file');
await file.click('double-click');
// Right-click
const item = await testdriver.find('menu item');
await item.click('right-click');
// Hover (same as element.hover())
const tooltip = await testdriver.find('info icon');
await button.click('hover');
Coordinate Click
Click at specific screen coordinates.
Syntax
await testdriver.click(x, y, action)
Parameters
X coordinate Y coordinate Type of click: `'click'`, `'double-click'`, `'right-click'`, `'mouseDown'`, `'mouseUp'`Returns
Promise<void>
Examples
// Click at coordinates
await testdriver.click(500, 300);
// Double-click at coordinates
await testdriver.click(500, 300, 'double-click');
// Right-click at coordinates
await testdriver.click(500, 300, 'right-click');
Click Actions
Regular Click
One left-click action.
const button = await testdriver.find('Login button');
await button.click();
Double Click
A double-click action. It usually opens files or selects text.
const file = await testdriver.find('document.pdf');
await file.click('double-click');
// Or use the dedicated method
await file.doubleClick();
Right Click
Right-click to open context menus.
const folder = await testdriver.find('Documents folder');
await folder.click('right-click');
// Or use the dedicated method
await folder.rightClick();
Mouse Down / Mouse Up
Use these for drag operations or custom click behavior.
const draggable = await testdriver.find('draggable item');
await draggable.click('mouseDown');
// Move to target
const dropZone = await testdriver.find('drop zone');
await dropZone.hover();
await dropZone.click('mouseUp');
// Or use dedicated methods
await draggable.mouseDown();
await dropZone.mouseUp();
Best Practices
**Prefer element clicks to coordinate clicks**A click on an element is more reliable. It does not depend on the resolution:
// ✅ Preferred
const button = await testdriver.find('submit button');
await button.click();
// ❌ Avoid (fragile)
await testdriver.click(500, 300);
**Make sure that TestDriver found the element**
const element = await testdriver.find('button');
if (!element.found()) {
throw new Error('Element not found');
}
await element.click();
**TestDriver must find the element before you click it**
The find() method finds elements automatically. But a click on an element that TestDriver did not find throws an error:
const element = await testdriver.find('button');
// This will throw if element wasn't found
await element.click();
Use Cases
```javascript const submitBtn = await testdriver.find('submit button'); await submitBtn.click();const cancelBtn = await testdriver.find('cancel button');
await cancelBtn.click();
```
```javascript
const file = await testdriver.find('report.pdf file icon');
await file.doubleClick();
```
```javascript
const item = await testdriver.find('file item');
await item.rightClick();
// Select menu option
const deleteOption = await testdriver.find('Delete option');
await deleteOption.click();
```
```javascript
const source = await testdriver.find('source item');
await source.mouseDown();
const target = await testdriver.find('target zone');
await target.hover();
await target.mouseUp();
```
Complete Example
import { beforeAll, afterAll, describe, it } from 'vitest';
import TestDriver from 'testdriverai';
describe('Click Interactions', () => {
let testdriver;
beforeAll(async () => {
client = new TestDriver(process.env.TD_API_KEY);
await testdriver.auth();
await testdriver.connect();
});
afterAll(async () => {
await testdriver.disconnect();
});
it('should perform various click actions', async () => {
await testdriver.focusApplication('Google Chrome');
// Regular click
const loginBtn = await testdriver.find('login button');
await loginBtn.click();
// Right-click for context menu
const profileIcon = await testdriver.find('profile icon');
await profileIcon.rightClick();
const settingsOption = await testdriver.find('Settings menu option');
await settingsOption.click();
// Double-click to edit
const nameField = await testdriver.find('name display field');
await nameField.doubleClick();
// Verify state
await testdriver.assert('name field is now editable');
});
it('should perform drag and drop', async () => {
const item = await testdriver.find('draggable item');
await item.mouseDown();
// Drag to new location
const dropTarget = await testdriver.find('drop area');
await dropTarget.hover();
await dropTarget.mouseUp();
// Verify
await testdriver.assert('item is in the drop area');
});
});
Related Methods
find()- Find elements to clickhover()- Put the cursor on the element without a clickdoubleClick()- The double-click methodrightClick()- The right-click method
Overview
The TestDriver client is the main entry point for the SDK. It does the authentication and the sandbox connection. It gives access to all test methods.
Constructor
const testdriver = new TestDriver(apiKey, options)
Parameters
Your TestDriver API key from the [dashboard](https://console.testdriver.ai/settings) The configuration options for the client. See [SDK Options](/options) for the full list, with defaults and examples for each option.Example
import TestDriver from 'testdriverai';
// API key is automatically loaded from TD_API_KEY in .env
const testdriver = new TestDriver({
os: 'windows',
resolution: '1920x1080',
logging: true,
analytics: true
});
// With AI config for stricter verification
const testdriver = new TestDriver({
ai: { temperature: 0, top: { p: 0.9, k: 40 } }
});
// Or pass API key explicitly
const testdriver = new TestDriver('your-api-key', {
os: 'windows'
});
Authentication
auth()
Authenticate with the TestDriver API.
await testdriver.auth()
Returns: Promise<string> - Authentication token
Example:
await testdriver.auth();
You must call `auth()` before `connect()`. Most examples call both sequentially.
Connection Management
connect()
Connect to a sandbox environment. This creates or reconnects to a virtual machine where your tests will run.
await testdriver.connect(options)
Parameters
Connection options Force creation of a new sandbox instead of reusing an existing one<ParamField path="sandboxId" type="string">
Existing sandbox ID to reconnect to
</ParamField>
<ParamField path="ip" type="string">
Direct IP address to connect to (for self-hosted sandboxes)
</ParamField>
<ParamField path="sandboxAmi" type="string">
AMI to use for the sandbox (AWS deployments)
</ParamField>
<ParamField path="sandboxInstance" type="string">
Instance type for the sandbox (AWS deployments)
</ParamField>
<ParamField path="preview" type="string" default="browser">
Preview mode for live test visualization:
- `"browser"` - Opens debugger in default browser (default)
- `"ide"` - Opens preview in IDE panel (VSCode, Cursor - requires TestDriver extension)
- `"none"` - Headless mode, no visual preview
</ParamField>
<ParamField path="headless" type="boolean" default="false">
**Deprecated**: Use `preview: "none"` instead. Run in headless mode without opening the debugger.
</ParamField>
<ParamField path="keepAlive" type="number" default="60000">
Keep sandbox alive for the specified number of milliseconds after disconnect. Set to `0` to terminate immediately on disconnect. Useful for debugging or reconnecting to the same sandbox.
</ParamField>
Returns: Promise<Object> - Sandbox instance details including instanceId, ip, vncPort, etc.
Examples
Basic connection:
await testdriver.connect();
Reconnect to existing sandbox:
const instance = await testdriver.connect({
sandboxId: 'existing-sandbox-id-123'
});
Self-hosted sandbox:
await testdriver.connect({
ip: '192.168.1.100'
});
disconnect()
Disconnect from the sandbox and clean up resources.
await testdriver.disconnect()
Returns: Promise<void>
Example:
afterAll(async () => {
await testdriver.disconnect();
});
Instance Information
getInstance()
Get the current sandbox instance details.
const instance = testdriver.getInstance()
Returns: Object | null - Sandbox instance information
Example:
const instance = testdriver.getInstance();
console.log('Instance ID:', instance.instanceId);
console.log('IP Address:', instance.ip);
getSessionId()
Get the current session ID for tracking and debugging.
const sessionId = testdriver.getSessionId()
Returns: string | null - Session ID
Example:
const sessionId = testdriver.getSessionId();
console.log('Session:', sessionId);
Logging & Events
setLogging()
Enable or disable console logging at runtime.
testdriver.setLogging(enabled)
Parameters:
enabled(boolean) - Whether to enable logging
Example:
// Disable logging for cleanup operations
testdriver.setLogging(false);
await testdriver.disconnect();
testdriver.setLogging(true);
getEmitter()
Get the event emitter for custom event handling.
const emitter = testdriver.getEmitter()
Returns: EventEmitter2 - Event emitter instance
Example:
const emitter = testdriver.getEmitter();
emitter.on('command:start', (data) => {
console.log('Command started:', data);
});
emitter.on('command:success', (data) => {
console.log('Command succeeded:', data);
});
emitter.on('command:error', (error) => {
console.error('Command failed:', error);
});
Complete Example
import { beforeAll, afterAll, describe, it } from 'vitest';
import TestDriver from 'testdriverai';
describe('My Test Suite', () => {
let testdriver;
beforeAll(async () => {
// Initialize client - API key loaded automatically from .env
testdriver = new TestDriver({
os: 'windows',
resolution: '1366x768',
logging: true
});
// Set up event listeners
const emitter = testdriver.getEmitter();
emitter.on('log:info', (msg) => console.log('[INFO]', msg));
// Authenticate and connect
await testdriver.auth();
const instance = await testdriver.connect();
console.log('Connected to sandbox:', instance.instanceId);
});
afterAll(async () => {
await testdriver.disconnect();
});
it('runs a test', async () => {
// Your test code here
});
});
Best Practices
Use `beforeAll`/`afterAll` to create one sandbox per test suite rather than per test. This significantly reduces execution time. Wrap `connect()` in a try-catch block to handle network issues or quota limits:```javascript
try {
await testdriver.connect();
} catch (error) {
console.error('Failed to connect:', error.message);
throw error;
}
```
Use `afterAll` or try-finally blocks to ensure `disconnect()` is called even if tests fail. This prevents orphaned sandboxes.
Never hardcode API keys. The SDK automatically loads `TD_API_KEY` from your `.env` file:
```bash .env
TD_API_KEY=your_api_key_here
```
```javascript
// API key is loaded automatically - no need to pass it!
const testdriver = new TestDriver();
```
Cloud pricing is based on device-seconds: the amount of time your tests run on our infrastructure.
- Zero Setup — Start testing immediately. No DevOps required.
- Free Tier — Get started with a limited preview at no cost.
- Pay As You Go — Only pay for the device-seconds you use.
Get Started
Cloud is the default when you follow the Quickstart guide.
<Card
title="Try the Quickstart"
icon="play"
href="/v7/quickstart"
Set your API key and start testing in minutes.
Parallel Testing Limits
Your account has a set number of license slots that determine how many devices can run simultaneously. You can view your available slots in the TestDriver Dashboard.
**When is a slot in use?** A license slot is occupied when a test client is connected. As soon as your device is destroyed the slot becomes available immediately.Avoiding Slot Conflicts
To prevent tests from failing due to exceeding your license slot limit, we recommend two key configurations:
Limit concurrent tests to match your available license slots:```javascript vitest.config.mjs
import { defineConfig } from 'vitest/config';
import { TestDriver } from 'testdriverai/vitest';
export default defineConfig({
test: {
testTimeout: 900000,
hookTimeout: 900000,
maxConcurrency: 5, // Set to your license slot limit
reporters: ['default', TestDriver()],
setupFiles: ['testdriverai/vitest/setup'],
},
});
```
<Tip>
Check your slot count at [console.testdriver.ai](https://console.testdriver.ai) and set `maxConcurrency` to that number or lower.
</Tip>
Prevent multiple workflow runs from competing for the same slots by using [GitHub's concurrency controls](https://docs.github.com/actions/writing-workflows/choosing-what-your-workflow-does/control-the-concurrency-of-workflows-and-jobs):
```yaml .github/workflows/test.yml
name: Tests
on:
push:
branches: [main]
pull_request:
# Prevent concurrent runs from competing for license slots
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Install dependencies
run: npm install
- name: Run tests
run: vitest run
env:
TD_API_KEY: ${{ secrets.TD_API_KEY }}
```
The `concurrency` block ensures:
- Only one workflow run per branch runs at a time
- New pushes cancel in-progress runs on the same branch
- Different branches/PRs can run in parallel (up to your slot limit)
When to Consider Self-Hosted
Cloud is perfect for getting started and for teams that want zero infrastructure management. However, you might consider Self-Hosted if you:
- Want to escape per-second billing with a flat license fee
- Require greater concurrency than offered in Cloud plans
- Need full control over your infrastructure and privacy
- Want to use your own AI API keys
- Require custom hardware configurations
- Have high test volumes that make self-hosting more economical
<Card title="Explore Self-Hosted" icon="server" href="/v7/self-hosted"
Learn about self-hosting for unlimited test execution at a flat rate.
TestDriver Options
Configure the TestDriver behavior with options that you give to the TestDriver() function:
const testdriver = TestDriver(context, {
// === Sandbox & Connection ===
newSandbox: true, // Force creation of a new sandbox (default: true)
reconnect: false, // Reconnect to last sandbox (default: false)
keepAlive: 60000, // Keep sandbox alive after disconnect in ms (default: 60000)
os: "linux", // 'linux' | 'windows' (default: 'linux')
resolution: "1366x768", // Sandbox resolution (e.g., '1920x1080')
ip: "203.0.113.42", // Direct IP for self-hosted sandbox
sandboxAmi: "ami-1234", // Custom AMI ID (AWS deployments)
sandboxInstance: "i3.metal", // EC2 instance type (AWS deployments)
// === Preview & Debugging ===
preview: "browser", // "browser" | "ide" | "none" (default: "browser")
headless: false, // @deprecated - use preview: "none" instead
debugOnFailure: false, // Keep sandbox alive on test failure for debugging
// === Caching ===
cache: true, // Enable element caching (default: true)
// Or use advanced caching config:
// cache: {
// enabled: true,
// thresholds: {
// find: { screen: 0.05, element: 0.8 },
// assert: 0.05
// }
// },
cacheKey: "my-test", // Cache key for element finding operations
// === Recording & Screenshots ===
dashcam: true, // Enable/disable Dashcam video recording (default: true)
autoScreenshots: true, // Capture screenshots before/after each command (default: false)
// === AI Configuration ===
ai: { // Global AI sampling configuration
temperature: 0, // 0 = deterministic, higher = more creative
top: {
p: 0.9, // Top-P nucleus sampling (0-1)
k: 40, // Top-K sampling (1 = most likely, 0 = disabled)
},
},
// === Screen Change Detection ===
redraw: true, // Enable redraw detection (default: true)
// Or use advanced redraw config:
// redraw: {
// enabled: true,
// thresholds: {
// screen: 0.05, // Pixel diff threshold (0-1), false to disable
// network: false, // Monitor network activity (default: false)
// }
// },
// === Logging & Analytics ===
logging: true, // Enable console logging output (default: true)
analytics: true, // Enable analytics tracking (default: true)
// === Advanced ===
apiRoot: "https://...", // API endpoint URL (for self-hosted deployments)
environment: {}, // Additional environment variables for the sandbox
});
Preview Mode
Control how test execution is visualized. The preview option determines where the live debugger view opens:
const testdriver = TestDriver(context, {
preview: "browser", // Opens in default browser (default)
});
| Value | Description |
|---|---|
"browser" | Opens debugger in default browser (default) |
"ide" | Opens preview in IDE panel (VSCode, Cursor - requires TestDriver extension) |
"none" | Headless mode, no visual preview |
IDE Preview
For the best development experience, use preview: "ide" with the TestDriver extension for VSCode or Cursor:
const testdriver = TestDriver(context, {
preview: "ide", // Opens preview in IDE panel
});
Headless Mode
Run tests without any visual preview. Useful for CI/CD pipelines:
const testdriver = TestDriver(context, {
preview: "none", // No visual preview (headless)
});
The legacy `headless: true` option still works for backward compatibility and maps to `preview: "none"`.
Debug on Failure
Keep the sandbox alive when a test fails so you can reconnect and debug interactively. The sandbox ID is printed to the console along with instructions for reconnecting via MCP.
const testdriver = TestDriver(context, {
debugOnFailure: true,
});
IP Target
If self-hosting TestDriver, use ip to specify the device IP. See Self-Hosting TestDriver for details.
const testdriver = TestDriver(context, {
ip: "203.0.113.42", // Your allowlisted IP
});
Operating System
Set the os property to run tests on a specific operating system. Available options are linux (default) and windows.
const testdriver = TestDriver(context, {
os: "windows", // Run on Windows sandbox
});
Using Environment Variables
You can make the operating system configurable via environment variables. This requires adding code to read from process.env in your test:
const testdriver = TestDriver(context, {
os: process.env.TD_OS || "linux", // Read from env, default to Linux
});
Then pass the variable when running tests:
# Run tests on Windows
TD_OS=windows vitest run
# Run tests on Linux (default)
TD_OS=linux vitest run
This pattern is useful for running the same test suite across multiple operating systems in CI/CD:
# Example GitHub Actions matrix
strategy:
matrix:
os: [linux, windows]
steps:
- run: TD_OS=${{ matrix.os }} vitest run
Dashcam Recording
Dashcam video recording is enabled by default. Disable it to skip recording:
const testdriver = TestDriver(context, {
dashcam: false,
});
Automatic Screenshots
Screenshots are automatically captured before and after every command (click, type, find, assert, etc.) by default. Each screenshot filename includes the line number from your test file.
Disable automatic screenshots:
const testdriver = TestDriver(context, {
autoScreenshots: false,
});
Caching
Element caching speeds up repeated find() and assert() calls. Enabled by default.
// Disable caching
const testdriver = TestDriver(context, {
cache: false,
});
// Advanced: custom thresholds
const testdriver = TestDriver(context, {
cache: {
enabled: true,
thresholds: {
find: { screen: 0.05, element: 0.8 },
assert: 0.05,
},
},
cacheKey: "my-test",
});
Redraw Detection
Redraw detection waits for the screen to stabilize before taking actions. Enabled by default.
// Disable redraw detection
const testdriver = TestDriver(context, {
redraw: false,
});
// Advanced: custom thresholds with network monitoring
const testdriver = TestDriver(context, {
redraw: {
enabled: true,
thresholds: {
screen: 0.05,
network: true,
},
},
});
AI Configuration
Control how the AI model generates responses for find() verification and assert() calls:
const testdriver = TestDriver(context, {
ai: {
temperature: 0, // 0 = deterministic
top: { p: 0.9, k: 40 },
},
});
Environment Variables
Pass additional environment variables to the sandbox:
const testdriver = TestDriver(context, {
environment: {
MY_VAR: "value",
DEBUG: "true",
},
});
Keepalive
By default, sandboxes stay alive for 60 seconds after disconnect. Customize this with keepAlive:
const testdriver = TestDriver(context, {
keepAlive: 300000, // Keep sandbox alive for 5 minutes after test
});
Set to 0 to terminate immediately:
const testdriver = TestDriver(context, {
keepAlive: 0, // Terminate sandbox immediately on disconnect
});
Connecting to an Existing Sandbox
Speed up test development by connecting to an existing sandbox instead of starting fresh each time. This lets you iterate quickly on failing steps without re-running the entire test from the beginning.
Split your test into two files: one for known-good steps that set up the desired state, and another for work-in-progress steps you want to debug.
const testdriver = TestDriver(context, {
keepAlive: 60000, // Keep sandbox alive for 60 seconds after test
});
// Second test file: experiment.test.mjs (run within keepAlive window)
const testdriver = TestDriver(context, {
keepAlive: 60000,
});
await testdriver.connect({ sandboxId: "sandbox-abc123" });
Then, you can run both tests in sequence:
vitest run -t known-good.test.mjs -t work-in-progress.test.mjs
And as you make changes to work-in-progress.test.mjs, you can re-run just that file to quickly iterate on the failing steps.
vitest run work-in-progress.test.mjs
Connecting to the same machine only works if run within the `keepAlive` window of the previous test.
Overview
Dashcam makes automatic video recordings. It also collects the logs for your tests. It captures screen recordings, application logs, and test details. You can review them later.
Basic Usage
With Presets
Most presets include Dashcam automatically:
import { test } from 'vitest';
import { chrome } from 'testdriverai/presets';
test('my test', async (context) => {
const { testdriver, dashcam } = await chrome(context, {
url: 'https://example.com'
});
// Test executes with recording automatically
await testdriver.find('login button').then(el => el.click());
// Dashcam URL available after test
console.log('Replay:', dashcam.url);
});
Manual Setup
For more control, make a Dashcam instance directly:
import TestDriver from 'testdriverai';
import Dashcam from 'testdriverai/lib/core/Dashcam.js';
const client = await TestDriver.create({ os: 'linux' });
const dashcam = new Dashcam(client, {
apiKey: process.env.DASHCAM_API_KEY
});
await dashcam.auth();
await dashcam.start();
// Run your tests
const url = await dashcam.stop();
console.log('Replay URL:', url);
Constructor
Create a new Dashcam instance:
new Dashcam(client, options)
Parameters
TestDriver client instance Configuration options Dashcam API key for authentication. Set via `TD_API_KEY` environment variable.<ParamField path="autoStart" type="boolean" default={false}>
Automatically start recording after authentication
</ParamField>
<ParamField path="logs" type="array" default={[]}>
Log configurations to add automatically
</ParamField>
Methods
auth()
Authenticate with Dashcam service:
await dashcam.auth(apiKey)
Override the API key set in constructor
Returns: Promise<void>
Example:
await dashcam.auth('your-api-key');
start()
Start recording:
await dashcam.start()
Returns: Promise<void>
Example:
await dashcam.start();
console.log('Recording started');
stop()
Stop recording and retrieve replay URL:
await dashcam.stop()
Returns: Promise<string|null> - Replay URL if available
Example:
const url = await dashcam.stop();
if (url) {
console.log('Watch replay:', url);
} else {
console.log('No replay URL available');
}
addFileLog()
Track a log file in the recording:
await dashcam.addFileLog(path, name)
Path to the log file
Display name for the log in Dashcam
Returns: Promise<void>
Example:
// Linux/Mac
await dashcam.addFileLog('/tmp/app.log', 'Application Log');
// Windows
await dashcam.addFileLog('C:\\logs\\app.log', 'Application Log');
addApplicationLog()
Track application-specific logs:
await dashcam.addApplicationLog(application, name)
Application name to track
Display name for the log
Returns: Promise<void>
Example:
await dashcam.addApplicationLog('Google Chrome', 'Browser Logs');
addWebLog()
Track web request logs by URL pattern:
await dashcam.addWebLog(pattern, name)
URL pattern to match (e.g., `"*example.com*"`)
Display name for the log
Returns: Promise<void>
Example:
await dashcam.addWebLog('*example.com*', 'Web Logs');
addLog()
Generic method to add any type of log:
await dashcam.addLog(config)
Log configuration
Display name for the log
<ParamField path="type" type="string" required>
Log type: `'file'`, `'application'`, or `'web'`
</ParamField>
<ParamField path="path" type="string">
File path (required for type='file')
</ParamField>
<ParamField path="application" type="string">
Application name (required for type='application')
</ParamField>
<ParamField path="pattern" type="string">
URL pattern to match (required for type='web', e.g., `"*example.com*"`)
</ParamField>
Returns: Promise<void>
Example:
await dashcam.addLog({
name: 'Test Output',
type: 'file',
path: '/tmp/test.log'
});
await dashcam.addLog({
name: 'Chrome Logs',
type: 'application',
application: 'Google Chrome'
});
await dashcam.addLog({
name: 'Web Logs',
type: 'web',
pattern: '*example.com*'
});
isRecording()
Check if currently recording:
await dashcam.isRecording()
Returns: Promise<boolean> - True if recording is active
Example:
if (await dashcam.isRecording()) {
console.log('Recording in progress');
}
Properties
recording
Current recording state:
dashcam.recording // boolean
apiKey
Configured API key:
dashcam.apiKey // string
client
Associated TestDriver client:
dashcam.client // TestDriver instance
Complete Examples
Basic Recording
import { test } from 'vitest';
import TestDriver from 'testdriverai';
import Dashcam from 'testdriverai/lib/core/Dashcam.js';
test('record test execution', async () => {
const client = await TestDriver.create({ os: 'linux' });
const dashcam = new Dashcam(client);
await dashcam.auth();
await dashcam.start();
// Run your test
await client.find('button').then(el => el.click());
const url = await dashcam.stop();
console.log('Replay:', url);
await client.cleanup();
});
With Log Tracking
test('record with logs', async () => {
const client = await TestDriver.create({ os: 'linux' });
const dashcam = new Dashcam(client);
await dashcam.auth();
// Add log files before starting
await dashcam.addFileLog('/tmp/testdriver.log', 'TestDriver Log');
await dashcam.addFileLog('/tmp/app.log', 'Application Log');
await dashcam.start();
// Test execution
await client.find('login button').then(el => el.click());
const url = await dashcam.stop();
console.log('Replay with logs:', url);
await client.cleanup();
});
Auto-start Configuration
test('auto-start recording', async () => {
const client = await TestDriver.create({ os: 'linux' });
const dashcam = new Dashcam(client, {
autoStart: true,
logs: [
{
name: 'App Log',
type: 'file',
path: '/tmp/app.log'
}
]
});
await dashcam.auth(); // Automatically starts recording
// Test execution
await client.find('submit button').then(el => el.click());
const url = await dashcam.stop();
console.log('Replay:', url);
await client.cleanup();
});
Using with Presets
import { chrome } from 'testdriverai/presets';
test('preset with dashcam', async (context) => {
const { testdriver, dashcam } = await chrome(context, {
url: 'https://example.com',
dashcam: true // Enabled by default
});
// Test runs with automatic recording
await testdriver.find('button').then(el => el.click());
// URL automatically available
console.log('Replay:', dashcam.url);
});
Disabling Dashcam in Presets
test('without dashcam', async (context) => {
const { testdriver } = await chrome(context, {
url: 'https://example.com',
dashcam: false // Disable recording
});
// Test runs without recording (faster)
await testdriver.find('button').then(el => el.click());
});
Platform Differences
Windows
On Windows, Dashcam uses PowerShell commands and installs via npm:
// Windows-specific paths
await dashcam.addFileLog(
'C:\\Users\\testdriver\\Documents\\testdriver.log',
'TestDriver Log'
);
Linux/Mac
On Linux/Mac, Dashcam uses shell commands:
// Unix-specific paths
await dashcam.addFileLog('/tmp/testdriver.log', 'TestDriver Log');
Overview
When a test fails, debug it. Look at what happened. Do not guess. TestDriver captures screenshots, video replays, and logs while your test runs. Thus you can replay the moment of the failure. You do not need to look at a stack trace. TestDriver MCP gives commands to view and examine the screenshots that TestDriver saved during the test. This lets you debug quickly, develop tests, and compare, without a manual open of image files.
**Automatic Screenshots (Default: Enabled)**: TestDriver captures screenshots automatically before and after each command. The screenshot name has the line number and the action. Thus it is easy to see which line of code made each screenshot. For example: `001-click-before-L42-submit-button.png`MCP Commands
list_local_screenshots
List and filter screenshots saved in the .testdriver/screenshots/ directory:
list_local_screenshots()
Filter Parameters:
Filter the screenshots by test file or subdirectory (for example, "login.test", "mcp-screenshots"). If you do not give this, the tool lists all screenshots. Filter by the exact line number from the test file (for example, 42 matches L42 in the filename). Filter by a line number range. Example: `{ start: 10, end: 20 }` matches screenshots from lines 10-20. Filter by action type: `click`, `find`, `type`, `assert`, `provision`, `scroll`, `hover`, etc. Filter by phase: `"before"` (state before action) or `"after"` (state after action). Regex pattern to match against filename. Example: `"login|signin"` or `"button.*click"`. Filter by exact sequence number. Filter by sequence range. Example: `{ start: 1, end: 10 }` matches first 10 screenshots. Maximum number of results to return (default: 50). Sort results by: `"modified"` (newest first, default), `"sequence"` (execution order), or `"line"` (line number).Returns:
Array of screenshot metadata including:
path- Full absolute path to the screenshot filerelativePath- Path relative to.testdriver/screenshots/name- Screenshot filenamesizeBytes- File size in bytesmodified- Last modification timestampsequence- Sequential number (from auto-screenshots)action- Action type (click, find, etc.)phase- Before/after phaselineNumber- Line number from test filedescription- Element or action description
Example Responses:
// Basic listing
[
{
"path": "/Users/user/project/.testdriver/screenshots/login.test/001-click-before-L42-submit-button.png",
"relativePath": "login.test/001-click-before-L42-submit-button.png",
"name": "001-click-before-L42-submit-button.png",
"sizeBytes": 145632,
"modified": "2026-01-23T10:00:00.000Z",
"sequence": 1,
"action": "click",
"phase": "before",
"lineNumber": 42,
"description": "submit-button"
}
]
view_local_screenshot
View a specific screenshot from the list:
view_local_screenshot({ path: "/full/path/to/screenshot.png" })
Parameters:
Full absolute path to the screenshot file (as returned by `list_local_screenshots`)Returns:
- Image content (displayed to both AI and user via MCP App)
- Screenshot metadata
- Success/error status
Common Workflows
Test Debugging After Failures
When a test fails, you don't have to wonder what went wrong — use powerful filtering to quickly find the screenshots that show exactly what happened:
1. Find screenshots at the failing line:
// If test failed at line 42
list_local_screenshots({ line: 42 })
// View before and after states at that line
view_local_screenshot({ path: ".testdriver/screenshots/login.test/005-click-before-L42-submit-button.png" })
view_local_screenshot({ path: ".testdriver/screenshots/login.test/006-click-after-L42-submit-button.png" })
2. See what happened leading up to the failure:
// Get screenshots from lines 35-45 to see context
list_local_screenshots({ directory: "login.test", lineRange: { start: 35, end: 45 } })
3. Find all assertion screenshots:
// See what the screen looked like during assertions
list_local_screenshots({ action: "assert" })
4. View the final state before failure:
// Get the last 5 screenshots in execution order
list_local_screenshots({ directory: "login.test", sortBy: "sequence", limit: 5 })
Finding Specific Actions
When debugging element interactions:
// Find all click actions
list_local_screenshots({ action: "click" })
// Find what the screen looked like BEFORE each click
list_local_screenshots({ action: "click", phase: "before" })
// Find screenshots related to a specific element using regex
list_local_screenshots({ pattern: "submit|button" })
// Find all type actions (for form filling issues)
list_local_screenshots({ action: "type" })
Understanding Test Flow
View screenshots in execution order to trace test behavior:
// Get screenshots in execution order
list_local_screenshots({ directory: "checkout.test", sortBy: "sequence" })
// Get just the first 10 actions
list_local_screenshots({ sequenceRange: { start: 1, end: 10 }, sortBy: "sequence" })
// Get just the last 10 actions
list_local_screenshots({ directory: "checkout.test", sortBy: "sequence", limit: 10 })
Interactive Test Development
While building tests using MCP tools, view screenshots to verify your test logic:
- After a test run, filter screenshots to see specific actions:
// See all assertions
list_local_screenshots({ action: "assert" })
// See what happened at a specific line you're debugging
list_local_screenshots({ line: 25 })
- Review key points in the test execution:
view_local_screenshot({ path: ".testdriver/screenshots/my-test.test/after-login.png" })
-
Verify element locations and states before adding assertions
-
Iterate - adjust your test code based on what you see in the screenshots
Comparison and Analysis
Compare screenshots to identify issues:
Using phase filtering for before/after comparison:
// See state before all clicks
list_local_screenshots({ action: "click", phase: "before" })
// See state after all clicks
list_local_screenshots({ action: "click", phase: "after" })
Using line-based debugging:
// Something went wrong around line 50
list_local_screenshots({ lineRange: { start: 45, end: 55 } })
Using regex patterns:
// Find screenshots related to login functionality
list_local_screenshots({ pattern: "login|signin|email|password" })
Best Practices
When saving screenshots in tests, use descriptive names to make them easier to identify:```javascript
await testdriver.screenshot("initial-page-load");
await testdriver.screenshot("after-login-click");
await testdriver.screenshot("dashboard-loaded");
```
Then when listing screenshots, you can quickly identify key moments without viewing every image.
Always call `list_local_screenshots` first to see what's available. The list is sorted by modification time (newest first), making it easy to find recent test runs.
When debugging a specific test, use the `directory` parameter to filter screenshots:
```
list_local_screenshots({ directory: "problematic-test.test" })
```
This avoids clutter from other tests.
When a test fails (especially with assertions), look at screenshots immediately before the failure. They show exactly what the AI or test "saw" at that moment, helping you understand why an assertion failed or why an element wasn't found.
TestDriver test reports include screenshots in the timeline. Use MCP screenshot viewing for interactive debugging during development, and test reports for post-run analysis and team sharing.
Remember that each test run clears its screenshot folder. If you need to preserve screenshots for comparison:
```bash
# Copy screenshots before next run
cp -r .testdriver/screenshots/my-test.test .testdriver/screenshots-backup/
```
Screenshot File Organization
Understanding the directory structure helps with efficient screenshot viewing:
.testdriver/
screenshots/
login.test/ # Test file name (without .mjs extension)
001-find-before-L15-email-input.png # Auto: before find() at line 15
002-find-after-L15-email-input.png # Auto: after find() at line 15
003-click-before-L16-email-input.png # Auto: before click() at line 16
004-click-after-L16-email-input.png # Auto: after click() at line 16
login-complete.png # Manual: screenshot("login-complete")
checkout.test/
001-find-before-L12-add-to-cart.png
002-find-after-L12-add-to-cart.png
...
Automatic Screenshot Naming Format
<seq>-<action>-<phase>-L<line>-<description>.png
| Component | Description | Example |
|---|---|---|
seq | Sequential number | 001, 002 |
action | Command name | click, type, find |
phase | Before, after, or error | before, after, error |
L<line> | Line number from test file | L42 |
description | Element/action description | submit-button |
Key Points
- Each test file gets its own subdirectory
- Automatic screenshots include line numbers for easy tracing
- Manual
screenshot()calls use custom names you provide - Folders are cleared at the start of each test run
- All screenshots are PNG format
- Disable automatic screenshots with
autoScreenshots: falseif needed
Interaction List Sidebar (Source of Truth)
When viewing a test run in the TestDriver console, the interaction list sidebar displays a screenshot for each interaction call (find, click, type, assert, etc.). These screenshots show exactly what was on the screen at the time each interaction was executed.
**The sidebar screenshots are the source of truth.** If a test is behaving unexpectedly, check the screenshot attached to the specific interaction in the sidebar — it shows precisely what the AI saw when making its decision. This is more reliable than inferring screen state from test logs or local screenshots alone.Use the interaction list to:
- Verify what the AI saw — confirm the correct page/state was visible when
find()orassert()ran - Debug misclicks — see whether the target element was actually on screen
- Identify timing issues — spot cases where the UI hadn't finished loading before an interaction fired
- Compare runs — review interaction screenshots across multiple runs to catch flaky behavior
Integration with Test Development
During MCP Interactive Development
When using TestDriver MCP tools (session_start, find_and_click, etc.), screenshots are automatically captured and displayed. Additionally, you can view previously saved screenshots:
# After test development session
list_local_screenshots({ directory: "my-new-test.test" })
view_local_screenshot({ path: ".testdriver/screenshots/my-new-test.test/login-page.png" })
This helps verify your test logic before running the full test file.
After Test Runs
When tests fail or behave unexpectedly, replay what happened step by step:
- Run the test with
vitest run tests/my-test.test.mjs - List screenshots using
list_local_screenshots - View relevant screenshots to diagnose the issue
- Update test code based on what you see
- Re-run and verify the fix
Troubleshooting
If `list_local_screenshots` returns an empty array:- Ensure your test includes `await testdriver.screenshot()` calls
- Verify the test actually ran (check test output)
- Check that `.testdriver/screenshots/` directory exists
- Confirm you're in the correct project directory
If `view_local_screenshot` returns an error:
- Verify the path is exactly as returned by `list_local_screenshots`
- Check file permissions - ensure the screenshot file is readable
- Confirm the file hasn't been deleted or moved
If you have hundreds of screenshots making it hard to find what you need, use filtering:
- Filter by test file: `list_local_screenshots({ directory: "my-test.test" })`
- Filter by line number: `list_local_screenshots({ line: 42 })` or `list_local_screenshots({ lineRange: { start: 40, end: 50 } })`
- Filter by action: `list_local_screenshots({ action: "click" })`
- Filter by phase: `list_local_screenshots({ phase: "before" })`
- Use regex: `list_local_screenshots({ pattern: "submit|login" })`
- Limit results: `list_local_screenshots({ limit: 10 })`
- Sort by line: `list_local_screenshots({ sortBy: "line" })`
- Clean up old folders: `rm -rf .testdriver/screenshots/*`
Remember that screenshot folders are cleared at the start of each test run. If you see old screenshots:
- The test may not have run recently
- Or the test failed before reaching the clearing logic
- Manually clear: `rm -rf .testdriver/screenshots/<test-name>/`
Where this fits in the Guide
Debugging is what you reach for when a Run goes sideways or a Validate assertion fails — the screenshots show you precisely what the AI saw before it acted. Once you've diagnosed the failure, the next step is to stop it from recurring.
Related
- screenshot() - Capture screenshots during test execution
- Dashcam - Record full test sessions with video and logs
- assert() - Make AI-powered assertions that benefit from screenshot context
Next
You've seen what went wrong — now keep it from happening again. Let auto-healing repair flaky tests automatically before they fail your suite.Provision methods are the starting point for most tests. They launch applications in your sandbox and prepare the environment for testing.
Chrome Browser
The most common starting point for web testing. Launches Chrome browser and navigates to a URL.
await testdriver.provision.chrome({
url: 'https://example.com',
});
Options
| Option | Type | Default | Description |
|---|---|---|---|
url | string | 'http://testdriver-sandbox.vercel.app/' | URL to navigate to |
maximized | boolean | true | Start browser maximized |
guest | boolean | false | Use guest mode (no profile) |
Example: Basic Web Test
import { describe, expect, it } from "vitest";
import { TestDriver } from "testdriverai/vitest/hooks";
describe("Login Flow", () => {
it("should log in successfully", async (context) => {
const testdriver = TestDriver(context);
await testdriver.provision.chrome({
url: 'https://myapp.com/login',
});
await testdriver.find("Email input").click();
await testdriver.type("user@example.com");
await testdriver.find("Password input").click();
await testdriver.type("password123");
await testdriver.find("Sign In button").click();
const result = await testdriver.assert("the dashboard is visible");
expect(result).toBeTruthy();
});
});
`provision.chrome()` automatically starts Dashcam recording and waits for Chrome to be ready before returning.
Chrome Extensions
Launch Chrome with a custom extension loaded. Supports both local extensions and Chrome Web Store extensions.
Load from Local Path
Clone or create an extension locally, then load it:
// First, get the extension onto the sandbox
await testdriver.exec(
'sh',
'git clone https://github.com/user/my-extension.git /tmp/my-extension',
60000
);
// Launch Chrome with the extension
await testdriver.provision.chromeExtension({
extensionPath: '/tmp/my-extension',
url: 'https://example.com'
});
Load from Chrome Web Store
Load any published extension by its Chrome Web Store ID:
await testdriver.provision.chromeExtension({
extensionId: 'cjpalhdlnbpafiamejdnhcphjbkeiagm', // uBlock Origin
url: 'https://example.com'
});
Find the extension ID in the Chrome Web Store URL. For example, `https://chrome.google.com/webstore/detail/ublock-origin/cjpalhdlnbpafiamejdnhcphjbkeiagm` → ID is `cjpalhdlnbpafiamejdnhcphjbkeiagm`
Options
| Option | Type | Default | Description |
|---|---|---|---|
extensionPath | string | - | Local path to unpacked extension directory |
extensionId | string | - | Chrome Web Store extension ID |
url | string | - | URL to navigate to after launch |
maximized | boolean | true | Start browser maximized |
Example: Testing a Chrome Extension
import { describe, expect, it } from "vitest";
import { TestDriver } from "testdriverai/vitest/hooks";
describe("Chrome Extension Test", () => {
it("should load and interact with extension", async (context) => {
const testdriver = TestDriver(context);
// Clone extension from GitHub
await testdriver.exec(
'sh',
'git clone https://github.com/user/my-extension.git /tmp/my-extension',
60000,
true
);
// Launch Chrome with extension loaded
await testdriver.provision.chromeExtension({
extensionPath: '/tmp/my-extension',
url: 'https://testdriver.ai'
});
// Click extensions puzzle icon
const extensionsButton = await testdriver.find("puzzle-shaped icon in Chrome toolbar");
await extensionsButton.click();
// Interact with your extension
const myExtension = await testdriver.find("My Extension in the dropdown");
await myExtension.click();
const result = await testdriver.assert("extension popup is visible");
expect(result).toBeTruthy();
});
});
Desktop Apps
Download and install desktop applications. Supports .deb, .rpm, .msi, .exe, .AppImage, .dmg, .pkg, and shell scripts.
const filePath = await testdriver.provision.installer({
url: 'https://example.com/app.deb',
appName: 'MyApp', // Focus this app after install
launch: true, // Auto-launch after install
});
Options
| Option | Type | Default | Description |
|---|---|---|---|
url | string | required | URL to download the installer from |
filename | string | auto-detected | Filename to save as |
appName | string | - | Application name to focus after install |
launch | boolean | true | Launch the app after installation |
Supported File Types
| Extension | OS | Install Method |
|---|---|---|
.deb | Linux | dpkg -i + apt-get install -f |
.rpm | Linux | rpm -i |
.AppImage | Linux | chmod +x |
.sh | Linux | chmod +x + execute |
.msi | Windows | msiexec /i /quiet |
.exe | Windows | Silent install (/S) |
.dmg | macOS | Mount + copy to Applications |
.pkg | macOS | installer -pkg |
Example: Install and Test a Desktop App
import { describe, expect, it } from "vitest";
import { TestDriver } from "testdriverai/vitest/hooks";
describe("Desktop App Test", () => {
it("should install and launch app", async (context) => {
const testdriver = TestDriver(context);
// Download and install
const installerPath = await testdriver.provision.installer({
url: 'https://github.com/sharkdp/bat/releases/download/v0.24.0/bat_0.24.0_amd64.deb',
});
// Verify installation
const output = await testdriver.exec('sh', 'bat --version', 5000);
expect(output).toContain('bat');
});
});
Example: Windows Installer
import { describe, expect, it } from "vitest";
import { TestDriver } from "testdriverai/vitest/hooks";
describe("Windows App Test", () => {
it("should install on Windows", async (context) => {
const testdriver = TestDriver(context, {
os: 'windows'
});
// Download MSI installer
const installerPath = await testdriver.provision.installer({
url: 'https://example.com/app.msi',
launch: false, // Don't auto-launch
});
// Custom installation if needed
await testdriver.exec(
'pwsh',
`Start-Process msiexec.exe -ArgumentList "/i", "${installerPath}", "/qn" -Wait`,
120000
);
// Verify installation
const result = await testdriver.assert("application is installed");
expect(result).toBeTruthy();
});
});
Manual Installation
Set launch: false to download without auto-installing:
const filePath = await testdriver.provision.installer({
url: 'https://example.com/custom-script.sh',
launch: false,
});
// Run custom install commands
await testdriver.exec('sh', `chmod +x "${filePath}"`, 5000);
await testdriver.exec('sh', `"${filePath}" --custom-flag`, 60000);
VS Code
Launch Visual Studio Code with optional workspace and extensions.
await testdriver.provision.vscode({
workspace: '/home/testdriver/my-project',
extensions: ['ms-python.python', 'esbenp.prettier-vscode'],
});
Options
| Option | Type | Default | Description |
|---|---|---|---|
workspace | string | - | Workspace folder to open |
extensions | string[] | [] | Extensions to install (by ID) |
Example: VS Code Extension Test
import { describe, expect, it } from "vitest";
import { TestDriver } from "testdriverai/vitest/hooks";
describe("VS Code Test", () => {
it("should open workspace with extensions", async (context) => {
const testdriver = TestDriver(context);
// Create a test project
await testdriver.exec('sh', 'mkdir -p /tmp/test-project && echo "print(1)" > /tmp/test-project/test.py', 10000);
// Launch VS Code
await testdriver.provision.vscode({
workspace: '/tmp/test-project',
extensions: ['ms-python.python'],
});
// Verify VS Code is ready
const result = await testdriver.assert("VS Code is open with the project");
expect(result).toBeTruthy();
// Open the Python file
await testdriver.find("test.py in the explorer").click();
});
});
Choosing the Right Provision Method
| Use Case | Method |
|---|---|
| Testing a website | provision.chrome |
| Testing a Chrome extension | provision.chromeExtension |
| Testing a desktop app (needs installation) | provision.installer |
| Testing VS Code or VS Code extensions | provision.vscode |
Overview
The doubleClick() method does a double-click action on an element. You can call it on an Element instance. Or you can use it with a selector.
Syntax
// Double-click on an element
await element.doubleClick();
// Double-click using a selector
await ai.doubleClick('selector');
Parameters
When called on an Element, no parameters are required.
When called directly on the AI client:
| Parameter | Type | Description |
|---|---|---|
selector | string | The selector describing the element to double-click |
Returns
It returns a Promise<void>. The promise resolves when the double-click action completes.
Examples
Double-Click on Found Element
const fileItem = await ai.find('README.md file');
await fileItem.doubleClick();
Direct Double-Click with Selector
await ai.doubleClick('README.md in the file list');
Opening Files in VS Code
import { test } from 'vitest';
import { vscode } from '@testdriver/sdk';
test('opens a file by double-clicking', async () => {
const { ai } = await vscode();
// Double-click to open a file in the explorer
await ai.doubleClick('package.json in the file explorer');
// Verify the file opened
const editor = await ai.find('text editor showing package.json');
expect(editor).toBeTruthy();
});
Opening Folders in File Manager
import { test } from 'vitest';
import { chrome } from '@testdriver/sdk';
test('navigates folders in Google Drive', async () => {
const { ai } = await chrome('https://drive.google.com');
// Double-click to open a folder
await ai.doubleClick('Documents folder');
// Wait for folder to open
await ai.find('breadcrumb showing Documents');
});
Selecting Text with Double-Click
// Double-click to select a word
await ai.doubleClick('word "TestDriver" in the paragraph');
// Verify selection
const selectedText = await ai.exec('window.getSelection().toString()');
expect(selectedText).toBe('TestDriver');
Related Methods
click()- Single click on an elementrightClick()- Right-click to open context menumouseDown()- Press mouse button without releasingmouseUp()- Release mouse buttonhover()- Move mouse over element without clicking
Overview
The TestDriver element finding system uses AI. It finds elements on the screen with natural language descriptions. The find() method returns an Element object. You can interact with the object.
Finding Elements
find()
Find an element on the screen with a natural language description.
const element = await testdriver.find(description)
Parameters:
description(string) - A natural language description of the element to find
Returns: Promise<Element> - The Element instance that TestDriver found
Example:
// Find a button
const submitButton = await testdriver.find('the submit button');
// Find an input field with context
const emailField = await testdriver.find('email input field in the login form');
// Find an element by visual characteristics
const redButton = await testdriver.find('red button in the top right corner');
Be specific in your descriptions. Include visual details, location context, or nearby text to make the accuracy better.
Element Class
The Element class represents a located (or to-be-located) UI element. It provides methods for interaction and properties for element information.
Methods
found()
Check if the element was successfully located.
element.found()
Returns: boolean - True if element coordinates were found
Example:
const element = await testdriver.find('login button');
if (element.found()) {
await element.click();
} else {
console.log('Element not found');
}
find()
Re-locate the element, optionally with a new description.
await element.find(newDescription)
Parameters:
newDescription(string, optional) - New description to search for
Returns: Promise<Element> - This element instance
Example:
// Re-locate if the UI changed
const element = await testdriver.find('submit button');
// ... page updates ...
await element.find(); // Re-locate with same description
// Or update the description
await element.find('blue submit button'); // Now looking for blue button
click()
Click on the element.
await element.click(action)
Parameters:
action(string, optional) - Type of click:'click'(default),'double-click','right-click','hover','mouseDown','mouseUp'
Returns: Promise<void>
Example:
const button = await testdriver.find('submit button');
await button.click(); // Regular click
const file = await testdriver.find('document.txt');
await file.click('double-click'); // Double-click
const menu = await testdriver.find('settings icon');
await menu.click('right-click'); // Right-click
The element must be found before clicking. The `find()` method automatically locates the element.
hover()
Hover over the element without clicking.
await element.hover()
Returns: Promise<void>
Example:
const tooltip = await testdriver.find('info icon');
await tooltip.hover();
// Wait to see tooltip
await new Promise(resolve => setTimeout(resolve, 1000));
doubleClick()
Double-click on the element.
await element.doubleClick()
Returns: Promise<void>
Example:
const file = await testdriver.find('README.txt file icon');
await file.doubleClick();
rightClick()
Right-click on the element to open context menu.
await element.rightClick()
Returns: Promise<void>
Example:
const folder = await testdriver.find('Documents folder');
await folder.rightClick();
mouseDown() / mouseUp()
Press or release mouse button on the element (for drag operations).
await element.mouseDown()
await element.mouseUp()
Returns: Promise<void>
Example:
// Drag and drop
const item = await testdriver.find('draggable item');
await item.mouseDown();
// Move to drop target (using coordinates or another element)
const target = await testdriver.find('drop zone');
await target.hover();
await target.mouseUp();
Properties
Element properties provide additional information about located elements. Properties are available after a successful find() call.
coordinates
Get the element's coordinates object containing all position information.
const coords = element.getCoordinates()
// or access directly
element.coordinates
Returns: Object | null - Coordinate object with { x, y, centerX, centerY }
Example:
const button = await testdriver.find('submit button');
const coords = button.coordinates;
if (coords) {
console.log(`Top-left: (${coords.x}, ${coords.y})`);
console.log(`Center: (${coords.centerX}, ${coords.centerY})`);
}
x, y, centerX, centerY
Direct access to coordinate values. Always available after successful find().
element.x // Top-left X coordinate (number)
element.y // Top-left Y coordinate (number)
element.centerX // Center X coordinate (number)
element.centerY // Center Y coordinate (number)
Example:
const button = await testdriver.find('submit button');
console.log(`Button at: (${button.x}, ${button.y})`);
console.log(`Button center: (${button.centerX}, ${button.centerY})`);
// Use for custom mouse operations
await testdriver.click(button.centerX, button.centerY);
width, height
Element dimensions in pixels. Available when AI detects element bounds.
element.width // Width in pixels (number | null)
element.height // Height in pixels (number | null)
Example:
const button = await testdriver.find('submit button');
if (button.width && button.height) {
console.log(`Button size: ${button.width}x${button.height}px`);
// Check if button is large enough
if (button.width < 50) {
console.warn('Button might be too small');
}
}
boundingBox
Complete bounding box information including position and dimensions.
element.boundingBox
Returns: Object | null - Bounding box with all dimension data
{
x: number, // Top-left X
y: number, // Top-left Y
width: number, // Width in pixels
height: number // Height in pixels
}
Example:
const element = await testdriver.find('dialog box');
if (element.boundingBox) {
const { x, y, width, height } = element.boundingBox;
console.log(`Dialog: ${width}x${height} at (${x}, ${y})`);
// Calculate if element is in viewport
const rightEdge = x + width;
const bottomEdge = y + height;
console.log(`Element extends to (${rightEdge}, ${bottomEdge})`);
}
screenshot
Base64-encoded PNG screenshot of the screen when element was found. Only available in DEBUG mode or when an error occurs.
element.screenshot
Returns: string | null - Base64-encoded PNG image
Example:
const element = await testdriver.find('error message');
if (element.screenshot) {
// Save screenshot to file
const fs = require('fs');
const base64Data = element.screenshot.replace(/^data:image\/\w+;base64,/, '');
fs.writeFileSync('element-screenshot.png', Buffer.from(base64Data, 'base64'));
console.log('Screenshot saved');
}
Screenshots can be large. They're automatically excluded from error messages to prevent memory issues.
text
Text content extracted from the element by AI (if available).
element.text
Returns: string | null - Element's text content
Example:
const message = await testdriver.find('notification message');
if (message.text) {
console.log('Message says:', message.text);
// Use text content in assertions
if (message.text.includes('success')) {
console.log('Success message detected');
}
}
// Another example - extracting button label
const button = await testdriver.find('blue button');
console.log('Button text:', button.text); // "Submit"
label
Accessible label or name of the element (if available). Useful for verifying accessibility.
element.label
Returns: string | null - Accessible label
Example:
const input = await testdriver.find('first input field');
if (input.label) {
console.log('Input label:', input.label); // "Email Address"
}
confidence
AI confidence score for the element match (0-1, where 1 is perfect confidence).
element.confidence
Returns: number | null - Confidence score between 0 and 1
Example:
const element = await testdriver.find('submit button');
if (element.confidence !== null) {
const percentage = (element.confidence * 100).toFixed(1);
console.log(`Match confidence: ${percentage}%`);
if (element.confidence < 0.8) {
console.warn('⚠️ Low confidence match - element might not be correct');
} else if (element.confidence > 0.95) {
console.log('✅ High confidence match');
}
}
Confidence scores below 0.8 may indicate the element description was ambiguous or the wrong element was found.
Property Availability
| Property | When Available |
|---|---|
x, y, centerX, centerY | ✅ Always after successful find() |
coordinates | ✅ Always after successful find() |
width, height | ⚠️ When AI detects element bounds |
boundingBox | ⚠️ When AI detects element bounds |
text | ⚠️ When AI extracts text content |
label | ⚠️ When element has accessible label |
confidence | ✅ Always after AI element finding |
screenshot | ⚠️ Only in DEBUG mode or on errors |
JSON Serialization
Element objects can be safely serialized using JSON.stringify() for logging, debugging, and data storage. Circular references are automatically removed:
const element = await testdriver.find('login button');
// Safe to stringify - no circular reference errors!
console.log(JSON.stringify(element, null, 2));
Serialized output includes:
{
"description": "login button",
"coordinates": { "x": 100, "y": 200, "centerX": 150, "centerY": 225 },
"found": true,
"threshold": 0.01,
"x": 100,
"y": 200,
"cache": {
"hit": true,
"strategy": "pixel-diff",
"createdAt": "2025-12-09T10:30:00.000Z",
"diffPercent": 0.0023,
"imageUrl": "https://cache.testdriver.ai/..."
},
"similarity": 0.98,
"confidence": 0.95,
"selector": "button#login",
"aiResponse": "Found the blue login button in the center of the form..."
}
Serialized properties:
| Property | Type | Description |
|---|---|---|
description | string | Element search description |
coordinates | object | Full coordinate object {x, y, centerX, centerY} |
found | boolean | Whether element was located |
threshold | number | Cache threshold used for this find |
x, y | number | Top-left coordinates |
cache.hit | boolean | Whether cache was used |
cache.strategy | string | Cache strategy (e.g., "pixel-diff") |
cache.createdAt | string | ISO timestamp when cache was created |
cache.diffPercent | number | Pixel difference from cached image |
cache.imageUrl | string | URL to cached screenshot |
similarity | number | Similarity score (0-1) |
confidence | number | AI confidence score (0-1) |
selector | string | CSS/XPath selector if available |
aiResponse | string | AI's explanation of what it found |
Use cases:
// Debugging element detection
const element = await testdriver.find('submit button');
if (!element.found()) {
console.error('Element not found:', JSON.stringify(element, null, 2));
}
// Logging cache performance
const data = JSON.parse(JSON.stringify(element));
if (data.cache.hit) {
console.log(`Cache hit! Diff: ${(data.cache.diffPercent * 100).toFixed(2)}%`);
}
// Sharing element data across processes
const elementData = JSON.stringify(element);
// Send to another process, log to file, etc.
Use JSON serialization when you need to log element data or when debugging why an element wasn't found. The serialized output excludes large binary data (screenshots) and circular references.
Examples
Basic Element Interaction
// Find and click
const submitButton = await testdriver.find('submit button');
await submitButton.click();
// Find, verify, then interact
const emailInput = await testdriver.find('email input field');
if (emailInput.found()) {
await emailInput.click();
await testdriver.type('user@example.com');
}
Working with Forms
// Fill out a multi-field form
const nameField = await testdriver.find('name input field');
await nameField.click();
await testdriver.type('John Doe');
const emailField = await testdriver.find('email input field');
await emailField.click();
await testdriver.type('john@example.com');
const submitButton = await testdriver.find('submit button');
await submitButton.click();
Conditional Interactions
// Check if element exists before interacting
const closeButton = await testdriver.find('close popup button');
if (closeButton.found()) {
await closeButton.click();
console.log('Popup closed');
} else {
console.log('No popup to close');
}
Re-locating Dynamic Elements
// Element that moves or changes
const notification = await testdriver.find('success notification');
// Do something that might cause it to move
await testdriver.scroll('down', 300);
// Re-locate the element
await notification.find();
if (notification.found()) {
await notification.click();
}
Best Practices
Include visual details, position context, and nearby text:```javascript
// ❌ Too vague
await testdriver.find('button');
// ✅ Specific
await testdriver.find('blue submit button below the email field');
```
Always verify elements were located before interacting:
```javascript
const element = await testdriver.find('submit button');
if (!element.found()) {
throw new Error('Submit button not found');
}
await element.click();
```
If you need to interact with the same element multiple times, reuse the reference:
```javascript
const input = await testdriver.find('search input');
await input.click();
await testdriver.type('first search');
await testdriver.pressKeys(['enter']);
// Re-use the same element reference
await input.click();
await testdriver.pressKeys(['ctrl', 'a']); // Select all
await testdriver.type('second search');
```
This page has moved to Self-Hosted.
Overview
TestDriver gives custom error classes with full debug information. The SDK exports them. You can use them for instanceof checks in your tests.
import TestDriver, { ElementNotFoundError, AIError } from 'testdriverai';
ElementNotFoundError
TestDriver throws this when find() cannot find an element on the screen. It also throws this when you call click() or hover() on an element that TestDriver did not find.
try {
await testdriver.find('nonexistent button').click();
} catch (error) {
if (error instanceof ElementNotFoundError) {
console.log(error.description); // "nonexistent button"
console.log(error.screenshotPath); // path to debug screenshot
console.log(error.pixelDiffPath); // path to pixel diff image
}
}
Properties
Always `"ElementNotFoundError"`. A message with a debug block. The block has the element description, the cache status, the similarity scores, and the AI response details. The original element description passed to `find()`. Absolute path to a debug screenshot saved at the time of failure. Written to `/testdriver-debug/screenshot-.png`. Absolute path to a pixel diff image showing the comparison between the cached and current screenshots. Written to `/testdriver-debug/pixel-diff-error-.png`. Only present when cache was involved. URL to the cached image that was compared against. Sanitized AI response object. Large binary fields (`croppedImage`, `screenshot`, `pixelDiffImage`) are removed. Contains cache metadata like `similarity`, `cacheHit`, `cacheStrategy`, `cacheDiffPercent`, and `threshold`. ISO 8601 timestamp of when the error was created.Enhanced Message
The error message is automatically enhanced with debugging information:
Element not found: "submit button"
=== Element Debug Info ===
Element: submit button
Cache Hit: false
Similarity: 0.23
Cache Strategy: pixel-diff
Threshold: 0.05
AI Response Element: null
Stack Trace Cleaning
Stack traces are automatically cleaned to remove internal SDK frames (Element.*, sdk.js internals), showing only your test code for easier debugging.
AIError
Thrown when act() exhausts all retry attempts.
try {
await testdriver.act('perform complex workflow', { tries: 3 });
} catch (error) {
if (error instanceof AIError) {
console.log(error.task); // "perform complex workflow"
console.log(error.tries); // 3
console.log(error.duration); // 15234 (ms)
console.log(error.cause); // underlying Error
}
}
Properties
Always `"AIError"`. Enhanced message with execution details block. The task description passed to `act()`. Number of attempts that were made. Maximum number of attempts configured. Total execution time in milliseconds. The underlying error that caused the final failure. ISO 8601 timestamp of when the error was created.Enhanced Message
AI failed: Element not found after 3 attempts
=== AI Execution Details ===
Task: perform complex workflow
Tries: 3 / 3
Duration: 15234ms
Cause: ElementNotFoundError: Element not found: "submit button"
Internal Error Classes
These errors are used internally by the agent and are not exported, but may appear as the cause of an AIError:
MatchError
Thrown when element matching fails (text, image, or assertion).
| Property | Type | Description |
|---|---|---|
fatal | boolean | If true, cannot be healed. Default: false |
attachScreenshot | boolean | Always true — a screenshot is attached to the error |
CommandError
Thrown for invalid arguments or unsupported operations.
| Property | Type | Description |
|---|---|---|
fatal | boolean | Always true |
attachScreenshot | boolean | Always false |
Soft Assert Mode
Inside act(), assertions run in soft assert mode. When an assertion fails, it returns the failure result instead of throwing, allowing the AI to process the failure and adjust its approach.
// Inside act(), assertion failures don't throw
await testdriver.act('verify the dashboard shows correct data', {
tries: 3,
});
// The AI can see assertion results and self-correct
This is automatic — you don't need to configure it. Regular assert() calls outside of act() will throw normally on failure.
Error Handling Patterns
Catching Specific Errors
import TestDriver, { ElementNotFoundError, AIError } from 'testdriverai';
try {
await testdriver.find('submit button').click();
} catch (error) {
if (error instanceof ElementNotFoundError) {
// Element wasn't found — check screenshot for debugging
console.log('Debug screenshot:', error.screenshotPath);
} else if (error instanceof AIError) {
// AI exhausted retries
console.log(`Failed after ${error.tries} tries in ${error.duration}ms`);
} else {
throw error; // Unexpected error
}
}
Using Debug Screenshots
try {
const el = await testdriver.find('checkout button');
await el.click();
} catch (error) {
if (error instanceof ElementNotFoundError) {
// Screenshot of what the screen looked like
console.log('Screen:', error.screenshotPath);
// Pixel diff showing cache comparison
console.log('Diff:', error.pixelDiffPath);
// Full AI response metadata
console.log('AI:', JSON.stringify(error.aiResponse, null, 2));
}
}
Types
class ElementNotFoundError extends Error {
name: 'ElementNotFoundError';
description: string;
screenshotPath: string | null;
pixelDiffPath: string | null;
cachedImagePath: string | null;
aiResponse: Record<string, any> | null;
timestamp: string;
}
class AIError extends Error {
name: 'AIError';
task: string;
tries: number;
maxTries: number;
duration: number;
cause?: Error;
timestamp: string;
}
Overview
TestDriver uses EventEmitter2 for its event system. Events use a namespace pattern with colons. They permit wildcard listeners.
Use the emitter through testdriver.emitter:
testdriver.emitter.on('command:start', (data) => {
console.log(`Running: ${data.command}`);
});
Configuration
The internal emitter is created with:
new EventEmitter2({
wildcard: true,
delimiter: ':',
maxListeners: 20,
verboseMemoryLeak: false,
ignoreErrors: false,
});
Wildcard Listeners
Use * to match one level. Use ** to match many levels:
// Match all log events
testdriver.emitter.on('log:*', (message) => {
console.log(message);
});
// Match all events in any namespace
testdriver.emitter.on('**', (...args) => {
console.log('Event:', this.event, args);
});
Event Reference
Command Events
Emitted during the execution of SDK commands (click, type, find, etc.).
| Event | Payload |
|---|---|
command:start | { command, depth, data, timestamp, sourcePosition } |
command:success | { command, depth, data, duration, response, timestamp, sourcePosition } |
command:error | { command, depth, data, error, duration, timestamp, sourcePosition } |
command:status | { command, status: "executing", data, depth, timestamp } |
command:progress | { command, status: "completed", timing, data, depth, timestamp } |
testdriver.emitter.on('command:start', ({ command, data }) => {
console.log(`Starting ${command}`, data);
});
testdriver.emitter.on('command:error', ({ command, error, duration }) => {
console.error(`${command} failed after ${duration}ms: ${error}`);
});
Step Events
Emitted for each AI reasoning step within a command.
| Event | Payload |
|---|---|
step:start | { stepIndex, prompt, commandCount, timestamp, sourcePosition } |
step:success | { stepIndex, prompt, commandCount, duration, timestamp, sourcePosition } |
step:error | { stepIndex, prompt, error, duration?, timestamp, sourcePosition? } |
testdriver.emitter.on('step:start', ({ stepIndex, prompt }) => {
console.log(`Step ${stepIndex}: ${prompt}`);
});
Test Events
Emitted when a test file execution starts.
| Event | Payload |
|---|---|
test:start | { filePath, timestamp } |
test:success | Emitted on test completion |
test:error | Emitted on test failure |
Log Events
Emitted for all log output from the SDK.
| Event | Payload |
|---|---|
log:log | (message: string) — general log message |
log:warn | (message: string) — warning message |
log:debug | (message: string) — debug output (only when VERBOSE/DEBUG/TD_DEBUG env set) |
log:info | (message: string) — informational message |
log:error | (message: string) — error message |
log:narration | (message: string, overwrite?: boolean) — in-place status line |
log:markdown | (markdown: string) — full static markdown content |
log:markdown:start | (streamId: string) — begin streaming markdown |
log:markdown:chunk | (streamId: string, chunk: string) — incremental chunk |
log:markdown:end | (streamId: string) — end streaming markdown |
// Capture all logs
testdriver.emitter.on('log:*', function (message) {
console.log(`[${this.event}]`, message);
});
Screen Capture Events
Emitted during screenshot capture.
| Event | Payload |
|---|---|
screen-capture:start | { scale, silent, display } |
screen-capture:end | { scale, silent, display } |
screen-capture:error | { error, scale, silent, display } |
Sandbox Events
Emitted for sandbox WebSocket lifecycle and communication.
| Event | Payload |
|---|---|
sandbox:connected | No payload — WebSocket connection established |
sandbox:authenticated | { traceId } — authentication successful |
sandbox:error | (err: Error | string) — connection or sandbox error |
sandbox:sent | (message: object) — WebSocket message sent |
sandbox:received | No payload — successful message reply received |
sandbox:progress | { step, message } — sandbox setup progress |
testdriver.emitter.on('sandbox:connected', () => {
console.log('Connected to sandbox');
});
testdriver.emitter.on('sandbox:progress', ({ step, message }) => {
console.log(`Sandbox: [${step}] ${message}`);
});
Redraw Events
Emitted during screen stability detection. See Redraw for more details.
| Event | Payload |
|---|---|
redraw:status | { redraw: { enabled, settled, hasChangedFromInitial, consecutiveFramesStable, diffFromInitial, diffFromLast, text }, network: { enabled, settled, rxBytes, txBytes, text }, timeout: { isTimeout, elapsed, max, text } } |
redraw:complete | { screenSettled, hasChangedFromInitial, consecutiveFramesStable, networkSettled, isTimeout, timeElapsed } |
testdriver.emitter.on('redraw:complete', (result) => {
if (result.isTimeout) {
console.warn('Redraw timed out after', result.timeElapsed, 'ms');
}
});
File Events
Emitted during file load/save operations in the agent.
| Event | Payload |
|---|---|
file:start | { operation: "load" | "save" | "run", filePath, timestamp } |
file:stop | { operation, filePath, duration, success, sourceMap?, reason?, timestamp } |
file:load | { filePath, size, timestamp } |
file:save | { filePath, size, timestamp } |
file:diff | { filePath, diff: { patches, sourceMaps, summary: { additions, deletions, modifications } }, timestamp } |
file:error | { operation, filePath, error, duration?, timestamp } |
Error Events
Emitted for errors at various severity levels.
| Event | Payload |
|---|---|
error:fatal | (error: string | Error) — terminates the process |
error:general | (message: string) — non-fatal error |
error:sandbox | (err: Error | string) — sandbox/WebSocket error |
testdriver.emitter.on('error:*', function (err) {
console.error(`[${this.event}]`, err);
});
SDK Events
Emitted for API request lifecycle.
| Event | Payload |
|---|---|
sdk:request | { path } — outgoing API request |
sdk:response | { path } — API response received |
sdk:retry | { path, attempt, error, delayMs } — request retry |
Other Events
| Event | Payload |
|---|---|
exit | (exitCode: number) — 0 for success, 1 for failure |
status | (message: string) — general status updates |
mouse-click | { x, y, button, click, double } — mouse click performed |
terminal:stdout | Terminal stdout output |
terminal:stderr | Terminal stderr output |
Practical Examples
Custom Test Reporter
const results = [];
testdriver.emitter.on('command:success', ({ command, duration }) => {
results.push({ command, duration, status: 'pass' });
});
testdriver.emitter.on('command:error', ({ command, duration, error }) => {
results.push({ command, duration, status: 'fail', error });
});
// After test completes
afterAll(() => {
console.table(results);
});
Progress Monitoring
testdriver.emitter.on('step:start', ({ stepIndex, prompt }) => {
process.stdout.write(`\r Step ${stepIndex}: ${prompt}`);
});
testdriver.emitter.on('command:progress', ({ command, timing }) => {
process.stdout.write(`\r ${command} completed in ${timing}ms`);
});
Debug Logging
// Log every event (verbose)
testdriver.emitter.on('**', function (...args) {
console.debug(`[EVENT] ${this.event}`, ...args);
});
Types
interface CommandStartEvent {
command: string;
depth: number;
data: Record<string, any>;
timestamp: number;
sourcePosition: SourcePosition;
}
interface CommandSuccessEvent {
command: string;
depth: number;
data: Record<string, any>;
duration: number;
response: any;
timestamp: number;
sourcePosition: SourcePosition;
}
interface CommandErrorEvent {
command: string;
depth: number;
data: Record<string, any>;
error: string;
duration: number;
timestamp: number;
sourcePosition: SourcePosition;
}
interface StepStartEvent {
stepIndex: number;
prompt: string;
commandCount: number;
timestamp: number;
sourcePosition: SourcePosition;
}
interface StepSuccessEvent {
stepIndex: number;
prompt: string;
commandCount: number;
duration: number;
timestamp: number;
sourcePosition: SourcePosition;
}
interface RedrawStatusEvent {
redraw: {
enabled: boolean;
settled: boolean;
hasChangedFromInitial: boolean;
consecutiveFramesStable: number;
diffFromInitial: number;
diffFromLast: number;
text: string;
};
network: {
enabled: boolean;
settled: boolean;
rxBytes: number;
txBytes: number;
text: string;
};
timeout: {
isTimeout: boolean;
elapsed: number;
max: number;
text: string;
};
}
interface RedrawCompleteEvent {
screenSettled: boolean;
hasChangedFromInitial: boolean;
consecutiveFramesStable: number;
networkSettled: boolean;
isTimeout: boolean;
timeElapsed: number;
}
interface SandboxProgressEvent {
step: string;
message: string;
}
interface SourcePosition {
file: string;
line: number;
column: number;
}
Overview
Run shell commands (Linux) or PowerShell commands (Windows) in the sandbox.
Syntax
await testdriver.exec(language, code, timeout, silent)
Parameters
The language to run: `'sh'` (Shell/Linux) or `'pwsh'` (PowerShell/Windows) The code or the command to run The timeout in milliseconds If `true`, TestDriver does not show the outputReturns
Promise<string> - The command output
Shell Execution (Linux)
Run shell commands in the Linux sandbox.
Basic Commands
// List files
const files = await testdriver.exec('sh', 'ls -la', 5000);
// Check current directory
const pwd = await testdriver.exec('sh', 'pwd', 5000);
// Run a script
await testdriver.exec('sh', './setup.sh', 60000);
File Operations
// Create a file
await testdriver.exec('sh', 'echo "Hello World" > test.txt', 5000);
// Read a file
const content = await testdriver.exec('sh', 'cat test.txt', 5000);
// Copy files
await testdriver.exec('sh', 'cp source.txt dest.txt', 5000);
// Delete files
await testdriver.exec('sh', 'rm test.txt', 5000);
PowerShell Execution (Windows)
Execute PowerShell commands in the Windows sandbox.
Software Installation
// Install npm package globally
await testdriver.exec('pwsh', 'npm install -g http-server', 30000);
// Install via Chocolatey
await testdriver.exec('pwsh', 'choco install firefox -y', 60000);
// Download and run installer
await testdriver.exec('pwsh', `
Invoke-WebRequest -Uri "https://example.com/setup.exe" -OutFile "C:\\setup.exe"
Start-Process -FilePath "C:\\setup.exe" -ArgumentList "/S" -Wait
`, 120000);
File Operations
// Create a file
await testdriver.exec('pwsh', `
Set-Content -Path "C:\\test.txt" -Value "Hello World"
`, 5000);
// Read a file
const content = await testdriver.exec('pwsh',
'Get-Content -Path "C:\\test.txt"',
5000
);
// Copy files
await testdriver.exec('pwsh',
'Copy-Item -Path "C:\\source.txt" -Destination "C:\\dest.txt"',
5000
);
// Delete files
await testdriver.exec('pwsh',
'Remove-Item -Path "C:\\test.txt"',
5000
);
// List directory
const files = await testdriver.exec('pwsh',
'Get-ChildItem -Path "C:\\Users\\testdriver\\Documents"',
5000
);
Process Management
// List running processes
const processes = await testdriver.exec('pwsh', 'Get-Process', 5000);
// Start application
await testdriver.exec('pwsh', `
Start-Process "C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe" -ArgumentList "--start-maximized", "https://example.com"
`, 5000);
// Kill a process
await testdriver.exec('pwsh', 'Stop-Process -Name "chrome" -Force', 5000);
// Wait for process
await testdriver.exec('pwsh', 'Start-Process notepad -Wait', 30000);
Environment Variables
// Set environment variable (session)
await testdriver.exec('pwsh', '$env:MY_VAR = "value"', 5000);
// Get environment variable
const value = await testdriver.exec('pwsh', '$env:MY_VAR', 5000);
// Set persistent environment variable
await testdriver.exec('pwsh', `
[Environment]::SetEnvironmentVariable("MY_VAR", "value", "User")
`, 5000);
Network Operations
// Test connectivity
const pingResult = await testdriver.exec('pwsh',
'Test-NetConnection google.com',
10000
);
// Download file
await testdriver.exec('pwsh', `
Invoke-WebRequest -Uri "https://example.com/file.zip" -OutFile "C:\\Downloads\\file.zip"
`, 30000);
// Check if port is open
const portOpen = await testdriver.exec('pwsh',
'Test-NetConnection -ComputerName localhost -Port 3000',
5000
);
Silent Execution
Suppress output for background operations:
// Silent installation
await testdriver.exec('pwsh', 'npm install -g some-package', 30000, true);
// Start background process
await testdriver.exec('pwsh', 'Start-Process notepad', 5000, true);
// Run setup script silently
await testdriver.exec('pwsh', '.\\setup.ps1', 60000, true);
Best Practices
**Use appropriate timeouts**// Quick operations: 5000ms
await testdriver.exec('sh', 'ls -la', 5000);
// Installations: 30000-60000ms
await testdriver.exec('pwsh', 'npm install -g package', 30000);
// Downloads or complex operations: 60000-120000ms
await testdriver.exec('pwsh', 'Install-Module Something', 120000);
**Handle errors gracefully**
try {
await testdriver.exec('pwsh', 'Some-Command', 5000);
} catch (error) {
console.error('Command failed:', error.message);
// Fallback or retry logic
}
**Use silent mode for background tasks**
// Silent install
await testdriver.exec('pwsh', 'npm install -g tool', 30000, true);
// Background service
await testdriver.exec('pwsh', 'Start-Service MyService', 5000, true);
**Escape strings properly in PowerShell**
Use proper escaping for special characters:
// Use backticks for newlines
await testdriver.exec('pwsh', `
Write-Host "Line 1\`nLine 2"
`, 5000);
// Use single quotes to avoid variable expansion
await testdriver.exec('pwsh',
"Write-Host 'Text with $special chars'",
5000
);
Complete Example
import { beforeAll, afterAll, describe, it } from 'vitest';
import TestDriver from 'testdriverai';
describe('Code Execution', () => {
let testdriver;
beforeAll(async () => {
client = new TestDriver(process.env.TD_API_KEY);
await testdriver.auth();
await testdriver.connect();
});
afterAll(async () => {
await testdriver.disconnect();
});
it('should execute shell commands on Linux', async () => {
// List directory
const files = await testdriver.exec('sh', 'ls -la', 5000);
console.log('Files:', files);
// Create a file
await testdriver.exec('sh', 'echo "Hello World" > test.txt', 5000);
// Read the file
const content = await testdriver.exec('sh', 'cat test.txt', 5000);
expect(content).toContain('Hello World');
});
it('should install and use tools on Windows', async () => {
// Install tool
await testdriver.exec('pwsh', 'npm install -g http-server', 30000, true);
// Create HTML file
await testdriver.exec('pwsh', `
Set-Content -Path "C:\\index.html" -Value "<h1>Test Page</h1>"
`, 5000);
// Start server in background
await testdriver.exec('pwsh', `
Start-Process pwsh -ArgumentList "-Command", "http-server C:\\ -p 8080"
`, 5000, true);
await new Promise(r => setTimeout(r, 3000));
// Launch browser
await testdriver.exec('pwsh', `
Start-Process chrome -ArgumentList "http://localhost:8080"
`, 5000);
await testdriver.focusApplication('Google Chrome');
});
});
Related Methods
focusApplication()- Focus apps before execfind()- Locate elements visuallytype()- Type text into inputs
Overview
Find UI elements on the screen with natural language descriptions and AI. This returns an Element object. You can interact with the object.
Syntax
const element = await testdriver.find(description)
const element = await testdriver.find(description, options)
Parameters
Natural language description of the element to find Optional configuration for finding and caching A custom cache key to store the element location. Use this to keep the cache clean when you use dynamic variables in prompts. You can also use it to share the cache between tests.<ParamField path="cacheThreshold" type="number" default={0.05}>
The similarity threshold (0-1) for a cache match. A lower value needs more similarity. Set it to -1 to disable the cache.
</ParamField>
<ParamField path="timeout" type="number" default={10000}>
The maximum time in milliseconds to poll for the element. TestDriver tries again each 5 seconds until it finds the element or the timeout ends. The default is `10000` (10 seconds). Set it to `0` to disable the poll and try one time.
</ParamField>
<ParamField path="confidence" type="number">
The minimum confidence threshold (0-1). If the confidence score of the found element is less than this value, the find is a failure (`element.found()` returns `false`). Use this to make sure of good matches in critical test steps.
</ParamField>
<ParamField path="type" type="string">
Element type hint that wraps the description for better matching. Accepted values:
- `"text"` — Wraps the prompt as `The text "..."`
- `"image"` — Wraps the prompt as `The image "..."`
- `"ui"` — Wraps the prompt as `The UI element "..."`
- `"any"` — No wrapping, uses the description as-is (default behavior)
</ParamField>
<ParamField path="zoom" type="boolean" default={false}>
A two-phase zoom mode for more precision in full UIs that have many similar elements. It is disabled by default.
</ParamField>
<ParamField path="verify" type="boolean" default={false}>
This enables AI verification of the found element. When `true`, a second AI call makes sure that the coordinates agree with the correct element. This catches incorrect positions. It is disabled by default for less latency. When you do not set it for each call, it uses the global `verify` option from the [SDK options](/options#ai-and-element-location).
</ParamField>
<ParamField path="ai" type="object">
AI sampling configuration for this find call (overrides global `ai` config from constructor).
<Expandable title="properties">
<ParamField path="temperature" type="number">
Controls randomness. `0` = deterministic. Default: `0` for find verification.
</ParamField>
<ParamField path="top" type="object">
Sampling parameters
<Expandable title="properties">
<ParamField path="p" type="number">
Top-P (nucleus sampling). Range: 0-1.
</ParamField>
<ParamField path="k" type="number">
Top-K sampling. `1` = most deterministic.
</ParamField>
</Expandable>
</ParamField>
</Expandable>
</ParamField>
Returns
Promise<Element> - The Element instance that TestDriver found automatically.
Examples
Basic Element Finding
// Find by role
const button = await testdriver.find('submit button');
const input = await testdriver.find('email input field');
// Find by text content
const link = await testdriver.find('Contact Us link');
const heading = await testdriver.find('Welcome heading');
// Find by visual appearance
const icon = await testdriver.find('red warning icon');
const image = await testdriver.find('company logo image');
Finding with Context
// Provide location context
const field = await testdriver.find('username input in the login form');
const button = await testdriver.find('delete button in the top right corner');
// Describe nearby elements
const input = await testdriver.find('input field below the email label');
const checkbox = await testdriver.find('checkbox next to "Remember me"');
// Describe visual position
const menu = await testdriver.find('hamburger menu icon in the top left');
Interacting with Found Elements
// Find and click
const submitBtn = await testdriver.find('submit button');
await submitBtn.click();
// Find and verify
const message = await testdriver.find('success message');
if (message.found()) {
console.log('Success message appeared');
}
// Find and extract info
const price = await testdriver.find('product price');
console.log('Price location:', price.coordinates);
console.log('Price text:', price.text);
Element Object
The Element object that TestDriver returns gives these:
Methods
found()- Make a check if TestDriver found the elementclick(action)- Click the elementhover()- Put the cursor on the elementdoubleClick()- Double-click the elementrightClick()- Right-click the elementfind(newDescription)- Find the element again with an optional new description
Properties
coordinates- Element position{x, y, centerX, centerY}x,y- Top-left coordinatescenterX,centerY- Center coordinatestext- Text content (if available)screenshot- Base64 screenshot (if available)confidence- AI confidence scorewidth,height- Element dimensionsboundingBox- Complete bounding box
See Elements Reference for complete details.
JSON Serialization
You can serialize elements safely with JSON.stringify() for logs and for debug. TestDriver removes circular references automatically:
const element = await testdriver.find('login button');
// Safe to stringify - no circular reference errors
console.log(JSON.stringify(element, null, 2));
// Output includes useful debugging info:
// {
// "description": "login button",
// "coordinates": { "x": 100, "y": 200, "centerX": 150, "centerY": 225 },
// "found": true,
// "threshold": 0.01,
// "x": 100,
// "y": 200,
// "cache": {
// "hit": true,
// "strategy": "pixel-diff",
// "createdAt": "2025-12-09T10:30:00Z",
// "diffPercent": 0.0023,
// "imageUrl": "https://..."
// },
// "similarity": 0.98,
// "confidence": 0.95,
// "selector": "button#login",
// "aiResponse": "Found the blue login button..."
// }
Use this for these:
- To debug problems with element detection
- To log the details of the test
- To share element data between processes
- To examine the cache performance
Best Practices
**Be specific in descriptions**More specific descriptions make the accuracy better:
// ✅ Good
await testdriver.find('blue submit button below the email field');
// ❌ Too vague
await testdriver.find('button');
**Always check if found**
Make sure that TestDriver found the elements before you interact with them:
const element = await testdriver.find('login button');
if (!element.found()) {
throw new Error('Login button not found');
}
await element.click();
**Include visual or positional context**
// Include color
await testdriver.find('red error icon');
// Include position
await testdriver.find('search button in the top navigation bar');
// Include nearby text
await testdriver.find('checkbox next to "I agree to terms"');
Confidence Threshold
Set a minimum confidence score for element matches. If the confidence is less than the threshold, find() makes the result "not found":
// Require at least 90% confidence
const element = await testdriver.find('submit button', { confidence: 0.9 });
if (!element.found()) {
// AI found something but wasn't confident enough
throw new Error('Could not confidently locate submit button');
}
await element.click();
Use this for these:
- Critical test steps. An incorrect click can cause more failures.
- To tell the difference between similar elements (for example, many buttons)
- To fail quickly when the UI changed
// Combine with timeout for robust polling with confidence gate
const element = await testdriver.find('success notification', {
confidence: 0.85,
timeout: 15000,
});
The `confidence` value is a float between 0 and 1 (for example, `0.9` = 90%). The AI returns its confidence with each find result. You can also read this from `element.confidence` after a good find.
## Element Type
Use the type option to show which kind of element you look for. This puts your description into a more specific prompt for the AI. It makes the match accuracy better, primarily when a description is short or not clear.
// Find text on the page
const label = await testdriver.find('Sign In', { type: 'text' });
// AI prompt becomes: The text "Sign In"
// Find an image
const logo = await testdriver.find('company logo', { type: 'image' });
// AI prompt becomes: The image "company logo"
// Find a UI element (button, input, checkbox, etc.)
const btn = await testdriver.find('Submit', { type: 'ui' });
// AI prompt becomes: The UI element "Submit"
// No wrapping — same as omitting the option
const el = await testdriver.find('the blue submit button', { type: 'any' });
| Type | Prompt sent to AI |
|---|---|
"text" | The text "..." |
"image" | The image "..." |
"ui" | The UI element "..." |
"any" | Original description (no wrapping) |
By default, find() polls for a maximum of 10 seconds (it tries again each 5 seconds) until it finds the element. You can change this with the timeout option:
// Uses default 10s timeout - polls every 5 seconds
const element = await testdriver.find('login button');
await element.click();
// Custom timeout - wait up to 30 seconds
const element = await testdriver.find('login button', { timeout: 30000 });
await element.click();
// Disable polling - single attempt only
const element = await testdriver.find('login button', { timeout: 0 });
The timeout option:
- Has a default of
10000(10 seconds) - Tries to find the element again each 5 seconds
- Stops when it finds the element or the timeout ends
- Logs the progress during the poll
- Returns the element (make a check with
element.found()if it does not throw an error on a failure) - Set it to
0to disable the poll and try one time
Zoom Mode
Zoom mode is disabled by default. It uses a two-phase method for more precision when it finds elements, primarily in full UIs that have many similar elements.
To enable zoom for a specific find call, give zoom: true:
// Enable zoom for better precision in crowded UIs
const extensionsBtn = await testdriver.find('extensions puzzle icon in Chrome toolbar', { zoom: true });
await extensionsBtn.click();
// Without zoom (default)
const largeButton = await testdriver.find('big hero button');
How Zoom Mode Works
- Phase 1: The AI finds the approximate location of the element.
- Phase 2: TestDriver makes a 30% crop of the screen around that location.
- Phase 3: The AI does the precise location on the cropped image.
- Result: TestDriver changes the coordinates back to the absolute screen position.
This two-phase method gives the AI a higher-resolution view of the target area. It makes the accuracy better when many similar elements are near together.
Enable zoom with `zoom: true` when: - You target small elements in full UIs that have many similar elements - You need more precision for UI elements that are near togetherVerify Mode
Verify mode is disabled by default. When it is enabled, a second AI call makes sure that the coordinates from find() agree with the correct element. This catches incorrect positions.
// Enable verification for critical interactions
const deleteBtn = await testdriver.find('delete account button', { verify: true });
await deleteBtn.click();
How Verify Mode Works
- Phase 1: The AI finds the element and returns coordinates.
- Phase 2: A second AI call looks at the screenshot at those coordinates. It makes sure that the element agrees with the description.
- Result: If the verification fails, TestDriver tries the find again or marks it "not found".
Combining Zoom and Verify
For the maximum accuracy, enable zoom and verify together. Use this for critical interactions. A click on the wrong element can cause more failures:
// Maximum accuracy: zoom for precision + verify to catch hallucinations
const element = await testdriver.find('small cancel icon next to the subscription', {
zoom: true,
verify: true,
});
await element.click();
Cache Options
When a test completes correctly, TestDriver caches the result of each find(). On later runs, TestDriver uses the cached match again. It does not make a new AI call. This finds the same element much more quickly. The cache is in your dashboard. TestDriver shares it between runs. Read the Cache page to see how the match, the thresholds, and the invalidation work.
Control the cache to make the performance better, primarily when you use dynamic variables in prompts.
Custom Cache Key
Use cacheKey to keep the cache clean when prompts have variables:
// ❌ Without cacheKey - creates new cache entry for each email value
const email = 'user@example.com';
await testdriver.find(`input for ${email}`); // Cache miss every time
// ✅ With cacheKey - reuses cache regardless of variable
const email = 'user@example.com';
await testdriver.find(`input for ${email}`, {
cacheKey: 'email-input'
});
// Also useful for dynamic IDs, names, or other changing data
const orderId = generateOrderId();
await testdriver.find(`order ${orderId} status`, {
cacheKey: 'order-status' // Same cache for all orders
});
Cache Threshold
Control how similar a cached result must be before TestDriver uses it again:
// Default: 95% similarity required
await testdriver.find('submit button');
// Strict threshold - 99% similarity required
await testdriver.find('submit button', {
cacheThreshold: 0.01
});
// Disable cache entirely for this call
await testdriver.find('submit button', {
cacheThreshold: -1
});
// Combine cacheKey with threshold
await testdriver.find('submit button', {
cacheKey: 'submit-btn',
cacheThreshold: 0.01
});
By default, TestDriver makes a cache key automatically from the SHA-256 hash of your test file. When you change your test file, the hash changes automatically. This makes the old cache entries not valid.
Manual Polling (Alternative)
If you need custom poll logic:
async function waitForElement(testdriver, description, timeout = 30000) {
const startTime = Date.now();
while (Date.now() - startTime < timeout) {
const element = await testdriver.find(description);
if (element.found()) return element;
await new Promise(r => setTimeout(r, 1000));
}
throw new Error(`Element "${description}" not found after ${timeout}ms`);
}
// Usage
const button = await waitForElement(testdriver, 'submit button', 10000);
await button.click();
Use Cases
```javascript const emailField = await testdriver.find('email input field'); await emailField.click(); await testdriver.type('user@example.com');const passwordField = await testdriver.find('password input');
await passwordField.click();
await testdriver.type('MyP@ssw0rd');
```
```javascript
const submitBtn = await testdriver.find('submit button');
await submitBtn.click();
const cancelLink = await testdriver.find('cancel link');
await cancelLink.click();
const menuIcon = await testdriver.find('hamburger menu icon');
await menuIcon.click();
```
```javascript
// Wait for loading to complete
let content;
for (let i = 0; i < 30; i++) {
content = await testdriver.find('results table');
if (content.found()) break;
await new Promise(r => setTimeout(r, 1000));
}
// Interact with loaded content
const firstRow = await testdriver.find('first row in the results table');
await firstRow.click();
```
```javascript
// Modals and dialogs
const modal = await testdriver.find('confirmation dialog');
if (modal.found()) {
const confirmBtn = await testdriver.find('confirm button in the dialog');
await confirmBtn.click();
}
// Dropdown menus
const dropdown = await testdriver.find('country dropdown');
await dropdown.click();
const option = await testdriver.find('United States option');
await option.click();
```
Complete Example
import { beforeAll, afterAll, describe, it, expect } from 'vitest';
import TestDriver from 'testdriverai';
describe('Element Finding', () => {
let testdriver;
beforeAll(async () => {
client = new TestDriver(process.env.TD_API_KEY);
await testdriver.auth();
await testdriver.connect();
});
afterAll(async () => {
await testdriver.disconnect();
});
it('should find and interact with elements', async () => {
await testdriver.focusApplication('Google Chrome');
// Find login form elements
const usernameField = await testdriver.find('username input field');
expect(usernameField.found()).toBe(true);
await usernameField.click();
await testdriver.type('testuser');
// Find with context
const passwordField = await testdriver.find('password input below username');
await passwordField.click();
await testdriver.type('password123');
// Find button
const submitBtn = await testdriver.find('green submit button');
expect(submitBtn.found()).toBe(true);
console.log('Button location:', submitBtn.centerX, submitBtn.centerY);
await submitBtn.click();
// Wait for success message
let successMsg;
for (let i = 0; i < 10; i++) {
successMsg = await testdriver.find('success notification');
if (successMsg.found()) break;
await new Promise(r => setTimeout(r, 1000));
}
expect(successMsg.found()).toBe(true);
});
});
Related Methods
click()- Click on found elementshover()- Hover over elementsassert()- Verify element states- Elements Reference - Complete Element API
findAll()
Locate all elements matching a description, rather than just one.
Syntax
const elements = await testdriver.findAll(description, options)
Parameters
Natural language description of elements to find Optional cache options (same as `find()`) Cache key for storing element location<ParamField path="cacheThreshold" type="number" default={-1}>
Similarity threshold (0-1) for cache matching. Set to -1 to disable cache.
</ParamField>
Returns
Promise<Element[]> - Array of Element instances
Examples
Basic Usage
// Find all matching elements
const buttons = await testdriver.findAll('button');
console.log(`Found ${buttons.length} buttons`);
// Interact with specific element
if (buttons.length > 0) {
await buttons[0].click(); // Click first button
}
// Iterate over all
for (const button of buttons) {
console.log(`Button at (${button.x}, ${button.y})`);
}
Finding Multiple Items
// Find all list items
const items = await testdriver.findAll('list item');
// Find specific item by index
const thirdItem = items[2];
await thirdItem.click();
// Check all items
for (let i = 0; i < items.length; i++) {
console.log(`Item ${i + 1}: ${items[i].text || 'No text'}`);
}
With Caching
// Cache element locations for faster subsequent runs
const menuItems = await testdriver.findAll('menu item', {
cacheKey: 'main-menu-items'
});
// First run: ~2-3 seconds (AI call)
// Subsequent runs: ~100ms (cache hit)
Empty Results
// Returns empty array if nothing found (doesn't throw error)
const errors = await testdriver.findAll('error message');
if (errors.length === 0) {
console.log('No errors found - test passed!');
} else {
console.log(`Found ${errors.length} errors`);
}
Differences from find()
| Feature | find() | findAll() |
|---|---|---|
| Return type | Single Element | Array of Element[] |
| If nothing found | Throws ElementNotFoundError | Returns empty array [] |
| Chainable | ✅ Yes: await find('button').click() | ❌ No (returns array) |
| Use case | One specific element | Multiple similar elements |
| Cache support | ✅ Yes | ✅ Yes |
Use Cases
```javascript // Find all rows in a table const rows = await testdriver.findAll('table row');// Click every row
for (const row of rows) {
await row.click();
await new Promise(r => setTimeout(r, 500)); // Wait between clicks
}
// Or click specific row
await rows[2].click(); // Click third row
```
```javascript
// Find all checkboxes
const checkboxes = await testdriver.findAll('checkbox');
// Check all boxes
for (const checkbox of checkboxes) {
await checkbox.click();
}
// Or select first unchecked
const unchecked = checkboxes[0];
await unchecked.click();
```
```javascript
// Find all navigation links
const navLinks = await testdriver.findAll('navigation link');
// Validate all are present
expect(navLinks.length).toBeGreaterThan(0);
// Click specific link by text
const homeLink = navLinks.find(link =>
link.text?.toLowerCase().includes('home')
);
if (homeLink) {
await homeLink.click();
}
```
```javascript
// Check if any error messages exist
const errors = await testdriver.findAll('error message');
if (errors.length > 0) {
console.log(`Found ${errors.length} validation errors`);
// Log each error location
errors.forEach((error, i) => {
console.log(`Error ${i + 1} at (${error.x}, ${error.y})`);
});
} else {
console.log('Form validation passed!');
}
```
Complete Example
import { test, expect } from 'vitest';
import { chrome } from 'testdriverai/presets';
test('select multiple items from list', async (context) => {
const { testdriver } = await chrome(context, {
url: 'https://example.com/products'
});
// Find all product cards
const products = await testdriver.findAll('product card');
expect(products.length).toBeGreaterThan(0);
console.log(`Found ${products.length} products`);
// Click first 3 products
const productsToSelect = Math.min(3, products.length);
for (let i = 0; i < productsToSelect; i++) {
await products[i].click();
console.log(`Selected product ${i + 1}`);
await new Promise(r => setTimeout(r, 500)); // Brief pause
}
// Verify selections
const selectedBadges = await testdriver.findAll('selected badge');
expect(selectedBadges.length).toBe(productsToSelect);
});
Best Practices
**Handle empty arrays gracefully**// ✅ Good - check length first
const items = await testdriver.findAll('list item');
if (items.length > 0) {
await items[0].click();
}
// ❌ Bad - may throw error
const items = await testdriver.findAll('list item');
await items[0].click(); // Error if array is empty!
**Use find() for single elements**
// ✅ Use find() when you need exactly one
const submitBtn = await testdriver.find('submit button');
await submitBtn.click();
// ❌ Unnecessary - findAll() returns array
const buttons = await testdriver.findAll('submit button');
await buttons[0].click(); // Extra array handling
**Cache for performance**
// First run - slow (AI call)
const items = await testdriver.findAll('menu item', {
cacheKey: 'menu-items'
});
// Subsequent runs - fast (cache hit)
// ~10-20x faster than without cache
Overview
Move a specific application window to the front. Make it the active window for interactions.
Syntax
await testdriver.focusApplication(name)
Parameters
The application name (for example, `'Google Chrome'`, `'Microsoft Edge'`, `'Notepad'`)Returns
Promise<string> - The result message
Examples
Common Applications
// Focus Chrome browser
await testdriver.focusApplication('Google Chrome');
// Focus Edge browser
await testdriver.focusApplication('Microsoft Edge');
// Focus Notepad
await testdriver.focusApplication('Notepad');
// Focus File Explorer
await testdriver.focusApplication('File Explorer');
// Focus Visual Studio Code
await testdriver.focusApplication('Visual Studio Code');
After Opening Applications
// Open Chrome and focus it
await testdriver.exec('pwsh', `
Start-Process "C:/Program Files/Google/Chrome/Application/chrome.exe" -ArgumentList "https://example.com"
`, 5000);
await new Promise(r => setTimeout(r, 2000)); // Wait for launch
// Focus the Chrome window
await testdriver.focusApplication('Google Chrome');
Best Practices
**Focus before UI interactions**Always focus the target application before interacting with its UI:
await testdriver.focusApplication('Google Chrome');
const button = await testdriver.find('submit button');
await button.click();
**Wait after launching apps**
Give applications time to open before focusing:
await testdriver.exec('pwsh', 'Start-Process notepad', 5000);
await new Promise(r => setTimeout(r, 1000)); // Wait for launch
await testdriver.focusApplication('Notepad');
**Use exact application names**
// ✅ Correct
await testdriver.focusApplication('Google Chrome');
// ❌ May not work
await testdriver.focusApplication('Chrome');
await testdriver.focusApplication('chrome.exe');
**Application must be running**
The application must already be running. focusApplication() won't launch applications, only bring existing windows to the foreground.
Use Cases
```javascript // Test workflow across multiple apps await testdriver.focusApplication('Google Chrome'); const data = await testdriver.extract('the order number');await testdriver.focusApplication('Notepad');
await testdriver.type(data);
await testdriver.pressKeys(['ctrl', 's']);
await testdriver.focusApplication('Google Chrome');
const nextButton = await testdriver.find('next button');
await nextButton.click();
```
```javascript
// Compare behavior in different browsers
await testdriver.focusApplication('Google Chrome');
await testdriver.assert('page loaded correctly in Chrome');
await testdriver.focusApplication('Microsoft Edge');
await testdriver.assert('page loaded correctly in Edge');
```
```javascript
// Launch and focus desktop app
await testdriver.exec('pwsh', 'Start-Process notepad', 5000);
await new Promise(r => setTimeout(r, 1000));
await testdriver.focusApplication('Notepad');
await testdriver.type('Test content');
```
```javascript
// Show desktop first
await testdriver.pressKeys(['winleft', 'd']);
// Click desktop icon
const icon = await testdriver.find('Chrome icon on desktop');
await icon.click();
await new Promise(r => setTimeout(r, 2000));
// Focus the opened window
await testdriver.focusApplication('Google Chrome');
```
Common Application Names
Browsers
'Google Chrome''Microsoft Edge''Mozilla Firefox''Safari'(macOS)
Office Applications
'Microsoft Word''Microsoft Excel''Microsoft PowerPoint''Microsoft Outlook'
Development Tools
'Visual Studio Code''Visual Studio''IntelliJ IDEA''Sublime Text'
System Applications
'Notepad''File Explorer''Command Prompt''Windows PowerShell''Task Manager'
Communication
'Microsoft Teams''Slack''Discord''Zoom'
Complete Example
import { beforeAll, afterAll, describe, it } from 'vitest';
import TestDriver from 'testdriverai';
describe('Multi-Application Workflow', () => {
let testdriver;
beforeAll(async () => {
client = new TestDriver(process.env.TD_API_KEY);
await testdriver.auth();
await testdriver.connect();
});
afterAll(async () => {
await testdriver.disconnect();
});
it('should work across multiple applications', async () => {
// Start in browser
await testdriver.focusApplication('Google Chrome');
// Get data from web page
const orderNumber = await testdriver.extract('the order number');
console.log('Order:', orderNumber);
// Open Notepad
await testdriver.exec('pwsh', 'Start-Process notepad', 5000);
await new Promise(r => setTimeout(r, 1500));
// Focus Notepad and save data
await testdriver.focusApplication('Notepad');
await testdriver.type(`Order Number: ${orderNumber}`);
await testdriver.type('\n');
await testdriver.type(`Date: ${new Date().toISOString()}`);
// Save file
await testdriver.pressKeys(['ctrl', 's']);
await new Promise(r => setTimeout(r, 500));
await testdriver.type('C:\\order-info.txt');
await testdriver.pressKeys(['enter']);
// Return to browser
await testdriver.focusApplication('Google Chrome');
const confirmButton = await testdriver.find('confirm order button');
await confirmButton.click();
await testdriver.assert('order confirmed');
});
it('should switch between browser tabs', async () => {
await testdriver.focusApplication('Google Chrome');
// Open new tab
await testdriver.pressKeys(['ctrl', 't']);
await new Promise(r => setTimeout(r, 500));
// Navigate to URL
await testdriver.pressKeys(['ctrl', 'l']);
await testdriver.type('https://example.com');
await testdriver.pressKeys(['enter']);
await new Promise(r => setTimeout(r, 2000));
// Ensure Chrome is still focused
await testdriver.focusApplication('Google Chrome');
await testdriver.assert('example.com page is loaded');
});
it('should handle dialog boxes', async () => {
await testdriver.focusApplication('Google Chrome');
const deleteButton = await testdriver.find('delete account button');
await deleteButton.click();
await new Promise(r => setTimeout(r, 500));
// Dialog appears - make sure it's focused
await testdriver.focusApplication('Google Chrome');
const confirmBtn = await testdriver.find('confirm deletion button');
await confirmBtn.click();
});
});
Related Methods
exec()- Launch applications with PowerShellpressKeys()- Use Alt+Tab to switch windowsfind()- Locate elements in the focused window
Start with an exploration of your app. Tests begin here: you describe a flow in plain English. Then the TestDriver vision agent clicks, types, and reads the screen to understand it. Then it writes the test for you. There are no selectors, no DOM, and no weak locators. Only describe what you want to test. Let the agent find the other parts.
There are two methods for an exploration. You can chat with your AI assistant through the TestDriver MCP server. Or you can give a coding agent our instructions file and tell it to generate a test.
Both paths need an MCP-capable AI coding assistant. If you don't have one yet, start with **GitHub Copilot** — there's a [free tier](https://github.com/features/copilot/plans), no credit card required. See [Run → Setup](/copilot/running-tests#setup) for the full install and sign-in walkthrough.Instructions for Coding Agents
Start with our quickstart. Then give your coding agent our agent instructions file.
Copy the current version of our agent instructions to provide your coding agent with up-to-date instructions on how to generate TestDriver tests.Then, you can prompt your coding agent to generate tests. Here is an example prompt:
Make me a TestDriver test that does the following steps:
Navigate to practicetestautomation.com
Type username student into Username field
Type password Password123 into Password field
Push Submit button
Verify new page contains expected text 'logged in'
Use explicit commands for production tests. They are at a lower cost, faster, and more reliable.
Start a Conversation
Use the TestDriver MCP server and your AI assistant (GitHub Copilot, Cursor, or Claude Desktop) to make tests. You chat with an AI agent. The agent starts a virtual machine, does actions, and writes the test code for you.
Open the chat of your AI assistant. If your project has no other agents, TestDriver uses the TestDriver agent by default. If not, select testdriver from the agent dropdown in the chat panel.
Describe what you want to test:
Create a test that logs into my app at https://myapp.com
The agent will:
- Start a new session and spawn a Linux virtual machine
- Launch Chrome and navigate to your URL
- Execute actions based on your instructions
- Write the test code to a
.test.mjsfile
Live Preview Panel
When the agent starts a session, a live preview panel opens in your editor. This lets you:
- Watch tests execute in real-time
- Interact with the VM — click, type, and navigate manually
- Debug issues — see exactly what the AI sees
Interactive Workflow
The recommended workflow for exploring and creating tests:
Tell the agent what you want to test:```
Test the checkout flow on https://shop.example.com
```
The agent will start executing. Watch the preview panel and provide guidance if needed:
```
The login modal appeared. Use test@example.com and password123
```
Ask the agent to verify specific conditions:
```
Assert that the order confirmation page is shown with the order number
```
The agent writes test code as it works. Review the generated `.test.mjs` file and make any adjustments.
Example Conversation
Here's a complete example of exploring and creating a login test:
You: Create a test for the login flow at https://console.testdriver.ai
Agent: I'll create a test for the login flow. Starting a session...
[Screenshot: Login page loaded]
I can see the login page. Let me find the email input.
[Screenshot: Email input highlighted]
Now I'll type the email and find the password field...
You: Use test@example.com for the email
Agent: Got it. Typing test@example.com...
[Screenshot: Email entered]
Now entering the password and clicking Sign In...
[Screenshot: Dashboard loaded]
✅ Login successful! I've written the test to tests/login.test.mjs
Generated Test Code
The agent generates standard Vitest test files:
import { describe, expect, it } from "vitest";
import { TestDriver } from "testdriverai/vitest/hooks";
describe("Login Flow", () => {
it("should log in successfully", async (context) => {
const testdriver = TestDriver(context);
await testdriver.provision.chrome({
url: "https://console.testdriver.ai"
});
const emailInput = await testdriver.find("email input field");
await emailInput.click();
await testdriver.type("test@example.com");
const passwordInput = await testdriver.find("password input field");
await passwordInput.click();
await testdriver.type("password123");
const signInButton = await testdriver.find("Sign In button");
await signInButton.click();
const result = await testdriver.assert("dashboard is visible");
expect(result).toBeTruthy();
});
});
Tips for Better Tests
Instead of "click the button", say "click the blue Sign In button in the header". More context helps the AI find the right element. If your app has animations or loading states, tell the agent to wait: ``` Wait for the loading spinner to disappear before continuing ``` Add assertions after each major action to catch regressions early: ``` Assert that the product was added to the cart ``` For long workflows, create the test incrementally and verify each step works before moving on.Next
Once the agent has explored your app, TestDriver caches what it discovers so your tests replay instantly without re-reasoning over the screen every time.The hosted price uses device-seconds: the time that your tests run on our infrastructure.
- Zero Setup. Start tests immediately. You do not need DevOps.
- Free Tier — Start with a limited preview at no cost.
- Pay As You Go — Pay only for the device-seconds that you use.
Hosted Plans
**$0/month**- 1 Concurrent Sandbox
- 60 Minutes Included
- 1 Team User
- Community Support
**$20/month per user**
- 2 Concurrent Sandboxes
- 120 Minutes Included
- Overage: $0.001/second
- Test Recordings
- Community Support
**$600/month**
- 8 Concurrent Sandboxes
- 10,000 Minutes Included
- Overage: $0.001/second
- 5 Team Users
- Test Recordings
- Private Support
- Test Analytics
- CPU, RAM, & Network Profiles
Get Started
Hosted is the default when you follow the Quickstart guide.
<Card
title="Try the Quickstart"
icon="play"
href="/quickstart"
Set your API key and start testing in minutes.
Parallel Testing Limits
Your account has a set number of license slots that determine how many devices can run simultaneously. You can view your available slots in the TestDriver Dashboard.
**When is a slot in use?** A license slot is occupied when a test client is connected. As soon as your device is destroyed the slot becomes available immediately.Avoiding Slot Conflicts
To prevent tests from failing due to exceeding your license slot limit, we recommend two key configurations:
Limit concurrent tests to match your available license slots:```javascript vitest.config.mjs
import { defineConfig } from 'vitest/config';
import TestDriver from 'testdriverai/vitest';
export default defineConfig({
test: {
testTimeout: 900000,
hookTimeout: 900000,
maxConcurrency: 5, // Set to your license slot limit
reporters: ['default', TestDriver()],
setupFiles: ['testdriverai/vitest/setup'],
},
});
```
<Tip>
Check your slot count at [console.testdriver.ai](https://console.testdriver.ai) and set `maxConcurrency` to that number or lower.
</Tip>
Prevent multiple workflow runs from competing for the same slots by using [GitHub's concurrency controls](https://docs.github.com/actions/writing-workflows/choosing-what-your-workflow-does/control-the-concurrency-of-workflows-and-jobs):
```yaml .github/workflows/test.yml
name: Tests
on:
push:
branches: [main]
pull_request:
# Prevent concurrent runs from competing for license slots
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Install dependencies
run: npm install
- name: Run tests
run: vitest run
env:
TD_API_KEY: ${{ secrets.TD_API_KEY }}
```
The `concurrency` block ensures:
- Only one workflow run per branch runs at a time
- New pushes cancel in-progress runs on the same branch
- Different branches/PRs can run in parallel (up to your slot limit)
When to Consider Self-Hosted
Hosted is perfect for getting started and for teams that want zero infrastructure management. However, you might consider Self-Hosted if you:
- Want to escape per-second billing with a flat license fee
- Require greater concurrency than offered in Cloud plans
- Need full control over your infrastructure and privacy
- Want to use your own AI API keys
- Require custom hardware configurations
- Have high test volumes that make self-hosting more economical
<Card title="Explore Self-Hosted" icon="server" href="/self-hosted"
Learn about self-hosting for unlimited test execution at a flat rate.
Overview
Move the mouse cursor onto elements or coordinates without a click. Use this to show tooltips, dropdowns, and hover effects.
Element Hover
Put the cursor on a found element.
Syntax
await element.hover()
Returns
Promise<void>
Examples
// Find and hover
const tooltip = await testdriver.find('info icon');
await tooltip.hover();
// Wait to see tooltip
await new Promise(r => setTimeout(r, 1000));
// Hover over menu to reveal submenu
const menu = await testdriver.find('Products menu');
await menu.hover();
const submenu = await testdriver.find('Laptops submenu item');
await submenu.click();
Coordinate Hover
Put the cursor at screen coordinates.
Syntax
await testdriver.hover(x, y)
Parameters
X coordinate Y coordinateReturns
Promise<void>
Examples
// Hover at coordinates
await testdriver.hover(500, 300);
// Hover and wait
await testdriver.hover(500, 300);
await new Promise(r => setTimeout(r, 1000));
Best Practices
**Prefer element hover over coordinates**// ✅ Preferred
const icon = await testdriver.find('info icon');
await icon.hover();
// ❌ Avoid
await testdriver.hover(500, 300);
**Wait after hovering for dynamic content**
const menuItem = await testdriver.find('Settings menu');
await menuItem.hover();
// Wait for submenu to appear
await new Promise(r => setTimeout(r, 500));
const subItem = await testdriver.find('Profile submenu');
await subItem.click();
**Verify element was found before hovering**
const element = await testdriver.find('tooltip trigger');
if (!element.found()) {
throw new Error('Element not found');
}
await element.hover();
Use Cases
```javascript // Hover to show tooltip const icon = await testdriver.find('help icon'); await icon.hover();await new Promise(r => setTimeout(r, 1000));
// Read tooltip content
const tooltipText = await testdriver.extract('the tooltip text');
console.log('Tooltip:', tooltipText);
```
```javascript
// Hover to reveal dropdown
const menuItem = await testdriver.find('Products menu item');
await menuItem.hover();
// Wait for dropdown animation
await new Promise(r => setTimeout(r, 500));
// Click submenu option
const category = await testdriver.find('Electronics category');
await category.click();
```
```javascript
// Hover over thumbnail to see preview
const thumbnail = await testdriver.find('product thumbnail');
await thumbnail.hover();
await new Promise(r => setTimeout(r, 800));
// Verify preview appears
await testdriver.assert('product preview is displayed');
```
```javascript
// Test hover state styling
const button = await testdriver.find('call to action button');
await button.hover();
await new Promise(r => setTimeout(r, 500));
// Verify hover effect
await testdriver.assert('button background changed to blue');
```
```javascript
// Use hover in drag and drop
const item = await testdriver.find('draggable item');
await item.mouseDown();
// Hover over drop zone
const dropZone = await testdriver.find('drop area');
await dropZone.hover();
await new Promise(r => setTimeout(r, 300));
await dropZone.mouseUp();
```
Complete Example
import { beforeAll, afterAll, describe, it } from 'vitest';
import TestDriver from 'testdriverai';
describe('Hover Interactions', () => {
let testdriver;
beforeAll(async () => {
client = new TestDriver(process.env.TD_API_KEY);
await testdriver.auth();
await testdriver.connect();
});
afterAll(async () => {
await testdriver.disconnect();
});
it('should show tooltip on hover', async () => {
await testdriver.focusApplication('Google Chrome');
// Find and hover over icon
const infoIcon = await testdriver.find('information icon');
await infoIcon.hover();
// Wait for tooltip to appear
await new Promise(r => setTimeout(r, 1000));
// Verify tooltip is visible
await testdriver.assert('tooltip is displayed');
// Extract tooltip text
const tooltipText = await testdriver.extract('the tooltip message');
console.log('Tooltip says:', tooltipText);
});
it('should navigate dropdown menu', async () => {
// Hover over main menu
const menu = await testdriver.find('Navigation menu');
await menu.hover();
// Wait for dropdown
await new Promise(r => setTimeout(r, 500));
// Hover over submenu item
const submenu = await testdriver.find('Account submenu');
await submenu.hover();
await new Promise(r => setTimeout(r, 300));
// Click nested menu item
const settings = await testdriver.find('Settings option');
await settings.click();
await testdriver.assert('settings page is displayed');
});
it('should preview images on hover', async () => {
// Hover over product image
const productImg = await testdriver.find('product 1 thumbnail');
await productImg.hover();
// Wait for preview
await new Promise(r => setTimeout(r, 800));
// Verify preview appeared
await testdriver.assert('large product preview is shown');
// Move away
const otherElement = await testdriver.find('page heading');
await otherElement.hover();
// Verify preview disappeared
await new Promise(r => setTimeout(r, 500));
await testdriver.assert('product preview is hidden');
});
});
Related Methods
find()- Locate elements to hoverclick()- Click after hoveringmouseDown()- Start drag operations
Overview
The TestDriver element finding system uses AI. It finds elements on the screen with natural language descriptions. The find() method returns an Element object. You can interact with the object.
Finding Elements
find()
Find an element on the screen with a natural language description.
const element = await testdriver.find(description)
Parameters:
description(string) - A natural language description of the element to find
Returns: Promise<Element> - The Element instance that TestDriver found
Example:
// Find a button
const submitButton = await testdriver.find('the submit button');
// Find an input field with context
const emailField = await testdriver.find('email input field in the login form');
// Find an element by visual characteristics
const redButton = await testdriver.find('red button in the top right corner');
Be specific in your descriptions. Include visual details, location context, or nearby text to make the accuracy better.
Element Class
The Element class represents a located (or to-be-located) UI element. It provides methods for interaction and properties for element information. For interaction methods like click() and hover(), see Interacting With Your App.
Methods
found()
Check if the element was successfully located.
element.found()
Returns: boolean - True if element coordinates were found
Example:
const element = await testdriver.find('login button');
if (element.found()) {
await element.click();
} else {
console.log('Element not found');
}
find()
Re-locate the element, optionally with a new description.
await element.find(newDescription)
Parameters:
newDescription(string, optional) - New description to search for
Returns: Promise<Element> - This element instance
Example:
// Re-locate if the UI changed
const element = await testdriver.find('submit button');
// ... page updates ...
await element.find(); // Re-locate with same description
// Or update the description
await element.find('blue submit button'); // Now looking for blue button
Properties
Element properties provide additional information about located elements. Properties are available after a successful find() call.
coordinates
Get the element's coordinates object containing all position information.
const coords = element.getCoordinates()
// or access directly
element.coordinates
Returns: Object | null - Coordinate object with { x, y, centerX, centerY }
Example:
const button = await testdriver.find('submit button');
const coords = button.coordinates;
if (coords) {
console.log(`Top-left: (${coords.x}, ${coords.y})`);
console.log(`Center: (${coords.centerX}, ${coords.centerY})`);
}
x, y, centerX, centerY
Direct access to coordinate values. Always available after successful find().
element.x // Top-left X coordinate (number)
element.y // Top-left Y coordinate (number)
element.centerX // Center X coordinate (number)
element.centerY // Center Y coordinate (number)
Example:
const button = await testdriver.find('submit button');
console.log(`Button at: (${button.x}, ${button.y})`);
console.log(`Button center: (${button.centerX}, ${button.centerY})`);
// Use for custom mouse operations
await testdriver.click(button.centerX, button.centerY);
width, height
Element dimensions in pixels. Available when AI detects element bounds.
element.width // Width in pixels (number | null)
element.height // Height in pixels (number | null)
Example:
const button = await testdriver.find('submit button');
if (button.width && button.height) {
console.log(`Button size: ${button.width}x${button.height}px`);
// Check if button is large enough
if (button.width < 50) {
console.warn('Button might be too small');
}
}
boundingBox
Complete bounding box information including position and dimensions.
element.boundingBox
Returns: Object | null - Bounding box with all dimension data
{
x: number, // Top-left X
y: number, // Top-left Y
width: number, // Width in pixels
height: number // Height in pixels
}
Example:
const element = await testdriver.find('dialog box');
if (element.boundingBox) {
const { x, y, width, height } = element.boundingBox;
console.log(`Dialog: ${width}x${height} at (${x}, ${y})`);
// Calculate if element is in viewport
const rightEdge = x + width;
const bottomEdge = y + height;
console.log(`Element extends to (${rightEdge}, ${bottomEdge})`);
}
screenshot
Base64-encoded PNG screenshot of the screen when element was found. Only available in DEBUG mode or when an error occurs.
element.screenshot
Returns: string | null - Base64-encoded PNG image
Example:
const element = await testdriver.find('error message');
if (element.screenshot) {
// Save screenshot to file
const fs = require('fs');
const base64Data = element.screenshot.replace(/^data:image\/\w+;base64,/, '');
fs.writeFileSync('element-screenshot.png', Buffer.from(base64Data, 'base64'));
console.log('Screenshot saved');
}
Screenshots can be large. They're automatically excluded from error messages to prevent memory issues.
text
Text content extracted from the element by AI (if available).
element.text
Returns: string | null - Element's text content
Example:
const message = await testdriver.find('notification message');
if (message.text) {
console.log('Message says:', message.text);
// Use text content in assertions
if (message.text.includes('success')) {
console.log('Success message detected');
}
}
// Another example - extracting button label
const button = await testdriver.find('blue button');
console.log('Button text:', button.text); // "Submit"
label
Accessible label or name of the element (if available). Useful for verifying accessibility.
element.label
Returns: string | null - Accessible label
Example:
const input = await testdriver.find('first input field');
if (input.label) {
console.log('Input label:', input.label); // "Email Address"
}
confidence
AI confidence score for the element match (0-1, where 1 is perfect confidence).
element.confidence
Returns: number | null - Confidence score between 0 and 1
Example:
const element = await testdriver.find('submit button');
if (element.confidence !== null) {
const percentage = (element.confidence * 100).toFixed(1);
console.log(`Match confidence: ${percentage}%`);
if (element.confidence < 0.8) {
console.warn('⚠️ Low confidence match - element might not be correct');
} else if (element.confidence > 0.95) {
console.log('✅ High confidence match');
}
}
Confidence scores below 0.8 may indicate the element description was ambiguous or the wrong element was found.
Property Availability
| Property | When Available |
|---|---|
x, y, centerX, centerY | ✅ Always after successful find() |
coordinates | ✅ Always after successful find() |
width, height | ⚠️ When AI detects element bounds |
boundingBox | ⚠️ When AI detects element bounds |
text | ⚠️ When AI extracts text content |
label | ⚠️ When element has accessible label |
confidence | ✅ Always after AI element finding |
screenshot | ⚠️ Only in DEBUG mode or on errors |
JSON Serialization
Element objects can be safely serialized using JSON.stringify() for logging, debugging, and data storage. Circular references are automatically removed:
const element = await testdriver.find('login button');
// Safe to stringify - no circular reference errors!
console.log(JSON.stringify(element, null, 2));
Serialized output includes:
{
"description": "login button",
"coordinates": { "x": 100, "y": 200, "centerX": 150, "centerY": 225 },
"found": true,
"threshold": 0.01,
"x": 100,
"y": 200,
"cache": {
"hit": true,
"strategy": "pixel-diff",
"createdAt": "2025-12-09T10:30:00.000Z",
"diffPercent": 0.0023,
"imageUrl": "https://cache.testdriver.ai/..."
},
"similarity": 0.98,
"confidence": 0.95,
"selector": "button#login",
"aiResponse": "Found the blue login button in the center of the form..."
}
Serialized properties:
| Property | Type | Description |
|---|---|---|
description | string | Element search description |
coordinates | object | Full coordinate object {x, y, centerX, centerY} |
found | boolean | Whether element was located |
threshold | number | Cache threshold used for this find |
x, y | number | Top-left coordinates |
cache.hit | boolean | Whether cache was used |
cache.strategy | string | Cache strategy (e.g., "pixel-diff") |
cache.createdAt | string | ISO timestamp when cache was created |
cache.diffPercent | number | Pixel difference from cached image |
cache.imageUrl | string | URL to cached screenshot |
similarity | number | Similarity score (0-1) |
confidence | number | AI confidence score (0-1) |
selector | string | CSS/XPath selector if available |
aiResponse | string | AI's explanation of what it found |
Use cases:
// Debugging element detection
const element = await testdriver.find('submit button');
if (!element.found()) {
console.error('Element not found:', JSON.stringify(element, null, 2));
}
// Logging cache performance
const data = JSON.parse(JSON.stringify(element));
if (data.cache.hit) {
console.log(`Cache hit! Diff: ${(data.cache.diffPercent * 100).toFixed(2)}%`);
}
// Sharing element data across processes
const elementData = JSON.stringify(element);
// Send to another process, log to file, etc.
Use JSON serialization when you need to log element data or when debugging why an element wasn't found. The serialized output excludes large binary data (screenshots) and circular references.
Best Practices
Include visual details, position context, and nearby text:```javascript
// ❌ Too vague
await testdriver.find('button');
// ✅ Specific
await testdriver.find('blue submit button below the email field');
```
Always verify elements were located before interacting:
```javascript
const element = await testdriver.find('submit button');
if (!element.found()) {
throw new Error('Submit button not found');
}
await element.click();
```
After a test runs, make sure that the app did the correct thing. Validation has two parts: to find the elements that you want to check, and to make assertions about the state of your app. TestDriver uses the AI as a judge. It returns a boolean and the reason. This shows if your app is in the correct state.
Locating Elements
Locating Single Elements
Use natural language to describe elements. A description must be specific enough to find the element. But it must not be so specific that it breaks with small UI changes. For example:
await testdriver.find('email input field');
await testdriver.find('first product card in the grid');
await testdriver.find('dropdown menu labeled "Country"');
TestDriver caches the found elements for better performance on later calls. Read more about element caching here.
Debugging Found Elements
After TestDriver finds an element, you can look at its properties for debug:
const button = await testdriver.find('submit button');
console.log(button);
This outputs all element properties:
{
description: 'submit button',
found: true,
x: 150,
y: 300,
coordinates: { x: 150, y: 300, centerX: 200, centerY: 320 },
threshold: 0.8,
confidence: 0.95,
similarity: 0.92,
selector: 'button[type="submit"]',
cache: {
hit: true,
strategy: 'pixel-diff',
createdAt: '2025-01-15T10:30:00Z',
diffPercent: 0.02,
imageUrl: 'https://...'
}
}
Working with Multiple Elements
Find and interact with multiple elements:
// Find all matching elements
const products = await testdriver.findAll('product card');
console.log(`Found ${products.length} products`);
// Interact with each
for (const product of products) {
const title = await product.find('title text');
console.log('Product:', title.text);
await product.find('add to cart button').click();
}
// Or find specific element
const firstProduct = products[0];
await firstProduct.click();
Making Assertions
Use AI-powered assertions to verify application state. TestDriver acts as a judge: it evaluates your natural-language statement against the current state of the app and returns a boolean plus reasoning explaining the verdict.
// Verify visibility
await testdriver.assert('login page is displayed');
await testdriver.assert('submit button is visible');
await testdriver.assert('loading spinner is not visible');
// Verify content
await testdriver.assert('page title is "Welcome"');
await testdriver.assert('success message says "Account created"');
await testdriver.assert('error message contains "Invalid email"');
// Verify state
await testdriver.assert('checkbox is checked');
await testdriver.assert('dropdown shows "United States"');
await testdriver.assert('button is disabled');
// Verify visual appearance
await testdriver.assert('submit button is blue');
await testdriver.assert('form has red border');
Assertions are not cached and always re-evaluated to ensure accuracy.
Next
Drive your app forward by performing actions on the elements you've located and validated.TestDriver MCP Workflow
Build automated tests by directly controlling a sandbox through MCP tools. Every action returns a screenshot AND the generated code to add to your test file.
When to Use This Skill
Use this skill when:
- You have access to TestDriver MCP tools (
session_start,find,click, etc.) - User asks to "write a test", "automate this workflow", "check if X works"
- You need to build tests iteratively with visual feedback
Overview
Use MCP tools to:
- Control the sandbox directly - Click, type, scroll in real-time
- See visual feedback - Every action shows a screenshot with overlays
- Get generated code - Each successful action returns the code to add to your test file
- Build tests incrementally - Append code to test files as you go
Quick Start
1. Start a Session
session_start({ type: "chrome", url: "https://your-app.com" })
This provisions a sandbox with Chrome and navigates to your URL. You'll see a screenshot and the provision code:
Add to test file:
await testdriver.provision.chrome({ url: "https://your-app.com" });
For local development (pointing to a custom API endpoint):
session_start({
type: "chrome",
url: "https://your-app.com",
apiRoot: "https://your-ngrok-url.ngrok.io"
})
For self-hosted AWS instances (your own Windows EC2):
session_start({
type: "chrome",
url: "https://your-app.com",
os: "windows",
ip: "1.2.3.4" // IP from your AWS instance
})
See AWS Setup Guide to deploy your own infrastructure.
2. Interact with the App
Find elements and interact with them. Each action returns a screenshot AND generated code:
find_and_click({ description: "Sign In button" })
→ Returns: screenshot with element highlighted
→ Add to test file: await testdriver.find("Sign In button").click();
type({ text: "user@example.com" })
→ Returns: screenshot showing typed text
→ Add to test file: await testdriver.type("user@example.com");
3. Check If Actions Succeeded
After performing actions, use check to verify they worked:
check({ task: "Was the text entered into the field?" })
→ Returns: AI analysis of whether the task completed, with screenshot
check({ task: "Did the button click navigate to a new page?" })
→ Returns: AI compares previous screenshot to current state
4. Make Assertions (for Test Files)
Use assert for boolean pass/fail conditions that get recorded in test files:
assert({ assertion: "the login form is visible" })
→ Returns: pass/fail with screenshot
→ Add to test file:
const assertResult = await testdriver.assert("the login form is visible");
expect(assertResult).toBeTruthy();
5. Write the Test File
As you perform actions, append the generated code to your test file:
/**
* Login Flow test
*/
import { describe, expect, it } from "vitest";
import { TestDriver } from "testdriverai/lib/vitest/hooks.mjs";
describe("Login Flow", () => {
it("should complete login", async (context) => {
const testdriver = TestDriver(context);
// Append generated code here as you go:
await testdriver.provision.chrome({ url: "https://app.example.com" });
await testdriver.find("email input field").click();
await testdriver.type("user@example.com");
// ... more code as you perform actions
});
});
6. Verify the Test
Run the test from scratch to validate it works:
verify({ testFile: "tests/login.test.mjs" })
Tools Reference
Session Management
| Tool | Description |
|---|---|
session_start | Start sandbox with browser/app, returns screenshot + provision code |
session_status | Check session health and time remaining |
session_extend | Add more time before session expires |
Element Interaction
Each tool returns a screenshot AND the generated code to add to your test file.
| Tool | Description |
|---|---|
find | Locate element by description, returns ref for later use |
click | Click on element ref |
find_and_click | Find and click in one action |
type | Type text into focused field |
press_keys | Press keyboard shortcuts (e.g., ["ctrl", "a"]) |
scroll | Scroll page (up/down/left/right) |
Verification & Display
| Tool | Description |
|---|---|
check | For AI to understand screen state. Analyzes current screen and tells you (the AI) whether a task/condition is met. Use this after actions to verify they worked. |
assert | AI-powered boolean assertion for test files (pass/fail for CI). Returns generated code. |
screenshot | For showing the user the screen. Captures and displays a screenshot. Does NOT return analysis to you (the AI). |
exec | Execute JavaScript, shell, or PowerShell in sandbox. Returns generated code. |
Test Validation
| Tool | Description |
|---|---|
verify | Run test file from scratch to validate it works |
Visual Feedback
Every tool returns a screenshot showing:
- Element highlights - Blue box around found elements
- Click markers - Red dot with ripple effect at click location
- Scroll indicators - Arrow showing scroll direction
- Action status - Success/failure with duration
- Session info - Time remaining before expiry
Workflow Best Practices
1. Work Incrementally
Don't try to build the entire test at once:
# Step 1: Get to login page
session_start({ url: "https://app.com" })
→ Add to test: await testdriver.provision.chrome({ url: "https://app.com" });
# Step 2: Verify you're on the right page
check({ task: "Is this the login page?" })
# Step 3: Fill in email
find_and_click({ description: "email input field" })
→ Add to test: await testdriver.find("email input field").click();
type({ text: "user@example.com" })
→ Add to test: await testdriver.type("user@example.com");
# Step 4: Check if email was entered
check({ task: "Was the email entered correctly?" })
# Step 5: Continue with password...
2. Use Check After Actions
After each action, use check to verify it worked:
find_and_click({ description: "Submit button" })
check({ task: "Was the form submitted?" })
The check tool compares the previous screenshot (from before your action) with the current state, giving you AI analysis of what changed and whether the action succeeded.
3. Understanding Screen State
For AI understanding: Use check to analyze the screen:
check({ task: "Did the form submit successfully?" })
→ Returns AI analysis you can read and understand
For user visibility: Use screenshot to show the user:
screenshot()
→ Displays to user, no analysis returned to you
Action tools (find, click, find_and_click) return screenshots automatically, which the user can see. But if you need to understand the state, use check.
4. Handle Timing Issues
If elements take time to appear, use find with timeout:
find({ description: "Loading complete indicator", timeout: 30000 })
5. Check Session Time
Sessions expire after 5 minutes by default. Use session_status to check time remaining and session_extend to add more time:
session_status()
→ "Time remaining: 45s"
session_extend({ additionalMs: 60000 })
→ "New expiry: 105s"
6. Write Code as You Go
After each successful action, append the generated code to your test file. This ensures you don't lose progress and makes the test easier to debug.
Error Recovery
Element Not Found
If find fails:
- Check the screenshot to see what's actually on screen
- Adjust the element description to be more specific
- Wait for page load with timeout:
find({ description: "...", timeout: 10000 }) - Scroll to find off-screen elements:
scroll({ direction: "down" })
Session Expired
If the session expires:
session_startagain with the same URL- Run the test with
verifyto get back to last state - Continue from where you left off
Test Verification Fails
If verify fails:
- Check the error message and screenshot
- Review the test file to see the generated code
- Start a new session and manually test the failing step
- Adjust element descriptions or add waits as needed
Dependencies
When creating a new test project, use these exact dependencies:
package.json:
{
"type": "module",
"devDependencies": {
"testdriverai": "canary",
"vitest": "^4.0.0"
},
"scripts": {
"test": "vitest"
}
}
Important: The package is testdriverai (NOT @testdriverai/sdk). Always install from the canary tag.
Test File Format
Create test files using this standard format. Append generated code inside the test function:
/**
* Login Flow test
*/
import { describe, expect, it } from "vitest";
import { TestDriver } from "testdriverai/lib/vitest/hooks.mjs";
describe("Login Flow", () => {
it("should complete Login Flow", async (context) => {
const testdriver = TestDriver(context);
// Append generated code from each action here:
await testdriver.provision.chrome({ url: "https://app.example.com" });
await testdriver.find("email input field").click();
await testdriver.type("user@example.com");
await testdriver.find("password field").click();
await testdriver.type("secret123");
await testdriver.find("Sign In button").click();
const assertResult = await testdriver.assert("dashboard is visible");
expect(assertResult).toBeTruthy();
});
});
Self-Hosted AWS Instances
You can use your own AWS-hosted Windows instances instead of TestDriver cloud. This gives you:
- Flat license fee - No device-second metering
- Your own AI keys - Use your OpenAI/Anthropic keys
- Custom configuration - Install custom software, configure networking
- Full debugging access - RDP into instances
Quick Setup
-
Deploy AWS infrastructure using CloudFormation
-
Spawn an instance:
AWS_REGION=us-east-2 \ AMI_ID=ami-0504bf50fad62f312 \ AWS_LAUNCH_TEMPLATE_ID=lt-xxx \ bash setup/aws/spawn-runner.shOutput:
PUBLIC_IP=1.2.3.4 -
Connect via session_start:
session_start({ type: "chrome", url: "https://example.com", os: "windows", ip: "1.2.3.4" }) -
Terminate when done:
aws ec2 terminate-instances --instance-ids i-xxx --region us-east-2
Environment Variable
You can also set TD_IP environment variable in your MCP config instead of passing ip to each session:
{
"mcpServers": {
"testdriver": {
"env": {
"TD_API_KEY": "your-key",
"TD_IP": "1.2.3.4"
}
}
}
}
Tips
-
Every action returns generated code - Look for "Add to test file:" in responses and append that code
-
Use
checkto understand the screen - This is how you (the AI) see and analyze the current state -
Use
screenshotto show the user - This displays the screen to the user, but does NOT return analysis to you -
Use
checkafter every action - Verify your actions succeeded before moving on -
Be specific with element descriptions - "the blue Sign In button in the header" is better than "button"
-
Use
checkfor verification,assertfor test files -checkgives detailed AI analysis,assertgives boolean pass/fail for CI -
Write code incrementally - Append generated code to your test file after each successful action
-
Extend session proactively - If you have complex workflows, extend before you run out of time
Overview
Overview
The mouseDown() method pushes the mouse button at the location of an element. It does not release the button. Use this for drag operations, custom gestures, or when you need precise control of the mouse events. You can call it on an Element instance. Or you can use it with a selector.
Syntax
// Mouse down on an element
await element.mouseDown();
// Mouse down using a selector
await ai.mouseDown('selector');
Parameters
When called on an Element, no parameters are required.
When called directly on the AI client:
| Parameter | Type | Description |
|---|---|---|
selector | string | The selector describing the element where the mouse button should be pressed |
Returns
It returns a Promise<void>. The promise resolves when TestDriver pushes the mouse button.
Examples
Basic Drag Operation
// Start dragging an item
const dragItem = await ai.find('file to drag');
await dragItem.mouseDown();
// Move to drop target
const dropTarget = await ai.find('folder to drop into');
await dropTarget.hover();
// Release
await ai.mouseUp();
Drag and Drop with Direct Selectors
await ai.mouseDown('draggable card');
await ai.hover('drop zone');
await ai.mouseUp();
Selecting Multiple Items
import { test } from 'vitest';
import { chrome } from '@testdriver/sdk';
test('selects multiple items with click and drag', async () => {
const { ai } = await chrome('https://app.example.com');
// Start selection at first item
await ai.mouseDown('first item in grid');
// Drag to last item
await ai.hover('last item in grid');
// Release to complete selection
await ai.mouseUp();
// Verify multiple items selected
const selectedItems = await ai.find('selected items count');
expect(selectedItems.text).toContain('5 items selected');
});
Custom Drawing Application
test('draws on canvas', async () => {
const { ai } = await chrome('https://drawing-app.example.com');
// Start drawing
await ai.mouseDown('canvas at top-left corner');
// Draw a line by moving mouse
await ai.hover('canvas at center');
await ai.hover('canvas at bottom-right corner');
// Stop drawing
await ai.mouseUp();
// Verify something was drawn
const canvas = await ai.exec('document.querySelector("canvas").toDataURL()');
expect(canvas).toBeTruthy();
});
Long Press Gesture
test('triggers long press menu', async () => {
const { ai } = await chrome('https://mobile-app.example.com');
// Press and hold
await ai.mouseDown('message in chat');
// Wait for long-press menu
await new Promise(resolve => setTimeout(resolve, 500));
// Verify menu appeared before releasing
const menu = await ai.find('message options menu');
expect(menu).toBeTruthy();
// Release
await ai.mouseUp();
});
Resizing UI Elements
test('resizes panel', async () => {
const { ai } = await vscode();
// Grab resize handle
await ai.mouseDown('sidebar resize handle');
// Drag to new position
await ai.hover('position 300 pixels from left edge');
// Release
await ai.mouseUp();
// Verify new size
const sidebar = await ai.find('sidebar');
expect(sidebar.width).toBeGreaterThan(250);
});
Important Notes
- Always pair
mouseDown()withmouseUp()to complete the gesture - The mouse button remains pressed until
mouseUp()is called - Use
hover()to move the mouse while the button is pressed - For simple drag operations, consider using
ai()with a natural language description like"drag file to folder"
Related Methods
mouseUp()- Release the mouse buttonhover()- Move mouse to elementclick()- Full click (mouseDown + mouseUp)doubleClick()- Double-click on elementrightClick()- Right-click for context menu
Overview
The mouseUp() method releases the mouse button. It completes a drag operation or a custom mouse gesture that you started with mouseDown(). Call it without parameters to release the button at the present mouse position.
Syntax
// Release mouse button at current position
await ai.mouseUp();
Parameters
None. The mouse button is released at the current cursor position.
Returns
It returns a Promise<void>. The promise resolves when TestDriver releases the mouse button.
Examples
Complete Drag and Drop
// Start dragging
await ai.mouseDown('file to drag');
// Move to drop location
await ai.hover('target folder');
// Complete the drop
await ai.mouseUp();
Selecting Multiple Files
import { test } from 'vitest';
import { vscode } from '@testdriver/sdk';
test('selects range of files', async () => {
const { ai } = await vscode();
// Click first file
await ai.click('first-file.js in explorer');
// Hold shift and click last file
await ai.pressKeys('Shift');
await ai.mouseDown('last-file.js in explorer');
await ai.mouseUp();
// Verify multiple files selected
const selectedCount = await ai.find('status bar showing file count');
expect(selectedCount.text).toContain('5 files');
});
Drawing Application
test('draws a shape', async () => {
const { ai } = await chrome('https://drawing-app.example.com');
// Select pencil tool
await ai.click('pencil tool');
// Draw a line
await ai.mouseDown('canvas near top-left');
await ai.hover('canvas center');
await ai.hover('canvas bottom-right');
await ai.mouseUp();
// Verify drawing exists
const strokes = await ai.exec('canvas.getContext("2d").getImageData(0,0,100,100)');
expect(strokes).toBeTruthy();
});
Resizing Window Panels
test('resizes editor panel', async () => {
const { ai } = await vscode();
// Grab the divider
await ai.mouseDown('panel resize divider');
// Drag to new position
await ai.hover('position 400 pixels from left');
// Release to complete resize
await ai.mouseUp();
// Verify new panel size
const panel = await ai.find('editor panel');
expect(panel.width).toBeGreaterThan(350);
});
Drag to Reorder List Items
import { test } from 'vitest';
import { chrome } from '@testdriver/sdk';
test('reorders tasks in list', async () => {
const { ai } = await chrome('https://todo-app.example.com');
// Start dragging first task
await ai.mouseDown('drag handle on first task');
// Move down to third position
await ai.hover('third task position');
// Drop the task
await ai.mouseUp();
// Verify new order
const thirdTask = await ai.find('third task in list');
expect(thirdTask.text).toContain('Original first task');
});
Text Selection with Mouse
test('selects text with mouse drag', async () => {
const { ai } = await chrome('https://document.example.com');
// Start selection at beginning of word
await ai.mouseDown('start of "TestDriver" word');
// Drag to end of word
await ai.hover('end of "TestDriver" word');
// Complete selection
await ai.mouseUp();
// Verify selection
const selection = await ai.exec('window.getSelection().toString()');
expect(selection).toBe('TestDriver');
});
Important Notes
mouseUp()must be preceded bymouseDown()to have an effect- Releases the button at the current cursor position
- Completes any drag or selection operation that was in progress
- For simple clicks, use
click()instead of mouseDown/mouseUp pair
Related Methods
mouseDown()- Press mouse button without releasinghover()- Move mouse to elementclick()- Complete click (mouseDown + mouseUp)doubleClick()- Double-click on elementrightClick()- Right-click for context menu
Overview
Parse the screen with OmniParser v2 to find all visible UI elements. TestDriver returns structured data. This includes element types, text content, interactivity levels, and bounding box coordinates.
This method examines all of the screen and returns each element that it finds. Use it for these:
- To understand the full UI layout of a screen
- To find all clickable or interactive elements
- To build custom element logic
- To debug which elements TestDriver can find
- To do an accessibility audit
Syntax
const result = await testdriver.parse()
Parameters
None.
Returns
Promise<ParseResult> - Object containing detected UI elements
ParseResult
| Property | Type | Description |
|---|---|---|
elements | ParsedElement[] | Array of detected UI elements |
annotatedImageUrl | string | URL of the annotated screenshot with bounding boxes |
imageWidth | number | Width of the analyzed screenshot |
imageHeight | number | Height of the analyzed screenshot |
ParsedElement
| Property | Type | Description |
|---|---|---|
index | number | Element index |
type | string | Element type (e.g. "text", "icon", "button") |
content | string | Text content or description of the element |
interactivity | string | Interactivity level (e.g. "clickable", "non-interactive") |
bbox | object | Bounding box in pixel coordinates {x0, y0, x1, y1} |
boundingBox | object | Bounding box as {left, top, width, height} |
Examples
Get All Elements on Screen
const result = await testdriver.parse();
console.log(`Found ${result.elements.length} elements`);
result.elements.forEach((el, i) => {
console.log(`${i + 1}. [${el.type}] "${el.content}" (${el.interactivity})`);
});
Find Clickable Elements
const result = await testdriver.parse();
const clickable = result.elements.filter(e => e.interactivity === 'clickable');
console.log(`Found ${clickable.length} clickable elements`);
clickable.forEach(el => {
console.log(`- "${el.content}" at (${el.bbox.x0}, ${el.bbox.y0})`);
});
Find and Click an Element by Content
const result = await testdriver.parse();
// Find a "Submit" button
const submitBtn = result.elements.find(e =>
e.content.toLowerCase().includes('submit') && e.interactivity === 'clickable'
);
if (submitBtn) {
// Calculate center of the bounding box
const x = Math.round((submitBtn.bbox.x0 + submitBtn.bbox.x1) / 2);
const y = Math.round((submitBtn.bbox.y0 + submitBtn.bbox.y1) / 2);
await testdriver.click({ x, y });
}
Filter by Element Type
const result = await testdriver.parse();
// Get all text elements
const textElements = result.elements.filter(e => e.type === 'text');
textElements.forEach(e => console.log(`Text: "${e.content}"`));
// Get all icons
const icons = result.elements.filter(e => e.type === 'icon');
console.log(`Found ${icons.length} icons`);
// Get all buttons
const buttons = result.elements.filter(e => e.type === 'button');
console.log(`Found ${buttons.length} buttons`);
Build Custom Assertions
import { describe, expect, it } from "vitest";
import { TestDriver } from "testdriverai/vitest/hooks";
describe("Login Page", () => {
it("should have expected form elements", async (context) => {
const testdriver = TestDriver(context);
await testdriver.provision.chrome({
url: 'https://myapp.com/login',
});
const result = await testdriver.parse();
// Assert expected elements exist
const textContent = result.elements.map(e => e.content.toLowerCase());
expect(textContent).toContain('email');
expect(textContent).toContain('password');
// Assert there are clickable elements
const clickable = result.elements.filter(e => e.interactivity === 'clickable');
expect(clickable.length).toBeGreaterThan(0);
});
});
Use Bounding Box Coordinates
const result = await testdriver.parse();
result.elements.forEach(el => {
// Pixel coordinates
console.log(`Element "${el.content}":`);
console.log(` bbox: (${el.bbox.x0}, ${el.bbox.y0}) to (${el.bbox.x1}, ${el.bbox.y1})`);
console.log(` size: ${el.boundingBox.width}x${el.boundingBox.height}`);
console.log(` position: left=${el.boundingBox.left}, top=${el.boundingBox.top}`);
});
View Annotated Screenshot
const result = await testdriver.parse();
// The annotated image shows all detected elements with bounding boxes
console.log('Annotated screenshot:', result.annotatedImageUrl);
console.log(`Image dimensions: ${result.imageWidth}x${result.imageHeight}`);
How It Works
- TestDriver captures a screenshot of the current screen
- The image is sent to the TestDriver API
- OmniParser v2 analyzes the image to detect all UI elements
- Each element is classified by type (text, icon, button, etc.) and interactivity
- Bounding box coordinates are returned in pixel coordinates matching the screen resolution
Best Practices
For locating and interacting with a specific element, prefer `find()` which uses AI vision. Use `parse()` when you need a complete inventory of all elements on screen.```javascript
// Prefer this for clicking a specific element
await testdriver.find("Submit button").click();
// Use parse() for full UI analysis
const result = await testdriver.parse();
const allButtons = result.elements.filter(e => e.type === 'button');
```
Use the `interactivity` field to distinguish between clickable and non-interactive elements.
```javascript
const result = await testdriver.parse();
const interactive = result.elements.filter(e => e.interactivity === 'clickable');
const static_ = result.elements.filter(e => e.interactivity === 'non-interactive');
```
If elements aren't being detected, the page may not be fully loaded. Add a wait first.
```javascript
// Wait for page to stabilize
await testdriver.wait(2000);
// Then parse
const result = await testdriver.parse();
```
The `annotatedImageUrl` provides a visual overlay showing all detected elements with their bounding boxes — great for debugging.
```javascript
const result = await testdriver.parse();
console.log('View annotated screenshot:', result.annotatedImageUrl);
```
Related
- find() - AI-powered element location
- assert() - Make AI-powered assertions about screen state
- screenshot() - Capture screenshots
- Elements Reference - Complete Element API
Real apps move, load, and change. Adapt your tests to these conditions.
You generated and learned your tests. Then you got them to run. Now the challenge is the real world: buttons show after a spinner, pages navigate, animations play, and content comes in over the network. To keep the tests reliable, do the correct actions and control the timing. Then your tests adapt to the true behavior of the UI. They do not break.
Performing Actions
TestDriver gives you many actions. You can click, type, hover, and scroll. For a full list, read the API Reference.
// Clicking
await testdriver.find('submit button').click();
await testdriver.find('file item').doubleClick();
await testdriver.find('text area').rightClick();
// Typing
await testdriver.find('email input').type('user@example.com');
await testdriver.find('password input').type('secret', { secret: true });
// Keyboard shortcuts
await testdriver.pressKeys(['enter']);
await testdriver.pressKeys(['ctrl', 'c']);
// Hovering
await testdriver.find('dropdown menu').hover();
// Scrolling
await testdriver.scroll('down', 500);
// Waiting
await testdriver.wait(2000); // Wait 2 seconds for animation/state change
// Extracting information from screen
const price = await testdriver.extract('the total price');
const orderNumber = await testdriver.extract('the order confirmation number');
Chaining Actions
TestDriver lets you chain methods for cleaner code:
// Chain find() with actions
const button = await testdriver.find('submit button').click();
Or keep the element reference for later use:
const button = await testdriver.find('submit button');
await button.click();
Waiting for Dynamic Content
By default, find() polls automatically for a maximum of 10 seconds. It tries again each 5 seconds until it finds the element. Thus TestDriver finds most elements that show after short async operations. You do not need more configuration.
For longer operations, make the timeout larger:
// Default behavior - polls for up to 10 seconds automatically
const element = await testdriver.find('Loading complete indicator');
await element.click();
// Wait up to 30 seconds for slower operations
const element = await testdriver.find('Loading complete indicator', { timeout: 30000 });
await element.click();
// Useful after actions that trigger loading states
await testdriver.find('submit button').click();
await testdriver.find('success message', { timeout: 15000 });
// Disable polling for instant checks
const toast = await testdriver.find('notification toast', { timeout: 0 });
Flake Prevention
TestDriver automatically waits for the screen and network to stabilize after each action using redraw detection. This prevents flaky tests caused by animations, loading states, or dynamic content updates.
Redraw detection adds a small delay after each action but significantly reduces test flakiness.For example, when clicking a submit button that navigates to a new page:
// Click submit - TestDriver automatically waits for the new page to load
await testdriver.find('submit button').click();
// By the time this runs, the page has fully loaded and stabilized
await testdriver.assert('dashboard is displayed');
await testdriver.find('welcome message');
Without redraw detection, you'd need manual waits or retries to handle the page transition. TestDriver handles this automatically by detecting when the screen stops changing and network requests complete.
You can disable redraw detection or customize its behavior:
// Disable redraw detection for faster tests (less reliable)
const testdriver = TestDriver(context, {
redraw: false
});
Here is an example of customizing redraw detection:
// Fine-tune redraw detection
const testdriver = TestDriver(context, {
redraw: {
enabled: true,
diffThreshold: 0.1, // Pixel difference threshold (0-1)
screenRedraw: true, // Monitor screen changes
networkMonitor: true, // Wait for network idle
}
});
Simple Delays with wait()
For simple pauses — waiting for animations, transitions, or state changes after an action — use wait():
// Wait for an animation to complete
await testdriver.find('menu toggle').click();
await testdriver.wait(2000);
// Wait for a page transition to settle
await testdriver.find('next page button').click();
await testdriver.wait(1000);
For waiting for specific **elements** to appear, prefer `find()` with a `timeout` option. Use `wait()` only for simple time-based pauses.
Once your tests can reliably act on a changing UI and assert the results, the next step is figuring out what happened when something does go wrong.
Use screenshots and run output to see exactly what your test saw and pinpoint failures.Overview
Push one or more keyboard keys at the same time. Use this for keyboard shortcuts, navigation, and special keys.
Syntax
await testdriver.pressKeys(keys)
Parameters
An array of the keys to push at the same timeReturns
Promise<void>
Common Keys
Special Keys
'enter','tab','escape','backspace','delete''space','up','down','left','right''home','end','pageup','pagedown'
Modifier Keys
'ctrl','alt','shift''command'(macOS),'win'(Windows)'ctrlleft','ctrlright','shiftleft','shiftright'
Function Keys
'f1'through'f24'
Examples
Navigation
// Tab to next field
await testdriver.pressKeys(['tab']);
// Shift+Tab to previous field
await testdriver.pressKeys(['shift', 'tab']);
// Arrow keys
await testdriver.pressKeys(['down']);
await testdriver.pressKeys(['up']);
await testdriver.pressKeys(['left']);
await testdriver.pressKeys(['right']);
// Home/End
await testdriver.pressKeys(['home']); // Start of line
await testdriver.pressKeys(['end']); // End of line
// Page navigation
await testdriver.pressKeys(['pagedown']);
await testdriver.pressKeys(['pageup']);
Keyboard Shortcuts
// Copy (Ctrl+C / Cmd+C)
await testdriver.pressKeys(['ctrl', 'c']);
// Paste (Ctrl+V / Cmd+V)
await testdriver.pressKeys(['ctrl', 'v']);
// Save (Ctrl+S)
await testdriver.pressKeys(['ctrl', 's']);
// Select All (Ctrl+A)
await testdriver.pressKeys(['ctrl', 'a']);
// Undo (Ctrl+Z)
await testdriver.pressKeys(['ctrl', 'z']);
// Redo (Ctrl+Y)
await testdriver.pressKeys(['ctrl', 'y']);
// Find (Ctrl+F)
await testdriver.pressKeys(['ctrl', 'f']);
// New tab (Ctrl+T)
await testdriver.pressKeys(['ctrl', 't']);
// Close tab (Ctrl+W)
await testdriver.pressKeys(['ctrl', 'w']);
// Refresh (F5 or Ctrl+R)
await testdriver.pressKeys(['f5']);
await testdriver.pressKeys(['ctrl', 'r']);
System Shortcuts
// Alt+Tab (Windows - switch apps)
await testdriver.pressKeys(['alt', 'tab']);
// Alt+F4 (Windows - close window)
await testdriver.pressKeys(['alt', 'f4']);
// Win+D (Windows - show desktop)
await testdriver.pressKeys(['winleft', 'd']);
// Win+L (Windows - lock screen)
await testdriver.pressKeys(['winleft', 'l']);
// Cmd+Tab (macOS - switch apps)
await testdriver.pressKeys(['command', 'tab']);
// Cmd+Q (macOS - quit app)
await testdriver.pressKeys(['command', 'q']);
Form Submission
// Submit form
await testdriver.pressKeys(['enter']);
// Cancel/Close
await testdriver.pressKeys(['escape']);
// Check checkbox
await testdriver.pressKeys(['space']);
Text Editing
// Delete selected text
await testdriver.pressKeys(['delete']);
// Backspace
await testdriver.pressKeys(['backspace']);
// Select all and delete
await testdriver.pressKeys(['ctrl', 'a']);
await testdriver.pressKeys(['delete']);
// Cut text
await testdriver.pressKeys(['ctrl', 'x']);
Best Practices
**Wait after shortcuts**Some keyboard shortcuts trigger animations or navigation:
await testdriver.pressKeys(['ctrl', 't']); // New tab
await new Promise(r => setTimeout(r, 500)); // Wait for tab
await testdriver.pressKeys(['ctrl', 'l']); // Focus URL bar
**Use Tab for form navigation**
Tab is more reliable than clicking multiple fields:
const firstField = await testdriver.find('email input');
await firstField.click();
await testdriver.type('user@example.com');
await testdriver.pressKeys(['tab']);
await testdriver.type('password123');
await testdriver.pressKeys(['tab']);
await testdriver.pressKeys(['enter']); // Submit
**Platform-specific keys**
Use the appropriate modifier key for the platform:
- Windows/Linux:
'ctrl' - macOS:
'command'
// For cross-platform, you might need to detect OS
const modKey = process.platform === 'darwin' ? 'command' : 'ctrl';
await testdriver.pressKeys([modKey, 'c']); // Copy
Use Cases
```javascript // Fill form using Tab const firstField = await testdriver.find('name field'); await firstField.click(); await testdriver.type('John Doe');await testdriver.pressKeys(['tab']);
await testdriver.type('john@example.com');
await testdriver.pressKeys(['tab']);
await testdriver.type('555-0123');
await testdriver.pressKeys(['tab']);
await testdriver.pressKeys(['enter']); // Submit
```
```javascript
const textArea = await testdriver.find('comment textarea');
await textArea.click();
// Select all existing text
await testdriver.pressKeys(['ctrl', 'a']);
// Copy it
await testdriver.pressKeys(['ctrl', 'c']);
// Type new text
await testdriver.type('New comment');
// Undo if needed
await testdriver.pressKeys(['ctrl', 'z']);
```
```javascript
// Open new tab
await testdriver.pressKeys(['ctrl', 't']);
await new Promise(r => setTimeout(r, 500));
// Focus address bar
await testdriver.pressKeys(['ctrl', 'l']);
await testdriver.type('https://example.com');
await testdriver.pressKeys(['enter']);
// Refresh page
await testdriver.pressKeys(['f5']);
// Close tab
await testdriver.pressKeys(['ctrl', 'w']);
```
```javascript
// Save document
await testdriver.pressKeys(['ctrl', 's']);
// Print
await testdriver.pressKeys(['ctrl', 'p']);
// Find in page
await testdriver.pressKeys(['ctrl', 'f']);
await testdriver.type('search term');
await testdriver.pressKeys(['escape']); // Close find
```
Complete Example
import { beforeAll, afterAll, describe, it } from 'vitest';
import TestDriver from 'testdriverai';
describe('Keyboard Navigation', () => {
let testdriver;
beforeAll(async () => {
client = new TestDriver(process.env.TD_API_KEY);
await testdriver.auth();
await testdriver.connect();
});
afterAll(async () => {
await testdriver.disconnect();
});
it('should navigate form with keyboard', async () => {
await testdriver.focusApplication('Google Chrome');
// Find first field
const emailField = await testdriver.find('email input');
await emailField.click();
await testdriver.type('user@example.com');
// Tab through fields
await testdriver.pressKeys(['tab']);
await testdriver.type('John');
await testdriver.pressKeys(['tab']);
await testdriver.type('Doe');
await testdriver.pressKeys(['tab']);
await testdriver.type('password123');
// Submit with Enter
await testdriver.pressKeys(['tab']);
await testdriver.pressKeys(['enter']);
await testdriver.assert('form submitted successfully');
});
it('should use keyboard shortcuts', async () => {
// Open new browser tab
await testdriver.pressKeys(['ctrl', 't']);
await new Promise(r => setTimeout(r, 500));
// Focus address bar
await testdriver.pressKeys(['ctrl', 'l']);
await testdriver.type('https://example.com');
await testdriver.pressKeys(['enter']);
await new Promise(r => setTimeout(r, 2000));
// Select all page content
await testdriver.pressKeys(['ctrl', 'a']);
// Copy
await testdriver.pressKeys(['ctrl', 'c']);
// Refresh page
await testdriver.pressKeys(['f5']);
});
});
Related Methods
Overview
The Provision API sets up applications in your sandbox before the tests run. It downloads, installs, and starts browsers, desktop apps, VS Code, Chrome extensions, and more.
Use the provision methods through testdriver.provision.*:
await testdriver.provision.chrome({ url: 'https://example.com' });
Methods
chrome()
Start Google Chrome with an optional URL.
await testdriver.provision.chrome(options?)
The URL to go to after Chrome starts.
<ParamField path="maximized" type="boolean" default={true}>
Start Chrome with the window at the maximum size.
</ParamField>
<ParamField path="guest" type="boolean" default={false}>
Start Chrome in the guest profile mode.
</ParamField>
// Basic
await testdriver.provision.chrome({ url: 'https://example.com' });
// With guest mode
await testdriver.provision.chrome({
url: 'https://example.com',
guest: true,
maximized: true,
});
chromeExtension()
Install and start a Chrome extension. You can install it from a local unpacked directory. Or you can install it from the Chrome Web Store by the extension ID.
await testdriver.provision.chromeExtension(options)
One of `extensionPath` or `extensionId` is required.
Local path to an unpacked extension directory. The extension files are uploaded to the sandbox.
<ParamField path="extensionId" type="string">
Chrome Web Store extension ID to install.
</ParamField>
<ParamField path="maximized" type="boolean" default={true}>
Launch Chrome in maximized window mode.
</ParamField>
// From local directory
await testdriver.provision.chromeExtension({
extensionPath: './my-extension',
});
// From Chrome Web Store
await testdriver.provision.chromeExtension({
extensionId: 'abcdefghijklmnop',
});
vscode()
Launch Visual Studio Code with an optional workspace and extensions.
await testdriver.provision.vscode(options?)
Path to a workspace folder or `.code-workspace` file to open.
<ParamField path="extensions" type="string[]" default={[]}>
Array of VS Code extension IDs to install before launching.
</ParamField>
await testdriver.provision.vscode({
workspace: '/home/testdriver/project',
extensions: ['ms-python.python', 'esbenp.prettier-vscode'],
});
installer()
Download and run an application installer. Supports .msi, .exe, .deb, .rpm, .appimage, .sh, .dmg, and .pkg formats.
await testdriver.provision.installer(options)
Download URL for the installer.
<ParamField path="filename" type="string">
Override the auto-detected filename from the URL.
</ParamField>
<ParamField path="appName" type="string">
Application name to focus after installation completes.
</ParamField>
<ParamField path="launch" type="boolean" default={true}>
Whether to focus/launch the app after installation.
</ParamField>
Behavior:
- Downloads the installer from the URL
- Auto-detects the install method based on file extension
- Runs the appropriate install command (e.g.,
msiexecfor.msi,dpkgfor.deb) - Optionally focuses the installed application
// Windows MSI installer
await testdriver.provision.installer({
url: 'https://example.com/app-setup.msi',
appName: 'MyApp',
});
// Linux DEB package
await testdriver.provision.installer({
url: 'https://example.com/app.deb',
appName: 'MyApp',
launch: true,
});
electron()
Launch an Electron application.
await testdriver.provision.electron(options)
Path to the Electron application directory or executable.
<ParamField path="args" type="string[]" default={[]}>
Additional command-line arguments to pass to the Electron app.
</ParamField>
await testdriver.provision.electron({
appPath: '/home/testdriver/my-electron-app',
args: ['--no-sandbox'],
});
dashcam()
Start Dashcam recording with custom options. Usually called automatically by other provision methods, but can be called directly for custom configurations.
await testdriver.provision.dashcam(options?)
Path to the TestDriver log file. Defaults to `/tmp/testdriver.log` (Linux) or `C:\Users\testdriver\testdriver.log` (Windows).
<ParamField path="logName" type="string" default="TestDriver Log">
Display name for the log file in the Dashcam replay.
</ParamField>
<ParamField path="webLogs" type="boolean" default={true}>
Enable web traffic log capture.
</ParamField>
<ParamField path="title" type="string">
Recording title for the Dashcam session.
</ParamField>
await testdriver.provision.dashcam({
title: 'Login Flow Test',
logPath: '/tmp/my-app.log',
logName: 'Application Log',
webLogs: true,
});
Types
interface ProvisionChromeOptions {
url?: string; // Default: "http://testdriver-sandbox.vercel.app/"
maximized?: boolean; // Default: true
guest?: boolean; // Default: false
}
interface ProvisionChromeExtensionOptions {
extensionPath?: string; // Local unpacked extension path
extensionId?: string; // Chrome Web Store ID
maximized?: boolean; // Default: true
}
interface ProvisionVSCodeOptions {
workspace?: string; // Workspace path
extensions?: string[]; // Extension IDs to install
}
interface ProvisionInstallerOptions {
url: string; // Download URL (required)
filename?: string; // Override filename
appName?: string; // App name to focus
launch?: boolean; // Default: true
}
interface ProvisionElectronOptions {
appPath: string; // Path to Electron app (required)
args?: string[]; // Additional args
}
interface ProvisionDashcamOptions {
logPath?: string; // Log file path
logName?: string; // Default: "TestDriver Log"
webLogs?: boolean; // Default: true
title?: string; // Recording title
}
Complete Example
import { describe, it, beforeAll, afterAll } from 'vitest';
import TestDriver from 'testdriverai';
describe('Chrome Extension Test', () => {
let testdriver;
beforeAll(async () => {
testdriver = new TestDriver({
os: 'linux',
resolution: '1920x1080',
});
await testdriver.ready();
// Install extension and open Chrome
await testdriver.provision.chromeExtension({
extensionPath: './my-extension',
});
// Start Dashcam with custom logs
await testdriver.provision.dashcam({
title: 'Extension Test',
webLogs: true,
});
});
afterAll(async () => {
await testdriver.disconnect();
});
it('tests the extension popup', async () => {
await testdriver.find('extension icon').click();
await testdriver.find('popup content').click();
});
});
TestDriver writes and runs computer-use tests for web apps, desktop apps, and browser extensions. Tests describe what a user sees and does in plain English, and run on real machines in the cloud. Pick the path that fits how you work:
No install. Mention `@testdriverai` in a PR or issue and it writes, runs, and commits tests for you. One command scaffolds a project and connects the TestDriver agent and MCP server to Cursor, Claude Code, VS Code, and more. Add TestDriver to an existing Vitest project file by file.Join our Discord if you have questions or get stuck.
Overview
The redraw system waits for the screen to become stable after an interaction. Then it continues. It finds when animations, page loads, and network requests are complete. This stops actions on a screen that changes.
**Redraw is disabled by default from v7.3.** Enable it if your tests interact with applications that have many animations or load states.How It Works
Redraw uses a two-phase detection method:
- Change Detection. TestDriver compares the present frame to the first screenshot from after the action. If the pixel diff is more than 0.1%, the screen changed.
- Stability Detection — TestDriver compares frames that come after each other with z-score analysis. When the diff between frames is less than 0.1%, or the z-score is negative (the present diff is less than the average), the screen is complete.
The screen is complete when both phases finish: the screen changed from the first state AND the frames that come after each other are now stable.
flowchart TD
A[Action performed] --> B{Phase 1: Change Detection\ndiffFromInitial > 0.1%?}
B -- "Yes (screen changed)" --> C{Phase 2: Stability\nz-score < 0 or\ndiffPercent < 0.1%?}
C -- "Yes (frames stable)" --> D[Screen settled ✓]
B -- "No (waiting...)" --> B
C -- "No (waiting...)" --> C
Polling
The system polls at 500ms intervals, comparing screenshot frames. This reduces WebSocket traffic while still providing responsive detection.
Pixel Comparison
Uses pixelmatch for per-pixel comparison with a threshold of 0.1 for pixel sensitivity. A frame diff above 0.1% of total pixels indicates the screen has changed.
Z-Score Analysis
Screen stability uses statistical analysis of the last 10 measurements:
- Calculate the mean and standard deviation of consecutive frame diffs
- Compute the z-score:
(currentDiff - mean) / stddev - Screen is stable when
diffPercent < 0.1%orz-score < 0(current diff is below the average)
This approach adapts to the specific animation patterns of your application rather than using a fixed threshold.
Per-Command Timeouts
Each command type has a specific redraw timeout:
| Command | Timeout | Reason |
|---|---|---|
click | 5000ms | Page navigations, modal openings |
hover (within click) | 5000ms | Same as click |
hover (standalone) | 2500ms | Tooltip animations |
scroll | 5000ms | Lazy-loaded content |
type | 5000ms | Autocomplete, validation |
pressKeys | 5000ms | Keyboard shortcuts may trigger changes |
focusApplication | 1000ms | Window focus animations |
If the timeout is reached before the screen settles, the command continues anyway. The timeout event is available via the redraw:complete event.
Configuration
Constructor Options
const testdriver = new TestDriver({
// Shorthand: enable/disable
redraw: true, // enable with defaults
redraw: false, // disable (default since v7.3)
// Full configuration
redraw: {
enabled: true,
screenRedraw: true, // enable screen pixel diff detection
networkMonitor: false, // enable network settling detection
},
});
Redraw configuration. Pass `true`/`false` for shorthand, or an object for fine-grained control.
Enable or disable the redraw system. Default changed to `false` in v7.3.
<ParamField path="screenRedraw" type="boolean" default={true}>
Enable pixel-diff-based screen change detection. If both `screenRedraw` and `networkMonitor` are `false`, redraw auto-disables.
</ParamField>
<ParamField path="networkMonitor" type="boolean" default={false}>
Enable network traffic monitoring for settling detection. Monitors WebSocket traffic on the sandbox to detect when network activity subsides.
</ParamField>
Per-Command Override
Override redraw settings for individual commands:
// Enable redraw for a specific click
await testdriver.find('load more').click({
redraw: { enabled: true },
});
// Disable redraw for a fast interaction
await testdriver.find('checkbox').click({
redraw: false,
});
Network Settling
When networkMonitor is enabled, the system also monitors sandbox network traffic:
- Polls for
totalBytesReceivedandtotalBytesSentfrom the sandbox - Keeps the last 60 measurements
- Calculates z-scores for both RX and TX byte rates
- Network is settled when both RX and TX z-scores are negative (traffic is below average)
- Has a 10-second timeout for network polling
- Non-critical: network errors are logged but never throw
The final settling condition requires both screen AND network to be settled (when both are enabled).
Events
The redraw system emits events through the SDK emitter. See Events for the full event reference.
| Event | Description |
|---|---|
redraw:status | Emitted on each poll with current screen diff, network stats, and timeout info |
redraw:complete | Emitted when redraw resolves (settled or timed out) |
testdriver.emitter.on('redraw:status', (status) => {
console.log(`Screen: ${status.redraw.text}`);
console.log(`Network: ${status.network.text}`);
console.log(`Timeout: ${status.timeout.text}`);
});
testdriver.emitter.on('redraw:complete', (result) => {
if (result.isTimeout) {
console.warn(`Redraw timed out after ${result.timeElapsed}ms`);
} else {
console.log(`Screen settled in ${result.timeElapsed}ms`);
}
});
When to Use Redraw
Enable redraw when:
- Testing single-page applications (SPAs) with route transitions
- Interacting with pages that lazy-load content on scroll
- Clicking buttons that trigger animations or modals
- Testing apps with significant network-driven UI updates
Keep redraw disabled when:
- Tests are already stable without it
- You want faster test execution
- Your application has minimal animations
- You're using explicit waits or assertions instead
Types
interface RedrawConfig {
enabled?: boolean; // Default: false (since v7.3)
screenRedraw?: boolean; // Default: true
networkMonitor?: boolean; // Default: false
}
interface RedrawStatusEvent {
redraw: {
enabled: boolean;
settled: boolean;
hasChangedFromInitial: boolean;
consecutiveFramesStable: number;
diffFromInitial: number;
diffFromLast: number;
text: string;
};
network: {
enabled: boolean;
settled: boolean;
rxBytes: number;
txBytes: number;
text: string;
};
timeout: {
isTimeout: boolean;
elapsed: number;
max: number;
text: string;
};
}
interface RedrawCompleteEvent {
screenSettled: boolean;
hasChangedFromInitial: boolean;
consecutiveFramesStable: number;
networkSettled: boolean;
isTimeout: boolean;
timeElapsed: number;
}
When your test suite becomes larger, put the common patterns into reusable code. This keeps the tests short, easy to read, and easy to keep.
Helper Functions
The most simple method is to put the common actions into helper functions. Make a helpers/ directory for the shared utilities:
export async function login(testdriver, { email, password }) {
const emailInput = await testdriver.find('email input');
await emailInput.click();
await testdriver.type(email);
const passwordInput = await testdriver.find('password input');
await passwordInput.click();
await testdriver.type(password);
const loginButton = await testdriver.find('login button');
await loginButton.click();
const result = await testdriver.assert('user is logged in');
return result;
}
export async function logout(testdriver) {
const userMenu = await testdriver.find('user menu');
await userMenu.click();
const logoutButton = await testdriver.find('logout button');
await logoutButton.click();
}
**Do not put dynamic values in element descriptions.** An element selector must describe the *type* of the element. It must not describe specific content that can change.
❌ Bad: await testdriver.find('profile name TestDriver in the top right')
✅ Good: await testdriver.find('user profile name in the top right')
Hardcoded values like usernames, product names, or prices will cause tests to fail when the data changes. Use generic descriptions that work regardless of the specific content displayed.
Now import and use these helpers in any test:
import { describe, expect, it } from "vitest";
import { TestDriver } from "testdriverai/vitest/hooks";
import { login } from './helpers/auth.js';
describe("Checkout", () => {
it("should complete checkout as logged in user", async (context) => {
const testdriver = TestDriver(context);
await testdriver.provision.chrome({
url: 'https://shop.example.com',
});
// Use the helper
await login(testdriver, {
email: 'user@example.com',
password: 'password123'
});
// Continue with checkout steps...
const cartButton = await testdriver.find('cart button');
await cartButton.click();
});
});
Page Objects
For larger test suites, the Page Object pattern encapsulates all interactions with a specific page or component:
export class LoginPage {
constructor(testdriver) {
this.td = testdriver;
}
async enterEmail(email) {
const input = await this.td.find('email input');
await input.click();
await this.td.type(email);
}
async enterPassword(password) {
const input = await this.td.find('password input');
await input.click();
await this.td.type(password);
}
async submit() {
const button = await this.td.find('submit button');
await button.click();
}
async login(email, password) {
await this.enterEmail(email);
await this.enterPassword(password);
await this.submit();
}
async assertError(message) {
return await this.td.assert(`error message shows "${message}"`);
}
async assertLoggedIn() {
return await this.td.assert('user dashboard is visible');
}
}
Use the page object in your tests:
import { describe, expect, it } from "vitest";
import { TestDriver } from "testdriverai/vitest/hooks";
import { LoginPage } from './pages/LoginPage.js';
describe("Authentication", () => {
it("should show error for invalid credentials", async (context) => {
const testdriver = TestDriver(context);
await testdriver.provision.chrome({
url: 'https://app.example.com/login',
});
const loginPage = new LoginPage(testdriver);
await loginPage.login('invalid@test.com', 'wrongpassword');
const hasError = await loginPage.assertError('Invalid credentials');
expect(hasError).toBeTruthy();
});
it("should redirect to dashboard on success", async (context) => {
const testdriver = TestDriver(context);
await testdriver.provision.chrome({
url: 'https://app.example.com/login',
});
const loginPage = new LoginPage(testdriver);
await loginPage.login('valid@test.com', 'correctpassword');
const isLoggedIn = await loginPage.assertLoggedIn();
expect(isLoggedIn).toBeTruthy();
});
});
Shared Test Fixtures
Create reusable fixtures for common test setup scenarios:
export const testUsers = {
admin: { email: 'admin@example.com', password: 'admin123' },
regular: { email: 'user@example.com', password: 'user123' },
guest: { email: 'guest@example.com', password: 'guest123' },
};
export const testUrls = {
staging: 'https://staging.example.com',
production: 'https://example.com',
};
export async function setupAuthenticatedSession(testdriver, user = testUsers.regular) {
const emailInput = await testdriver.find('email input');
await emailInput.click();
await testdriver.type(user.email);
const passwordInput = await testdriver.find('password input');
await passwordInput.click();
await testdriver.type(user.password);
const loginButton = await testdriver.find('login button');
await loginButton.click();
await testdriver.assert('user is logged in');
}
import { describe, expect, it } from "vitest";
import { TestDriver } from "testdriverai/vitest/hooks";
import { testUsers, testUrls, setupAuthenticatedSession } from './fixtures/index.js';
describe("Admin Panel", () => {
it("should access admin settings", async (context) => {
const testdriver = TestDriver(context);
await testdriver.provision.chrome({
url: `${testUrls.staging}/login`,
});
await setupAuthenticatedSession(testdriver, testUsers.admin);
const settingsLink = await testdriver.find('admin settings link');
await settingsLink.click();
const result = await testdriver.assert('admin settings panel is visible');
expect(result).toBeTruthy();
});
});
Suggested Project Structure
| Folder | Purpose |
|---|---|
fixtures/ | Test data and setup utilities |
helpers/ | Reusable helper functions |
pages/ | Page object classes |
specs/ | Test files |
Agents 1
TestDriver Expert
You are an expert at writing automated tests using the TestDriver library. Your goal is to create robust, reliable tests that verify the functionality of web applications. You work iteratively, verifying your progress at each step.
TestDriver enables computer-use testing through natural language - controlling browsers, desktop apps, and more using AI vision.
Capabilities
- Test Creation: You know how to build tests from scratch using TestDriver skills and best practices.
- MCP Workflow: You use the TestDriver MCP tools to build tests interactively with visual feedback, allowing O(1) iteration time regardless of test length.
- Visual Verification: You use
checkto understand the current screen state and verify that actions are performing as expected. - Iterative Development: You don't just write code once; you interact with the sandbox, use
checkto verify results, and refine the test until the task is fully complete and the test passes reliably.
Context and examples
Use this agent when the user asks to:
- "Write a test for X"
- "Automate this workflow"
- "Debug why this test is failing"
- "Check if the login page works"
Workflow
- Analyze: Understand the user's requirements and the application under test.
- Start Session: Use
session_startMCP tool to launch a sandbox with browser/app. SpecifytestFileto track where code should be written. - Interact: Use MCP tools (
find,click,type, etc.) - each returns a screenshot AND generated code. - ⚠️ WRITE CODE IMMEDIATELY: After EVERY successful action, append the generated code to the test file RIGHT AWAY. Do NOT wait until the end.
- Verify Actions: Use
checkafter actions to verify they succeeded (for YOUR understanding only). - Add Assertions: Use
assertfor test conditions that should be in the final test file. - ⚠️ RUN THE TEST YOURSELF: Use
vitest run <testFile>to run the test - do NOT tell the user to run it. Iterate until it passes. NEVER usenpx vitest- always usevitestdirectly. - ⚠️ SHARE THE TEST REPORT: After EVERY test run, find the
TESTDRIVER_RUN_URLin the output (e.g.,TESTDRIVER_RUN_URL=https://console.testdriver.ai/runs/...) and share it with the user so they can view the recording and results.
Prerequisites
Quick Start - Creating Your First TestDriver Test
For new projects, use the init command to automatically set up everything:
CLI:
npx testdriverai init
MCP (via this agent):
// apiKey is optional - if not provided, user adds it to .env manually after init
init({ directory: "." })
// Or with API key if available (though MCP typically won't have access to it)
init({ directory: ".", apiKey: "your_api_key" })
Note: The apiKey parameter is optional. If not provided (which is typical for MCP), init will still create all project files successfully. The user can manually add TD_API_KEY=... to the .env file afterward.
The init command creates:
- ✅
package.jsonwith proper dependencies - ✅ Example test files (
tests/example.test.js,tests/login.js) - ✅
vitest.config.jswith correct timeouts - ✅
.gitignorewith.env - ✅ GitHub Actions workflow (
.github/workflows/testdriver.yml) - ✅ VSCode MCP config (
.vscode/mcp.json) - ✅ TestDriver skills and agents in
.github/ - ✅
.envfile (user adds API key manually if not provided to init)
After running init:
- User adds their API key to
.env:TD_API_KEY=... - Test the setup:
vitest run - Start building custom tests using the examples as templates
API Key Setup
The user must have a TestDriver API key set in their environment:
# .env file
TD_API_KEY=your_api_key_here
Get your API key at: https://console.testdriver.ai/team
Manual Installation
If not using init, install TestDriver:
npm install --save-dev testdriverai
Test Runner
TestDriver only works with Vitest. Tests must use the .test.mjs extension and import from vitest:
import { describe, expect, it } from "vitest";
import { TestDriver } from "testdriverai/vitest/hooks";
Vitest Configuration
TestDriver tests require long timeouts for both tests and hooks (sandbox provisioning, cleanup, and recording uploads). Always create a vitest.config.mjs with these settings:
import { defineConfig } from "vitest/config";
import { config } from "dotenv";
config();
export default defineConfig({
test: {
testTimeout: 900000,
hookTimeout: 900000,
},
});
Important: Both
testTimeoutandhookTimeoutmust be set. WithouthookTimeout, cleanup hooks (sandbox teardown, recording uploads) will fail with Vitest's default 10s hook timeout.
Basic Test Structure
import { describe, expect, it } from "vitest";
import { TestDriver } from "testdriverai/vitest/hooks";
describe("My Test Suite", () => {
it("should do something", async (context) => {
// Initialize TestDriver - screenshots are captured automatically before/after each command
const testdriver = TestDriver(context);
// Start with provision - this launches the sandbox and browser
await testdriver.provision.chrome({
url: "https://example.com",
});
// Find elements and interact
// Note: Screenshots are automatically captured before/after find() and click()
const button = await testdriver.find("Sign In button");
await button.click();
await testdriver.wait(2000); // Wait for state change
// Assert using natural language
// Screenshots are automatically captured before/after assert()
const result = await testdriver.assert("the dashboard is visible");
expect(result).toBeTruthy();
});
});
**Automatic Screenshots**: TestDriver captures screenshots before and after every command by default. Screenshots are saved with descriptive names like `001-click-before-L42-submit-button.png` that include the line number from your test file.
Provisioning Options
Most tests start with testdriver.provision.
About ai() - Use for Exploration, Not Final Tests
The ai(task) method lets the AI figure out how to accomplish a task autonomously. It's useful for:
- Exploring how to accomplish something when you're unsure of the steps
- Discovering element descriptions and UI flow
- Last resort when explicit methods fail repeatedly
However, prefer explicit methods (find, click, type) in final tests because:
- They're more predictable and repeatable
- They're faster (no AI reasoning loop)
- They're easier to debug when they fail
// ✅ GOOD: Explicit steps (preferred for final tests)
const emailInput = await testdriver.find("email input field");
await emailInput.click();
await testdriver.type("user@example.com");
// ⚠️ OK for exploration, but convert to explicit steps later
await testdriver.ai("fill in the email field with user@example.com");
Element Properties (for debugging)
Elements returned by find() have properties you can inspect:
const element = await testdriver.find("Sign In button");
// Debugging properties
console.log(element.x, element.y); // coordinates
console.log(element.centerX, element.centerY); // center coordinates
console.log(element.width, element.height); // dimensions
console.log(element.confidence); // AI confidence score
console.log(element.text); // detected text
console.log(element.boundingBox); // full bounding box
Element Methods
const element = await testdriver.find("button");
await element.click(); // click
await element.hover(); // hover
await element.doubleClick(); // double-click
await element.rightClick(); // right-click
await element.mouseDown(); // press mouse down
await element.mouseUp(); // release mouse
element.found(); // check if found (boolean)
Automatic Screenshots (Enabled by Default)
TestDriver automatically captures screenshots before and after every command by default. This creates a complete visual timeline without any additional code. Screenshots are named with the line number from your test file, making it easy to trace issues:
.testdriver/screenshots/login.test/
001-find-before-L15-email-input.png
002-find-after-L15-email-input.png
003-click-before-L16-email-input.png
004-click-after-L16-email-input.png
005-type-before-L17-userexamplecom.png
006-type-after-L17-userexamplecom.png
Filename format: <seq>-<action>-<phase>-L<line>-<description>.png
Note: The screenshot folder for each test file is automatically cleared when the test starts.
Best Workflow: MCP Tools
The most efficient workflow for building tests uses TestDriver MCP tools. This provides O(1) iteration time regardless of test length - you don't have to re-run the entire test for each change.
Key Advantages
- No need to restart - continue from current state
- Generated code with every action - each tool returns the code to add to your test
- Use
checkto verify - understand screen state without explicit screenshots
⚠️ CRITICAL: Write Code Immediately & Run Tests Yourself
Every MCP tool response includes "ACTION REQUIRED: Append this code..." - you MUST write that code to the test file IMMEDIATELY before proceeding to the next action.
When ready to validate, RUN THE TEST YOURSELF using vitest run. Do NOT tell the user to run it. NEVER use npx vitest.
Step 1: Start a Session
session_start({ type: "chrome", url: "https://your-app.com/login", testFile: "tests/login.test.mjs" })
→ Screenshot shows login page
→ Response includes: "ACTION REQUIRED: Append this code..."
→ ⚠️ IMMEDIATELY write to tests/login.test.mjs:
await testdriver.provision.chrome({ url: "https://your-app.com/login" });
This provisions a sandbox with Chrome and navigates to your URL. You'll see a screenshot of the initial page.
Note: Screenshots are captured automatically before/after each command. The generated code no longer includes manual
screenshot()calls.
Step 2: Interact with the App
Find elements and interact with them. Write code to file after EACH action:
find_and_click({ description: "email input field" })
→ Returns: screenshot with element highlighted
→ ⚠️ IMMEDIATELY append to test file:
await testdriver.find("email input field").click();
type({ text: "user@example.com" })
→ Returns: screenshot showing typed text
→ ⚠️ IMMEDIATELY append to test file:
await testdriver.type("user@example.com");
Note: Screenshots are automatically captured before/after each command. Each screenshot filename includes the line number (e.g.,
001-click-before-L42-email-input.png).
Step 3: Verify Actions Succeeded (For Your Understanding)
After actions, use check to verify they worked. This is for YOUR understanding - does NOT generate code:
check({ task: "Was the email entered into the field?" })
→ Returns: AI analysis comparing previous screenshot to current state
Step 4: Add Assertions (Generates Code)
Use assert for pass/fail conditions. This DOES generate code for the test file:
assert({ assertion: "the dashboard is visible" })
→ Returns: pass/fail with screenshot
→ ⚠️ IMMEDIATELY append to test file:
const assertResult = await testdriver.assert("the dashboard is visible");
expect(assertResult).toBeTruthy();
Step 5: Run the Test Yourself
⚠️ YOU must run the test - do NOT tell the user to run it. NEVER use npx vitest - always use vitest directly:
vitest run tests/login.test.mjs
Analyze the output, fix any issues, and iterate until the test passes.
⚠️ ALWAYS share the test report link with the user. After each test run, look for TESTDRIVER_RUN_URL in the test output (e.g., TESTDRIVER_RUN_URL=https://console.testdriver.ai/runs/...) and share it with the user so they can view the recording and results. This is CRITICAL - users need to see the visual recording to understand test behavior.
MCP Tools Reference
| Tool | Description |
|---|---|
session_start | Start sandbox with browser/app, returns screenshot + provision code |
session_status | Check session health and time remaining |
session_extend | Add more time before session expires |
find | Locate element by description, returns ref for later use |
click | Click on element ref |
find_and_click | Find and click in one action |
type | Type text into focused field |
press_keys | Press keyboard shortcuts (e.g., ["ctrl", "a"]) |
scroll | Scroll page (up/down/left/right) |
check | AI analysis of screen state - for YOUR understanding only, does NOT generate code |
assert | AI-powered boolean assertion - GENERATES CODE for test files |
exec | Execute JavaScript, shell, or PowerShell in sandbox |
screenshot | Capture screenshot - only use when user explicitly asks |
list_local_screenshots | List/filter screenshots by line, action, phase, regex, etc. |
view_local_screenshot | View a local screenshot (returns image to AI + displays to user) |
Debugging with Local Screenshots
After test runs (successful or failed), you can view saved screenshots to understand test behavior.
Screenshot filename format: <seq>-<action>-<phase>-L<line>-<description>.png
Example: 001-click-before-L42-submit-button.png
1. List all screenshots from a test:
list_local_screenshots({ directory: "login.test" })
2. Filter by line number (find what happened at a specific line):
// Find screenshots from line 42
list_local_screenshots({ line: 42 })
// Find screenshots from lines 10-20
list_local_screenshots({ lineRange: { start: 10, end: 20 } })
3. Filter by action type:
// Find all click screenshots
list_local_screenshots({ action: "click" })
// Find all assertions
list_local_screenshots({ action: "assert" })
4. Filter by phase (before/after):
// See state BEFORE actions (useful for debugging what was visible)
list_local_screenshots({ phase: "before" })
// See state AFTER actions (useful for verifying results)
list_local_screenshots({ phase: "after" })
5. Filter by regex pattern:
// Find screenshots related to login
list_local_screenshots({ pattern: "login|signin" })
// Find button-related screenshots
list_local_screenshots({ pattern: "button.*click" })
6. Filter by sequence number:
// Find screenshots 1-5 (first 5 actions)
list_local_screenshots({ sequenceRange: { start: 1, end: 5 } })
7. Sort results:
// Sort by execution order (useful for understanding flow)
list_local_screenshots({ sortBy: "sequence" })
// Sort by line number (useful for tracing back to code)
list_local_screenshots({ sortBy: "line" })
// Sort by modified time (default - newest first)
list_local_screenshots({ sortBy: "modified" })
8. Combine filters:
// Find click screenshots at line 42
list_local_screenshots({ directory: "checkout.test", line: 42, action: "click" })
// Find all "before" screenshots in lines 10-30
list_local_screenshots({ lineRange: { start: 10, end: 30 }, phase: "before" })
9. View a screenshot:
view_local_screenshot({ path: ".testdriver/screenshots/login.test/001-click-before-L42-submit-button.png" })
When to use screenshot viewing:
- After test failures - View screenshots to see exactly what the UI looked like when the test failed
- Debugging element finding issues - See if elements are actually visible or have different appearances than expected
- Comparing test runs - View screenshots from multiple runs to identify flaky behavior
- Verifying test logic - Before running a test, view screenshots from previous runs to understand the UI flow
Debugging workflow example:
# Test failed at line 42, let's see what happened
list_local_screenshots({ line: 42 })
# View the before/after state at that line
view_local_screenshot({ path: ".testdriver/screenshots/checkout.test/005-click-before-L42-submit-button.png" })
view_local_screenshot({ path: ".testdriver/screenshots/checkout.test/006-click-after-L42-submit-button.png" })
# Check what the screen looked like before the failing action
list_local_screenshots({ directory: "checkout.test", phase: "before", limit: 10 })
Tips for MCP Workflow
- ⚠️ Write code IMMEDIATELY - After EVERY action, append generated code to test file RIGHT AWAY
- ⚠️ Run tests YOURSELF - Use
vitest run(NEVERnpx vitest) - do NOT tell user to run tests - ⚠️ Add screenshots liberally - Include
await testdriver.screenshot()after every significant action for debugging - ⚠️ Use screenshot viewing for debugging - When tests fail, use
list_local_screenshotsandview_local_screenshotto understand what went wrong - Work incrementally - Don't try to build the entire test at once
- Use
checkafter actions - Verify your actions succeeded before moving on (for YOUR understanding) - Use
assertfor test verifications - These generate code that goes in the test file - Be specific with element descriptions - "the blue Sign In button in the header" is better than "button"
- Extend session proactively - Sessions expire after 5 minutes; use
session_extendif needed
Recommended Development Workflow
- Write a few steps - Don't write the entire test at once
- Run the test - See what happens on the sandbox
- Inspect outputs - Use element properties to debug
- Assert/expect - Verify the step worked
- Iterate - Add more steps incrementally
// Development workflow example
// Note: Screenshots are automatically captured before/after each command!
it("should incrementally build test", async (context) => {
const testdriver = TestDriver(context);
await testdriver.provision.chrome({ url: "https://example.com" });
// Automatic screenshot: 001-provision-after-L3-chrome.png
// Step 1: Find and inspect
const element = await testdriver.find("Some button");
console.log("Element found:", element.found());
console.log("Coordinates:", element.x, element.y);
console.log("Confidence:", element.confidence);
// Automatic screenshot: 002-find-after-L7-some-button.png
// Step 2: Interact
await element.click();
// Automatic screenshot: 003-click-after-L13-element.png
// Step 3: Assert
const result = await testdriver.assert("Something happened");
console.log("Assertion result:", result);
expect(result).toBeTruthy();
// Automatic screenshot: 004-assert-after-L17-something-happened.png
// Then add more steps...
});
TestDriver Options Reference
const testdriver = TestDriver(context, {
newSandbox: true, // Create new sandbox (default: true)
preview: "browser", // "browser" | "ide" | "none" (default: "browser")
reconnect: false, // Reconnect to last sandbox (default: false)
keepAlive: 30000, // Keep sandbox alive after test (default: 30000ms / 30 seconds)
os: "linux", // 'linux' | 'windows' (default: 'linux')
resolution: "1366x768", // Sandbox resolution
cache: true, // Enable element caching (default: true)
cacheKey: "my-test", // Cache key for element finding
autoScreenshots: true, // Capture screenshots before/after each command (default: true)
});
Preview Modes
| Value | Description |
|---|---|
"browser" | Opens debugger in default browser (default) |
"ide" | Opens preview in IDE panel (VSCode, Cursor - requires TestDriver extension) |
"none" | Headless mode, no visual preview |
Common Patterns
Typing in Fields
await testdriver.find("Email input").click();
await testdriver.type("user@example.com");
Keyboard Shortcuts
await testdriver.pressKeys(["ctrl", "a"]); // Select all
await testdriver.pressKeys(["ctrl", "c"]); // Copy
await testdriver.pressKeys(["enter"]); // Submit
Waiting and Polling
// Use timeout option to poll until element is found (retries every 5 seconds)
const element = await testdriver.find("Loading complete indicator", {
timeout: 30000,
});
await element.click();
Scrolling
⚠️ Important: Ensure proper focus before scrolling
Scrolling requires the page or frame to be focused, not an input field or other interactive element. If an input is focused, scroll commands may not work as expected.
// If you've been typing in an input, click elsewhere first
await testdriver.find("page background").click();
// Or press Escape to unfocus
await testdriver.pressKeys(["escape"]);
// Now scroll
await testdriver.scroll("down");
// If scroll is not working, try using Page Down key directly
await testdriver.pressKeys(["pagedown"]);
Executing Code in Sandbox
// Shell (Linux)
const output = await testdriver.exec("sh", "ls -la", 5000);
// PowerShell (Windows)
const date = await testdriver.exec("pwsh", "Get-Date", 5000);
Capturing Screenshots
Screenshots are captured automatically before and after each SDK command (click, type, find, assert, etc.). Each screenshot filename includes:
- Sequential number for chronological ordering
- Action name (e.g.,
click,find,assert) - Phase (
beforeorafter) - Line number from your test file
- Description from the command
Example filenames:
001-provision-after-L8-chrome.png002-find-before-L12-login-button.png003-click-after-L12-element.png
Screenshots are saved to .testdriver/screenshots/<test-file>/.
To disable automatic screenshots:
const testdriver = TestDriver(context, { autoScreenshots: false });
For manual screenshots (e.g., with mouse cursor visible):
await testdriver.screenshot(1, false, true);
Tips for Agents
- ⚠️ WRITE CODE IMMEDIATELY - After EVERY successful MCP action, append the generated code to the test file RIGHT AWAY. Do NOT wait until the session ends.
- ⚠️ RUN TESTS YOURSELF - Do NOT tell the user to run tests. YOU must run the tests using
vitest run <testFile>(NEVER usenpx vitest). Analyze the output and iterate until the test passes. - ⚠️ SHARE THE TEST REPORT URL - After EVERY test run, find
TESTDRIVER_RUN_URL=https://console.testdriver.ai/runs/...in the output and share it with the user. This is CRITICAL - users need to view the recording to understand what happened. - Screenshots are automatic - TestDriver captures screenshots before/after every command by default. Each screenshot filename includes the line number (e.g.,
001-click-before-L42-submit-button.png) making it easy to trace issues. - ⚠️ USE SCREENSHOT VIEWING FOR DEBUGGING - When tests fail, use
list_local_screenshotsandview_local_screenshotMCP commands to see exactly what the UI looked like. The filenames tell you which line of code triggered each screenshot. - Use
wait()for simple delays - Useawait testdriver.wait(ms)when you need a pause (e.g., after actions, for animations). For waiting for specific elements, preferfind()with atimeoutoption. - Use MCP tools for development - Build tests interactively with visual feedback
- Always check
sdk.d.tsfor method signatures and types when debugging generated tests - Look at test samples in
node_modules/testdriverai/testfor working examples - Use
checkto understand screen state - This is how you verify what the sandbox shows during MCP development. - Use
checkafter actions,assertfor test files -checkgives detailed AI analysis (no code),assertgives boolean pass/fail (generates code) - Be specific with element descriptions - "blue Sign In button in the header" > "button"
- Start simple - get one step working before adding more
- Always
awaitasync methods - TestDriver will warn if you forget, but for TypeScript projects, add@typescript-eslint/no-floating-promisesto your ESLint config to catch missingawaitat compile time:
// eslint.config.js (for TypeScript projects)
{
"rules": {
"@typescript-eslint/no-floating-promises": "error"
}
}
MCP servers 1
Sponsors
Details
- Marketplace
- testdriverai/testdriverai
- Source
- ai
- Category
- Community