Computer Controls

OS-level mouse, keyboard, and screen control for precise browser interaction.

When to Use

Computer controls provide OS-level interaction with the browser VM, operating at the system level rather than through browser APIs. Use computer controls when:

• Playwright/CDP isn't sufficient: When you need to interact with elements that are difficult to target via DOM selectors

• Testing real user behavior: Simulating actual mouse movements, clicks, and keyboard input as a human would perform them

• Interacting with browser UI: Clicking on browser chrome elements (address bar, extensions, menus) that aren't accessible via page automation

• Handling complex interactions: Drag-and-drop operations, hover effects, or interactions that require precise coordinate-based control

• Capturing visual output: Taking screenshots of the entire browser viewport or specific screen regions

• Working with canvas/WebGL: Interacting with canvas elements or games where DOM-based automation doesn't work

• Bypassing automation detection: Some anti-bot systems detect Playwright/CDP usage but not OS-level input

When NOT to use computer controls:

• For standard web automation (form filling, clicking buttons) - use kernel browsers playwright execute instead for better reliability and speed

• When you need to access page content or execute JavaScript - use Playwright execution

• For headless browsers - computer controls require a GUI environment

Prerequisites

Load the kernel-cli skill for Kernel CLI installation and authentication.

Screenshots

Full Screenshot

kernel browsers computer screenshot <session_id> --to screenshot.png

Region Screenshot

kernel browsers computer screenshot <session_id> --to region.png --x 0 --y 0 --width 800 --height 600

Mouse Actions

Click

Left click

kernel browsers computer click-mouse <session_id> --x 100 --y 200

Right click (double)

kernel browsers computer click-mouse <session_id> --x 100 --y 200 --button right --num-clicks 2

Move

kernel browsers computer move-mouse <session_id> --x 500 --y 300

Drag

kernel browsers computer drag-mouse <session_id> --point 100,200 --point 200,300 --button left

Scroll

kernel browsers computer scroll <session_id> --x 300 --y 400 --delta-y 120

Keyboard Actions

Type Text

Fast typing

kernel browsers computer type <session_id> --text "Hello, World!"

Slow typing (100ms delay between chars)

kernel browsers computer type <session_id> --text "Slow typing" --delay 100

Press Keys

Key names follow X11 keysym definitions. Common keys include: Return , Tab , Escape , BackSpace , Delete , Home , End , Page_Up , Page_Down , Up , Down , Left , Right , Shift_L , Control_L , Alt_L , etc.

Single key

kernel browsers computer press-key <session_id> --key Return

Key combination

kernel browsers computer press-key <session_id> --key Control_L+t

Complex combination with held keys

kernel browsers computer press-key <session_id> --key Control_L+Shift_L+Tab --hold-key Alt_L

Common Pattern: Navigate and Screenshot

SESSION=$(kernel browsers create -o json | jq -r '.session_id')

Navigate using Playwright

kernel browsers playwright execute $SESSION 'await page.goto("https://kernel.sh")'

Take screenshot

kernel browsers computer screenshot $SESSION --to kernel-homepage.png

Cleanup

kernel browsers delete $SESSION --yes

MCP Tool: Use kernel:execute_playwright_code and kernel:take_screenshot for playwright execution and screenshots.