AI-authored checks

Describe what you want to watch in plain words. An AI writes a JavaScript check script for it, tests it, and you save it. From then on the plain script runs on the workers like any other js check, with zero AI calls.

When the target changes its pages and the script breaks, SolidPing can propose a repair. It never repairs a real outage.

The feature is off unless the server has an AI provider (see Configuration). Without one, the "Describe it" button is hidden and the endpoints answer 404.

Describe it

On the checks page or the New check page, click Describe it.

1. Write what the check should watch, for example "log in to https://app.acme.com with the test account, the dashboard shows at least one project". Add plain parameters (env, e.g. BASE_URL) and secrets (secrets, e.g. PASSWORD) if the script needs them.
2. SolidPing proposes a contract: the ordered list of assertions the script will check. Edit it until it says what you want.
3. Generate. The AI explores the target (fetch_page, browser_snapshot), writes a script and runs it (run_script) until a run returns up, or the turn cap is hit. The page shows each step live: what the AI says, the pages it opens and the scripts it tests. You see the script and its last run. When no script passes, you see why: the last test run, the last script and the AI's last message.
4. Save. The check is a normal js check with an ai block in its config.

The AI may run probe scripts while it explores the target. A probe is never saved, even when it returns up: only the finished script, the one that implements every line of the contract, ends the generation. When the AI answers with a script it did not test, SolidPing tests it as the finished script. When it stops without one, it is sent back to work, up to twice.

Ask for a screenshot in the description ("then take a screenshot") and the script calls page.screenshot() at that point, and again before any down. The capture follows the js check rules: it is kept on down and timeout runs and on Capture now, and dropped on other up runs.

The AI never receives a secret value. It only sees the secret names, and the script reads them as secrets.NAME. A test run only gets the real values when every host the script contacts appears in your description or parameters. A run holding real values can only connect to those hosts, checked on every request, redirect and socket while it runs, so a URL the script builds at runtime cannot reach anywhere else. Such a run cannot open a browser or run a solidping.* sub-check: both connect outside that check. Secret values are scrubbed from everything handed back to the AI. The scrubbing matches the literal value, so a script that prints an encoded form (base64, URL-encoded) of a secret to the console would not be caught: never log secrets. Once a generation or repair has made one run with the real values, every later fetch_page, browser_snapshot and blank-secrets run of that session is held to the same hosts (no browser), so the AI has nowhere else to send what it read. A refused host built at runtime is not named in the error.

Edit and regenerate

The check's edit page shows its prompt, contract and repair mode at the top of the script section. Change the prompt or the contract, then click Regenerate the script: the AI writes and tests a new script, which replaces the one in the editor. The test run uses the check's stored secrets, unless you typed new ones in the form. Nothing changes until you save, and the previous script stays in the check's history. Saving without regenerating keeps the current script and only updates the prompt, contract or repair mode.

The ai block

type: js
config:
  script: |
    ...
  ai:
    prompt: "the acme api lists at least one project"
    contract:
      - "GET /api answers 200"
      - "the list has at least one project"
    model: glm-5.3-flash
    generated_at: "2026-10-03T10:00:00Z"
    repair: propose

Field	Meaning
prompt	Your description
contract	The assertions you confirmed. Required when prompt is set
model	The model that wrote the current script
generated_at	When it was written
repair	off, propose (default) or auto
repair_hosts, repair_grant	Written by repair: auto only: the hosts the applied version may reach (see below)

The script stays editable, exports with sp export and runs unchanged on cloud workers and private agents. Workers never see the provider or the key.

Saving a generated script records the version with origin ai_generate in the check history.

Drift-only repairs

Generated scripts tag their failures: output.failure = "assertion" when the service misbehaves, output.failure = "drift" when the script cannot find what it expected (a selector, a field, a page structure).

A repair attempt starts only when all of these hold:

• the failure is drift: a script exception, or output.failure == "drift". A clean down tagged assertion, a timeout or a connection error never qualifies;
• 3 consecutive failing runs;
• the target looks healthy: a GET on the script's base URL answers 2xx or 3xx;
• at most one attempt per check per 24 h, and at most 20 per organization per UTC day.

The AI gets the old script, the contract, the failure output and a fresh look at the target. Before a candidate is kept, mechanical guards run:

• it contacts no host the previous version did not. The text is checked first, then every run holding the secrets (test runs and the verification run) is held to the previous version's hosts while it runs, so no candidate sends the secrets to a new host, even through a URL built at runtime;
• it adds no try/catch;
• it can still report down, and uses no new protocol API;
• a verification run returns up.

With repair: propose, a candidate that passes is stored as a proposed version (origin ai_repair). The check stays down, the incident stays open. Approve or reject it from the check's History page, with the diff and a one-line reason. A candidate failing a guard is dropped.

With repair: auto, a candidate that passed every guard and its verification run is applied directly as a new version. The incident resolves on the next run, and the author of the replaced version gets an email with the diff. Undo it with a normal version restore. These fall back to a proposal instead:

• a candidate failing a guard or the verification;
• a candidate that can reach a host its text does not show: a network call whose URL or host is not a string literal or an env read (a variable, a string built from pieces, a template, eval, a sub-check), or more network calls than the previous version;
• a second repair within 24 h.

A version repair: auto applied can never reach, on schedule, a host the version before it could not. It stores those hosts in its ai block (repair_hosts, with repair_grant binding them to that script and env). Every scheduled run of that version is held to them, whether its credentials sit in secrets or come from a ${param:…} or ${env:…} reference. A connection to any other host fails the run with an error, and the check goes back to repair: propose, so the next repair needs a human. Editing the script or the env drops the grant: a version you wrote is not held.

Whether a run is held is decided on the stored script and env, before any ${param:…}, ${env:…} or ${var:…} reference is resolved. A reference does not lift the grant, and repair: auto proposes, never applies, a candidate that adds a reference the previous version did not have. A grant that no longer matches its script or env holds the run to no host at all.

Every attempt is on the check's timeline (check.ai_repair_attempted, check.ai_repair_proposed, check.ai_repair_applied, and check.ai_repair_revoked when a refused run sets the check back to propose).

Bring your own agent

The MCP server exposes the same probes through its probe tool (kind: script, fetch or browser), so any external agent can author js checks. It needs no AI provider on the server.

Configuration

Variable	Default	Description
SP_AI_PROVIDER	-	openai or anthropic. Empty = off
SP_AI_BASE_URL	driver default	Endpoint base URL
SP_AI_API_KEY	-	Provider key, server only
SP_AI_MODEL	-	Model ID, passed verbatim
SP_AI_MAX_TURNS	12	Agent loop cap
SP_AI_TIMEOUT	120s	Per-call timeout

The openai driver speaks Chat Completions (POST {base_url}/chat/completions with tools), which covers OpenAI, BytePlus ModelArk, Mistral, Groq, DeepSeek, Gemini's OpenAI endpoint, OpenRouter, LiteLLM, Ollama and vLLM. A thinking model's reasoning_content (DeepSeek, GLM) is sent back with its turn, as those APIs require when tools are in play. The anthropic driver uses the Messages API with prompt caching on the system prompt.

BytePlus ModelArk example:

SP_AI_PROVIDER=openai
SP_AI_BASE_URL=https://ark.ap-southeast.bytepluses.com/api/v3
SP_AI_MODEL=glm-5.3-flash
SP_AI_API_KEY=...          # activate the model in the Ark console first

Every LLM call is logged with the organization, the check, the purpose (contract, generate, repair) and the token usage, every tool call with its duration and outcome, and each call is recorded as a check.ai_usage event. On the SaaS, the maxAiTokensPerDay entitlement caps the tokens an organization spends per UTC day (429 once spent). Self-hosted is unlimited: your key pays.