Docs-as-tests validation

Documentation drifts out of date the moment the systems it describes change: a link rots, an API endpoint retires, a copied code sample no longer parses, a documented step stops working. Docs-as-tests treats your published documentation like a test suite and reports what no longer holds up.

Two systems carry that job, and they behave differently:

Both are configured in Settings > Connections, in the Built-in checks and CI test ingestion (Doc Detective) cards, and both turn what they find into fix-it items in your Demand worklist. Neither ever edits your content.

Docs-as-tests is available on every plan, including Free.

Built-in checks #

What is scanned #

Only topics that are published and not archived. A topic archived after publication is skipped, because it is no longer live for readers.

What each check does #

Each run performs four independent checks.

External links #

The scan requests each external link with HEAD first. When the server answers 401, 403, 404, 405, or 429, the scan retries with GET before deciding anything: many servers, CDNs, and bot walls refuse HEAD on URLs that GET serves normally. Only a link that still fails after that retry is flagged. Requests are batched with a delay between batches, and requests to private or internal network addresses are blocked.

The finding records the failing URL and the status code, or "unreachable" for a connection error or timeout.

Internal cross-references #

Every link to another topic is checked against the database, with no HTTP request involved. Two things count as broken:

API endpoints #

This check runs only on topics whose type is reference, typically API documentation, including topics imported from an OpenAPI spec. It finds full URLs containing an api or version segment such as /v1/ and probes each one exactly as written.

Two rules keep it quiet:

• Templated URLs are skipped. Anything carrying a placeholder such as {id}, <token>, :userId, or * is never probed, because a placeholder path 404s by design.

• Auth walls are not failures. Only a connection failure (API host unreachable) or a 404 or 410 (Documented API endpoint gone) raises a finding. A 401, 403, 405, or 429 means the endpoint still exists and merely refused an unauthenticated call, so nothing is reported.

Code samples #

Fenced code blocks with a declared language are parsed by a real parser, never by a pattern-matching guess.

The YAML check catches bad indentation, tab characters, duplicate keys, and broken flow syntax, and the finding quotes the parser's own error with its line and column.

Code blocks in any other language, or with no declared language, are not checked. XML and HTML validation was removed: it counted tags to guess whether they balanced, which flagged legitimate fragments and self-closing elements as errors.

When the built-in checks run #

• Weekly, Mondays at 03:00 UTC.

• On demand, when you click Run checks now in the Built-in checks card. The button queues a run and confirms with "Checks queued — new issues appear in your Demand worklist when the run finishes."

Each run reconciles the previous results rather than replacing them wholesale: a problem that is still there keeps its existing finding, and a problem that is fixed disappears on the next run.

Where built-in findings appear #

Every built-in finding is a warning severity, and it appears in three places:

• The editor's Findings panel for the affected topic

• The Content health tab of Analytics

• The Demand worklist

Each finding includes the offending value (the URL, the target topic, or a preview of the code) so you can locate and fix it without re-running the scan.

CI test ingestion (Doc Detective) #

Built-in checks test what your documentation references. Doc Detective tests what it instructs: it walks the documented steps in the live product and compares screenshots. Connect the CI that runs those tests, and a failure becomes a fix-it item on the topic that carries the broken instruction.

Connect a test runner #

1. Go to Settings > Connections.

2. In the CI test ingestion (Doc Detective) card, enter a Name for the pipeline, for example, "GitHub Actions".

3. Click Add test runner. The setup panel opens automatically, because setup is still required.

4. Copy the Secret. In your code repository, add it as an Actions secret named TOPICARY_DOCTEST_SECRET, under Settings > Secrets and variables > Actions.

5. Copy the Workflow file and commit it to .github/workflows/doc-tests.yml.

The generated workflow checks out the repository, runs npx doc-detective@latest runTests, and posts the report back to Topicary with an HMAC-SHA256 signature, the commit, the branch, the run URL, and up to 30 screenshots. It runs on every push to main and on a weekly schedule (Mondays, 06:00 UTC).

Read a runner's state #

Each runner row shows one status and its last result:

Below the name, the row reads either "No test runs yet" or a summary of the last run: passed, failed, and warning counts.

Use Secret, workflow & name to reopen the setup panel, rename the runner, or copy the secret and workflow again. Remove test runner deletes the connection.

Where CI failures appear #

CI failures do not go to the Findings panel, and they are not content findings. They appear in the Demand worklist.

Opening one from Demand opens the affected topic with a From Demand banner. Click What failed in that banner to expand the whole failure:

Mark fixed sits in the banner itself, so you can close the item without going back to the worklist.

See also #

• Content health indicators: the structural and freshness checks that run continuously

• Automated lifecycle recommendations: the nightly agent that flags stale, unverified, and duplicate topics

• Content statuses: only Published topics are scanned by the built-in checks