> For the complete documentation index, see [llms.txt](https://k-ai.gitbook.io/knowledge-ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://k-ai.gitbook.io/knowledge-ai/sources-and-ingestion/connectors/web-crawler.md).

# Web crawler

K-AI ingests public web pages via the **K-AI Web Crawler** service — a headless-browser crawler that fetches pages, classifies them, extracts structured content, and detects changes via a hash diff on the extracted content. The crawler is used for ingesting public marketing sites, regulatory portals, supplier documentation, and similar URL-addressable corpora.

URL bases are managed in the platform application at `app.kai-studio.ai`, under **Govern → Web crawler** — the same portal as the rest of the Govern surface, scoped to the selected organization. K-AI exposes the crawler as a connector, and the Instance pulls extracted content via the crawler's read-only API.

{% hint style="info" %}
The web crawler previously had its own separate admin interface. It is now part of the platform application — same account, same organization switcher, same roles. Existing URL bases, their URLs, and their API keys are unchanged.
{% endhint %}

## Architecture

The web crawler is a managed service: URL bases are administered from the **Govern → Web crawler** screens of the platform portal, and a read-only API serves the extracted content the K-AI Instance pulls from. Workers are stateless and scale horizontally. The K-AI Instance treats the crawler as a source through the generic ingestion pathway: the connector pulls successful URLs via the crawler's public API and hands them to the [indexation pipeline](/knowledge-ai/sources-and-ingestion/indexation-pipeline.md).

### Multi-stage extraction

Each URL is loaded in a headless browser, classified (CAPTCHA / paywall / login-wall pages exit early), cleaned of boilerplate (cookie banners, navigation, footers, ads), and converted to structured Markdown chunks with breadcrumb context preserved. A quality-scoring pass filters out boilerplate and applies selective LLM refinement to weak chunks. Change detection uses a content hash over the structured output, so heading or page-title changes trigger a re-index while ads and timestamps are ignored.

## Authentication

Two layers:

* **Source-side** (between crawler and the web) — anonymous for public web pages. Basic auth and cookie headers are supported for protected sites; configured per URL base.
* **K-AI side** (between Instance and crawler) — one API key per URL base, generated from the **Public API access** card on the URL base page (**Govern → Web crawler** → base). The raw key is displayed once at generation; it can be regenerated or revoked at any time, and revocation is immediate. The K-AI Instance authenticates with an `X-API-Key` header. Generating, regenerating, and revoking that key is restricted to **Org Owner** and **Administrator** — see [Access control](#access-control) below.

## Configuration

| Field     | Type   | Required | Description                                                                                      |
| --------- | ------ | -------- | ------------------------------------------------------------------------------------------------ |
| `api_key` | string | yes      | Public API key issued by the crawler for a specific URL base. Encrypted at rest by the platform. |

The URL base itself — its URL list and its re-crawl interval — is configured under **Govern → Web crawler**, not in the K-AI Instance. There is no allow-list, deny-list, or crawl depth: each URL is crawled in isolation, so the set of pages to crawl is exactly the list you enter.

Fields available on a URL base:

| Field                    | Description                                                                                  |
| ------------------------ | -------------------------------------------------------------------------------------------- |
| `name`                   | URL base name (organization-scoped).                                                         |
| `description`            | Free-text description.                                                                       |
| `urls`                   | List of URLs to crawl. Importable via CSV. Each URL is crawled independently — no recursion. |
| `recrawl_interval_hours` | Cadence for re-crawling each URL. Range: 3–720.                                              |

## Access control

Every URL base belongs to one organization. Access is granted by the organization roles described in [Access & roles](/knowledge-ai/the-k-ai-platform/access-control.md), and enforced on the API as well as in the portal — a URL base of another organization is not readable, even by a user who is a member of both.

| Action                                                                     | Roles                                                      |
| -------------------------------------------------------------------------- | ---------------------------------------------------------- |
| View URL bases, crawl results, and extracted content                       | Org Owner, Administrator, Data Authority, Document Steward |
| Create, edit, or delete a URL base; add or remove URLs; trigger a re-crawl | Org Owner, Administrator, Data Authority, Document Steward |
| Generate, regenerate, or revoke the URL base API key                       | Org Owner, Administrator                                   |

The API key lifecycle sits on a narrower bar than the crawl operations next to it: the key is a long-lived bearer credential onto the base's extracted content, so it is held to the same standard as [Organization API keys](/knowledge-ai/authentication/organization-api-keys.md).

## Document types ingested

Each URL produces a single document containing the extracted text content. Native MIME type ingested:

* HTML pages (the primary case).
* PDFs linked from a page — fetched and routed through the [indexation pipeline](/knowledge-ai/sources-and-ingestion/indexation-pipeline.md). PDFs hosted directly at a URL (not behind an HTML wrapper) follow the same path.

Images, video, and binary media are not ingested by the crawler; references are recorded as metadata pointers.

## Sync mode

Scheduled re-crawl. Each URL has its own `recrawl_interval_hours` and is re-fetched independently. Workers pick up URLs whose last successful crawl is older than the interval.

Change detection is a content hash on the structured output. The K-AI Instance treats a hash change as a content modification and re-runs the [indexation pipeline](/knowledge-ai/sources-and-ingestion/indexation-pipeline.md). Unchanged content is a no-op.

Tombstone logic: a URL that returns `404` for several consecutive crawls is flipped to `status='gone'` and removed from the public API. A subsequent `200` brings it back automatically. RFC 9110 `410 Gone` tombstones immediately.

## ACL handling

Web pages have no native ACL semantics — they are public by definition. The connector applies the configured **fallback group** at registration time (typically a `public_web` group readable by everyone, or a more restrictive group when the customer wants to gate access to ingested web content). The mapping is one-shot at the source level and identical for every document under the URL base.

For sites behind basic auth or cookie sessions, the same fallback rule applies; access control is enforced upstream by the source site, and the K-AI Instance simply mirrors the resulting allowlist.

## Rate limits

Worker concurrency, per-host politeness delay, and page size cap are configurable per deployment. Pages exceeding the size cap are flagged in the crawl result and not extracted.

## Known limitations

* **JavaScript-heavy SPAs** may need extended wait windows; the adaptive wait covers most cases but some apps require manual tuning per URL base.
* **CAPTCHA / WAF** challenges (DataDome, Cloudflare, hCaptcha) cause the page to be flagged at the detect stage and exit before extraction. No CAPTCHA-solving is performed.
* **Deep recursive crawls** are not supported — each URL is crawled in isolation. To ingest a site tree, the URLs must be enumerated upfront (manual entry or CSV import).
* **Dynamic per-user content** is not supported; the crawler authenticates as one identity and ingests the resulting view.
* **LLM-refined chunks** are recorded as such in document metadata for downstream audit.
* **Cost** — the refine stage triggers LLM calls, billed per organization (see [Monitoring](/knowledge-ai/operate/monitoring.md#cost-events)). High-volume bases should be sized accordingly.

## Setup walkthrough

1. **Sign in** to `https://app.kai-studio.ai`, select the target organization, and open **Govern → Web crawler**.
2. **Create a URL base**: name, description, recrawl interval. Add URLs one at a time, or import a list from a CSV / text file (one URL per line).
3. **Wait for the initial crawl** to complete (URLs are picked up on the next worker poll). Open a URL from the base to inspect its crawl result and extracted content, and confirm extraction quality.
4. **Generate the API key** from the **Public API access** card on the URL base page (one key per base; regenerable, revocable). Capture the key — it is displayed once at generation. Requires Org Owner or Administrator.
5. **Add the source** under **Govern → Sources**: select Web crawler, paste the key. The connector resolves the URL base and verifies the credential before the source is saved.
6. **Trigger a test sync** from the K-AI Instance. Confirm the document count in the indexing statistics of the instance page in the K-AI Studio portal.