> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pepline.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Keeping knowledge fresh

> One source per origin — sync it with external_id upserts, bring in whole websites, refresh anything with one verb.

## Sources are origins

A knowledge source is one **origin**: a paste of text, a document, a single page — or a whole website. Website sources keep their pages as **entries** beneath one source (`GET /knowledge_sources/{id}/entries`); every refresh of any kind writes an **update run** (`GET /knowledge_sources/{id}/updates`). You sync origins; pepline keeps the grain beneath them fresh.

## The one sync pattern

Give every piece of content a stable `external_id` (your CMS page id, your database key — anything unique in your system) and `POST` it on every change:

```bash theme={null}
curl https://app.pepline.ai/api/v1/knowledge_sources \
  -H "Authorization: Bearer pep_sk_..." -H "Content-Type: application/json" \
  -d '{ "kind": "text", "external_id": "notion-8f3a", "title": "FAQ", "body": "...", "agents": ["intake"] }'
```

* First call → `201`, source created and processed.
* Content changed → `200`, source updated and re-processed.
* Nothing changed → `200`, no re-processing (retries and cron re-runs are free).

Your system never needs to remember pepline ids, diff content, or track state. Fire and forget — idempotence is the API's job.

For byte-exact retry semantics (e.g. a queue worker that might redeliver), add an `Idempotency-Key` header: the same key replays the original response without re-executing for 24 hours; the same key with a different payload answers `409 idempotency_conflict`. If you always send `external_id`, you rarely need it — upserts converge anyway.

## Whole websites

```bash theme={null}
curl https://app.pepline.ai/api/v1/knowledge_sources \
  -H "Authorization: Bearer pep_sk_..." -H "Content-Type: application/json" \
  -d '{ "kind": "website", "url": "https://fullpicture.studio", "max_pages": 30, "external_id": "www", "agents": ["intake"] }'
```

pepline crawls the site from its navigation — a site's own link structure is how it ranks its pages — and ingests up to `max_pages` of them. It also maps the rest of the site, so any pages the crawl didn't reach surface in the dashboard, ready to add. Each ingested page becomes an **entry under the one website source** — page-level citations in agent replies, per-page failure visibility, one source row for the whole site.

Follow along on the source itself:

```json theme={null}
{
  "object": "knowledge_source", "kind": "website", "status": "updating",
  "entries": { "total": 24, "ingested": 21, "pending": 2, "failed": 1 },
  "last_update_at": "2026-07-10T08:12:00Z"
}
```

`status` aggregates the entries: `ready` (everything read), `updating` (work in flight — poll), `attention` (usable, some pages failed — reasons under `/entries`), `failed` (nothing usable). The run's debrief (pages discovered, discovery method, duration) lives under `/updates`.

## Refresh: one verb for every kind

```bash theme={null}
curl -X POST https://app.pepline.ai/api/v1/knowledge_sources/42/refresh -H "..."
```

Re-fetches the page (`url`), re-discovers and re-fetches the site (`website` — entries upsert by page URL, so re-runs converge instead of duplicating), re-processes stored content (`text`/`file`). `202` — poll the source until it settles. Every refresh writes an update run.

## Failures are loud

A page that fetched but yielded no readable text fails with `failure_reason: "page has no readable content"` — the page loaded but there was nothing to index (an empty or image-only page, or one that failed to render). The rest of the site is unaffected; fix the page, then refresh.

Limits: 50 pages per website source, 5 website creations per hour per key, 2MB per page (10MB for PDFs).

## Zapier / Make / n8n

Use the generic HTTP/webhook module:

* **URL**: `https://app.pepline.ai/api/v1/knowledge_sources`
* **Method**: POST · **Auth**: Bearer token (your `pep_sk_…` key)
* **Body**: map `external_id` to the trigger record's id, `title` and `body` to its fields, `agents` to your agent slugs.

That's the entire connector.

## Files and hosted documents

* Hosted PDFs: just push their URL as `kind: "url"` — pepline fetches and extracts them.
* Local files: `kind: "file"` with `filename` and strict-base64 `data` (txt, md, pdf, docx; 10MB max decoded).

## Deletions

When content is deleted in your system, archive its source:

```bash theme={null}
curl -X POST https://app.pepline.ai/api/v1/knowledge_sources/42/archive -H "..."
```

Find the id by your key first: `GET /knowledge_sources?external_id=notion-8f3a`. Archived sources leave the agents immediately but keep their history — and a later upsert with that `external_id` answers `409` until you restore it, so a resync can't silently undo a deliberate removal.