Skip to main content

​
Sources are origins

A knowledge source is one origin: a paste of text, a document, a single page — or a whole website. Website sources keep their pages as entries beneath one source (GET /knowledge_sources/{id}/entries); every refresh of any kind writes an update run (GET /knowledge_sources/{id}/updates). You sync origins; pepline keeps the grain beneath them fresh.

​
The one sync pattern

Give every piece of content a stable external_id (your CMS page id, your database key — anything unique in your system) and POST it on every change:
  • First call → 201, source created and processed.
  • Content changed → 200, source updated and re-processed.
  • Nothing changed → 200, no re-processing (retries and cron re-runs are free).
Your system never needs to remember pepline ids, diff content, or track state. Fire and forget — idempotence is the API’s job. For byte-exact retry semantics (e.g. a queue worker that might redeliver), add an Idempotency-Key header: the same key replays the original response without re-executing for 24 hours; the same key with a different payload answers 409 idempotency_conflict. If you always send external_id, you rarely need it — upserts converge anyway.

​
Whole websites

pepline crawls the site from its navigation — a site’s own link structure is how it ranks its pages — and ingests up to max_pages of them. It also maps the rest of the site, so any pages the crawl didn’t reach surface in the dashboard, ready to add. Each ingested page becomes an entry under the one website source — page-level citations in agent replies, per-page failure visibility, one source row for the whole site. Follow along on the source itself:
status aggregates the entries: ready (everything read), updating (work in flight — poll), attention (usable, some pages failed — reasons under /entries), failed (nothing usable). The run’s debrief (pages discovered, discovery method, duration) lives under /updates.

​
Refresh: one verb for every kind

Re-fetches the page (url), re-discovers and re-fetches the site (website — entries upsert by page URL, so re-runs converge instead of duplicating), re-processes stored content (text/file). 202 — poll the source until it settles. Every refresh writes an update run.

​
Failures are loud

A page that fetched but yielded no readable text fails with failure_reason: "page has no readable content" — the page loaded but there was nothing to index (an empty or image-only page, or one that failed to render). The rest of the site is unaffected; fix the page, then refresh. Limits: 50 pages per website source, 5 website creations per hour per key, 2MB per page (10MB for PDFs).

​
Zapier / Make / n8n

Use the generic HTTP/webhook module:
  • URL: https://app.pepline.ai/api/v1/knowledge_sources
  • Method: POST · Auth: Bearer token (your pep_sk_… key)
  • Body: map external_id to the trigger record’s id, title and body to its fields, agents to your agent slugs.
That’s the entire connector.

​
Files and hosted documents

  • Hosted PDFs: just push their URL as kind: "url" — pepline fetches and extracts them.
  • Local files: kind: "file" with filename and strict-base64 data (txt, md, pdf, docx; 10MB max decoded).

​
Deletions

When content is deleted in your system, archive its source:
Find the id by your key first: GET /knowledge_sources?external_id=notion-8f3a. Archived sources leave the agents immediately but keep their history — and a later upsert with that external_id answers 409 until you restore it, so a resync can’t silently undo a deliberate removal.