// automation

Data pipelines

A data pipeline runs one cheap structured call per item instead of a whole agent session per row. When the work is uniform and the list is thousands long, that difference is the whole point: same classification, a fraction of the spend, and a run log that tells you which rows failed and why.

one call per itempush-fed inboxdata jobs module

01// what a data job is

A recipe, an inbox and a drain

A pipeline is not agentic. It is a recipe that reads a batch of items from stdin and prints one result per item, run over and over as producers push work onto its inbox. Use it to qualify, classify, enrich or extract across many items. Reach for an agent instead when each item needs judgement, tools and a conversation.

the recipe contractjson

// stdin — one batch
{ "items": [ { "id": "itm_1", "payload": { "row_id": "..." } } ], "params": {} }

// stdout — one NDJSON line per item, written as that item settles
{ "item": { "id": "itm_1", "status": "done", "result": { "tier": "A" } } }
{ "item": { "id": "itm_2", "status": "failed", "error": "no matching row" } }

Two knobs control throughput: batch size (how many items are drained into one sandbox) and concurrency (how many batches run at once). The recipe picks a model per call, defaulting to the cheapest one, and can be given a per-job tool allowlist and a write-back table so results land in the workspace database without a second pass.

02// creating and running

Validate the recipe, then feed it

01
Dry-run the recipe
Pipelines are created from a validated code id, never from pasted source. Run the recipe through the TypeScript dry run first: a green run returns the id, and create and update resolve the validated bytes by that id. A model that re-emits source corrupts it, which is exactly what this closes off.
02
Create the pipeline
Name it, hand it the validated code id, and set the model, batch size and concurrency if the defaults do not fit. Runs bill credits, so confirm the scope before you create something that will chew through a table.
03
Push items onto it
Submitting is the primary way work arrives: up to 1000 payloads per call, rejected outright if the pipeline is paused or archived. To seed from rows already in the workspace database, backfill from a table with filters or from a read-only query. Backfill is capped at 5000 rows and tells you when it truncated, so narrow the filter and call again.
04
Watch the first run
Read the run back before you push the rest. A recipe that is wrong on item one is wrong on item ten thousand, and the items are already billed by then.

agnt_data_job_* from an MCP clienttext

// 1. validate the recipe first: create takes the validated id, never source
agnt_typescript_skill_dry_run({ /* … */ })   // → validated_code_id

// 2. create the pipeline
agnt_data_job_create({
  name: "Qualify new leads",
  validated_code_id: "<from the green dry run>",
  model: "haiku",
  write_back: { table: "leads", column: "qualification" }
})

// 3. push work onto it
agnt_data_job_submit({
  data_job_id: "<pipeline id>",
  payloads: [{ row_id: "lead_1" }, { row_id: "lead_2" }]
})

// or seed it from rows already in the workspace database
agnt_data_job_backfill({
  data_job_id: "<pipeline id>",
  table: "leads",
  filters: ["status.eq.new"]
})

Alongside those, agnt_data_job_list returns every pipeline with its latest run and live inbox health, and agnt_data_job_read returns one pipeline with its full, untruncated recipe source, agnt_data_job_update swaps in a new validated recipe or moves the status, and agnt_data_job_inspect_run is the debugging read. Deleting cascades to every run and every queued item and cannot be undone, so pause when you only want it to stop.

In the dashboard, the Data Pipelines page lists each pipeline with a run button, a pause and resume control, and a delete that confirms first.

03// runs and items

Two clocks, read both

A run is one pass of the drain; an item is one unit of work waiting in the inbox. They fail independently, and the trap is reading only the first: every run can read completed while a wall of items sits pending because nothing is draining them. The pipeline reports that case explicitly as a stalled queue with a reason.

Run status	Meaning
`queued`	Accepted, not started.
`running`	Batches in flight.
`completed`	Every item in the run finished.
`partial`	The run finished with some items failed. Worth a look even though it is not a failure.
`failed`	The run itself broke.

Item status	Meaning
`pending`	In the inbox, waiting for a drain.
`processing`	Claimed by a batch.
`done`	Processed successfully.
`failed`	Retried a few times, then given up on. Keeps its error.

When a run goes wrong, the inspect read is the one call that explains it: the run itself, the recipe’s stdout and stderr, the failed items with their payloads, and a whole-pipeline rollup that groups failures by error class so one read explains a failure spread across thousands of rows.

Spend is reported per pipeline and split two ways: AI credits for the structured calls the recipe makes, and data credits for any data source it reaches. The dashboard shows both alongside rows processed and the failure count, which is how you find the pipeline that is cheap per item and expensive in aggregate.

04// feeding a pipeline from outside

Bind an inbound webhook to it

An inbound webhook endpoint can target a pipeline instead of an agent. Every delivery then enqueues exactly one item with the raw JSON body as its payload, and the pipeline drains it like anything else you pushed. That is the right shape for a SaaS that fires one webhook per record: high volume, nothing conversational about it, and no agent session per event.

Bind it at creation time or afterwards with agnt_webhooks_link_data_job. An endpoint targets a pipeline or an agent, never both, so binding one clears the other. See Receiving webhooks for the ingest contract and the signature scheme.

a paused pipeline still receives

While the pipeline is paused or archived the delivery is still recorded, but the enqueue is rejected and the delivery is marked failed. Pausing a pipeline does not pause its producers, so check the delivery log after you resume.

05// requirements and pricing

The Data Jobs module

Async bulk pipelines — one cheap structured call per item instead of an agent session each. It requires the Database & Canvas module, which is added automatically when you take it: pipelines read and write the workspace database, so there is nothing to run without one. 3 rungs, each with a 7-day trial.

Tier	Price	Includes
starter	$29/mo	3 active pipelines · 10,000 items / mo
growth	$59/mo	15 active pipelines · 75,000 items / mo
scale	$119/mo	50 active pipelines · 400,000 items / mo

allowances, not caps

Module tier quantities are published allowances rather than enforced ceilings. The subscription buys the capability; the per-item AI credits and any data-source credits a recipe spends are billed separately against the workspace balance.

06// related

automation