Configuring Exports

An export is the data source in a Celigo integration. It connects to an external system and pulls data into the pipeline. Exports serve two roles:

Source -- the starting point that fetches the primary batch of records
Lookup -- a mid-flow enrichment step (
```
isLookup: true
```
) that fetches additional data per-record during processing

Both roles are used across flows, APIs, and tools.

Beyond fetching data, exports also handle post-retrieval processing before records enter the pipeline:

Output filter -- expression-based filtering to skip records that don't match criteria
Transform -- Transformation 2.0 expression rules to reshape/flatten response data before mapping
preSavePage hook -- JavaScript processing on the full page of records before they enter the pipeline
One-to-many -- when used as a lookup, fan out child records from a parent. Set
```
oneToMany: true
```
and
```
pathToMany
```
to the child array path so each child triggers a separate lookup. Once fanned out, the array element itself is the record -- see One-to-many fan-out
Response mapping -- when used as a lookup, extract fields from the lookup response back into the record. Configured on the flow's
```
pageProcessors[]
```
entry, but planned when building the lookup export. The response contains a
```
data
```
array and an
```
errors
```
array. Use
```
data[0].fieldName
```
when you expect a single result (e.g., fetching one order by ID); use
```
data[*].fieldName
```
when multiple results are expected. Response mapping uses Transformation 1.0 syntax (extract/generate pairs), not the newer expression-based transforms
postResponseMap hook -- JavaScript processing after response mapping merges the lookup response back into the record. Configured on the flow's
```
pageProcessors[]
```
entry, but planned when building the lookup export

Export Execution Pipeline

When a flow runs, each export executes this pipeline in strict order:

API request / query / file read -- fetches raw data from the external system
Response parsing --
```
resourcePath
```
extracts the record array from the response body or file (e.g.,
```
http.response.resourcePath
```
for HTTP,
```
file.json.resourcePath
```
for JSON files, XPath for XML)
Transformation (optional) --
```
transform
```
reshapes individual records after extraction (Transformation 2.0)
Output filter (optional) -- discards records that don't match filter expression rules
preSavePage hook (optional) -- JavaScript processing on the full page of records

Key distinction:

resourcePath

tells the export WHERE to find records in the response. Transforms reshape WHAT each record looks like after extraction. When a user says "extract records from X" or "treat each X as a separate record", that's almost always a

resourcePath

change, not a transform. Use transforms when you need to flatten nested objects, rename fields, or restructure individual records.

Three Categories of Export

Not all exports work the same way. Before building, understand which category you need:

Listeners

Receive data pushed to Celigo from an external system. No polling, no scheduling -- the source system sends data when events happen.

```
WebhookExport
```
-- inbound HTTP listener (no connection required)
```
AS2Export
```
-- AS2 EDI file reception
Distributed exports (
```
type: "distributed"
```
) -- real-time event-driven push for NetSuite (via SuiteScript) and Salesforce (via streaming API). The platform installs listeners in the source system that fire when records change.
Change data capture (
```
type: "stream"
```
) -- MongoDB change streams that tail the oplog for real-time record changes.

When to use: The source system supports outbound webhooks, push notifications, or change data capture and you want real-time processing.

File Transfers

Read files from a remote location, then either parse them into records or transfer them as blobs.

```
FTPExport
```
/
```
S3Export
```
/
```
FileSystemExport
```
-- fetch files from FTP/SFTP, S3, or local filesystem
```
HTTPExport
```
with
```
http.type: "file"
```
-- fetch files over HTTP from cloud storage APIs (Google Drive, Box, Dropbox, Azure Blob Storage). The HTTP connector handles auth; the
```
file{}
```
config handles parsing.
```
NetSuiteExport
```
with
```
netsuite.type: "file"
```
-- fetch and parse files (CSV, JSON, XLSX, XML, EDI) from the NetSuite file cabinet
Parsed mode (
```
file.output: "records"
```
) -- CSV, XML, JSON, XLSX, EDI files are parsed into individual records
Blob mode (
```
type: "blob"
```
) -- binary files transferred as-is without parsing. Supported on HTTPExport, NetSuiteExport, SalesforceExport, FTPExport, and S3Export.

When to use: The source system drops files (CSV, EDI, XML, etc.) into a directory, bucket, file cabinet, or cloud storage rather than exposing a record-based API.

Record-Based Exports

Actively fetch batches of records from an API or database on a schedule.

```
HTTPExport
```
-- REST/GraphQL APIs
```
NetSuiteExport
```
-- saved searches, restlets, SuiteQL
```
SalesforceExport
```
-- SOQL/Bulk queries
```
RDBMSExport
```
-- SQL SELECT queries
```
MongodbExport
```
,
```
JDBCExport
```
,
```
DynamodbExport
```
-- other databases
```
WrapperExport
```
-- custom stack (Walmart, BigCommerce)

When to use: You need to poll an API or query a database for records on a schedule (full fetch or delta/incremental).

Quick Reference

Adaptor Decision Matrix

Your data comes from...	Use adaptorType	Category	Read schema
REST or GraphQL API	`HTTPExport`	Record-based	http.yml
Files over HTTP (Google Drive, Box, Dropbox, Azure Blob)	`HTTPExport` with `http.type: "file"`	File transfer	http.yml + file.yml
NetSuite (any method)	`NetSuiteExport`	Record-based	netsuite.yml
Salesforce objects	`SalesforceExport`	Record-based	salesforce.yml
SQL database	`RDBMSExport`	Record-based	rdbms.yml
MongoDB	`MongodbExport`	Record-based	mongodb.yml
JDBC database	`JDBCExport`	Record-based	jdbc.yml
DynamoDB	`DynamodbExport`	Record-based	dynamodb.yml
Files on FTP/SFTP	`FTPExport`	File transfer	ftp.yml + file.yml
Files on S3	`S3Export`	File transfer	s3.yml + file.yml
Webhooks / push events	`WebhookExport`	Listener	webhook.yml
AS2 EDI messages	`AS2Export`	Listener	as2.yml
Manual file upload	`SimpleExport`	File transfer	simple.yml
Local filesystem	`FileSystemExport`	File transfer	filesystem.yml + file.yml
Pre-built stack connector	`WrapperExport`	Record-based	wrapper.yml

Raw HTTP is the fallback, not the default. Pick the most specific match, in order:

Native adaptor -- if the application has its own row (NetSuite, Salesforce, databases, FTP/S3), use it. Do not build an
```
HTTPExport
```
against that app's REST API.
Pre-built HTTP connector -- for any other REST/GraphQL app, check the 550+ connector catalog before writing HTTP config (see Check for a pre-built connector). The step is still an
```
HTTPExport
```
, but it runs on a connector-backed connection and takes its endpoint config from the connector.
Manual HTTP -- hand-write the config from public API docs only when no connector exists or it doesn't cover the endpoint you need.

adaptorType

is case-sensitive:

HTTPExport

, not

httpExport

Minimum Required Fields

Every export needs at minimum:

```
name
```
-- human-readable label
```
adaptorType
```
-- from the matrix above
```
_connectionId
```
-- except
```
WebhookExport
```
and
```
SimpleExport
```

Adaptor config block --

http{}

netsuite{}

ftp{}

salesforce{}

rdbms{}

, etc.

Which Schemas to Read

Always: request.yml (base fields for all exports)
Plus: the adaptor-specific file from the matrix above (e.g.,
```
http.yml
```
for HTTPExport)
If file-based: also file.yml (CSV, XML, JSON, XLSX, EDI parsing config)
If delta/incremental: check delta.yml or Handlebars URI pattern (
```
{{{lastExportDateTime}}}
```
)
If cloning: clone-request.yml, clone-response.yml

Schema Index

All schemas are in references/schemas/:

Base fields (all exports): request.yml
Response shape: response.yml
Adaptor-specific config:
- http.yml -- HTTP/REST/GraphQL
- netsuite.yml -- NetSuite (restlet, saved search, SuiteQL, file cabinet)
- salesforce.yml -- Salesforce (SOQL, bulk)
- ftp.yml -- FTP/SFTP
- s3.yml -- Amazon S3
- rdbms.yml -- SQL databases
- mongodb.yml -- MongoDB
- jdbc.yml -- JDBC databases
- dynamodb.yml -- DynamoDB
- as2.yml -- AS2 EDI
- wrapper.yml -- custom stack connectors
- filesystem.yml -- local filesystem
- simple.yml -- data loader / manual upload
File parsing: file.yml (CSV, XML, JSON, XLSX, EDI)
Operational modes: delta.yml, webhook.yml, distributed.yml, once.yml
Mock output: mock-output.yml
Clone: clone-request.yml, clone-response.yml

Related Skills

configuring-connections > Quick Reference -- connection types, auth methods, iClients
writing-mappings > Transformation 2.0 -- reshape export output before mapping
writing-scripts > Data Pipeline Hooks -- preSavePage, postResponseMap hooks
writing-handlebars > Quick Reference -- dynamic values in URIs, filters, delta tokens
building-flows > How to Build a Flow -- wiring exports into flows
troubleshooting-flows > Diagnostic Workflow -- diagnosing export-related failures

How to Build an Export

1. Identify the target application

What system are you pulling data from? This determines everything -- adaptor type, connection type, and configuration shape.

2. Check for existing patterns

Before building from scratch, look at what already exists:

bash

# Search across the entire account for related resources
celigo account search "<keyword>"

# Show what an existing export uses (connection) and what uses it (flows)
celigo account dependencies export <id>

# Find orphaned exports not referenced by any flow
celigo account lint

# Check if a similar export already exists in the account
celigo exports list | grep -i "<application-name>"

# Search the marketplace for pre-built integration templates
celigo templates marketplace

# Preview a template to see its export configuration
celigo templates preview <id> --model Export
celigo templates preview <id> --summary

The account index auto-refreshes when stale (>4 hours). Force a fresh snapshot with

celigo account snapshot

Existing exports in the account are the best reference -- they show proven patterns for that specific customer's setup. Marketplace templates may provide a complete pre-built integration you can install rather than building from scratch.

3. Check for a pre-built connector

Always run this check before writing any HTTP config. Celigo maintains 550+ HTTP connector definitions and 590+ trading partner connectors. These provide pre-configured auth, base URLs, and endpoint definitions for common applications. Connectors are set on the connection, not the export -- but they determine what the export can do. Hand-write a manual

HTTPExport

from public API docs only when this search comes up empty or the connector doesn't cover the endpoint you need.

bash

# Search HTTP connectors (REST APIs: Shopify, Stripe, HubSpot, etc.)
celigo http-connectors list | grep -i "<application-name>"
celigo http-connectors get <id> --full    # see endpoints, resources, auth config

# Drill into the endpoints the connector defines for exports
celigo http-connectors catalog <id> --resource-type export --published-only
celigo http-connectors endpoint-detail <id> --resource-type export --resource-id <rid> --endpoint-id <epid>

# Search trading partner connectors (EDI, AS2, VAN)
celigo tp-connectors list

If an HTTP connector exists for your target app, create the connection from it (

http._httpConnectorId

-- see configuring-connections > Check for a pre-built connector and global iClient) and take the export's

relativeURI

, method, pagination, and response paths from the connector's endpoint metadata rather than reconstructing them from public API docs. The connector-reference fields on the export itself (

http._httpConnectorEndpointId

http._httpConnectorVersionId

http._httpConnectorResourceId

) are read-only -- the platform sets them; what you control is the connection and the endpoint config you copy from the connector.

If a trading partner connector exists (EDI/AS2), reference it on the export via

ftp._tpConnectorId

(FTP exports) or

as2._tpConnectorId

(AS2 exports). You may also need to set

_ediProfileId

on the export for EDI document validation.

4. Query metadata for the target system

For NetSuite, Salesforce, and RDBMS connections, you can discover available record types and fields directly from the live system:

bash

# List available record types / sObjects / tables
# NetSuite also returns saved searches alongside record types
celigo metadata types <connectionId>

# List fields for a specific entity type
celigo metadata fields <connectionId> <entityType>

This tells you what data is available to export before you write any configuration.

NetSuite:
```
metadata types
```
returns both record types and saved searches (with IDs you need for
```
netsuite.restlet.searchId
```
).
```
metadata fields
```
returns field IDs, names, types, and group — including sublist fields you'll need for
```
mapping.lists[].generate
```
on the import side.
Salesforce:
```
metadata types
```
returns sObjects with queryable/createable flags.
```
metadata fields
```
returns fields, types, and relationship names — use these to discover child objects for
```
distributed.relatedLists[]
```
and relationship field names for cross-object queries.
RDBMS:
```
metadata types
```
returns table names.
```
metadata fields
```
returns column names and types for a given table — use these when writing SQL queries or building field mappings.

5. Determine the category

Is this a listener (real-time push from the source), a file transfer (fetch and parse/transfer files), or a record-based export (poll an API or query a database)? This narrows which adaptor types and modes apply.

6. Choose the right adaptor type

Use the Adaptor Decision Matrix in Quick Reference above to select the correct

adaptorType

for your target system.

7. Build the export JSON

Use the Schema Index and Which Schemas to Read in Quick Reference above. Read

request.yml

for base fields, then the adaptor-specific schema, plus

file.yml

if file-based and

delta.yml

if incremental.

Export Design Decisions

A few design choices recur when building exports. Each has a defensible default once the framing is clear.

Delta vs one-time vs full sync

The export's

type

field selects the sync behavior:

Delta (
```
type: "delta"
```
) -- pulls only records created or modified since the last successful run. The default for ongoing scheduled syncs when the source exposes a usable "last modified" timestamp. Non-HTTP adaptors set the timestamp field via
```
delta.dateField
```
; HTTP exports instead embed
```
{{{lastExportDateTime}}}
```
in the
```
relativeURI
```
or body. See delta.yml.
One-time (
```
type: "once"
```
) -- processes each record exactly once via a tracking flag: each run selects records where
```
once.booleanField
```
is
```
false
```
, then sets it to
```
true
```
after a page succeeds so later runs skip them. Use for backfills and migrations, or when the source has no reliable timestamp but its records can carry a processed flag. See once.yml.
Full (neither
```
delta
```
nor
```
once
```
mode) -- re-pulls the entire dataset every run. Use when the source has no usable modification timestamp, the dataset is small enough that re-pulling is cheap, or business logic requires a fresh snapshot each run.

When the request is vague ("sync customers"), confirm which kind of sync is intended before building. Delta is a reasonable default when the source exposes a timestamp field; full is reasonable for small static datasets.

Listener/webhook vs scheduled export

Both are starting steps (see Three Categories of Export); the choice is driven by what the source supports and the latency budget, not preference:

Reach for a listener (
```
WebhookExport
```
, or NetSuite/Salesforce
```
type: "distributed"
```
) when the source pushes events and the flow needs to react quickly ("when X happens, do Y").
Reach for a scheduled export when the source has no push mechanism, or when batch timing at off-peak hours is acceptable.

NetSuite and Salesforce support both for many record types. Mixing them on one flow is a common, good pattern -- a listener handles low-latency reactions while a scheduled export runs as a safety net for backfills, end-of-day reconciliation, and catching up after a webhook outage.

Lookup export vs separate scheduled export

The distinguishing question is when the data is needed:

A lookup export (
```
isLookup: true
```
) runs per in-flight record, mid-pipeline, keyed off the upstream record -- fetching the customer for a specific order, or inventory for a specific SKU.
A scheduled export runs once per flow run as a starting point, producing the first batch of records the flow processes.

If the request is "for each X, look up Y", it's a lookup. If it's "every hour, pull all Y", it's a scheduled export.

One-to-many fan-out -- the array element IS the record

With

oneToMany: true

and

pathToMany

set to a child array path, each element of that array triggers its own lookup. Once fanned out, the element becomes the record: templates reference the element's own fields as

{{record.variantId}}

-- not

{{variantId}}

, and not

{{record.lineItems.variantId}}

. The array wrapper is gone; you are inside one element.

Three consequences worth knowing before you debug the wrong thing:

The build-time preview warning is expected. Previewing a fanned-out lookup in isolation reports "
```
<field>
```
not defined in the model" because no upstream record is bound yet. That is not a broken template -- don't "fix" a correct
```
{{record.X}}
```
reference because of it.
Response mapping merges per element automatically. To get looked-up values back onto each element, author a normal top-level
```
fields
```
response mapping; Celigo merges each result into its corresponding fanned-out element.
Two anti-patterns. Don't target the array with a
```
lists
```
entry (that nests a new array inside each element), and don't attempt the per-element merge in
```
postResponseMap
```
(it sees the page of parent records, not per-element results).

Source-side transform vs destination-side mapping

Both reshape data, but in opposite directions:

A transform on a source export reshapes records as they enter the flow -- flattening nested responses, or aligning multiple sources to a common shape (see Export Execution Pipeline).
A mapping on a downstream import reshapes records as they leave the flow toward a destination.

Don't add a transform to "match a destination" -- that's the destination import mapping's job. Transforms are for entry reshaping; mappings are for exit reshaping.

Async APIs (submit, poll, fetch)

Most APIs return data in the same call and need none of this. Some APIs only acknowledge a request (an HTTP 202, a job ticket, a feed or document id) and process it in the background -- Amazon SP-API feeds, large report generators, bulk extract and file-conversion jobs. For those, attach an async helper to the export via

http._asyncHelperId

. The helper teaches the step the submit-poll-fetch pattern; it is part of the export, not something managed on its own, and bundles three pieces:

A status export (required) -- run on each poll to ask "is it done yet?". Configure the status path to read in the response, the case-sensitive in-progress / done / done-without-data / error value lists (taken from the API's docs), and the initial wait and poll wait intervals in minutes.
A result export (optional, usually present) -- fetches the final payload once status reports done.
Initial-submission handling -- where to find the job ticket in the first acknowledgement: "same as status" when the acknowledgement is itself shaped like a status response, otherwise a resource path (plus transform rules for non-JSON acknowledgements, e.g. Amazon's XML).

Two constraints shape the design: the status and result exports must be ordinary synchronous exports (an async helper cannot nest another), and the async-configured step cannot carry its own transform, output filter, or preSavePage hook -- put any reshaping or filtering on the dedicated result export instead. The same pattern applies symmetrically to imports writing to asynchronous destinations (

_asyncHelperId

on the import).

Reach for an async helper only when the API genuinely forces the fire-and-check-back shape. Adding one to a synchronous API is pure overhead -- extra polling plus a status and result export to maintain.

CLI Commands

bash

# CRUD
celigo exports list
celigo exports get <id>
celigo exports create < export.json
celigo exports update <id> < export.json
celigo exports set <id> key=value [key2=value2 ...]
celigo exports delete <id>

# Invoke (test-run an export, see what data comes back)
celigo exports invoke [id] [--all]

# Clone and connection management
echo '{"connectionMap":{"oldConnId":"newConnId"}}' | celigo exports clone <id>
celigo exports replace-connection <id> <newConnectionId>

# Discovery
celigo account search "<keyword>"
celigo templates marketplace
celigo templates preview <id> --model Export
celigo templates preview <id> --summary
celigo http-connectors list
celigo http-connectors catalog <id> --resource-type export --published-only
celigo http-connectors endpoint-detail <id> --resource-type export --resource-id <rid> --endpoint-id <epid>
celigo tp-connectors list
celigo metadata types <connectionId>
celigo metadata fields <connectionId> <entityType>

# Debug
celigo exports enable-debug <id> [--duration <minutes>]
celigo exports disable-debug <id>

Pre-Submit Checklist

Before creating or updating an export, verify:

adaptorType
is exact -- case-sensitive, matches the Adaptor Decision Matrix (e.g.,
```
HTTPExport
```
, not
```
httpExport
```
or
```
HttpExport
```
)
Pre-built connector was checked -- for HTTP exports,
```
celigo http-connectors list
```
found no connector for the app (or the connector lacks the endpoint) before any hand-written
```
relativeURI
```
_connectionId
is valid -- points to an existing, online connection of the correct type. Not needed for
```
WebhookExport
```
or
```
SimpleExport
```
Adaptor config block is present --
```
http{}
```
,
```
netsuite{}
```
,
```
ftp{}
```
, etc. matches the
```
adaptorType
```
resourcePath
or query is correct -- wrong path silently returns 0 records with no error
Pagination is configured -- for HTTP exports, set
```
http.paging
```
if the API returns paginated results
Delta/incremental is configured -- if using delta, check
```
delta.dateField
```
or Handlebars
```
{{{lastExportDateTime}}}
```
in the URI
File parsing matches the format -- if file-based,
```
file.type
```
matches the actual file format (csv, json, xml, xlsx, edi)

mockOutput
format is correct --

{ "page_of_records": [{ "record": {...} }] }

, not a plain array

No
rest:
block --
```
rest:
```
creates a legacy RESTExport. Use only
```
http:
```
for new exports
Output filter syntax is valid -- if using an output filter expression, test it against sample data
Lookup config is complete -- if
```
isLookup: true
```
, ensure response mapping is planned for the flow's
```
pageProcessors[]
```
entry

Gotchas

PUT erases omitted fields. Always GET first, modify, then PUT. The
```
set
```
command handles this.
Including a
rest:
block creates a legacy RESTExport. Use only
```
http:
```
for new exports.
Wrong
resourcePath
produces 0 records with no error. First thing to check when an export succeeds but returns nothing.

mockOutput
format is
{ "page_of_records": [{ "record": {...} }] }
. Not a plain array.

HTTP delta exports use Handlebars (

{{{lastExportDateTime}}}

relativeURI

), not

delta.dateField

NetSuite saved searches need
netsuite.restlet.searchId
. Use

celigo metadata types <connectionId>

to find the search ID.

File exports require the
file{}
block. Without it, file-based exports return raw bytes instead of parsed records.
Webhook exports have no
_connectionId
. Setting one causes validation errors.
Distributed exports require
type: "distributed"
on the export AND
distributed: true
on the connection.
type: "once"
needs a dedicated, writeable tracking flag.
```
once.booleanField
```
must be writeable by the export's connection, and no other process may update the same field -- a shared flag causes records to be skipped.
An async-helper export cannot carry its own transform, output filter, or preSavePage hook. Build that processing into the helper's result export instead. The status and result exports must themselves be plain synchronous exports -- an async helper cannot nest another. See Async APIs (submit, poll, fetch).

Common Errors

Error	Likely Cause	Fix
`404 Not Found` on export invoke	Wrong `relativeURI` or `resourcePath`	Verify the endpoint path against the API docs; check for missing path parameters
`401 Unauthorized`	Connection credentials expired or invalid	Run `celigo connections ping <connId>` ; re-authorize OAuth connections
`0 records exported` (no error)	Wrong `resourcePath` , empty date range, or overly restrictive filter	Check `resourcePath` , widen delta window, test without output filter
`Cannot read property of undefined` in preSavePage	Script assumes a field exists that is missing from some records	Add null checks: `if (record.field)` before access
`mockOutput is invalid`	Wrong format -- used array instead of object	Use `{ "page_of_records": [{ "record": {...} }] }`
`Invalid adaptorType`	Case mismatch or typo	Use exact casing from the Adaptor Decision Matrix
`Connection is offline`	Connection failed health check	Fix credentials, re-authorize, then `celigo connections ping <id>`
`Rate limit exceeded` / `429`	Too many concurrent requests to the source API	Lower `concurrencyLevel` on the connection; add retry config
`Timeout` on large exports	Query returns too much data or API is slow	Add pagination, narrow the date range, or increase timeout settings
`File parsing error`	`file.type` doesn't match actual file format, or delimiter/encoding mismatch	Verify `file.type` , check `file.csv.columnDelimiter` , ensure correct encoding

configuring-exports

NPX Install

Tags

SKILL.md Content

Configuring Exports

Export Execution Pipeline

Three Categories of Export

Listeners

File Transfers

Record-Based Exports

Quick Reference

Adaptor Decision Matrix

Minimum Required Fields

Which Schemas to Read

Schema Index

Related Skills

How to Build an Export

1. Identify the target application

2. Check for existing patterns

3. Check for a pre-built connector

4. Query metadata for the target system

5. Determine the category

6. Choose the right adaptor type

7. Build the export JSON

Export Design Decisions

Delta vs one-time vs full sync

Listener/webhook vs scheduled export

Lookup export vs separate scheduled export

One-to-many fan-out -- the array element IS the record

Source-side transform vs destination-side mapping

Async APIs (submit, poll, fetch)

CLI Commands

Pre-Submit Checklist

Gotchas

Common Errors