---
title: Pipelines
description: Pipelines run your own JavaScript when an external system pushes data to the platform, turning uploaded files and payloads into stored Files and events without an Edge Server.
source: https://documentation.controlcomtech.com/platform/pipelines
---

# Pipelines

Not everything that matters arrives through an Edge Server. Power quality meters email event files, utilities export CSVs, contractors upload test reports. A pipeline is a small JavaScript program you write inside the platform that receives those pushes over HTTPS, runs your parsing logic, and turns the result into stored files and events, with no server of your own to host.

> **Workflow Stage: Collect**

## Overview

Pipelines is reached from Settings: open **Settings**, find the **Pipelines** card ("Transform and route incoming data with custom processing pipelines.") and click **Manage**, which lands on the list at `/administration/pipelines`.

Each pipeline is a named endpoint plus the code behind it. An external system POSTs a JSON payload or a file to the pipeline's URL; the platform runs your handler in an isolated serverless function and executes the actions it returns. What comes out the other end is ordinary platform data: files on the [Files](https://documentation.controlcomtech.com/platform/files) page and events in [Alarm History](https://documentation.controlcomtech.com/platform/alarms#alarm-history).

![The Pipelines list in ControlCom Connect showing three pipelines with their execution counts](https://documentation.controlcomtech.com/images/pipelines/01-pipelines-list.png)

*The Pipelines list, with how often each has run*

## How a Pipeline Runs

The flow is push, run, act. A sender POSTs to the pipeline's endpoint with an API key. The platform invokes your handler with the request payload: JSON fields arrive as fields, and an uploaded file arrives with its filename, content and type. Your code does whatever parsing it needs and returns a list of actions; the platform then performs them, storing files and writing events. Every invocation, success or failure, increments the pipeline's execution count, and a run counts as failed when the handler throws or returns something that is not a valid action list.

Runs are stateless and short-lived: each request gets a fresh execution with a 30 second limit, so pipelines suit event files and periodic exports rather than continuous streams. Continuous device data belongs to the Edge Server path in the Connect and Collect stages.

## The Pipelines List

The list shows each pipeline's name, its **Execution Count**, and **Last Modified**. The search box (**Search pipelines…**) filters by name, the column and density toggles adjust the table, and the export button writes the rows out. Clicking a row, or **Open** in its actions, opens the pipeline's configuration; there is no delete, so a retired pipeline is switched off with its enable toggle instead.

## Creating a Pipeline

Click **New Pipeline** to open the **Create Pipeline** dialog. Give the pipeline a name, and note the code editor arrives pre-filled with a complete working example handler, which is the fastest way to learn the contract: it parses an uploaded file's name, stores the file, and writes an event bound to it.

![The Create Pipeline dialog with the starter handler code in the editor](https://documentation.controlcomtech.com/images/pipelines/02-create-pipeline.png)

*A new pipeline starts from a working example*

**Create** deploys the pipeline and opens its configuration page. New pipelines start disabled; flip **Enable Pipeline** when the code is ready to receive real traffic. Saving from the configuration page redeploys the code, and the change is live on the next request.

## Writing Pipeline Code

The editor is a real multi-file JavaScript environment. The **Files** panel on the left lists the module files; `index.mjs` is the entry point, always present, and cannot be renamed or removed. The **New file** button adds helper modules (names get `.mjs` appended automatically), so a parser, a report generator and a lookup table can live in their own files and be imported from the entry file with standard ES module imports.

![The pipeline configuration page with the file list beside the code editor](https://documentation.controlcomtech.com/images/pipelines/03-pipeline-configuration.png)

*A multi-file pipeline: the entry file plus its helper modules*

The contract is a single exported function:

```js
export const handler = async (event) => {
    const body = event.body ?? {};
    const file = body.file; // { filename, value (base64), mimetype } when a file was uploaded

    // ...parse, transform, decide...

    return [
        {
            action: 'STORE_FILE',
            base64file: file.value,
            filename: file.filename,
            mimetype: file.mimetype,
            directory: 'power-quality/meter-12',
        },
        {
            action: 'WRITE_EVENT',
            message: 'Power quality event recorded',
            isFileBoundToEvent: true,
        },
    ];
};
```

The return value must be an array of actions, and it is the only way a pipeline affects the platform: no actions, no side effects. Code runs on Node.js 22 with 256 MB of memory and the 30 second limit above.

## Actions

**Store File** (`STORE_FILE`) saves a file into the Organization's storage. It takes the content (`base64file`), a `filename`, a `mimetype` and an optional `directory`; the directory becomes the folder shown on the Files page, so the folder tree there mirrors what your pipelines write.

**Write Event** (`WRITE_EVENT`) records a manual event, the same kind that appears in Alarm History. It takes a `message` and optional `dateTime`, can attach the event to an Asset with `assetId`, can bind the file stored by the same run with `isFileBoundToEvent`, and can notify an alarm group by email with `notifyAlarmGroupId`, optionally attaching the file with `attachFileToEmail`. That last combination is how "meter 12 recorded a sag, report attached" reaches the on-call inbox with no one watching a screen.

**No Action** (`NA`) is the explicit way to do nothing, for payloads your code decides to ignore.

A single run can return several actions, and typically does: store the evidence, then write the event that points at it.

## Layers

Pipeline code cannot install its own npm packages; shared dependency bundles called layers provide them instead. The **Layers** selector on the pipeline form attaches any of the published bundles: a base utility layer (HTTP requests, lodash, date handling with Luxon), a file layer (CSV parsing with Papa Parse), and a PDF layer (PDF generation and charting). Attach only what the code imports, and expect a selected layer to display its version; an outdated version is flagged on the form after the platform publishes a newer one.

## Sending Data to a Pipeline

Each pipeline listens at its own execute URL:

```
POST /p/v1/organizations/{organizationId}/pipelines/{endpointId}/execute?apiKeyId={id}&apiKeySecret={secret}
```

The `endpointId` is a short identifier generated for the pipeline when it is created. Requests authenticate with an Organization API key, created under **Settings → API keys** (see [Settings](https://documentation.controlcomtech.com/platform/settings)), passed as the `apiKeyId` and `apiKeySecret` query parameters.

The body is either JSON (`application/json`) or a file upload (`multipart/form-data`, one file per request, up to 50 MB). Form fields arrive in your handler alongside the file, which is how a sender passes context like which meter or site the upload belongs to. A disabled pipeline rejects requests until it is enabled again.

## Permissions

| Permission                 | Grants                        |
| -------------------------- | ----------------------------- |
| `Pipelines.ReadPipelines`  | Opening the Pipelines pages   |
| `Pipelines.CreatePipeline` | The **New Pipeline** button   |
| `Pipelines.UpdatePipeline` | Editing and saving a pipeline |

## Related Workflow Stages

Pipelines are the Collect stage's second door: the Edge Server pulls readings from equipment on site, while a pipeline lets outside systems push what they have. Its output lands in the Store stage, as documents on [Files](https://documentation.controlcomtech.com/platform/files) and events beside your Alarms, and the API keys that secure it belong to the Integrate stage covered in [Settings](https://documentation.controlcomtech.com/platform/settings) and the [API reference](https://documentation.controlcomtech.com/api-reference). For continuous device data, start instead with the Edge Server in [Device Administration](https://documentation.controlcomtech.com/platform/devices/administration).