Skip to main content

Pipelines

Pipelines automate and schedule notebook execution, letting you chain notebooks into production workflows that run on a schedule or are triggered on demand via the API. Each pipeline consists of multiple elements, each of which contains a notebook and specific configuration settings for its execution.

Creating a pipeline​

You can create a pipeline in the Workspace > Pipelines section of your project or by clicking on the + button in the File System Explorer, located in the top right-hand corner. Then, select New... and choose Pipeline.

Configuration​

Upon accessing the detail page of a pipeline, you'll find three tabs:

General settings​

Here you can configure the basic settings of the pipeline:

OptionTypeDescriptionDefault
Execution timeoutnumberThe maximum time in seconds that a pipeline is allowed to run. Counted from the moment the worker starts executing; time spent waiting for a slot or for the worker to start does not count.300
Input variablesstring/JSONJSON object that defines input variables shared by all pipeline elements.{}
Webhook URLstring(Optional) A valid URL to be called at the end of the execution, see Webhook Integration.
Worker imagestringThe worker image used to run the pipeline. The default is to use the project's base image.<default_project_image>
Worker environmentstringYou can select the available machine resource options (RAM, CPU, GPU) that will be used to run the pipeline.
Max concurrencynumberMaximum number of concurrent executions allowed for this pipeline, see Max Concurrency<default_pricing_plan>
Active PRObooleanWhether the automatic executions are enabled for the pipeline.false
Crontab expression PROstringThe crontab expression that defines the schedule of the pipeline. See Crontab.guru

Under the form options, you have a button to 'See logs' and another to 'Clear logs'. Pipelines logs work in the same way as in Notebooks.

Elements​

Pipelines are composed of elements that will get executed in a sequence. Each element contains a Notebook and some Settings:

OptionTypeDescriptionDefault
NotebookstringThe notebook to be executed. You can choose from all the notebooks in your project.
Parametersstring/JSONA JSON object containing the parameters of the item.{}
OrdernumberThe order in which pipeline elements are executed. Items with a lower execution order are executed first.

You can also delete, clone or edit elements in this element table.

Executions​

info

In the Free Plan you can only manually run pipelines by clicking the Execute button in the top right hand corner of the pipeline detail page.

The Executions tab displays a table of pipeline execution statistics, filterable by status (Pending, Queued, Running, Completed, Errored, Timeout or Cancelled) or creation date. From the Actions column, you can relaunch or cancel an execution based on its current status. Select several rows to use the Relaunch execution(s) and Cancel execution(s) bulk actions.

Possible Status values include:

  • Pending: The execution is waiting for a free slot, see Max Concurrency.
  • Queued: The execution has a slot and its worker is starting.
  • Running: The pipeline execution is currently in progress.
  • Completed: The pipeline execution finished successfully.
  • Errored: The pipeline execution encountered an error.
  • Timeout: The execution exceeded the allowed time limit.
  • Cancelled: The execution was cancelled before completion.

Only executions in a final state (Completed, Errored, Timeout or Cancelled) can be relaunched. Any other execution can be cancelled.

Resources and quotas​

Each execution runs on its own worker, so it uses the same resources as a notebook session:

  • The worker appears in the Active sessions of the user who launched the execution while it runs.
  • The execution counts towards your plan's concurrent session limit and CPU/GPU hours.
  • If no quota is available when the execution is about to start, it ends as Errored and the logs show the reason (for example Insufficient quota to deploy a new worker or You have reached your session limit). Relaunch the execution once quota is available.

Output​

To store and access the results of pipeline executions, use the MINEO_PIPELINE_OUTPUT variable in your notebooks. This variable ensures that the execution results are stored and can be retrieved via the API. The output will be available as soon as the execution is complete.

Public API​

The API allows users to run pipelines programmatically through a REST API interface. This feature is designed to provide data pipeline users with better integration and connectivity with third-party applications and processes.

All endpoints are secured and require a valid token for access. You must add the token to the Authorization header as follows:

Authorization: Token YOUR_ACCESS_TOKEN

You can access the API tab within a specific pipeline to generate a token and get an example of how to run the pipeline.

Execute​

POST /api/project/{project_node_uuid}/pipeline/{node_uuid}/execute/

Starts executing a Pipeline.

Body parameters​

  • input_variables: (optional) - A JSON object containing key-value pairs for the input variables to be accessed in the all pipeline elements.
  • webhook_url: (optional) - A valid URL to be called at the end of the execution. See Webhook Integration.

Example response​

Returns a json with urls accessible using GET method:

  • detail: Returns the details of the pipeline execution triggered by this execution.
  • list: Returns a list of all pipeline executions for the specific pipeline.
{
    "execution_status_urls": {
        "method": "GET",
        "detail": "https://b.mineo.app/api/project/{project_node_uuid}/pipeline/{node_uuid}/executions/{execution_uuid}/",
        "list": "https://b.mineo.app/api/project/{project_node_uuid}/pipeline/{node_uuid}/executions/"
    }
}

Get execution​

Returns the details of the pipeline execution triggered by this execution.

GET /api/project/{project_node_uuid}/pipeline/{node_uuid}/executions/{execution_uuid}/

Get a specific Pipeline execution.

Returns​

  • uuid: The UUID of the pipeline execution.
  • pipeline_uuid: The UUID of the pipeline.
  • elapsed_time: The elapsed time of the pipeline execution.
  • creation_datetime: The creation datetime of the pipeline execution.
  • status: The status of the pipeline execution.
  • exception: The exception message if the pipeline execution failed.
  • input: The input variables passed to the pipeline execution.
  • output: The output of the pipeline execution.
  • webhook_http_status: The HTTP status code returned by the webhook call, if applicable.

Example response​

{
    "uuid": "execution_uuid",
    "pipeline_uuid": "pipeline_uuid",
    "elapsed_time": "00:30:00",
    "creation_datetime": "2023-10-02T15:20:30Z",
    "status": "Running",
    "exception": "",
    "input": "{input_variables}",
    "output": "{MINEO_PIPELINE_OUTPUT}",
    "webhook_http_status": 403
}

List executions​

Returns a paginated list of all pipeline executions for the specified pipeline.

GET /api/project/{project_node_uuid}/pipeline/{node_uuid}/executions/

Parameters​

  • page: (optional) The page of results to retrieve. Defaults to the first page if not specified.
  • page_size: (optional) The number of results per page. Defaults to 20 if not specified. The maximum allowed value is 100.

Example URL with Custom Page Size and Page:

GET /api/project/{project_node_uuid}/pipeline/{node_uuid}/executions/?page_size=10&page=2

Returns​

  • items: A list of pipeline executions for the specified pipeline.
  • count: The total number of pipeline executions available.
  • next: A URL to the next page of results, or null if no further pages are available.
  • previous: A URL to the previous page of results, or null if there are no previous pages.

Each item in items contains:

  • uuid: The UUID of the pipeline execution.
  • pipeline_uuid: The UUID of the pipeline.
  • elapsed_time: The elapsed time of the pipeline execution.
  • creation_datetime: The creation datetime of the pipeline execution.
  • status: The status of the pipeline execution.
  • exception: The exception message if the pipeline execution failed.
  • input: The input variables passed to the pipeline execution.
  • webhook_http_status: The HTTP status code returned by the webhook call, if applicable.

Example response​

{
  "items": [
    {
      "uuid": "execution_uuid1",
      "pipeline_uuid": "pipeline_uuid",
      "elapsed_time": "00:15:00",
      "creation_datetime": "2023-10-02T12:20:30Z",
      "status": "Running",
      "exception": "",
      "input": "{input_variables}",
      "webhook_http_status": 403
    },
    {
      "uuid": "execution_uuid2",
      "pipeline_uuid": "pipeline_uuid",
      "elapsed_time": "00:20:00",
      "creation_datetime": "2023-10-02T13:20:30Z",
      "status": "Pending",
      "exception": "Some error occurred",
      "input": "{input_variables}",
      "webhook_http_status": 200
    }
  ],
  "count": 26,
  "next": "/api/project/{project_node_uuid}/pipeline/{node_uuid}/executions/?page=2",
  "previous": null
}

Max concurrency​

The Max concurrency setting controls the maximum number of simultaneous executions allowed for a pipeline.

  • Accepted Values: Any integer ≥ 1, up to the limit defined by your Pricing Plan.
  • Purpose: Enables multiple instances of the same pipeline to run in parallel. Ideal for processing different datasets or handling concurrent events.
  • Limit Handling: If the limit is exceeded, new executions stay in a Pending state until a running one finishes.

Adjusting this value helps optimize throughput and resource usage.

Webhook integration​

Pipelines support an optional webhook integration feature to enhance external communication and automation processes. The webhook is a user-configured URL endpoint that can be triggered automatically during pipeline execution.

Configuring the webhook URL​

  • Optional Feature: Configuring a webhook URL is entirely optional. If no URL is specified, the webhook will not be triggered.
  • How It Works: Once configured, the webhook will be called using a HTTP GET request during the pipeline execution process.
  • Endpoint Configuration: Users have the flexibility to define custom query parameters within their webhook URL. For example, you can configure the endpoint as https://example.com/endpoint?param=mypipeline to pass static identifiers or parameters.
  • Via API: The webhook URL can also be specified when executing the pipeline via the API. This allows for dynamic webhook configurations based on the execution context.

Monitoring webhook calls​

When a webhook URL is specified, its execution status is logged within the pipeline execution records:

  • Execution Table: The status of the webhook call can be reviewed in the "Executions" tab of the pipeline detail page. This helps in diagnosing issues or confirming successful communication with the external system.
  • Logs: Both successful and failed webhook calls generate log messages, detailing the result of the call. This helps keep track of the webhook activity and provides easy access to troubleshooting errors if they arise.

By leveraging the webhook integration, users can facilitate extensive workflows and communications, enabling automation and real-time data exchanges with third-party applications or systems.

See also​