Pipelines

Pipelines are a specialised type of node designed to efficiently schedule and manage notebook execution within your workflow. Each pipeline consists of multiple elements, each of which contains a notebook and specific configuration settings for its execution.

Creating a Pipeline

You can create a pipeline in any folder of your project by clicking on the + button in the File System Explorer, located in the top right-hand corner. Then, select New.... and choose Pipeline.

Configuration

Upon accessing the detail page of a pipeline, you'll find three tabs:

General settings

Here you can configure the basic settings of the pipeline:

Option	Type	Description	Default
Execution timeout	number	The maximum time in seconds that a pipeline is allowed to run.	300
Input variables	string/JSON	JSON object that defines input variables shared by all pipeline elements.	{}
Worker image	string	The worker image used to run the pipeline. The default is to use the project's base image.	<default_project_image>
Worker environment	string	You can select the available machine resource options (RAM, CPU, GPU) that will be used to run the pipeline.
Active	boolean	Whether the automatic executions are enabled for the pipeline.	false
Crontab expression	string	The crontab expression that defines the schedule of the pipeline. See Crontab.guru

Under the form options, you have a button to 'See logs' and another to 'Clear logs'. Pipelines logs work in the same way as in Notebooks.

Elements

Pipelines are composed of elements that will get executed in a sequence. Each element contains a Notebook and some Settings:

Option	Type	Description	Default
Notebook	string	The notebook to be executed. You can choose from all the notebooks in your project.
Parameters	string/JSON	A JSON object containing the parameters of the item.	{}
Order	number	The order in which pipeline elements are executed. Items with a lower execution order are executed first.

You can also delete, clone or edit elements in this element table.

Executions

info

In the Free Plan you can only manually run pipelines by clicking the Execute button in the top right hand corner of the pipeline detail page.

The Executions tab displays a table of pipeline execution statistics, filterable by status (Completed, Errored, Pending, Running or OTHER) or creation date. From the Actions column, you can restart or cancel tasks based on their current status.

Output

To store and access the results of pipeline executions, use the MINEO_PIPELINE_OUTPUT variable in your notebooks. This variable ensures that the execution results are stored and can be retrieved via the API. The output will be available as soon as the execution is complete.

Public API

The API allows users to run pipelines programmatically through a REST API interface. This feature is designed to provide data pipeline users with better integration and connectivity with third-party applications and processes.

All endpoints are secured and require a valid token for access. You must add the token to the Authorisation header as follows:

Authorization: Token YOUR_ACCESS_TOKEN

You can access the API tab within a specific pipeline to generate a token and get an example of how to run the pipeline:

Execute

POST /api/project/{project_node_uuid}/pipeline/{node_uuid}/execute/

Starts executing a Pipeline.

• Body Parameters:
  • input_variables: (optional) - A JSON object containing key-value pairs for the input variables to be accessed in the all pipeline elements.

Response:

Returns a json with urls accessible using GET method:

• detail: Returns the details of the pipeline execution triggered by this execution.
• list: Returns a list of all pipeline executions for the specific pipeline.

{
    "execution_status_urls": {
        "method": "GET",
        "detail": "https://b.mineo.app/api/project/{project_node_uuid}/pipeline/{node_uuid}/executions/{execution_uuid}/",
        "list": "https://b.mineo.app/api/project/{project_node_uuid}/pipeline/{node_uuid}/executions/"
    }
}

Get Execution

Returns the details of the pipeline execution triggered by this execution.

GET /api/project/{project_node_uuid}/pipeline/{node_uuid}/executions/{execution_uuid}/

Get a specific Pipeline execution. Returns:

• uuid: The UUID of the pipeline execution.
• pipeline_uuid: The UUID of the pipeline.
• elapsed_time: The elapsed time of the pipeline execution.
• creation_datetime: The creation datetime of the pipeline execution.
• status: The status of the pipeline execution.
• exception: The exception message if the pipeline execution failed.
• input: The input variables passed to the pipeline execution.
• output: The output of the pipeline execution.

Response:

{
    "uuid": "execution_uuid",
    "pipeline_uuid": "pipeline_uuid",
    "elapsed_time": "00:30:00",
    "creation_datetime": "2023-10-02T15:20:30Z",
    "status": "Running",
    "exception": "",
    "input": "{input_variables}",
    "output": "{MINEO_PIPELINE_OUTPUT}"

}

List Executions

Returns a list of all pipeline executions for the specific pipeline.

GET /api/project/{project_node_uuid}/pipeline/{node_uuid}/executions/

Returns:

• items: A list of all pipeline executions for the specific pipeline.
  • uuid: The UUID of the pipeline execution.
  • pipeline_uuid: The UUID of the pipeline.
  • elapsed_time: The elapsed time of the pipeline execution.
  • creation_datetime: The creation datetime of the pipeline execution.
  • status: The status of the pipeline execution.
  • exception: The exception message if the pipeline execution failed.
  • input: The input variables passed to the pipeline execution.
  • output: The output of the pipeline execution.

Example response:

{
  "items": [
    {
        "uuid": "execution_uuid1",
        "pipeline_uuid": "pipeline_uuid",
        "elapsed_time": "00:15:00",
        "creation_datetime": "2023-10-02T12:20:30Z",
        "status": "Running",
        "exception": "",
        "input": "{input_variables}",
        "output": "{MINEO_PIPELINE_OUTPUT}"
    },
    {
        "uuid": "execution_uuid2",
        "pipeline_uuid": "pipeline_uuid",
        "elapsed_time": "00:20:00",
        "creation_datetime": "2023-10-02T13:20:30Z",
        "status": "Pending",
        "exception": "Some error occurred",
        "input": "{input_variables}",
        "output": "{MINEO_PIPELINE_OUTPUT}"
    }
  ]
}