Pipeline Execute

Overview

The PipeLine Execute Snap executes a pipeline in a specific Snaplex with the specified parameters. You can use this Snap to execute multiple child pipelines from a single parent pipeline. Configure the Snap to execute the pipeline and data input.

Tip:

To execute a SnapLogic pipeline that is exposed as a REST service, use the HTTP Client Snap.

Flow-type Snap
Works in Ultra Tasks

Key Features

The Pipeline Execute Snap enables you to:

Structure complex pipelines into smaller segments through child pipelines
Initiate parallel data processing using the pooling option
Orchestrate data processing across nodes, in the Snaplex or across Snaplexes
Distribute global values through pipeline parameters across a set of child pipeline Snaps

Supported Modes for Pipelines


Modes	Description
Standard mode (default)	A new child pipeline is launched for each input document in the standard mode. If you set the pool size to n (where n is any number with the default setting value of 1, then n concurrent child pipelines can run simultaneously. Each child pipeline processes one document from the parent and then completes.
Reuse mode	In reuse mode, child pipelines are started, and each child pipeline instance can process multiple input documents from the parent. If you set the pool size to n (default 1), then n number of child pipelines are started and they will process the input document in a streaming manner. The child pipeline needs to have an unlinked input view for use in reuse mode. Reuse mode is more performant, but it has the restriction that the child pipeline has to be a streaming pipeline. In reuse mode if a child pipeline fails, it causes the parent pipeline to fail.
Resumable Child pipeline	The resumable pipeline does not support the Pipeline Execute Snap. A regular mode pipeline can use the Pipeline Execute Snap to call a Resumable mode pipeline. If the child pipeline is a Resumable pipeline, then the batch size cannot be greater than one.
Ultra Task pipelines	Works in Ultra Tasks Pipelines with the following exceptions: The reuse option works in the Ultra pipeline if the child pipelines run on the same node. Running on a different node in the same Snaplex fails with a lineage error. To ensure that the child pipelines run on the same node as the parent pipeline, set Execute on to LOCAL_NODE. For more information, refer to Snap Support for Ultra Tasks. Without reuse enabled, the one-in-one-out requirement for Ultra means batching is not supported. A runtime check will fail the parent pipeline if the batch size is set to greater than 1. This would be similar to the current behavior in DB insert and other Snaps in Ultra mode.
ETL Mode	A pipeline can only use the Pipeline Execute Snap to call another pipeline with ELT Snaps in Standard Mode.
Pooling Enabled	The pool size and batch size can both be set to greater than one, in which case the input documents are spread across the child pipeline in a round-robin method to ensure that if the child pipeline does any slow processing external calls, then the processing is spread across the children in parallel. The limitation of this option is that the document order sent to the child pipeline is not maintained.

Note:

Replaces Deprecated Snaps: This Snap replaces the ForEach and Task Execute Snaps and the Nested pipeline mechanism.

Limitations

If there are insufficient Snaplex nodes to execute the pipeline, the Snap waits until the resources become available. In this scenario, a message appears in the execution statistics dialog.
Only the last 100 completed child pipeline runs are saved for inspection in the Dashboard because this Snap generates many pipeline runtimes.
The PipeLine Execute Snap cannot exceed a depth of 64 child pipelines before they begin to fail.

The child pipelines do not display data preview details. However, you can view the data preview for any child pipeline after the PipeLine Execute Snap completes execution in the parent pipeline. Ultra Pipelines do not support batching.
Unlike the Group By N Snap, when you configure the Batch field, the documents are processed one by one by the PipeLine Execute Snap and then transferred to the child pipeline when the parent pipeline receives it. The child pipeline closes when the batch or input stream ends.
Error handling for child pipelines inside a parent Ultra Pipeline are not supported. Instead, configure error handling in the Ultra Pipeline with the PipeLine Execute Snap.
Parent pipeline fails to capture errors from child pipeline:
- Workaround: When the parent pipeline fails to capture errors from the child pipeline, you can utilize error views or an error pipeline within the parent pipeline's PipeLine Execute snap. This configuration enables errors from the child pipeline to be directed to the parent pipeline’s error view, allowing for logging or further processing instead of halting or failing the parent pipeline entirely.
- Design pipelines to handle errors through error views or dedicated error pipelines, which allow the parent to continue processing even if some child executions fail. Refactor pipelines to route all errors explicitly, avoiding default failure behavior.

Snap views


View	Description	Examples of upstream and downstream Snaps
Input	The document or binary data to send to the child pipeline. Retry is not supported if the input view is a Binary data type.	Mapper Copy
Output	If the child pipeline has an unlinked output, then documents or binary data from that view are transferred out of this view. If the child pipeline has a linked view then the output document or binary data is generated for successful runtimes with the run_id field. Unsuccessful pipeline runs write a document to the error view. If Reuse executions to process documents is deselected, the original input or binary data is added to the output document. For Pool Size Usage: If the Pool Size field value is greater than one, then the order of records sent to the child pipeline in the output is not guaranteed to be the same as that of the input.	Mapper Router
Error	Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab. The available options are: Stop Pipeline Execution Stops the current pipeline execution when an error occurs. Discard Error Data and Continue Ignores the error, discards that record, and continues with the remaining records. Route Error Data to Error View Routes the error data to an error view without stopping the Snap execution. Learn more about Error handling in Pipelines.

Snap settings

Legend:

Expression icon (): Allows using pipeline parameters to set field values dynamically (if enabled). SnapLogic Expressions are not supported. If disabled, you can provide a static value.
SnapGPT (): Generates SnapLogic Expressions based on natural language using SnapGPT. Learn more.
Suggestion icon (): Populates a list of values dynamically based on your Snap configuration. You can select only one attribute at a time using the icon. Type into the field if it supports a comma-separated list of values.
Upload : Uploads files. Learn more.

Learn more about the icons in the Snap settings dialog.


Field / Field set	Type	Description
Label	String	Required. Specify a unique name for the Snap. Modify this to be more appropriate, especially if more than one of the same Snaps is in the pipeline. Default value: Pipeline Execute Example: ExecuteCustomerUpdate Pipeline
Pipeline	Dropdown list/Expression	Required. Specify the child pipeline's absolute or relative (expression-based) path to run. If you specify only the pipeline name, Snap searches for the pipeline in the following folders in this order: Current project Shared project-space Shared folder of the Org You can specify the absolute path to the target project using the Org/project_space/project notation. You can also dynamically choose the pipeline to run by entering an expression in this field when Expressions are enabled. For example, to run all of the pipelines in a project, you can connect the SnapLogic List Snap to this Snap to retrieve the list of pipelines in the project and run each one.
Execute On	Dropdown list	Select one of the following Snaplex options to specify the target Snaplex for the child pipeline: LOCAL_NODE. Runs the child pipeline on the same node as the parent pipeline. This option is recommended when the calls between the child and parent pipeline are frequent. Use this option for most regular child Pipeline executions. LOCAL_SNAPLEX. Runs the child pipeline on one of the available nodes in the same Snaplex as the parent pipeline. This option is is most effective when the child pipelines process intensive workloads - such as, a parent pipeline listing files and calling a child to process data from one of those files. SNAPLEX_WITH_PATH. Runs the child pipeline on a user-specified Snaplex, and the Snaplex Path field appears. It's important to note that this process is highly dependent on the network. The network communication is optimized for batch data processing since the child Pipeline is on a remote Snaplex. It is recommended to use this option only when the child Pipeline needs to run on a different Snaplex because of endpoint connectivity restrictions or for workload distribution purposes. For more information, refer to the Best Practicesto be updated. Default value: SNAPLEX_WITH_PATH Example: groundplex4-West
Snaplex Path	Dropdown list/Expression	Appears when you select SNAPLEX_WITH_PATH* for Execute On.* Enter the name of the Snaplex on which you want the child pipeline to run. Click to select from the list of Snaplex instances available in your Org. Default value: N/A Example: DevPlex-1
Execution Label	Dropdown list/Expression	Specify the label to display in the Pipeline view of the Dashboard. You can use this field to differentiate each pipeline execution. Default value: N/A Example: NetSuite-Create-Credit-Memo
Pipeline Parameters	Use this fieldset to define the Pipeline Parameters for the pipeline selected in the Pipeline field. When you select Reuse executions to process documents, you cannot change parameter values from one Pipeline invocation to the next.
Parameter Name	String/Suggestion	*Debug mode* Enter the name of the parameter. Select the defined Pipeline Parameters in the Pipeline field. Default value: N/A Example: Postal_code
Parameter Value	Dropdown list/Expression	Enter the value for the Pipeline Parameter, which can be an expression based on incoming documents or a constant. If you configure the value as an expression based on the input, then each incoming document or binary data evaluates against that expression when to you invoke the pipeline. The result of the expression is JSON-encoded if it is not a string. The child Pipeline then needs to use the `JSON.parse()` expression to decode the parameter value. When Reuse executions to process documents is enabled, the parameter values cannot change from one invocation to the next. Default value: N/A Example: 9402
Reuse executions to process documents	Checkbox	Select this checkbox to start a child pipeline and pass multiple inputs to the pipeline. Reusable executions continue to live until all of the input documents to this Snap are fully processed. Important: When you select this checkbox, and the pipeline parameters use expressions, the expressions are evaluated with the first document. The parameter value in the child pipeline does not change across documents. If you do clear this checkbox, a new pipeline execution is created for each input document. Reuse mode does not support batching. Default status: Deselected
Batch size	Integer/Expression	Required. Specify the number of documents in the batch size. If Batch Size is set to N, then N input documents are sent to each child pipeline that is started. After N documents, the child pipeline input view is closed until the child pipeline completes its execution. The output of the child pipeline (one or more documents) passes to the Pipeline Execute output view. New child pipelines are started after the original pipeline is complete. Note: Batching is not supported for reuse executions. This option is does not support reusable executions. Default value: 1 Example: 2
Pool Size	Integer/Expression	Required. Specify an execution pool size to process multiple input documents or binary data concurrently. When the pool size is greater than one, the Snap starts Pipeline executions as needed up to the specified pool size. When you select Reuse executions to process documents, the Snap starts a new execution only if either all executions are busy working on documents or binary data and the total number of executions is below the pool size. Default value: 1 Example: 4
Timeout (in seconds)	Integer/Expression	Specify the number of seconds for which the Snap must wait for the child pipeline to complete the runtime. If the child pipeline does not complete the runtime before the timeout, the execution process stops and is marked as failed. Default value: N/A Example: 10
Retry limit	Integer/Expression	Specify the maximum number of retry attempts that the Snap must make in the case of a failure. If the child pipeline does not execute successfully, an error document is written to the error view. If the child pipeline is not in a completed state, then it will retry. The pipeline failure at the application level could have various causes, including network failures. Important: Usage Guidelines This feature is incompatible with reusable executions. Enabling Retry overrides the Pool Size setting. If a task fails and retries are enabled, a new instance starts regardless of the configured pool size. Retry is not supported if the input view is a Binary data type. Default value: N/A Example: 3
Retry interval	Integer/Expression	Specify the minimum number of seconds the Snap must wait between two retry requests. A retry happens only when the previous attempt results in an error. Warning: This feature is incompatible with reusable executions. Retry is not supported if the input view is a Binary data type. Default value: N/A Example: 10
Snap execution	Dropdown list	Choose one of the three modes in which the Snap executes. Available options are: Validate & Execute: Performs limited execution of the Snap and generates a data preview during pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during pipeline runtime. Execute only: Performs full execution of the Snap during pipeline execution without generating preview data. Disabled: Disables the Snap and all Snaps that are downstream from it. Default value: Validate and Execute Example: Execute only

Troubleshooting


Error	Reason	Resolution
Account validation failed	The Pipeline ended before the batch could complete execution because of a connection error.	Verify that the Refresh token field is configured to handle the inputs properly. If you are not sure when the input data is available, configure this field as zero so the connection is always open.

Access pipelines in the Pipeline Catalog in the Designer to create a Child pipeline

You can now browse the pipeline catalog for the target child pipeline and then select, drag, and drop it into Canvas. The Designer automatically adds the child pipeline using the Pipeline Execute Snap. You can preview a child pipeline by hovering over a Pipeline Execute Snap while the parent pipeline is open on the Designer.

Return child pipeline output to the parent pipeline

A common use case for the PipeLine Execute Snap is to run a child Pipeline whose output is immediately returned to the parent pipeline for further processing. You can achieve this return with the following Pipeline design for the child Pipeline.

In this example, the document in this child Pipeline is sent to the parent Pipeline through output1 of the child pipeline. Any unconnected output view is returned to the parent Pipeline. You can use any Snap that completes execution in this way.

Execution States

When a PipeLine Execute Snap activates its child pipeline, you can view the status as it executes on the SnapLogic Designer canvas.

A child pipeline also reports its execution to the parent pipeline. In the Monitor overview and Dashboard, you can hover over the status shown in the Status column for a pipeline with a PipeLine Execute Snap, and the following messages are displayed for the following scenarios:

If the parent Pipeline has the status Failed shown in the Status column, the following message is displayed: One of the child Pipelines Failed.
If the parent Pipeline has the status Completed with Errors shown in the Status column, the following message is displayed: One of the child Pipelines completed with Errors.

These execution state messages apply even when the child pipeline does not appear on the Dashboard because of child pipeline execution limits.

The Monitor Execution overview does not include the Completed with Warnings status as a searchable status.