SQS Consumer

Overview

This Snap consumes the messages from the Amazon Simple Queue Service (SQS).

The number of messages to be returned is configured in Message count. The maximum number of messages (up to 10) can be configured, however a particular message from the queue cannot be requested.


Amazon SQS Producer Overview

Prerequisites

  • Valid AWS Account and SQS access.
  • Authentication such as IAM Role access, Access key, and Security Access key.

Known issues

The AWS extended library uses S3 only when the message size is more than 256 KB. However, you cannot enable the Client-side buffering checkbox when you select the Enable Large Message Support checkbox in your SQS Account.

Snap views

View Description Examples of upstream and downstream Snaps
Input This Snap has at most one document input view.

The input document must contain the keys ReceiptHandle and QueueURL.
Output

This Snap has exactly one document output view.

This Snap generates a Binary output that contains the Message body and a header. The header contains the queue URL, receipt handle, and message attributes.

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab. The available options are:

  • Stop Pipeline Execution Stops the current pipeline execution when an error occurs.
  • Discard Error Data and Continue Ignores the error, discards that record, and continues with the remaining records.
  • Route Error Data to Error View Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap settings

Legend:
  • Expression icon (): Allows using pipeline parameters to set field values dynamically (if enabled). SnapLogic Expressions are not supported. If disabled, you can provide a static value.
  • SnapGPT (): Generates SnapLogic Expressions based on natural language using SnapGPT. Learn more.
  • Suggestion icon (): Populates a list of values dynamically based on your Snap configuration. You can select only one attribute at a time using the icon. Type into the field if it supports a comma-separated list of values.
  • Upload : Uploads files. Learn more.
Learn more about the icons in the Snap settings dialog.
Field / Field set Type Description
Label String

Required. Specify a unique name for the Snap. Modify this to be more appropriate, especially if more than one of the same Snaps is in the pipeline.

Default value: SQS Consumer

Example: SQS Consumer
Region String/Expression/ Suggestion Specify the AWS region where the application is implemented.

Default: None

Example: us-east-1, us-west-1
Queue name String/Expression/ Suggestion
Specify the name of the target Queue.
Warning: The FIFO Queue must end with a '.fifo'. suffix.

Default value: None.

Example: SQS Consumer
Queue URL String/Expression

Specify the URL for the Queue. This can be found through the AWS console.

When a new queue is created with a unique name, the Amazon SQS assigns each queue with an identifier called a queue URL that includes the queue name and other SQS components. This URL will override the queue specified in Queue name property.

The URL must be in this format: https//sqs.<region>.amazonaw...t-id>/<queue-name>;

Default value: None

Example: https://sqs.amazonnaws.com
Maximum retries Integer/Expression
Specify the maximum number of retry attempts for a failed request.
Important:
  • If the value is larger than 0, the Snap first downloads the target file into a temporary local file. If any error occurs during the download, the Snap waits for the time specified in the Retry interval and attempts to download the file again from the beginning. When the download is successful, the Snap streams the data from the temporary file to the downstream Pipeline. All temporary local files are deleted when they are no longer needed.
  • Ensure that the local drive has sufficient free disk space to store the temporary local file.

Default value: 3

Example: 3
Wait timeout (Second) Integer/Expression Specify the maximum time in seconds the Snap waits for each fetching. A zero wait timeout means that short polling is performed. Otherwise, long polling is performed. If the wait timeout is set to 0, the queue attribute, Receive Message Wait Time, is used.

Default value: 0

Example: 2
Message count Integer/Expression Required. Specify the number of messages to read before consuming stops. If this value is set to 0, the Snap reads all available messages and returns. If the count is less than 0, the Snap waits indefinitely.

Default value: -1
Visibility timeout Integer/Expression The SQS sets a visibility timeout to prevent other consumers from re-processing the same message from a queue. Visibility timeout is a period of time during which Amazon SQS prevents other consumers from receiving and processing the message. Click here for more information.

Default value: 0
Auto acknowledge Checkbox Select this checkbox to automatically acknowledge the message after it is received by the Snap.

Default status: Deselected
Client-side buffering Checkbox Select this checkbox to enable the Snap buffer up to 10 outbound requests and send them in batch, instead of sending each request separately. It allows pre-fetching of messages when consuming. The Amazon SQS Buffered Asynchronous Client currently does not support the FIFO queues.

Default status: Deselected
Buffer configurations Appears when you select Client-side buffering checkbox.

The configurations for the client-side buffer.
Max buffer size Integer/Expression Specify the maximum number of messages that are batched together in a single request. The higher the setting, the fewer batches are required to carry out the same number of requests. The maximum batch size is 10.

Default value: 10

Example: 5
Max buffer open time (ms) Integer/Expression Specify the maximum buffer wait time in milliseconds before sending out the request.

Default value: 200

Example: 10
Max outbound batches in parallel Integer/Expression Specify the maximum number of concurrent batches for each type of outbound request. The greater the number, greater the throughput that can be achieved (at the expense of consuming more threads).

Default value: 5
Max batches prefetching Integer/Expression Specify the number of receive batches the Snap pre-fetches for client-side buffering.

Default value: 10

Example: 6
Max inbound batches in parallel Integer/Expression Specify the maximum number of inbound batches that are processed in parallel. The higher the setting, the more messages can be received (at the expense of consuming more threads). If the property is set to 0, the pre-fetching is disabled.

Default value: 10

Example: 15
Snap execution Dropdown list
Choose one of the three modes in which the Snap executes. Available options are:
  • Validate & Execute: Performs limited execution of the Snap and generates a data preview during pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during pipeline runtime.
  • Execute only: Performs full execution of the Snap during pipeline execution without generating preview data.
  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Default value: Execute only

Default value: Validate & Execute

Examples