SQS Producer

Overview

This Snap sends messages to the Amazon Simple Queue Service (SQS) for consumption. The Snap supports standard queues as well as FIFO queues. Multiple messages can be sent using the Snap by batching as well as by client-side buffering.

This Snap loads the messages
  • Extract Binary data (including header) from the upstream Snap containing the message to be sent to the queue.
  • Transform the input Binary into the message to be sent.
  • Load the data into the Producer Snap and send the message to the target queue.


    Amazon SQS Producer Overview

Prerequisites

  • Valid AWS account
  • Access Key and Secret Key
  • IAM Role access

Known issues

The Snap conforms to the limits within SQS. Hence, the following must be kept in mind when configuring the Snap:

  • The maximum allowed size for the message, including message attributes, is 256KB. For a workaround to overcome this please refer to Messages larger than 256 KB
  • The maximum number of inflight messages per queue are:
    • Standard queue - 120,000 inflight messages per queue.
    • FIFO queue - 20,000 inflight messages per queue.
  • The queue name for a FIFO queue can be a maximum of 80 characters long, including the ".fifo" suffix.
  • Queue names are case-sensitive.

    For more information on message-related limits refer to Message-limits

The AWS extended library uses S3 only if the message size is more than 256 KB. However, you cannot enable the Client-side buffering option in Amazon SQS Snaps if you select the Enable Large Message Support checkbox in your SQS account.

Snap views

View Description Examples of upstream and downstream Snaps
Input This Snap has exactly one document input view.

The input includes binary data containing the message to be sent to the queue.

Output This Snap has at most one document output view.

The output contains the ID of the message.

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab. The available options are:

  • Stop Pipeline Execution Stops the current pipeline execution when an error occurs.
  • Discard Error Data and Continue Ignores the error, discards that record, and continues with the remaining records.
  • Route Error Data to Error View Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap settings

Legend:
  • Expression icon (): Allows using pipeline parameters to set field values dynamically (if enabled). SnapLogic Expressions are not supported. If disabled, you can provide a static value.
  • SnapGPT (): Generates SnapLogic Expressions based on natural language using SnapGPT. Learn more.
  • Suggestion icon (): Populates a list of values dynamically based on your Snap configuration. You can select only one attribute at a time using the icon. Type into the field if it supports a comma-separated list of values.
  • Upload : Uploads files. Learn more.
Learn more about the icons in the Snap settings dialog.
Field / Field set Type Description
Label String

Required. Specify a unique name for the Snap. Modify this to be more appropriate, especially if more than one of the same Snaps is in the pipeline.

Default value: SQS Producer

Example: SQS Producer
Region String/Expression/ Suggestion Specify the AWS region where the application is running. This is a suggestible field and suggestions will be populated based on the Account settings.
Important:

FIFO Queues - Regions

FIFO Queue is available only in US East (N. Virginia), US East (Ohio), US West (Oregon), and EU (Ireland) regions.

Default: None

Example: us-east-1
Queue name String/Expression/ Suggestion
Specify the name of the target queue. This is a suggestible field and suggestions will be populated based on the region selected.
Important:

FIFO Queues - Queue name

When using FIFO queues, it is important that the queue name be suffixed with ".fifo". The maximum character limit for the queue name is 80 characters (including the ".fifo" suffix.

Default value: None.

Example: snapqatest
Queue URL String/Expression

Conditional. Specify the URL for the Queue, this can be found through the AWS console under the Details section. When this property is configured, the queue specified in Queue nameproperty will be overridden.

The URL must be in this format: https//sqs.<region>.amazonaw...t-id>/<queue-name>;

Default value: None.

Example: https://sqs.amazonnaws.com
Maximum retries Integer/Expression
Specify the maximum number of retries to be made in case of connection failures.
Important:
  • If the value is larger than 0, the Snap first downloads the target file into a temporary local file. If any error occurs during the download, the Snap waits for the time specified in the Retry interval and attempts to download the file again from the beginning. When the download is successful, the Snap streams the data from the temporary file to the downstream Pipeline. All temporary local files are deleted when they are no longer needed.
  • Ensure that the local drive has sufficient free disk space to store the temporary local file.

Default value: 3

Example: 2
Delay Second Integer/Expression Specify the time duration in seconds for which the message is visible to consumers after being added to the queue.
Important: This property is not applicable to FIFO queues.

Default value: 0

Example: 1
Message group ID Integer/Expression Appears when you select a FIFO queue.

Specify a tag which indicates the messages group the message is associated with. The field is used and required by FIFO queue.

For more information refer to FIFO Queue Recommendations - Message Group Id

Default value: None.

Example: 5
Deduplication ID Integer/Expression Appears when you select a FIFO queue.

Specify the token used for de-duplication of sent messages in FIFO queue. The property is used by FIFO queue only.

For more information refer to Recommendations - Deduplicate Id.

Default value: None.

Example: 2
Message attributes Specify the table of the message attributes. Message attributes are structured metadata for the message to help the consumer identify how to deal with the message without having to read the message body. Each message can have up to 10 attributes. The value type is string.

Value: Specify the value for the message attribute. By default this field is expression enabled.

Example: snap.instanceId
Name

Specify the name for the message attribute. By default this field is expression enabled.

Default: None

Example: snap-runtime
Value

Specify the value for the message attribute. By default this field is expression enabled.

Default: None

Example: snap.instanceId
Batch size Integer/Expression Specify the number of Documents the Snap should batch before processing. The maximum number is 10. This is ignored if client-side buffering is enabled.

Default value: 10

Example: 5
Client-side buffering Checkbox Select this checkbox to enable the Snap to automatically buffer a maximum 10 outbound requests (or less as configured in the Max buffer size property) and send them as a batch and allow message pre-fetching when consuming. The subsequent properties need to be configured if this property is enabled.
Important: This property is not applicable to FIFO queues.

Default status: Deselected
Buffer configurations Appears when you select the Client-side buffering checkbox

Configure the settings for the client-side buffer.
Max buffer size Integer/Expression Specify the maximum number of messages that are batched together in a single request. The higher the max buffer size, the fewer batches are required to carry out the same number of requests. The maximum batch size is 10.
Important: This property is not applicable to FIFO queues.

Default value: 10

Example: 4
Max buffer open time (ms) Integer/Expression Specify the maximum time in millisecond the buffer would wait before sending message batches.

Default value: 200

Example: 200
Max outbound batches in parallel Integer/Expression Specify the maximum number of concurrent batches for each type of outbound request. The greater the number, greater the throughput that can be achieved (at the expense of consuming more threads).

Default value: 5

Example: 3
Snap execution Dropdown list
Choose one of the three modes in which the Snap executes. Available options are:
  • Validate & Execute: Performs limited execution of the Snap and generates a data preview during pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during pipeline runtime.
  • Execute only: Performs full execution of the Snap during pipeline execution without generating preview data.
  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Default value: Execute only

Default value: Validate & Execute

Examples