Parquet Parser

Overview

You can use this Snap to read the binary (Parquet) data from the input and write the document data to the output.



Prerequisites

None.

Snap views

View Description Examples of upstream and downstream Snaps
Input Requires binary (Parquet) input to convert it into the document.
Output Writes the document data as output.
Note: The second output view provides the schema model of the parquet file.
Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab. The available options are:

  • Stop Pipeline Execution Stops the current pipeline execution when an error occurs.
  • Discard Error Data and Continue Ignores the error, discards that record, and continues with the remaining records.
  • Route Error Data to Error View Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap settings

Legend:
  • Expression icon (): Allows using pipeline parameters to set field values dynamically (if enabled). SnapLogic Expressions are not supported. If disabled, you can provide a static value.
  • SnapGPT (): Generates SnapLogic Expressions based on natural language using SnapGPT. Learn more.
  • Suggestion icon (): Populates a list of values dynamically based on your Snap configuration. You can select only one attribute at a time using the icon. Type into the field if it supports a comma-separated list of values.
  • Upload : Uploads files. Learn more.
Learn more about the icons in the Snap settings dialog.
Field / Field set Type Description
Label String

Required. Specify a unique name for the Snap. Modify this to be more appropriate, especially if more than one of the same Snaps is in the pipeline.

Default value: Parquet Parser

Example: Transform Parquet Parser
Ignore empty data Checkbox

When you select this checkbox, the Snap does not generate any output. If you deselect this checkbox, the Snap generates an empty output document when you pass either an empty input stream or an input stream with CSV headers.

Default status: Selected
Snap execution Dropdown list
Choose one of the three modes in which the Snap executes. Available options are:
  • Validate & Execute: Performs limited execution of the Snap and generates a data preview during pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during pipeline runtime.
  • Execute only: Performs full execution of the Snap during pipeline execution without generating preview data.
  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Default value: Validate and Execute

Example: Execute only

Example Schema Model for the Parquet Parser Second Output View

[
  {
    "schemaMap": {
      "AUTOSYNC_PRIMARYKEY": {
        "logicalType": "STRING",
        "repetition": "OPTIONAL",
        "primitiveType": "BINARY",
        "typeLength": 0
      },
      "AUTOSYNC_VALUEHASH": {
        "logicalType": "STRING",
        "repetition": "OPTIONAL",
        "primitiveType": "BINARY",
        "typeLength": 0
      },
      "AUTOSYNC_CURRENTRECORDFLAG": {
        "logicalType": "STRING",
        "repetition": "OPTIONAL",
        "primitiveType": "BINARY",
        "typeLength": 0
      },
      "AUTOSYNC_EFFECTIVEBEGINTIME": {
        "logicalType": "TIMESTAMP(MILLIS,true)",
        "repetition": "OPTIONAL",
        "primitiveType": "INT64",
        "typeLength": 0
      },
      "AUTOSYNC_EFFECTIVEENDTIME": {
        "logicalType": "TIMESTAMP(MILLIS,true)",
        "repetition": "OPTIONAL",
        "primitiveType": "INT64",
        "typeLength": 0
      },
      "ID1": {
        "logicalType": null,
        "repetition": "OPTIONAL",
        "primitiveType": "DOUBLE",
        "typeLength": 0
      },
      "ID2": {
        "logicalType": "STRING",
        "repetition": "OPTIONAL",
        "primitiveType": "BINARY",
        "typeLength": 0
      },
      "ID3": {
        "logicalType": "STRING",
        "repetition": "OPTIONAL",
        "primitiveType": "BINARY",
        "typeLength": 0
      },
      "ID4": {
        "logicalType": "STRING",
        "repetition": "OPTIONAL",
        "primitiveType": "BINARY",
        "typeLength": 0
      },
      "ID5": {
        "logicalType": "STRING",
        "repetition": "OPTIONAL",
        "primitiveType": "BINARY",
        "typeLength": 0
      },
      "ID6": {
        "logicalType": "STRING",
        "repetition": "OPTIONAL",
        "primitiveType": "BINARY",
        "typeLength": 0
      },
      "ID7": {
        "logicalType": "STRING",
        "repetition": "OPTIONAL",
        "primitiveType": "BINARY",
        "typeLength": 0
      },
      "ID8": {
        "logicalType": null,
        "repetition": "OPTIONAL",
        "primitiveType": "BINARY",
        "typeLength": 0
      },
      "ID9": {
        "logicalType": null,
        "repetition": "OPTIONAL",
        "primitiveType": "DOUBLE",
        "typeLength": 0
      },
      "ID10": {
        "logicalType": null,
        "repetition": "OPTIONAL",
        "primitiveType": "DOUBLE",
        "typeLength": 0
      },
      "ID11": {
        "logicalType": null,
        "repetition": "OPTIONAL",
        "primitiveType": "DOUBLE",
        "typeLength": 0
      },
      "ID12": {
        "logicalType": null,
        "repetition": "OPTIONAL",
        "primitiveType": "DOUBLE",
        "typeLength": 0
      },
      "ID13": {
        "logicalType": null,
        "repetition": "OPTIONAL",
        "primitiveType": "DOUBLE",
        "typeLength": 0
      },
      "ID14": {
        "logicalType": null,
        "repetition": "OPTIONAL",
        "primitiveType": "DOUBLE",
        "typeLength": 0
      },
      "ID15": {
        "logicalType": "DATE",
        "repetition": "OPTIONAL",
        "primitiveType": "INT32",
        "typeLength": 0
      },
      "ID16": {
        "logicalType": "TIMESTAMP(MILLIS,true)",
        "repetition": "OPTIONAL",
        "primitiveType": "INT64",
        "typeLength": 0
      },
      "ID17": {
        "logicalType": "TIMESTAMP(MILLIS,true)",
        "repetition": "OPTIONAL",
        "primitiveType": "INT64",
        "typeLength": 0
      },
      "ID18": {
        "logicalType": "TIMESTAMP(MILLIS,true)",
        "repetition": "OPTIONAL",
        "primitiveType": "INT64",
        "typeLength": 0
      },
      "ID100": {
        "logicalType": null,
        "repetition": "OPTIONAL",
        "primitiveType": "DOUBLE",
        "typeLength": 0
      }
    },
    "schema": "message document {\n  optional binary AUTOSYNC_PRIMARYKEY (STRING);\n  optional binary AUTOSYNC_VALUEHASH (STRING);\n  optional binary AUTOSYNC_CURRENTRECORDFLAG (STRING);\n  optional int64 AUTOSYNC_EFFECTIVEBEGINTIME (TIMESTAMP(MILLIS,true));\n  optional int64 AUTOSYNC_EFFECTIVEENDTIME (TIMESTAMP(MILLIS,true));\n  optional double ID1;\n  optional binary ID2 (STRING);\n  optional binary ID3 (STRING);\n  optional binary ID4 (STRING);\n  optional binary ID5 (STRING);\n  optional binary ID6 (STRING);\n  optional binary ID7 (STRING);\n  optional binary ID8;\n  optional double ID9;\n  optional double ID10;\n  optional double ID11;\n  optional double ID12;\n  optional double ID13;\n  optional double ID14;\n  optional int32 ID15 (DATE);\n  optional int64 ID16 (TIMESTAMP(MILLIS,true));\n  optional int64 ID17 (TIMESTAMP(MILLIS,true));\n  optional int64 ID18 (TIMESTAMP(MILLIS,true));\n  optional double ID100;\n}\n"
  }
]

Examples