Execute PySpark script

The following example pipeline demonstrates how the PySpark Snap executes the PySpark script.

• File Reader
• HDFS Writer
• PySpark

JavaScript and Python

These example script illustrates basic scripts in both languages.

• Script

Script Snap Configuration Using Python, JS Script, and Ruby Script

The following pipeline demonstrates how the Script Snap executes in all three supported languages. This pipeline uses a simple JSON file with First Name, Last Name, and Birthday.

• JSON Generator
• JSON Splitter
• Data
• Copy

Script Snap Execution Using Pipeline Parameters

The following example demonstrates the execution of the Script Snap using the pipeline parameters; the script file in the SLDB is passed as a Pipeline parameter. The pipeline demonstrated above is modified to accept pipeline parameters by configuring the Script file property. For the scripts to be passed as a pipeline parameter, the script file should be present in the project folder in SnapLogic. Confirm that the script files are present in the Files section inside the Manager; if not, upload them by clicking on the '+' icon.

• JSON Generator
• JSON Splitter
• Data
• Copy

Read CSV data with PySpark

The following example pipeline demonstrates how to execute a PySpark script on Windows Groundplex to read and display CSV data. It takes a CSV file path as an argument, reads the file using Spark DataFrame operations, and displays the data content. The pipeline is configured to run locally on Windows Groundplex.

• PySpark

Word count using PySpark script

The following example pipeline demonstrates how to execute a PySpark script on Linux Grouplex to count the number of words in an application that processes text files to count the frequency of each word. It uses the PySpark framework to distribute the computation across a Spark cluster, reading text files, splitting them into words, and counting occurrences of each word using map-reduce operations.

• PySpark