Boost your ETLs with Data in The Fast Lane - DataFriday 1x23

Description:
Extract, transformations, and load (ETL) operations can become quite complex. In this short video, I'll go over what is an ETL process and how you can simplify those processes with a new tool: Data in the Fast Lane (DFL).

On 10/19/2020, Advance Auto Parts announced the Advance Open Source Garage, and part of this announcement, a new tool has been released. Data in the Fast Lane (DFL) is an easy-to-use ETL based on Apache Spark. See it as a way to script Spark.

I will first remind everyone of what is an ETL, then I will go over two demos: the first demo simply displays the content of a CSV file as well as some other information like the schema and the number of records. The second demo will take two CSV files and upload them in a PostgreSQL database.

DFL leverages simple recipes written in YAML, allowing you to easily write data pipelines.

Data in the Fast Lane can be downloaded from GitHub: https://github.com/AdvanceAutoParts/datafastlane.

Demo #1:
GitHub link: https://github.com/AdvanceAutoParts/datafastlane/blob/master/doc/demo/recipe-demo01-display-csv.yaml

Demo #2:
GitHub link: https://github.com/AdvanceAutoParts/datafastlane/blob/master/doc/demo/recipe-demo02-load-data-into-pg.yaml

Advance Open Source Garage is on Twitter at @AdvanceOSS. More information and documentation are coming to http://medium.com/AdvanceOSS.

This event is supported by La French Tech Raleigh-Research Triangle (https://www.linkedin.com/company/frenchtechraleigh).

#DataFriday #LifeInAdvance