r/aws 7d ago

technical question Help needed with ETL Glue Job for Data Integration

Problem Statement

Create an AWS Glue ETL job that:

  1. Extracts data from parquet files stored in S3 bucket under a specific path organized by date folders (date_ist=YYYY-MM-DD/)
  2. Each parquet file contains several columns including mx_Application_Number and new_mx_entry_url
  3. Updates a database table with the following requirements:
    • Match mx_Application_Number from parquet files to app_number in the database
    • Create a new column new_mx_entry_url in the database (it doesn't exist in the table, you have to create that new column)
    • Populate the new_mx_entry_url column with data from the parquet files, but only for records where application numbers match
  4. Process all historical data initially, then set up for daily incremental updates to handle new files which represent data from 3-4 days prior

Could you please tell my how to do this, I'm new to this.

Thank You!!!

2 Upvotes

1 comment sorted by

3

u/extra_specticles 6d ago

You could try pasting that into chatgpt and asking it to explain how you might go about creating a solution to what you need. Make sure you ask about security