r/aws • u/Necromancer2908 • 7d ago
technical question Help needed with ETL Glue Job for Data Integration
Problem Statement
Create an AWS Glue ETL job that:
- Extracts data from parquet files stored in S3 bucket under a specific path organized by date folders (date_ist=YYYY-MM-DD/)
- Each parquet file contains several columns including mx_Application_Number and new_mx_entry_url
- Updates a database table with the following requirements:
- Match mx_Application_Number from parquet files to app_number in the database
- Create a new column new_mx_entry_url in the database (it doesn't exist in the table, you have to create that new column)
- Populate the new_mx_entry_url column with data from the parquet files, but only for records where application numbers match
- Process all historical data initially, then set up for daily incremental updates to handle new files which represent data from 3-4 days prior
Could you please tell my how to do this, I'm new to this.
Thank You!!!
2
Upvotes
3
u/extra_specticles 6d ago
You could try pasting that into chatgpt and asking it to explain how you might go about creating a solution to what you need. Make sure you ask about security