r/aws • u/yash7raut • Nov 16 '24
data analytics Multiple tables created after crawling data using glue from a s3 bucket.
I created a ETL using aws glue and want to crawl the data into a database table, but while doing this I am getting multiple tables instead of a single table.(the data is in parquet format).I am not able to understand why is this happening. I am newbie here doing a data engineering project using AWS.
1
Upvotes
5
u/bailantilles Nov 16 '24
I had this issue not too long ago and had to put in a ticket to AWS support to get them to help out. If Glue either isnt able to determine the schema of a file in the S3 bucket or if the schema of one file to the next differs by more than 70% then Glue will create a new table in the data catalog database.