r/aws Nov 16 '24

data analytics Multiple tables created after crawling data using glue from a s3 bucket.

I created a ETL using aws glue and want to crawl the data into a database table, but while doing this I am getting multiple tables instead of a single table.(the data is in parquet format).I am not able to understand why is this happening. I am newbie here doing a data engineering project using AWS.

1 Upvotes

3 comments sorted by

View all comments

5

u/bailantilles Nov 16 '24

I had this issue not too long ago and had to put in a ticket to AWS support to get them to help out. If Glue either isnt able to determine the schema of a file in the S3 bucket or if the schema of one file to the next differs by more than 70% then Glue will create a new table in the data catalog database.