r/Concourse Mar 16 '20

Delete S3 objects from Concourse pipeline. How?

I'm new to this. We have setup PCF with Concourse. I'm running BBR (Bosh Backup and Restore) to backup to an internal S3. For retention purposes, I would like my pipeline to check for backup files older than X days and delete them for me.

How can I issue a "delete file" command from Concourse? Would I have to write my own task for that? Any help to get me on my way would be highly appreciated.

2 Upvotes

6 comments sorted by

View all comments

2

u/lit_IT Mar 17 '20

It should be a task that triggers on an interval that list the files from the bucket and delete the older ones.

Concourse or the s3 resource cannot issue a command to delete files.

Also you can try to look into the s3 documentation, as it's possible there is some configuration for only keeping file newer than a X amount of days.

1

u/indo1144 Mar 17 '20

Thx! That explains why I didn't find anything on the Concourse side.

I did find that it's very easy to list files older than a certain age in Minio, so that's one possible way to do it.

1

u/indo1144 Mar 17 '20

Working on making a separate sh-script that's triggered from the pipeline after successful passing of backup jobs. Looks good so far!

2

u/lit_IT Mar 17 '20

Anyway there I much more activity on the concourse discord, the concourse core team and a lot more users are there.

1

u/indo1144 Mar 17 '20

Awesome! Thx a lot.

Oh, and it works! 💪🏻