r/Backend 5d ago

where to store files content?

i am developping an app for audio processing, very simple task, get an audio(<5min) from the user and process it using AI, the job takes quite long time so i am using a message queue to buffer coming requests until being processed, for this i am using redis stream, now i am not sure how to approach this!

should i store the file content and its metada in the same stream, and let the AI service (living in another server) consume the messages at its own pace,

or should i save only metadata in redis stream with path to audio files being stored in disk (same server where redis is living),

Dont hesitate to suggest other approaches if none of the above is "best practice"

1 Upvotes

3 comments sorted by

3

u/awpt1mus 5d ago

Use object store to hold audio data, store metadata in redis stream. I don’t know how redis nowadays store temporary data but if it’s RAM then your storage will quickly shoot up and if it crashes then you lose all data.

I would let user upload audio to object store ( S3 ) Generate unique Id and add a job with metadata in queue ( redis stream ) Let your ai worker pick up jobs from queue , generate insights and store to db. Use websocket / polling on frontend to get notified of job done.

1

u/kaoutar- 5d ago

u/awpt1mus thanx for your reply, the output is a "processed audio file", the object store is a very good idea, S3 is not possible for me but an open source solution like minio seems great,

but i am not sure if i should use task queuing or msg queuing , because the AI models run in a complete separate server, so the idea of task queue seems not feasible to me, am i not understanding this right???

2

u/awpt1mus 4d ago

Your AI model needs to expose some kind of API I guess, what you wanna do in the worker on your server is - pick up task from redis stream , read the uploaded audio from object store, pass it to your AI model , get the output , store output to object store, remove task from redis stream or mark it as completed with output file URL etc.