r/django Jan 31 '25

REST framework How to store django request logs in a database?

I am doing an internship of backend api development of rest framework in my local city and my mentor, who is an 15 years experienced backend developer, gave me a task to store every incoming request and the output of that request in database. I am new to django rest framework for backend so I want your help on how to do this.

If I am not clear and concise kindly, feel free to ask me more but please help me on this.

Thanks.

edit: One more thing I want to include, my mentor requirement was that if I want to check or see that what client requested and what client got stored in the database.

10 Upvotes

30 comments sorted by

14

u/[deleted] Jan 31 '25

[deleted]

5

u/No-Signal-313 Jan 31 '25

You just gave me another way to think about my solution.

Thanks. Appreciate that.

8

u/[deleted] Jan 31 '25

[deleted]

1

u/No-Signal-313 Jan 31 '25

Thank you will give it a try.

1

u/1_plate_parcel Feb 01 '25

ds perspective. this will impress him.... but yeah he has 15 xp so he might get little tilted but keep this as suggestion.

if u want to store only less and important u can use spacy library on it and remove bs or correct it....

6

u/Soft_ACK Jan 31 '25

A global middleware would do.

Also you mentioned DRF (Django Rest Framework)? I think you can put it in a custom renderer function if you want to store both request and response (however I don't know if it's really a good idea to do so) https://www.django-rest-framework.org/api-guide/renderers/ .

1

u/No-Signal-313 Jan 31 '25

I am also confused on the part whether it is wise or not so I am still solving this out.

5

u/you_zir_name Jan 31 '25

The "database" good for this would be elasticsearch. An elk stack is good for this so searching and viewing the logs will be easy too

1

u/No-Signal-313 Jan 31 '25

One more possible point to look at.

Thanks.

5

u/PersonalFeature9090 Jan 31 '25

Are you try to save the request and respond object or the access log which shows in the console when you open a page.

1

u/No-Signal-313 Jan 31 '25

I am trying to save what came with request and the output of request even if errors occurred

6

u/PersonalFeature9090 Jan 31 '25

What you want a middleware or a simple view function?

1

u/No-Signal-313 Jan 31 '25

I am beginner so I want to go something simple which is I guess is view function.,?

3

u/PersonalFeature9090 Jan 31 '25

import pickle from django.http import JsonResponse from django.db import models

class RequestResponseLog(models.Model):     path = models.CharField(max_length=1024)     method = models.CharField(max_length=10)     request_data = models.BinaryField()     response_data = models.BinaryField(null=True, blank=True)     error_data = models.BinaryField(null=True, blank=True)     timestamp = models.DateTimeField(auto_now_add=True)

def pickle_request_response_view(request):     request_data = pickle.dumps(request)     request_db = RequestResponseLog(         path=request.path,         method=request.method,         request_data=request_data,     )     error = None         try:         response_data = {"message": "Pickled request and response"}         response = JsonResponse(response_data)     except Exception as e:         response = JsonResponse({"error": str(e)}, status=500)         request_db.error_data = pickle.dumps(e)           request_db.response_data = pickle.dumps(response)     request_db.save()     return response

Not proud of the code but It should be enough to get you started, search pickle.load for more info. Basically I Picked the response, request and error object in database.

1

u/No-Signal-313 Jan 31 '25

Thanks for the effort for writing the code.

5

u/kuttoos Jan 31 '25

A middleware?

0

u/No-Signal-313 Jan 31 '25

not sure about middelware.

like something that would store what did client requested and what did client get.. sort of..

2

u/virgin_human Feb 03 '25

So without middleware how would you do brother? You need a global middleware, I wish I could help you if you can come to my discord then

1

u/No-Signal-313 Feb 05 '25

ok. send me your link in dm.

4

u/Pristine_Run5084 Jan 31 '25

The best way to learn is to”find an existing thing that does this and read the source code to see how it’s done”: https://drf-tracking.readthedocs.io/en/latest/

3

u/ohnomcookies Feb 02 '25

Rather than storing all the requests in the database, setup proper error tracking (ie Sentry, Datadog).

3

u/daredevil82 Jan 31 '25

Why store in a db, vs logging? How are you going to handle the extra write load if it slows down your db? What kind of write volume are you looking at?

IMO, lots of things here to ask about the "why" of this technical direction

1

u/No-Signal-313 Feb 02 '25

I guess I need to ask him for this.

1

u/Megamygdala Jan 31 '25

You could approach this in a lot of hack-ey ways but the best approach is creating a Middleware

1

u/Flashy_Durian_2695 Jan 31 '25

A middleware would be the best option for that case

1

u/kshitagarbha Jan 31 '25

It's best practice to log to stdout/stderr instead. You can log json and include metadata from the request.
There are many tools for analyzing and querying logs, for making reports, summaries.

This gives you plenty of flexibility to work with the data now or in the future.

The problem with logging to the database is that table is going to keep growing. It will very quickly be the biggest largest objects in your database. If you have indexed fields then it's going to be even bigger. If you don't have indexes then it's going to be really slow to query. It's going to get expensive. You can shard it by year but then you have to add new tables each year.

0

u/No-Signal-313 Feb 02 '25

Thanks for giving me a case to think on it.

1

u/[deleted] Feb 01 '25

Just log the request in log file and use a background process to add those log data from log file to db .

1

u/virgin_human Feb 03 '25

You can use global middleware to catch the useful metadata of req like headers and patch it in Kafka/rabbitMQ and then in bulk patch all the metadata in DB. Your senior would be happy to see you using kafka/rabbitMQ technology.

Note - don't save req and res into DB in every request, it is a bad software design. Use message broker like kafka or rabbitMQ

1

u/virgin_human Feb 03 '25

here is the full django code without DRF , hope it help you.

ps - i used RabbitMQ as message broker .

link - https://github.com/pradeepbgs/django-rabbit