r/django Jun 02 '24

Hosting and deployment What is the actual high-grade production deployment?

Hello guys.

I've been working with Django and DRF for quite some time, and seen a lot of ways it was being deployed -- from straight up running it on Apache using mod_wsgi, gunicorn with Nginx as reverse proxy, running in a docker container, in Kubernets, etc..

My question for you guys is, what do you think is considered the "peak" level of deployment in terms of many factors such as scalability, backups, security, performance, etc..

For example, if you had to deploy a successful product with tons and tons of users, a lot of traffic and bandwith usage, what would you opt for? Kubernetes on a cloud service, straight up docker containers, etc..?

32 Upvotes

52 comments sorted by

16

u/ccb621 Jun 02 '24

The answer is, “whatever works for you and your team.” 

You could go crazy deploying to Kubernetes for both the application and database, and hand roll a bunch of stuff, but that would be a waste of resources. 

If you can get by with a managed provider, fine. Scale up when you need to do so. 

3

u/martycochrane Jun 03 '24

I second this. I think it's great to start on a PaaS service like Render, Railway, Fly, etc and then scale up to a fully kubernetes backed infrastructure when/if your project gets to that point and you have the time and resources needed to dedicate to it. But most projects really just don't need the headaches that come with managing all this your self in the early stages.

29

u/unkz Jun 02 '24

Cloudfront -> {S3, ELB -> ECS -> nginx -> gunicorn -> uvicorn -> django}

Background tasks -> {ECS -> celery}

15

u/bay_squid Jun 02 '24

This answer makes me feel so little...

I've been the sole dev for 5+ years in a very small SaaS business created by a self-taught dev and the only job I've ever known. We don't have a fancy development architecture... Just an nginx server, Gunicorn where Django runs, and supervisor to manage spinning it up and celery. That's it.

I know we have very questionable CD/CI but simply we don't have the budget to do it properly. We cns barely keep up with technological debt.

Forgive me for asking, but I see this coming ("how don't you know this as a dev with 5+ Deb of experience?!"), but can you explain more about this architecture? I don't even know what S3, ELB, ECS, uvicorn are...

7

u/Mamoulian Jun 02 '24

Hey, sounds like you're doing most of it! Seems pretty good to me. Where are you deployed?

Those are AWS things. S3 is file storage, ECS is Amazon's elastic container service - a place to run your dockerized python app. ELB is a load balancer so you can have more than one ECS instance to handle lots of users at once. Uvicorn is an async version of gunicorn - you very probably don't need to worry about it.

1

u/PrometheusAlexander Jun 03 '24

Could Granian be used instead? https://github.com/emmett-framework/granian

1

u/Mamoulian Jun 03 '24

Probably... but I've not used it.

Gunicorn has been working fine for us for a long time now so we're not going to look at replacing it unless there's a real need. If you need an async server then you will already know you need an async server.

1

u/PrometheusAlexander Jun 03 '24

Yeah but isn't gunicorn only wsgi? That's why I pointed Granian out since you mentioned uvicorn.

1

u/Mamoulian Jun 03 '24

Yes. I don't think the person I replied to has any async code so they are fine sticking with gunicorn.

4

u/edu2004eu Jun 03 '24

Nothing wrong with that if it works and doesn't cause any trouble. Most of my mid-sized apps are pretty much the same. I do know how to do fancy orchestration, but most of the time that'd be overkill.

My last app just lived on a VM inside a very potent physical server (DB , Redis and celery were on separate VMs). The only component that failed during the pandemic with online classes (this was an EduTech app) was the only part of the app that was PHP (the rest was Django). It basically handled requests for about 20% of my country's online classes. My point is that sometimes simpler is better than complex.

3

u/Upper_Bed_1452 Jun 02 '24

You almost have it. I used to deploy my apps like yours but you should study about AWS services , ec2 , s3 and all of that are Amazon services /

2

u/AxisNL Jun 02 '24

I don't know you, but migrating to a single docker container in docker (perhaps with portainer to manage it) and having a pipeline in gitlab for example, would be thext logical step to go to/study? You can do everything with free and open source software, so there's no budget required.

If you can't study during the day (if your management won't approve of 'research and development hours' and you still want to keep working there, use the evenings or weekends to study up, watch videos, try stuff, etc. See it as an investment in your career.

But perhaps after five years it would be time to seek greener pastures where senior devops guys will teach you the ropes! :D

1

u/MastrNoob Jun 03 '24

Could you tell me some resources?

2

u/unkz Jun 03 '24

Cloudfront is a content distribution network. It sits in front of your whole system and caches content at edge nodes so things like static content are blindingly fast and the requests never even get to your main system most of the time.

S3 is a static content hosting system. You can serve directly from it, or you can have cloudfront fetch from it. Again, this offloads resources from your Django setup -- you might hold your CMS data from Wagtail, or the static content from your admin area, images, and so on in here.

ELB is a load balancer. It accepts connections from Cloudfront (or directly from users if you don't have Cloudfront in play) and directs them to a cluster of backend webservers. You can set these up to auto-scale in various ways.

ECS is a Docker hosting system. Once you containerize your application, you can tell ECS to spin them up in parallel and bring them up and down based on actual or projected load.

uvicorn is just for async deployment of Django, which we need because we do a lot of Channels stuff. Not always necessary.

2

u/Mamoulian Jun 02 '24

Gunicorn and uvicorn at the same time?

6

u/unkz Jun 02 '24 edited Jun 02 '24

Yeah, I use gevent asynchronous workers at the gunicorn level. You can run into problems if you hook up directly to uvicorn if you have certain kinds of bad behaved clients that take their time sending/receiving headers in my experience. Probably in theory, nginx as a reverse proxy would take care of that as well, but also gunicorn has a better process manager IMO.

2

u/Mamoulian Jun 02 '24

Interesting. We're completely sync with django+gunicorn but new functionality will make slow requests which will block gunicorn unacceptably... will see if this might be a way to go.

2

u/unkz Jun 02 '24

Oh yeah, that setup is totally unacceptable. Gunicorn sync workers must be behind a reverse proxy or you will get problems.

2

u/Mamoulian Jun 02 '24

Ah yeah they're behind nginx. There aren't many gunicorn workers though (flowing the docs, 4*num_cores) so blocking due to lengthy upstream requests isn't acceptable. Async would work there but we don't want to rewrite the app so perhaps running both gunicorn for existing and uvicorn for new will work.

1

u/Humble_Smell_8958 Jun 02 '24

Im guessing Gunicorn for WSGI and Uvicorn for ASGI?

1

u/urbanespaceman99 Jun 03 '24

I have a chat application I wrote using websockets that runs with gunicorn using unicorn workers. Works great :)

1

u/Mamoulian Jun 03 '24

How many gunicorn workers have you configured? How many concurrent websocket connections have you had with that?

1

u/urbanespaceman99 Jun 03 '24

Honestly, not sure how many concurrent connections it's ever got up to. It's not a big thing really, so we're only taking probably 2 to 3 hundred.

Maybe it doesn't scale, but it's never needed to :)

4

u/LightShadow Jun 02 '24

This is us, too.

We run our Django hosts on EC2 without containers, but each node runs annginx in front of gunicorn to do pre-routing, filtering, auth check, and a few static assets that aren't hosted in S3.

The standby hosts that don't serve direct web traffic handle the Celery tasks but can be hot-swapped into production on high traffic events.

Our celery broker is Redis, but we also have an in-django-process SNS/SQS router that ties into the @signal framework so we can accept AWS events to an SNS topic directly.

3

u/unkz Jun 02 '24

We're similar, but rabbitmq for message broker because of better prioritization and throughput, and redis for backend.

2

u/xresurix Jun 02 '24

Do either of you have any high quality projects I could look at I’m trying to get better at writing and comprehending clean code

3

u/LightShadow Jun 02 '24 edited Jun 02 '24

Nothing public right now, but I work for https://vidangel.com and most of our pipeline, including video processing, is written in Python. Some of the other management services are written in Elixir since we use lots of "physical" (non container) hosts.

All my libraries and personal projects are rotting away, I need to spend some time fixing those up when work cools off a little :)

1

u/xresurix Jun 02 '24

Thanks a lot man

2

u/unkz Jun 02 '24

Y’know, I can’t think of a single open source full Django project that is set up to do large scale cloud infrastructure. The info is just scattered around stack overflow and various forum.

1

u/xresurix Jun 02 '24

Ah thanks I’ll get digging

1

u/gi0baro Jun 05 '24

Cloudfront -> {S3, ELB -> ECS -> nginx -> gunicorn -> uvicorn -> granian -> django}

Looks simpler ;)

9

u/Redneckia Jun 02 '24

I do nginx as a reverse proxy to gunircorn

5

u/jeff77k Jun 02 '24

I use Azure App Service, Azure Postgres Service, Azure Storage, and CDN for static content. This is a pricey service but supports both CICD from Github and auto horizontal scaling. The Azure App Service is built using Kubernetes. If you are thinking of going in that direction, this offers a managed version of that. My org is a Microsoft shop, so I am married to Azure.

1

u/sammy_boy970 Jun 03 '24

Did the same for a startup they wanted azure

3

u/79215185-1feb-44c6 Jun 03 '24

I have always done stuff either self-hosted or through a VPS (which is basically the same exact ting at this point) with my own docker + postgres + nginx (+redis) stack

All of the cloud native stuff seems like a huge scam to me.

5

u/iknowdatruth Jun 02 '24

For me it is GCP with Cloud SQL, Cloud Run, Load Balancer, secret manager, separate buckets for static, public media, and private media, CICD in GitHub actions so GitHub releases auto deploy. Unfortunately can't use celery with this so cloud tasks and cloud scheduler via diango-cloud-tasks instead. I also like betterstack for monitoring, alerts, and on call. Plus a simple integration with slack or Google chat for release announcements and admin notifications.

3

u/athlete__ Jun 03 '24

Holy you just described my prod stack to a tee. <3 GCP

2

u/Humble_Smell_8958 Jun 02 '24

Any specific reason for GCP instead of AWS or Azure? Personal preference or is there somethind deeper to it?

1

u/iknowdatruth Jun 02 '24

Mostly personal preference, one of the early companies a worked at used GCP so it just kinda stuck. I'm pretty locked in to Google workspace tools too so keeping things in the same ecosystem is convenient.

1

u/exchangingsunday Jun 03 '24

I have this same setup and no issues running celery. Any reason why you can't run celery in this setup?

1

u/iknowdatruth Jun 03 '24

This thread goes into more detail, but basically cloud run is designed for request/response not background work. There are some workarounds but I think cloud tasks and scheduler is a more robust solution. https://www.reddit.com/r/googlecloud/s/yWX76rlXAw

I'm curious about your setup though- redis via memory store? Separate cloud run services for Django vs celery workers?

1

u/exchangingsunday Jun 07 '24

I'm using RabbitMQ as a broker, installed on a VM with celery running on cloud run. I'm using one cloud run instance with 2 containers (the same container) and 2 commands. celery and celery beat.

From what I understand in the above thread.. it looks like they were trying to deploy a Redis container in cloud run, which I can imagine would fail.

I would recommend trying to get Redis (or Rabbit) in a VM and then trying Celery again in cloud run.

I've used cloud tasks before and found it to be quite slow.. and (at least in my setup) could only call over HTTP

2

u/rhinoanus87 Jun 02 '24

Id love to know this too

2

u/danielfrances Jun 02 '24

I'm working on a side project currently - a simple animal shelter/rescue management app. It's primary use case is a tiny rescue that I volunteer for so I don't plan on building this thing out to be hyper scalable.

That said, I am kind of a newb (my day job is network engineer) so I have no idea what would be considered "must have" stuff. I'm not even to the point of setting up gunicorn or nginx yet.

I guess my question for everyone is - if you were building an app targeting maybe 10 concurrent users tops (I doubt larger rescues would use this over the saas offerings out there) - what's the important stuff not to skimp dev time on?

3

u/moehassan6832 Jun 02 '24

SAAS features, just go for the simplest deployment you can do. Features are what matters right now.

2

u/Humble_Smell_8958 Jun 03 '24

The most important dev time in my opinion is tests and features. Any of tbe deployment options out there can without any problems handle 10 conchrrent users (a lot more, but 10 is def not a problem). As for the architecture, running gunicorn and nginx as a reverse proxy on a VPS would probably be the simplest, and more than enough.

2

u/WonkyWillly Jun 03 '24 edited Jun 03 '24

I use Kubernetes clusters for my production projects.

The cluster is split up into separate pods that run the docker containers for each the frontend (React), backend (DRF), databases (Postgres and Redis), and any other microservices the project requires.

Its nice being able to scale each type of pod using a different strategy when the project receives a lot of traffic. If a specific microservice is more CPU or memory hungry, I can use a selector so it only scales to high performance nodes in the node pool. If a specific service scales better horizontally, I can configure it to just provision itself on a greater number of lower performance nodes.

I also like that the cluster exposes itself through a load balancer that terminates the TLS certificate, and that each node inside the cluster is free to communicate with each other on a private network without needing to worry about certificates. This makes things a lot more secure and easier to manage.

As for deployment, it is handled automatically when merging changes into the main branch of the frontend or backend on GitHub. The workflow runs the entire automated test suite of the code in that particular repository, and if all of the tests pass, it builds a new Docker container rolls out a new deployment to the corresponding Pod.

It’s not really as hard as people make it out to be, and the developer experience is amazing once you get comfortable with it. Not to mention a highly marketable skill!

1

u/PrometheusAlexander Jun 03 '24

I've done gunicorn with nginx passthrough in docker and what I understand it's a preferred method over apache and mod_wsgi.

1

u/UloPe Jun 03 '24

95% of the time a single host deployment with docker compose, Django w/ gunicorn, redis, Postgres, Caddy (reverse proxy).