Does Django scale?

1. "What are the largest sites built on Django today?"

   There isn't any single place that collects information about traffic on Django built sites, so I'll have to take a stab at it using data from various locations. First, we have a list of Django sites on the front page of the main Django project page and then a list of Django built sites at djangosites.org. Going through the lists and picking some that I know have decent traffic we see:

   • Instagram: What Powers Instagram: Hundreds of Instances, Dozens of Technologies.

   • Pinterest: Alexa rank 37 (21.4.2015) and 70 Million users in 2013

   • Bitbucket: 200TB of Code and 2.500.000 Users

   • Disqus: Serving 400 million people with Python.

   • curse.com: 600k daily visits.

   • tabblo.com: 44k daily visits, see Ned Batchelder's posts Infrastructure for modern web sites.

   • chesspark.com: Alexa rank about 179k.

   • pownce.com (no longer active): alexa rank about 65k. Mike Malone of Pownce, in his EuroDjangoCon presentation on Scaling Django Web Apps says "hundreds of hits per second". This is a very good presentation on how to scale Django, and makes some good points including (current) shortcomings in Django scalability.

   • HP had a site built with Django 1.5: ePrint center. However, as for novemer/2015 the entire website was migrated and this link is just a redirect. This website was a world-wide service attending subscription to Instant Ink and related services HP offered (*).

2. "Can Django deal with 100,000 users daily, each visiting the site for a couple of hours?"

   Yes, see above.

3. "Could a site like Stack Overflow run on Django?"

   My gut feeling is yes but, as others answered and Mike Malone mentions in his presentation, database design is critical. Strong proof might also be found at www.cnprog.com if we can find any reliable traffic stats. Anyway, it's not just something that will happen by throwing together a bunch of Django models :)

There are, of course, many more sites and bloggers of interest, but I have got to stop somewhere!

Blog post about Using Django to build high-traffic site michaelmoore.com described as a top 10,000 website. Quantcast stats and compete.com stats.

_{(*) The author of the edit, including such reference, used to work as outsourced developer in that project.}

We're doing load testing now. We think we can support 240 concurrent requests (a sustained rate of 120 hits per second 24x7) without any significant degradation in the server performance. That would be 432,000 hits per hour. Response times aren't small (our transactions are large) but there's no degradation from our baseline performance as the load increases.

We're using Apache front-ending Django and MySQL. The OS is Red Hat Enterprise Linux (RHEL). 64-bit. We use mod_wsgi in daemon mode for Django. We've done no cache or database optimization other than to accept the defaults.

We're all in one VM on a 64-bit Dell with (I think) 32Gb RAM.

Since performance is almost the same for 20 or 200 concurrent users, we don't need to spend huge amounts of time "tweaking". Instead we simply need to keep our base performance up through ordinary SSL performance improvements, ordinary database design and implementation (indexing, etc.), ordinary firewall performance improvements, etc.

What we do measure is our load test laptops struggling under the insane workload of 15 processes running 16 concurrent threads of requests.

Scaling Web apps is not about web frameworks or languages, is about your architecture. It's about how you handle you browser cache, your database cache, how you use non-standard persistence providers (like CouchDB), how tuned is your database and a lot of other stuff...

Playing devil's advocate a little bit:

You should check the DjangoCon 2008 Keynote, delivered by Cal Henderson, titled "Why I hate Django" where he pretty much goes over everything Django is missing that you might want to do in a high traffic website. At the end of the day you have to take this all with an open mind because it is perfectly possible to write Django apps that scale, but I thought it was a good presentation and relevant to your question.

The largest django site I know of is the Washington Post, which would certainly indicate that it can scale well.

Good design decisions probably have a bigger performance impact than anything else. Twitter is often cited as a site which embodies the performance issues with another dynamic interpreted language based web framework, Ruby on Rails - yet Twitter engineers have stated that the framework isn't as much an issue as some of the database design choices they made early on.

Django works very nicely with memcached and provides some classes for managing the cache, which is where you would resolve the majority of your performance issues. What you deliver on the wire is almost more important than your backend in reality - using a tool like yslow is critical for a high performance web application. You can always throw more hardware at your backend, but you can't change your users bandwidth.

I was at the EuroDjangoCon conference the other week, and this was the subject of a couple of talks - including from the founders of what was the largest Django-based site, Pownce (slides from one talk here). The main message is that it's not Django you have to worry about, but things like proper caching, load balancing, database optimisation, etc.

Django actually has hooks for most of those things - caching, in particular, is made very easy.

I'm sure you're looking for a more solid answer, but the most obvious objective validation I can think of is that Google pushes Django for use with its App Engine framework. If anybody knows about and deals with scalability on a regular basis, it's Google. From what I've read, the most limiting factor seems to be the database back-end, which is why Google uses their own...

Yes it can. It could be Django with Python or Ruby on Rails. It will still scale.

There are few different techniques. First, caching is not scaling. You could have several application servers balanced with nginx as the front in addition to hardware balancer(s). To scale on the database side you can go pretty far with read slave in MySQL / PostgreSQL if you go the RDBMS way.

Some good examples of heavy traffic websites in Django could be:

• Pownce when they were still there.
• Discus (generic shared comments manager)
• All the newspaper related websites: Washington Post and others.

You can feel safe.

The developer advocate for YouTube gave a talk about scaling Python at PyCon 2012, which is also relevant to scaling Django.

YouTube has more than a billion users, and YouTube is built on Python.

My experience with Django is minimal but I do remember in The Django Book they have a chapter where they interview people running some of the larger Django applications. Here is a link. I guess it could provide some insights.

It says curse.com is one of the largest Django applications with around 60-90 million page views in a month.

Note that if you're expecting 100K users per day, that are active for hours at a time (meaning max of 20K+ concurrent users), you're going to need A LOT of servers. SO has ~15,000 registered users, and most of them are probably not active daily. While the bulk of traffic comes from unregistered users, I'm guessing that very few of them stay on the site more than a couple minutes (i.e. they follow google search results then leave).

For that volume, expect at least 30 servers ... which is still a rather heavy 1,000 concurrent users per server.

Another example is rasp.yandex.ru, Russian transport timetable service. Its attendance satisfies your requirements.

You can definitely run a high-traffic site in Django. Check out this pre-Django 1.0 but still relevant post here: http://menendez.com/blog/launching-high-performance-django-site/

Check out this micro news aggregator called EveryBlock.

It's entirely written in Django. In fact they are the people who developed the Django framework itself.