I love the Linux development model. It works like this: you have two classes of developers. First, a distributed group of developers focus exclusively on the Linux kernel. All communication and design decisions are made through a channel known as Linux Kernel Mailing List. Occasionally, they also have conferences where collaborators meet face to face. It’s fascinating to watch this evolutionary process at work.
Second, you have a rich ecosystem of distribution managers who package the kernel with a select user land. Each distribution has a target user base in mind. Distributions like Ubuntu cater to the more novice user and focus on ease of use and convention over configuration. Distributions like Gentoo or Arch Linux focus on the power user and expect users to spend some effort understanding the system. Yet another class of distributions like Red Hat Enterprise Linux (and its offshoot CentOS) guarantee stability.
Somehow it all works.
I feel a similar approach can work on web frameworks. Let’s look at how a web application is typically developed.
Take One: Enter the Monolithic App
Frameworks like Django or Rails lets a small team rapidly build and iterate web apps. Great for building your first Minimum Viable Product. These frameworks come with batteries included:
- a templating system to render HTML
- an admin interface
- a URL router
- an ORM
- other niceties like sessions, authentication, etc.
This is the simplest way to build web apps. As you grow in terms of users and the size of the development team, you find that the monolithic architecture is slowing you down. It’s getting harder for your team to add features without stepping on each others’ toes. Scaling is also a bottleneck: there are limits to how much hardware you can throw at the problem.
One approach to solve this problem is to break up your monolithic app into multiple smaller, more tightly focussed services. Let’s look into this a little more.
Take Two: Service Oriented Architecture
An architecture built upon services differs from a monolithic one in many ways:
- your entire application is broken into smaller units of functionality known as services
- each service conforms to an interface and all implementation details are abstracted away
- each service talks to other services through a standard protocol such as HTTP
- each service has its own resources: queues, databases, etc.
- each service is deployed and monitored independently
- each service has its own unit and integration tests
While conceptually this architecture can be easy to understand, there’s a dearth of knowledge and off-the-shelf frameworks to help you make this transition. Each team has to burn engineering cycles building the same building blocks and patching together adhoc, poorly tested solutions.
I believe there’s room for a new framework to fill this gap.
Why Flask isn’t it
Flask is a popular microframework in the Python world. It intentionally keeps a small core and leaves the choices of object-relational mappers, user authentication, and database upto you. There’s a rich ecosystem of extensions to make Flask featureful, though at times it takes a bit of research to find the right extension.
A problem with Flask in my experience is that it requires some work to structure your application. Without careful thought, your models, views and templates can get out of hand and messy. Sure, you can create internal rules to structure your code, but this increases the time it take to onboard a new developer. Django wins in this area because there’s an accepted way to layout an application.
While building a full range application, you need user authentication, sessions, password resets, etc. It can quickly get tiresome to re-build or find extensions in the Flask world.
A New Distribution
I’m proposing a new distribution of Django that eases the transition from a monolithic architecture to a service oriented architecture.
To make this happen, a few things will have to fall in place:
View Dispatch: Today the dispatcher system takes an incoming URL request, matches it against a list of regular expressions, and routes the request to that view. With a plugable system, you can inject a RPC mechanism. Instead of dispatching to a function or class, it makes another request to a remote service to process the request and return a result.
Non-blocking: A big challenge with a distributed system is the latency variance and failure modes. Therefore, all service calls need to be non-blocking. I’m excited about the upcoming asyncio features in upcoming Python version.
Templates: The standard use-case of Django returns a reponse upon rendering a template. This should be extended to simply accept Python native objects. Then depending upon configuration, the Python object is serialized to JSON or a more space efficient binary format.
Testing: The Django testing framework is designed for a monolithic application. Beyond the current unit testing methods, you’ll need ways to do integration tests across services.
What do you think? Is this of interest to anyone?
Photo Credit: gagilas.
