Extreme Web UI Scalability – Part 2

Yesterday I did a massive post about separating HTML layout from user data and then using a JavaScript templating engine to bind up JSON data object which contains user-specific information.  I also talked about setting the HTTP expiration policy on the HTML files and using CDNs to scale out your web tier without lots of additional hardware.

In thinking about the solution a little bit further it occurred to me that this could facilitate a RESTful layout of both HTML and JSON URLs.  Specifically, a sample HTML URL would look like the following:

/account/history

While the corresponding JSON data file would have the following URL:

/account/history.js

Then, through URL rewriting, the JSON file would use the session state identifier in the request to map the a directory on the file system, e.g.:

/<session-state-id>/account/history.js

which would use hard links to map to the real (but hidden) user directory on the server:

/<randomly-generated-never-disclosed-user-specific-guid>/account/history.js

I know this sounds like way too much overhead, but as I said before, this solution is not mutually exclusive.  You can do this where it makes sense and when you have significant load on certain pages.  Elsewhere you can do your traditional request that goes all the way to the server, hits a database, and renders a page.

The last thought that I had was that the JSON files need not be stored on the file system.  You could simply expose a cluster of CouchDB instances (with appropriate security controls, of course).  The CouchDB instance(s) would be responsible for handling the raw request, mapping the session state to the underling user ID, and then pushing out the appropriate JSON file for the requested view.  In this way, you could potentially index the JSON files and make them queryable if you desire.

Postulating Extreme Web UI Scalability

I’ve been working with CQRS for almost a year.  Ever since I first heard about it, I was immediately convinced that it had the ability to vastly simplify a lot of unnecessary development overhead.  In this particular post, I will not be discussing event sourcing which is often mingled with CQRS.  Instead, I will be focusing on the query side of the equation.

Please note that the ideas contained herein are merely conjectures of what might be rather than any kind of working, concrete implementation.  I’ve thought my way through some aspects and ramifications of implementing what I mention below but new insights will undoubtedly merit additional consideration.  Furthermore the solutions outlined below need not be considered mutually exclusive with more traditional ways of doing things.  There is definitely room to implement portions of what I mention for specific views while doing things in the traditional way in other views.

ViewModel Tables

Several weeks ago I blogged about having one table per view.  Since that time Udi Dahan has mentioned it as well, although independently of my post.  Querying view model tables directly, which comes as a positive consequence of implementing CQRS, enables us to avoid having numerous mapping layers between the database and the view.  Further, it allows us to avoid projecting off of our domain model and keeps the domain model focused solely on behavior.  Lastly, it allows us to scale the read database independently of the write database.

Content Delivery Networks

Think for a moment about the pure static resources that are handled on a given website.  They just sit there with the exact same content for each request.  These are the exact kinds of resources for which we could leverage a content-delivery network.  In other words, we could push these static elements to a CDN and have the CDN deliver them for us thus offloading  traffic from our servers and freeing our web tier to perform more interesting tasks—such as processing update-type commands.  Not only does this offload work from our web tier, but the user has a better and faster browsing experience.

HTML + JSON

In ASP.NET as well as other server-based page-rendering technologies we typically have the request come all the way to our server and then we render some HTML but with a few modifications depending upon the user.  Could there be a way to somehow render static layout HTML but render the parts that vary separately?  Why not have the pages render identical static HTML regardless of user and then, on a separate request, render the user-specific content and then stitch the two together using JavaScript?

Let me say that another way.  Our MVC view or our ASP.NET “Page” could render pure HTML with absolutely no server-side markup.  Inside of that HTML we would have the URL of an embedded resource: <script src=”/scripts/account-history/”></script>.  The request to this embedded resource would call our MVC controller and then render JSON data with user-specific information for the currently logged-in user for that view.  We could then use a client-side JavaScript templating engine such as jTemplates to combine the JSON with the static HTML.

Static HTML Views + CDNs

The above has some interesting ramifications.  Why couldn’t we just put the static HTML for each view into an HTML file and push those HTML files to a CDN?  The static HTML files would have <script> tags inside which would make HTTP requests to our controller actions.  Of course we would need to figure out a way to render each view with its associated header/footer HTML or perhaps use JavaScript to get the various HTML parts of the page and assemble them on the client.

By doing this, our web tier is now responsible for exactly two things: outputting user-specific data in JSON format and handling of mutating/update-type POST commands from the browser.

CDNs + JSON

Here’s where things get interesting.  What if we didn’t have to query the database through our controller/service layer to render the JSON?  What if the request for the user-specific data never reached the web server?  What if the user-specific JSON files were already on the CDN?  Think about that for a minute.  If that was somehow possible, our web servers would only be responsible for handling POSTs which contained task-specific commands.  Wouldn’t that make the web tier massively scalable?  Wouldn’t it be easier to handle exponentially larger loads on the web tier without the need for additional hardware?  Even if we didn’t use a CDN and we did everything ourselves, Apache and IIS are really, really good at serving up static files when compared to the expense of going through the ASP.NET pipeline.  [This applies to any server-side technology such as Ruby, PHP, Perl, Java, etc.]

How is this accomplished?  In addition to what has been mentioned throughout this post of splitting up the HTML and the JSON requests, there are two additional pieces.  First, you would want to have a component subscribing to events being published by your domain model.  This component would then render the user-specific JSON for each view or page.   Second, we would need a way to map the incoming request for a particular JSON file at a “general” location, e.g. “/scripts/account-history” to a user-specific location, e.g. “/scripts/user-12345/account-history.js”.

URL Rewriting + Session State ID + Hard Links

That last part is the tricky part because we want to ensure that the user’s information is protected and cannot be accessed by others.  The way to do it is with URL rewriting and file system hard links/junction points.  The user-specific JSON file would be written to a known/fixed location, but that location couldn’t be accessed and would never be disclosed over the web.  Instead what we would do is rewrite the URL to be: “/scripts/<session-state-id>/account-history.js”.  This could be accomplished through either server-side rewriting or even client-side rewriting.

If you recall, the session state ID is very, very temporary.  When a user logs out of their account or the session expires, the session state is nullified and disappears.  But we don’t typically think like that about directories.  This is where hard links or junction points come into play.

When a user logs in, a hard link is created which points to the user-specific (but never disclosed) directory.  For example “/scripts/<session-state-id>/account-history.js” would map to: “/scripts/<random-guid>/account-history.js”.  When the login takes place, a small, server-side lookup is performed to get the GUID for the particular user—not the user identifier, but another GUID that is never publicly disclosed.  We then have a server-side process create a hard link between the temporary session state identifier and the GUID.  Conversely, when the user logs out or the session expires, the hard link is removed.  In this way we can protect the user data from prying eyes—there would be no way to access the user data without knowing the session state identifier.

Conclusion

All of this is highly experimental, but conceptually makes sense.  The development tools currently available bring us some but not all of the way there.  Further, there is an amount of overhead in developing such a solution, but the benefits could be significant, especially the separation of concerns aspect and the ability to scale without significant additional effort.  Amazon, eBay, Google, and other big players currently implement some or even most of the above techniques but I don’t think I’ve ever seen all of them put together in quite this way.

There are three other benefits that I just thought of.  First, if your web servers are completely offline, because of HTTP content caching with expirations, your users wouldn’t even notice—until they tried to perform an update/POST.  In this way, your website remains highly available—even if your servers aren’t online!  The second benefit, although much smaller, is that you could more easily render browser-specific implementations of your site, such as those for mobile devices and browsers (which you can already do this using different views in MVC).  Lastly, Google can index your site much more quickly because everything is effectively static, which will soon have a positive effect on your PageRank.

For those of you that have read thus far, I’m curious to hear your comments.

Persistence Ignorance and ActiveRecord

Jimmy Bogard recently tweeted that Persistence Ignorance does not apply to the ActiveRecord pattern.  I couldn’t agree more because ActiveRecord is about creating a strong coupling to the database schema.

Persistence Ignorance is about keeping your business objects ignorant of the underlying mechanism used to persist them and the schema in which they are persisted.  In the case of ActiveRecord, it doesn’t apply because these objects are intimately coupled to the schema itself.  ActiveRecord objects may implement a special base class, have lots of decorator attributes and other such things.  But ActiveRecord objects should also be simple DTOs--ones that can be easily serialized across the wire if necessary.

Just because we are tightly coupled to a database schema does not mean we have to be coupled to any particular database product or vendor.  In other words, we should still be able to swap out a relational database using another one from another vendor without having to modify our application code.

Using CQRS this becomes less than trivial because the only place where we use a relational database is for reporting.  Incidentally, this happens to be the ideal location for a relational database because they’re really, really good at querying data.

By using the proper abstraction to isolate your ActiveRecord-based technology, you should be able to swap out not only your database product, but also your data access technology and your application shouldn’t even notice.

The Software Simplist

Udi Dahan is known as the “Software Simplist”.  What does he mean by simple?  Is he saying that software cannot be used to tackle complex problems?  Or does he mean that we work at a problem until a simple, clean, and concise solution presents itself?  And what about “the simplest thing that could possibly work”?

Here are a few of my favorite quotes regarding simplicity:

"I wouldn't give a fig for the simplicity on this side of complexity; I would give my right arm for the simplicity on the far side of complexity" ~ Attributed to Oliver Wendell Holmes Jr.

“Simplicity does not precede complexity, but follows it.” Alan Perlis

"Complexity means distracted effort. Simplicity means focused effort." ~ Edward de Bono

From these statements we learn that true elegance and simplicity only come after diving head first into a problem and then seeing all the various nuances and angles of the problem.

That’s why I tend to gravitate towards Udi’s down-to-earth approach to tackling software.  It’s when you’ve been through the fire that you gain the perspective and understanding to solve a problem in a simple way.

The next time you’re convinced you’ve found a simple solution, take a step back and consider if you’re on the near or far side of complexity.

CQRS: Reporting Database Access Strategies

One of the things that I love about CQRS is that it completely eliminates the impedance mismatch that you normally experience with traditional ORMs. It is incredibly compelling to be able query and retrieve all values necessary for a particular report or screen in a single database call because of a denormalized database schema.

Over the past year there has been significant debate regarding Microsoft’s LINQ to SQL, Entity Framework, and ADO.NET Data Services data access strategies. Further, NHibernate with the 2.0+ release has become the de-facto standard in many ALT.NET shops. Other solutions such as LLBLGen and Subsonic are also gaining more and more acceptance and traction.

With all of these options, the question becomes: which one should we use? The answer from a CQRS perspective is…it doesn’t matter. That’s right. It doesn’t matter. One of the primary reasons this works is because the only place you need a relational database is in your reporting context and it follows the ActiveRecord pattern where the class structure mimics the table schema. Most data access mechanisms handle this scenario very, very well. But you’ve got to expose your data access through the correct abstraction.

In a project I am working on, we simply created an abstraction of our data access mechanism through an IAccessStorage interface which had a single property with the following signature:

IQueryable<T> Items<T> { get; }

It could be queried using either a “SQL-like” LINQ expression or like this:

storage.Items<User>.Where(user => user.Id == 12345)

As a proof of concept, I created a class to abstract each persistence mechanism, e.g. NHibernateStorage, LinqToSqlStorage, EntityFrameworkStorage, and SubsonicStorage. Using dependency injection, I was easily able to swap out which storage mechanism I used and my application code was none the wiser.

The caveat in all of this is that each one must have a LINQ provider. While I’ve run into some quirks in NHibernate’s LINQ provider related to “joins”, this isn’t really an issue for the reporting database because you don’t really want to be doing joins anyway.

An additional issue that you may notice concerns the DTO/class files generated by the above technologies. This may be a small issue because the “User” object generated by Subsonic is not the same one generated LINQ to SQL or the Entity Framework. In the end, it isn’t that big of a deal because these objects are simple DTOs with zero behavior. As long as they don’t have behavior, you’re okay, so you may end up with a dependency on the DTOs. If you really wanted to maintain purity, you could potentially use a solution like AutoMapper to map the generated objects to your own POCOs. [Update: The “dynamic” keyword in C# is another possible resolution to this.]

Conclusion

With all of the arguing that is happening regarding the various persistence mechanisms and their relative strengths and weaknesses, using CQRS allows us to completely avoid dependencies on any of the aforementioned projects thus allowing us to freely choose whichever one we see fit at that time. And if we make a mistake or discover additional capabilities in another ORM, we can swap implementations with virtually no effort. Thus, we are now doubly insulated from the database and we can swap both database and data access technologies at will.

DDDD: Why I Love CQRS

There are a number of significant advantages to using an emerging pattern known as Command Query Responsibility Segregation (CQRS), formally known as Command Query Separation [Young].

vs. ActiveRecord

ActiveRecord is an incredibly quick and powerful way to create an application to deliver business value as evidenced by the abrupt emergence and fast-growing popularity of Ruby on Rails. Specifically, its power revolves around displaying and mutating objects matching a database schema, thus avoiding such technical complexity as the impedance mismatch problem.

A critical deficiency to ActiveRecord is when business logic reaches a certain level of complexity. Once at that point, ActiveRecord begins to “break down” and it becomes difficult to ensure that all business logic is properly maintained in a single location and handled in a consistent fashion.

Furthermore, all business state is governed by the database. In other words, the database becomes the single point of failure and performance bottleneck. Possible resolutions might include sharding the database but referential integrity and distributed transactions could become an issue. These could be abandoned but that would require more technical complexity to ensure consistent application state. Further, reconstructing a report or screen would then require retrieval of data from disparate sources.

vs. Traditional Domain Model

Domain-Driven Design should be used when business complexity merits the overhead and difficulty of discovering the correct domain model. Otherwise, ActiveRecord or another type of pattern should be used. Even so, DDD has the advantage of isolating all business logic into a single location. Even with "anemic" domain models the logic is still in one place.

A significant downside of traditional DDD is found in the common practice of using an object-relational mapper or similar data access strategy. Specifically, the domain objects may not necessarily be truly “POCO” or “POJO”. For example, in C# you end up having to declare your methods as “virtual” or you may have to create unused, protected constructors. Some ORMs even require you to use a certain base class in order for the ORM to perform its magic.

In addition, many DDDers query their domain objects through a repository for data to display on a report or screen. Many also have their domain objects return a DTO that can be databound to the screen. The problem here is that domain objects are not meant for display—they’re behavioral entities--they have behavior rather than shape. This often results in additional properties on domain model entities to support information needed on a particular report or screen. Lastly, because the domain object is now used for display, there is an extra level of coupling that must be addressed when refactoring domain objects.

CQRS: A Class By Itself

The CQRS style of programming isolates domain complexity into a particular location like its traditional DDD counterpart, but it doesn’t expose state—only behavior through methods that can be invoked and that return "void". In this way, the domain model is strictly behavioral and can more easily be refactored towards deeper insight.

There are a number of significant, positive consequences when using CQRS that you don’t get by default with the aforementioned patterns.

1. Distributed systems capabilities: CQRS facilitates spreading your application across multiple physical machines using messaging patterns.
2. High availability: Because the application is distributed, each piece can continue to function in the absence or failure of any other piece. Further, the distributed nature has a “load leveling” effect whereby spikes in demand are leveled using messaging.
3. External systems integration: Messaging patterns  facilitate the ability to easily replicate both the intent as well as the data of domain events to external systems.
4. NoSQL: CQRS doesn’t even require a database to support the domain model. It only needs a relational database if you choose to use one for reporting. This prevents the reporting database from being a single point of failure or even a bottleneck because it can be scaled independently from the rest of the application.
5. Auditing and historical tracing: I’ve done a lot of work with temporal databases in an attempt to have a complete and accurate audit. (That’s what attracted me to this pattern in the first place.) The problem with temporal DBs and schemas is that the technical complexity quickly overwhelms the business complexity as the two become easily mixed. In CQRS, we capture each state transition explicitly which gives us a complete and accurate historical record that we can fast forward or rewind to any point in time.
6. One repository: There has been some debate over the last few months about the viability of the repository pattern. The argument is that a repository cannot handle all of the querying needs of the application. This is absolutely, 100% correct, which is why we should only use the repository in the domain model to query by ID in order to get a single domain object back. When we need data for a particular screen or report we should use the correct tool for the job and go straight to the database. This is what databases are designed for and CQRS appropriately pushes us in that direction.
7. Screen-based reporting: Rather than trying to bind domain objects to the screen, CQRS tells us to query the database directly and to have simple DTOs which are screen-based objects. This allows us to query all the information necessary for a particular screen in a single request rather than 3, 4, 5, or even 15 DB requests per screen. This also makes development using MVC patterns significantly easier.

Conclusion

While reading technical blog posts or listening to technical podcasts, I am constantly amazed at how elegantly and beautifully CQRS solves a multitude of deficiencies found in more established or “more traditional” approaches.

DDD: Entity Injection

In a former life, I wouldn’t think twice about using dependency injection to put behavior into a domain object.  But the more I learned about traditional DDD, the more I found that it was considered “taboo” to inject things into your entity objects.  Obviously injecting behavior into domain services was okay, but entities were special.  Of course, this topic is still a matter of debate among DDDers and there are people on both sides of the line.

Last night, while I was reviewing some CQRS principles, I decided to watch the original video recording that introduced me to DDDD.  I’ve seen it a few times, but something stuck out at me regarding DI and entity objects.  Specifically at 15:30 in the video Greg talks about DI: “My rule of thumb when it comes to things like dependency injection is that coupling is okay if it’s at the same layer.”

This does not mean that we are free to inject anything that we want into our entity objects.  For example, it is not okay do inject an EmailGatewayService or something like that into our entity—because EmailGatewayService wouldn’t have anything to do with our domain—it’s a purely infrastructure concern.  Instead we should send a message to the EmailGatewayService.  It does mean that we can inject *domain services* for the domain into an entity object.

ORMs vs. DI

I appears that one of the biggest deterrents to using DI with entity objects is that, in traditional DDD, your entity objects are hydrated using an ORM.   Hooking into the hydration process isn’t very fun and can be a little sticky depending on your ORM of choice.  I’ve worked through this before and, while possible, isn’t a particularly elegant solution and very often feels like a hack.

But in DDDD, on the other hand, dependency injection is quite easy.  Because you’re dealing with true POCO/POJO object and you control the full object lifecycle and rehydration via the event stream, you can easily inject domain services into your entities.

Flip Flop?

Does this represent a shift in my previous thinking?  Absolutely.  But the worst thing that we can do as developers is to blindly cling to a convention.  As part of our growth we should be able to re-evaluate decisions in light of new understanding and be able to make more educated decisions as a result.  In other words, yes, I changed my mind--but only because of a new understanding, a new point of view, a new paradigm.

“And the end of all our wanders will be to arrive where you started and to know the place for the first time.”