Clustering Design EssentialsThe first step in breaking services into clusters that work, regardless of the details of the implementation, is to make sure that an application can be used in a clustered setup. Every time I give a conference talk, I am approached by a self-deprecating developer who wants to know the secret to building clustered applications. The big secret is that there is no secret: Building applications that don't break when run in a cluster is not terribly complex. This is the critical assumption that is required for clustered applications: Never assume that two people have access to the same data unless it is in an explicitly shared resource. In practical terms, this generates a number of corollaries:
Planning to FailOne of the major reasons for building clustered applications is to protect against component failure. This isn't paranoia; Web clusters in particular are often built on so-called commodity hardware. Commodity hardware is essentially the same components you run in a desktop computer, perhaps in a rack-mountable case or with a nicer power supply or a server-style BIOS. Commodity hardware suffers from relatively poor quality control and very little fault tolerance. In contrast, with more advanced enterprise hardware platforms, commodity machines have little ability to recover from failures such as faulty processors or physical memory errors. The compensating factor for this lower reliability is a tremendous cost savings. Companies such as Google and Yahoo! have demonstrated the huge cost savings you can realize by running large numbers of extremely cheap commodity machines versus fewer but much more expensive enterprise machines. The moral of this story is that commodity machines fail, and the more machines you run, the more often you will experience failuresso you need to make sure that your application design takes this into account. These are some of the common pitfalls to avoid:
Working and Playing Well with OthersIt is critical to design for cohabitation, not for exclusivity. Applications shrink as often as they grow. It is not uncommon for a project to be overspecified, leaving it using much more hardware than needed (and thus higher capital commitment and maintenance costs). Often, the design of the architecture makes it impossible to coalesce multiple services onto a single machine. This directly violates the scalability goal of being flexible to both growth and contraction. Designing applications for comfortable cohabitation is not hard. In practice, it involves very little specific planning or adaptation, but it does require some forethought in design to avoid common pitfalls. Always Namespace Your FunctionsWe have talked about this maxim before, and with good reason: Proper namespacing of function, class, and global variable names is essential to coding large applications because it is the only systematic way to avoid symbol-naming conflicts. In my code base I have my Web logging software. There is a function in its support libraries for displaying formatted errors to users: function displayError($entry) {
//... weblog error display function
}
I also have a function in my general-purpose library for displaying errors to users: function displayError($entry) {
//... general error display function
}
Clearly, I will have a problem if I want to use the two code bases together in a project; if I use them as is, I will get function redefinition errors. To make them cohabitate nicely, I need to change one of the function names, which will then require changing all its dependent code. A much better solution is to anticipate this possibility and namespace all your functions to begin with, either by putting your functions in a class as static methods, as in this example: class webblog {
static function displayError($entry) {
//...
}
}
class Common {
static function displayError($entry) {
//...
}
}
or by using the traditional PHP4 method of name-munging, as is done here: function webblog_displayError($entry) {
//...
}
function Common_displayError($entry) {
//...
}
Either way, by protecting symbol names from the start, you can eliminate the risk of conflicts and avoid the large code changes that conflicts often require. Reference Services by Full Descriptive NamesAnother good design principal that is particularly essential for safe code cohabitation is to reference services by full descriptive names. I often see application designs that reference a database called dbhost and then rely on dbhost to be specified in the /etc/hosts file on the machine. As long as there is only a single database host, this method won't cause any problems. But invariably you will need to merge two services that each use their own dbhost that is not in fact the same host; then you are in trouble. The same goes for database schema names (database names in MySQL): Using unique names allows databases to be safely consolidated if the need arises. Using descriptive and unique database host and schema names mitigates the risk of confusion and conflict. Namespace Your System ResourcesIf you are using filesystem resources (for example, for storing cache files), you should embed your service name in the path of the file to ensure that you do not interfere with other services' caches and vice versa. Instead of writing your files in /cache/, you should write them in /cache/www.foo.com/. Distributing Content to Your ClusterIn Chapter 7, "Enterprise PHP Management," you saw a number of methods for content distribution. All those methods apply equally well to clustered applications. There are two major concerns, though:
The first point is addressed in Chapter 7. The most complete way to ensure that you do not have mismatched code is to shut down a server while updating code. The reason only a shutdown will suffice to be completely certain is that PHP parses and runs its include files at runtime. Even if you replace all the old files with new files, scripts that are executing at the time the replacement occurs will run some old and some new code. There are ways to reduce the amount of time that a server needs to be shut down, but a shutdown is the only way to avoid a momentary inconsistency. In many cases this inconsistency is benign, but it can also cause errors that are visible to the end user if the API in a library changes as part of the update. Fortunately, clustered applications are designed to handle single-node failures gracefully. A load balancer or failover solution will automatically detect that a service is unavailable and direct requests to functioning nodes. This means that if it is properly configured, you can shut down a single Web server, upgrade its content, and reenable it without any visible downtime. Making upgrades happen instantaneously across all machines in a cluster is more difficult. But fortunately, this is seldom necessary. Having two simultaneous requests by different users run old code for one user and new code for another is often not a problem, as long as the time taken to complete the whole update is short and individual pages all function correctly (whether with the old or new behavior). If a completely atomic switch is required, one solution is to disable half of the Web servers for a given application. Your failover solution will then direct traffic to the remaining functional nodes. The downed nodes can then all be upgraded and their Web servers restarted while leaving the load-balancing rules pointing at those nodes still disabled. When they are all functional, you can flip the load-balancer rule set to point to the freshly upgraded servers and finish the upgrade. This process is clearly painful and expensive. For it to be successful, half of the cluster needs to be able to handle full traffic, even if for only a short time. Thus, this method should be avoided unless it is an absolutely necessary business requirement. Scaling HorizontallyHorizontal scalability is somewhat of a buzzword in the systems architecture community. Simply put, it means that the architecture can scale linearly in capacity: To handle twice the usage, twice the resources will have to be applied. On the surface, this seems like it should be easy. After all, you built the application once; can't you in the worst-case scenario build it again and double your capacity? Unfortunately, perfect horizontal scalability is almost never possible, for a couple reasons:
The guiding principle in horizontally scalable services is to avoid specialization. Any server should be able to handle a number of different tasks. Think of it as a restaurant. If you hire a vegetable-cutting specialist, a meat-cutting specialist, and a pasta-cooking specialist, you are efficient only as long as your menu doesn't change. If you have a rise in the demand for pasta, your vegetable and meat chefs will be underutilized, and you will need to hire another pasta chef to meet your needs. In contrast, you could hire general-purpose cooks who specialize in nothing. While they will not be as fast or good as the specialists on any give meal, they can be easily repurposed as demand shifts, making them a more economical and efficient choice. Specialized ClustersLet's return to the restaurant analogy. If bread is a staple part of your menu, it might make sense to bring in a baking staff to improve quality and efficiency. Although these staff members cannot be repurposed into other tasks, if bread is consistently on the menu, having these people on staff is a sound choice. In large applications, it also sometimes make sense to use specialized clusters. Sometimes when this is appropriate include the following:
|