For many years now when building scalable and highly available computing infrastructures I've been doing something I call Factoring Complexity. I think a lot of this is very well served by using agile project methods and ITSM (such as ITIL) concepts. There may be better names for this but I'll stick with it for now.
I do this to achieve the least common denominator systems architecture to efficiently provide the required business service. This includes processes, people, computers, and code. To understand what I mean be this and how I approach two cross-discipline definitions are required. I borrowed these from some of my math classes from long ago.
From www.algebrahelp.com a definition for factoring reads as follows:
Factoring is an important process in algebra which is used to simplify expressions, simplify fractions, and solve equations.
I typically aim to Factor complexity out of infrastructure and systems or software in general for the lowest number of interconnections between components that is able to reliably perform the task at hand, like serve a website, that provides appropriate scalability and performance.
In the context of building web systems Factoring Complexity serves several purposes. Two that stand out are resource application (time and money) and maintainability. From a business perspective it is important to use the appropriate level of resources to solve any particular business problem. From a practical point of view, things should be maintainable. The successful output of a round of Factoring Complexity is usually less connections between components. It is the least complex system that can adequately provide for any particular business need.
From algebrahelp.com a definition for factoring reads as follows:
Factoring is an important process in algebra which is used to simplify expressions, simplify fractions, and solve equations.
From wikipedia the least common denominator is seen to be defined as follows:
Q: The term is used figuratively to refer to the "lowest"—least useful, least advanced, or similar—member of a class or set which is common to things that relate to members of that class.
When attempting to factor complexity I strive to focus on a few key things. They are documentation, relationships, and advance planning. I'll take each of these in turn. But first, let me show a very simple and visual example of what I am talking about.
Figure 1 is a rather complex system with a lot of interconnections. Figure 2 is a less complex system with many less interconnections. Figure 3 is a very simple system with only 1 interconnection. I illustrate this in this way to explain something that is often missed. The complexity is not as much in the nodes themselves but in how they interact and interconnect. The complexity increases dramatically every single time you add a node. Nodes can be software programs, development frameworks, servers, people, network connections, anything really; anything that might interact in some way with some other node.
In the context of refactoring existing web infrastructure, one way to attack overly complex systems is to focus on the interconnections. If you can begin to eliminate the need for interconnections without negatively impacting performance, scalability, availability, and capacity then you have done an excellent thing. You have reduced the overall complexity of the system while maintaining or even improving the systems manageability and capability as a whole. Sometimes people don't believe this is possible when they first see the concept. But, I assure you that is because I have personally used this concept successfully many times professionally and personally. One might equate the number of interconnections to the level of overall effectiveness of any given environment.
Documentation is a critical factor in getting any environment under control and getting to a position of relative stability and predictability. In the context of Factoring Complexity I am primary talking about first documenting all of the known and discoverable components of a giving system as much as possible and at an appropriate level of detail. Then, even more importantly, documenting the relationships of one component to another in as much detail as possible. These relationships are very important for the next steps.
If you want to get started factoring complexity in your compute environments then there are three key things to document. One, physical components. Two, abstract services. Three, relationships of these items. In ITIL these are called CI's, Configuration Items. The relationships between CI's is what we are looking for here. The are tracked in a CMDB. CMDB is a nasty four letter word in some places because they can be very challenging to implement. The CMDB, or Configuration Management DataBase, is in essence a social graph. There are some great pieces of software finally emerging that can easily handle the kinds of complexity and number of interconnections that many IT environments present. One of the more interesting ones to me is Neo4j. It is, in the sites words,
Neo4j is a graph database. It is an embedded, disk-based, fully transactional Java persistence engine that stores data structured in graphs rather than in tables. A graph (mathematical lingo for a network) is a flexible data structure that allows a more agile and rapid style of development.
But, that's all for now, how to make Neo4j into a killer CMDB is just a idea for now as far as I know. It makes a lot of sense though doesn't it?