scale

, , ,

Gluster 3.1 GA Release

, , ,

Over the past couple of months I was taking a really close look at GlusterFS for potential use on a virtualization project.  Today I saw the notice the version 3.1 was released.  That's good news.  They call it a scale out NAS platform which it is but it's also a bit more than that too.

I had the chance to speak at length with Anand Babu (AB) Periasamy and a few members of his team at VMWorld recently about 3.1 prior to release and it was genuinely interesting and exciting. I've been following the Gluster project for years and it really just seems to keep getting better and better.  Not only that, they seem pretty passionate about what they do which is always a good thing.

Of particular interest in 3.1 is that you are now supposed to be able to add and remove nodes to the cluster without impacting the applications using the cluster at all.  This is CRITICAL and was a major barrier to adoption perviously.  Previously you actually had to restart the cluster to expand.

One of the things that can be challenging is large scale file sharing to many, and sometimes varying numbers of, application servers in large scale web environments.  I could see GlusterFS 3.1 being very useful in this scenario.  One recently published example of this is the way that Acquia uses GlusterFS for scaling Drupal.  

Of course, other options exist such as Swift from open stack, MongoDB w/ GridFS, Riak perhaps in smaller file size senarios, and perhaps Ceph which just released.  The file / storage space is hot right now with change and even *gasp* innovation.  It is pretty exciting and more choice over the last few years has been a very good thing.

I suspect I'll be writing more about this in the future assuming I can get some of the testing I want to do completed.  As usual, my lab in my secret lair is under powered and over utilized. *sigh*

, ,

Building for the Cloud is Building for Scalability

, ,

Even today with all the hoorah about cloud computing there are very few applications that can really call themselves cloud computing applications outside of the sciences and even there, I suspect there aren't that many.  So, I began trying to piece together a list of the features that I think a cloud computing application must have.  Otherwise, it's not really a cloud computing application at all but perhaps on its way to being one or not at all.  What I realized quickly was that being a cloud application is often about three things. 

  • Scalability
  • Availability 
  • Costs 

Part of what prompted me to write this is that I have installed and scaled countless websites and applications over the years.  I've done Drupal, Wordpress, Clearspace, ModX, Expression Engine, Custom CMS applications, X Cart, Magento, Django, and every language you can think of from ruby to PHP to java to python.  Thing is, they all have a very serious problem and it baffels me that it doesn't appear to be addressed well yet.  Every single one of these applications was NOT built for the cloud.  They just do not scale well without serious work and complexity to make them scale.  I did not say they do not scale as in most cases, one way or another, they almost certainly can be made to scale.  For example, I've built Drupal sites that could handle 1.5 to 2.0 billion page views per month for example.  But, it was forced and very complex all in all.  

But, what is important to understand is that very, very few applications can natively and easily scale to what we call cloud scale or web scale as they are out of the box.  They simply cannot.  But, if you do want you application to be able to utilize all the benefits of cloud computing such as:

  • Automated provisioning and scale - Using tools like Chef and Puppet to code your infrastructure
  • Rapid Deployment - You should be able to deploy fast and often
  • Lower Cost to Market - You should be able to get up and running for less money in infrastructure than ever before (less capital expenditures on your balance sheet)
  • High server to admin ratios and reduced operational costs over time - You can do more with less and it's all about quality over quantity
  • Client satisfaction - Happy clients are clients that never even know your service grew 10x last quarter, they just know it works
  • Shareholder/Partner Satisfaction - sleep better, make more money, run a better business

Here are some items that I think are must-haves if you are going to build a true cloud computing application. They are in no particular order but all are important.  The take away from all this for me is that building something to scale is very much about building something for the cloud.

  • Be Distributed - It must be able to do whatever it does over a dispersed network of nodes(servers/services) nearly as easily as it does on a single server
  • Be Asynchronous - Think message based event driven frameworks
  • Be Monitorable (aka Be Instrumented) - Just like your developers should implement unit tests they should implement monitoring instrumentation at the application level
  • Be Self healing and Resiliant - The system should be able to handle faults and route around it when needed. A degraded state is often just fine so long as transaction can keep flowing to some degree.
  • Be multi-provider - Do not put all your eggs in one basket (see my post on configuration management and deployment automation w/ Chef for example)
  • Be multi-site - your site should be active-active to some degree across mutiple geographic locations.  There is no reason any longer not to do this.
  • Be able to Scale Out - Designing a distributed, stateless and asynchronous application will go a long way here
  • Be Stateless - This one is not always possible for many reasons it is worth doing or at least minimizing in clever ways, think hard about this one and don't abuse state.
  • Use the Cloud for your Infrasturucture - (note that I did not say virtualize only) - 'Nuff Said.
  • Be able to rollback anything - If you deploy it, you should be able to un-deploy it just as easy.  Oh database schema changes how I love thee.
  • Test, test, test - Make sure your test process is solid.
  • Be highly available (N+1) - If you need 5 servers to provide the capacity you need then you should have at least 6 servers at any given time. This is a simpler version of more complex capacity planning that you can do on a per server/service basis that's very, very important
  • Use the right technology for the job - If you need a relational database by all means use one. If you only need a key/value store then by all means do NOT use a relational database.  If you need data persistence to disk and complete data durablility then don't use memcached (not picking on memcached of course.. great tech there)  Use other technologies properly configured to give you what you need.  Put in non-technical terms, don't use a spoon to dig the foundation for a house.  That's just silly.
  • Build in failsafes (aka circuit breakers) - When something breaks like access from the application server to the database then the application should be able to handle it gracefully.  Build this in!

I'm sure this list can be tuned and improved.  I'm sure I will do so.  What I'm after is laying down the rules for building truly cloud native applications.  There has been some phenomenal work in this area over the years but it's only just beginning to be better understood and move more into the mainstream.  This is both frustrating and exciting!

, , ,

Practical Matters When Building Servers in the Cloud - Configuration Management

, , ,
For some time now I’ve been thinking about and reading about tools like Chef and Puppet.  A couple of years ago a got a couple of small projects off the ground with Puppet for a job I was working as well.  But, with the way the cloud is developing and my general belief that if you are really deploying a cloud computing application but then you find yourself logging into a server command prompt for some reason during normal operations then something has either gone wrong or you are doing something wrong.

The issue of scripted server build and configuration management is hardly new.  There are numerous other resources I’m sure you can search to learn the history of configuration management tools both commercial and open source.  For my part, I’ve been doing a number of experiments and have chosen to work with the Opscode Chef platform.  What follows in this article are a few of the things I’ve learned along the way.

Knowing some ruby helps a LOT!  Opscode chef is going to be challenging to get the hang of if you do not know the first thing about the ruby programming language.  If you are in that camp, you might just want to invest a bit of time w/ a good basic ruby programming language tutorial first.  A great deal of the flexibility and power of the chef tools is about being able to use ruby effectively.  This is not a major barrier because you do not need to be a l33t hax0r by any means.  But, it will help a great deal if you know how variables work, how to do a loop, case statements, and other basic constructs of the language.

I have been used to building and deploying things with push methods a lot in the past.  With a tool like chef things are kind of turned around the other way.  You need to put yourself the the shoes of the server you are trying to configure and think more about pulling data to yourself.  This is essentially what happens after you bootstrap a chef configured server.  It pulls a lot of data to it after it registers itself with a chef server somewhere.  It then uses this data(recipes and roles) to turn itself into the server it is told it should be more or less.

Why would I bother with all this you might be thinking!?  Well, assuming I have set up my environment properly, defined my roles, created the proper recipes assigned to that role, then with a command that looks a bit like the following:

knife rackspace server create 'role[serverrole]' --server-name myawesomeserver --image 49 --flavor 1

NOTE: This example uses the knife tools to interact w/ chef.  Knife is a command-line utility used to interact with a Chef server directly through the RESTful API.

I can start a server (a node) running the software I want running on the rackspace cloud with the code I want on it in about 4-5 minutes.  That’s 4-5 minutes from the time I hit the <enter> key on my keyboard!  Race ya!

Now, if I’m building one server, this might not seem very worthwhile.  But, if I am building 100 or 1000... or if I’m going to be building them and tearing them down constantly by the dozens or hundreds per day then yes, this makes ALL THE SENSE IN THE WORLD!  But WAIT! It gets better.

With this command I can launch THE SAME server on a different cloud in 4-5 minutes (AWS US-East-1c in this case)

knife ec2 server create -G WebServers,default --flavor m1.small -i ami-2d4aa444 -I /Users/me/Downloads/mykey.pem -S servername -x ubuntu 'role[serverrole]' -Z us-east-1c

Just think about this for a moment.  From my laptop (macbookpro in this case) I can launch a server that is exactly the same on two different cloud computing providers in well under 10 minutes without ever touching a cloud GUI like the AWS console or the Rackspace Cloud console (which would only slow me down).

Now, it wasn’t exactly trivial to set all this up so that it works.  But, the fact is, it wasn’t that bad either and I learned a TON along the way.

So, this was just a little intro article.  There are LOADS of great resources about chef out there.  I will warn you about one other thing I learned.  It’s a bit hard to search for information about this software because it’s called “chef” and it has “recipes” which means a lot of the time you’ll end up with search results from FoodTV network.  I like food so I don’t mind sometimes but it can be annoying.
I've worked with Puppet in the past, love it.  I'm working with Chef now, love it.  I'll almost certainly be using both for projects where they are best appropriate in the future.

Have good configurating and I'm sure I'll be writing more about this in the near future.

 

, ,

Dynamic DNS Rocks, More Sites Should Use It!

, ,

 

I was doing some thinking about DNS today and in particular, Dynamic DNS.  I'm still surprised more people haven't heard of and do not use this type of service.  DNS is one of those things that, in my opinion and if at all possible, you should outsource to people who can and will do it better than you.  Yes, that includes internal and external DNS.
In short, dynamic DNS services allow you to provide and programitcally control things like multiple load balanced A records or CNAMES for a single domain or web service.  This can be especially important in the context of an elastic cloud computing hosted service were certain things can sometimes be ephermal or come and go very quickly (like an IP address or a compute node).  Just like every other part of your infrastructure, your DNS needs to be elastic and programmable too.
Some of the reasons you might use Dynamic DNS:
  • Load Balancing - A Smarter version of round robin more or less
  • CDN Management
  • Site Migrations
  • Disaster Recovery
  • It'll make you all the rage at parties
I made a short list of some of the Dynamic DNS services I know about. Here they are:
Dynect Platform - http://dyn.com/enterprise-dynect-platform
Used this extensively over the years and have met the team.  It's a great service run by an excellent team. I highly recommend.
Used this one a couple of times and it worked out well.  Their interaface was a bit odd but I haven't used it for a couple of years.
Have not personally used this one so I can't provide much more information at the moment.  Will update in the future if that changes.
For further reading, WikiPedia Says...
http://en.wikipedia.org/wiki/Dynamic_DNS
"Dynamic DNS provers provide a software client program that automates the discovery and registration of client's public IP addresses. The client program is executed on a computer or device in the private network. It connects to the service provider's systems and causes those systems to link the discovered public IP address of the home network with a hostname in the domain name system. Depending on the provider, the hostname is registered within a domain owned by the provider or the customer's own domain name. These services can function by a number of mechanisms. Often they use an HTTP service request since even restrictive environments usually allow HTTP service. This group of services is commonly also referred to by the term Dynamic DNS, although it is not the standards-based DNS Update method. However, the latter might be involved in the providers systems."
So, while you are thinking about DNS I'll leave you with the following related tip...
Your DNS registrar is not the same as your dynamic DNS provider necessarily.  Your DNS Registrar should not necessarily be the same as your Dynamic DNS provider and it most definately should NEVER be your ISP/hosting provider.  Although, I have used www.dyndns.com and Dynect together for various reasons.  This is serious business if things go south w/ your hosting provider.  I have actually seen companies held hostage pending litigation over trivial matters when the wrong provider had registrar control.  Your domains are an asset.  Control them yourself and delegate control of them securely to someone you trust to get the help you do the work need.