At the site highscalabilty.com Todd Hoff has written an article1 that is kind of a summary of some of the challenges and lessons learned from the Twitter team in building a large scale web applicaiton. There were a couple of surprises in there for me for sure. For example, twitter hasn't yet done any sharding at the DB layer and they only have a single 8 core MySQL master. This is likely even possible due to the caching and database optimizations they've done. I've been seeing simliar results myself in my own work by using memcached to scale PHP based sites. One of the parts of this particular article I liked best was the "Lessons Learned" section. That is particularly nice to read. My favorite though was, "Scale changes what can be stupid." Amen! I say.
In any event, it's a good article with plenty of take away. So enjoy!
Resources:
1- Scaling Twitter - http://highscalability.com/scaling-twitter-making-twitter-10000-percent-faster