QCon SF Wrap Up – Facebook, NASA Moves to the Cloud, Agile
I had to opportunity to attend QCon San Francisco last week, along with our new owners at CollabNet who sponsored the event. The only downside was that there were too many talks I wanted to go to with conflicting time slots, but I managed to catch Jason Sobel discussing Facebook’s backend infrastructure and PHP, Brian Zeichick speaking to CollabNet’s Agile development process, and Khawaja Shams from NASA Jet Propulsion Laboratory sharing their use of EC2 and cloud computing.
Jason Sobel – Facebook Infrastructure
With more than 500 million active users, the infrastructure team at Facebook have some serious traffic to deal with. While this isn’t hot news, it’s always cool to hear directly from the people working on the product and supporting infrastrucure. Jason Sobel, Engineering manager at Facebook, gave a good talk on how they decided to stick with the technology that Facebook originally used – MySQL and PHP.
As most product managers know, a product will hit an upper limit threshold at some point and you’ll be faced with a decision to either (1) re-write using a different, or more modern language and gain improvements but risk setting back the product several years while you rewrite (this varies depending on the size of the project of course), or (2) stick with the product as-is and put up with whatever is causing problems and work around the issues. The team at facebook did both, they stuck with PHP and when they hit upper limits…what did they do? Write a compiler of course!
The compiler turns PHP into C++ and according to Jason, increased website performance 4-5x. This is major out of the box thinking, and requires a lot of technical ability just to write a compiler alone. Ultimately this allows Facebook’s web developers to keep iterating on the site at the same rate, using the same language without any impact on their internal development processes.
This brings us to the backend of Facebook, where they had to add a layer on top of MySQL to alleviate database load and allow them to scale their web and database servers horizontally. They have open sourced this technology as memcached, and are currently working on an improved version called, ‘Son of Memcached’.
The challenges Facebook has been forced to deal with are unique, a direct result of their popularity. With website uptime and response time ever more important on ‘Today’s Internet’, its great to see there are some real innovators at work behind the scenes, part of why Facebook is the hugh success it is.
Khawaia Shams – NASA Jet Propulsion Lab moved to the Cloud
I decided to check out one of the talks in the cloud computing stream next with Khawaja Shams from NASAs Jet Propulsion Laboratory. He spoke about how they are leveraging the cloud to cope with the vast amount of image and raw data collected from their Mars Rover’s- Sprit and Opportunity. They recently migrated their compute infrastructure away from an ‘in house’ solution to Amazon AWS. By leveraging on-demand compute power in the cloud they are able to process massive volumes of image data and display it in a meaningful way to their scientists in a fraction of time compared to their ‘in house’ solution.
With the lag time in communication between Earth and Mars, mission control must plan a day ahead by reviewing data from the previous day, making decisions about what to do the next day, and communicating the instructions back to the rovers for the next 12 hours (the rovers operate only 12 hrs a day). The scientists must make complex and important decisions all within a 1 hr window each day, so the quicker the images and data can be processed, the more time the scientists and mission controllers have to make these important decisions. It’s great to see that cloud computing is hitting the main stream, with organisations like NASA realizing the benefits on-demand compute power can deliver. Check out the Mars rover site for more details.
Brian Zeichick – CollabNet Agile development
Agile is a hot topic today, with everyone wanting to improve their software delivery process and time to market. We use the Agile process here at Codesion, and understand the benefits it provides to us and ultimately to our customers. Brian Zeichick from CollabNet explained how Agile is something that’s easily adopted by new and emerging startups, but for larger enterprises can be a real challenge, especially with embedded software development practices such as waterfall, or some hybrid of waterfall.
Brian’s key point is that Agile is all about enabling regular communication amongst developers and project managers (Scrum masters). If people in enterprise are thinking about Agile but are unsure about its benefits, they should understand that the most common reason software projects fail or break down is because of lack of communication amongst the team. This is core to what the Agile methodology addresses.
Did you go to QCon SF this year? What were your favorite talks?