The Facebook Blog

Almost two million new users from around the world sign up for Facebook each week—and we couldn't be happier. It's tremendously rewarding to see so many people find what we work on useful and fun. As we continue to add new users and features, however, the load on our thousands of servers continues to increase at a pretty astounding rate. A few weeks ago we reached full capacity in our California datacenters. In the past we handled this problem by purchasing a few dozen servers, hooking them up, and getting on with our lives, but this time we... didn't have it so easy. We'd actually run out of space in our datacenters for new machines.

Fortunately we saw this problem coming a long time ago and started work on a new datacenter in Virginia. Now, we identify whether a user would be better off talking to the east coast datacenter or a west coast data center. For people in Europe and the eastern half of the US, it's noticeably faster to talk to a server in Virginia than in California. For these users we direct them to Virginia whenever they're browsing the site and not making any changes.

Whenever that person goes to change some data—uploading a photo album, or changing profile info for example—we send them off to California so that all our modifying operations happen in the same location. This decision was made to prevent two or more modifications from conflicting with each other and messing up our data. It might sound like we're forcing our users to go to California a lot but only about 10% of our traffic causes a modifying operation. MySQL has a great replication feature that allows us to, in real time, stream all the modifications happening on a California MySQL server to another one in Virginia. Replication happens so fast, even across the country, that the Virginia servers are almost never more than one or two seconds behind the California servers.

Even though all of the modification happens in California and streams instantly to Virginia, we were faced with another problem. Although Facebook's data is stored in MySQL database servers, we use a large number of memcached servers to store copies of the data. Memcached is much faster and able to keep up with requests quicker than the databases themselves can keep up. We had to figure out a way for memcached servers to replicate data concurrently with the MySQL databases. Because of various technical limitations of our architecture there was no easy way to do so.

Fortunately MySQL is open source software, meaning we can actually change the way it works by modifying the code. We did just that—embedding extra information in to the MySQL replication stream that allows us to properly update memcached in Virginia. This ensures that the cache and the database are always in sync. Over the last seven months a great team of Facebook employees has been building new software and setting up new servers like I described above. Over Thanksgiving we finally flipped the switch and since then almost 30% of our traffic has been served from Virginia.

The east coast datacenter is a great first step towards keeping Facebook fast and reliable as the site grows. Going forward we have lots of exciting plans to expand our infrastructure and improve performance so no user ever has to sit around waiting for a page to load.


Jason, a Facebook engineer, is all about the east coast.

Recently, chain letters have started to spread across Facebook with all sorts of misinformation. Let's get some information out there about how we communicate to you about Facebook. Right now, there are only four ways that we try to get information to users.

1. Home Page Announcements: Home Page announcements are the big boxes that push down your News Feed. These announcements contain information that we really want you to see; changes to your account, big new products we think you might like, and any information that is required for you to use... Facebook effectively.


Product Stories and the What's New page: You may occasionally see stories about Facebook inside your News Feed. These contain useful tips and fun information about Facebook, but are not of the same importance as Home page announcements.


The Blog: Since you're here reading, you know what the blog is. This is a good place to get extended information about what's going on at Facebook.


Through Pages and Updates: This is fairly new with the launch of Pages, but notice that these will appear in your "Updates" section of your Inbox, not mixed in with messages from friends. These should be few and far between—the Update you saw from any Politicians you support that came from Facebook is a one-time occurrence.

About a month ago, we released a new feature called Beacon to try to help people share information with their friends about things they do on the web. We've made a lot of mistakes building this feature, but we've made even more with how we've handled them. We simply did a bad job with this release, and I apologize for it. While I am disappointed with our mistakes, we appreciate all the feedback we have received from our users. I'd like to discuss what we have learned and how we have improved Beacon.

When we first thought of Beacon, our goal was... to build a simple product to let people share information across sites with their friends. It had to be lightweight so it wouldn't get in people's way as they browsed the web, but also clear enough so people would be able to easily control what they shared. We were excited about Beacon because we believe a lot of information people want to share isn't on Facebook, and if we found the right balance, Beacon would give people an easy and controlled way to share more of that information with their friends.

But we missed the right balance. At first we tried to make it very lightweight so people wouldn't have to touch it for it to work. The problem with our initial approach of making it an opt-out system instead of opt-in was that if someone forgot to decline to share something, Beacon still went ahead and shared it with their friends. It took us too long after people started contacting us to change the product so that users had to explicitly approve what they wanted to share. Instead of acting quickly, we took too long to decide on the right solution. I'm not proud of the way we've handled this situation and I know we can do better.

Facebook has succeeded so far in part because it gives people control over what and how they share information. This is what makes Facebook a good utility, and in order to be a good feature, Beacon also needs to do the same. People need to be able to explicitly choose what they share, and they need to be able to turn Beacon off completely if they don't want to use it.

This has been the philosophy behind our recent changes. Last week we changed Beacon to be an opt-in system, and today we're releasing a privacy control to turn off Beacon completely. You can find it here. If you select that you don't want to share some Beacon actions or if you turn off Beacon, then Facebook won't store those actions even when partners send them to Facebook.

On behalf of everyone working at Facebook, I want to thank you for your feedback on Beacon over the past several weeks and hope that this new privacy control addresses any remaining issues we've heard about from you.

Thanks for taking the time to read this.

Mark