Update on Monday, March 9: We have completed the fix on the earlier photo storage problems, and access to impacted photos is now fully restored.
You may have noticed in the past day that some photos aren't appearing or are displaying a "question mark" graphic when you go to view them. We have experienced some problems with our photo storage that affected between 10 to 15 percent of already uploaded photos. Don't worry: Your photos are safe, and we are working to make them available again as soon as possible. We've already repaired about... one-third of affected photos and expect to complete repairs on another third tonight.
Here's what happened, and what we're doing to fix the problem: During an otherwise routine software upgrade on Friday night, we ran into some problems with our photo storage and a few of the hard drives where we store photos apparently failed all at once. We're trying to fully understand what happened, since simultaneous hardware failures like this are rare.
We still have all your photos because we store them in a way that maintains multiple copies of the data in case of hardware failures like this. However, even though your photos are safe, we can't serve photos off the affected storage volumes until they're repaired. We're working on them right now, but it will take some time because there's so much data on them and the repair process largely involves copying huge amounts of data to new drives. This is why some photos aren't showing up right now.
We're restoring photos as we repair the hard drives, so some should be working again today and we should be back to normal by early next week. New photo uploads will continue to work properly during the repairs, because we write them to different storage volumes. Thanks for bearing with us while we return things to normal.
Evan, an engineer, is working alongside Doug Beaver, Myles Uyema and the rest of the photo storage team to restore your photos as quickly as possible.
See MoreYou may have noticed in the past day that some photos aren't appearing or are displaying a "question mark" graphic when you go to view them. We have experienced some problems with our photo storage that affected between 10 to 15 percent of already uploaded photos. Don't worry: Your photos are safe, and we are working to make them available again as soon as possible. We've already repaired about... one-third of affected photos and expect to complete repairs on another third tonight.
Here's what happened, and what we're doing to fix the problem: During an otherwise routine software upgrade on Friday night, we ran into some problems with our photo storage and a few of the hard drives where we store photos apparently failed all at once. We're trying to fully understand what happened, since simultaneous hardware failures like this are rare.
We still have all your photos because we store them in a way that maintains multiple copies of the data in case of hardware failures like this. However, even though your photos are safe, we can't serve photos off the affected storage volumes until they're repaired. We're working on them right now, but it will take some time because there's so much data on them and the repair process largely involves copying huge amounts of data to new drives. This is why some photos aren't showing up right now.
We're restoring photos as we repair the hard drives, so some should be working again today and we should be back to normal by early next week. New photo uploads will continue to work properly during the repairs, because we write them to different storage volumes. Thanks for bearing with us while we return things to normal.
Evan, an engineer, is working alongside Doug Beaver, Myles Uyema and the rest of the photo storage team to restore your photos as quickly as possible.
