RAM Sled: Facebook wants to replace the leaves and run it on a RAM sled with between 128 GB and 512 GB of memory, and for $500 to $700 per sled. Only a basic CPU would be needed. Total queries would be 450,000 to 1 million key queries per second.http://slashdot.org/topic/datacenter/how-facebook-will-power...
Not really any way to know what Facebook is up to here: 512 GB is normally done via 4 sockets, 4 channels, 2 dimms per channel, 16GB dimms; whether FB intends on disrupting any of the factors in this equation for getting mass memory on a single system will be interesting to see.
It has been rather shocking to me that FB-DIMMS & other approaches which allow a lot of RAM to be chained to one another- really deep channels- hasn't seen any widescale success. RAM is cheap enough: would that we could plug in a lot of it.
If you just care about capacity and cost you can build a system with a $20 ARM SoC for every two DIMMs. The ultimate is BlueGene-style packaging with an SoC on each DIMM, but Facebook's probably not ready for that.
Similarly, I've been waiting to see an extended modern version of the iRam Box[0], but it doesn't seem to be coming. The 4GB limit is useless now, but imagine the same with hundreds of gigabytes.
Somewhat relevant, but on a much smaller scale I suppose:
I had to geocode 100million addresses and used PostGIS to get it done. It was painfully slow on disk, so I rented some big-ram servers, created ramdisks, configured postgres to run from those ramdisks, and it was off to the races.
What would have taken days of processing was taken down to several hours. The speed improvement was amazing. Even SSD couldn't touch it.
It is intended to be durable and available despite host crashes and localized hardware failures. They discuss getting 64GB back into DRAM in 0.6seconds from secondary backup store. The trick is to have many hosts share the burden, so that when one host goes down, 1/100 of its memory is quickly sucked into each of 100 machines from another 100 disk/flash backups. Neat.
The idea is not that weird. Lots of applications run from RAM most of the time, needing long cache warmups before performing well. If you keep your entire storage in RAM this isn't necessary, and you can do away with the caching layer.
This looks like the "RAM SAN" products, where there's a bunch of memory and some spinning disks with enough local battery backup to flush to disk. I think the main difference is that they want the data aggregated across multiple machines, which I don't think that previous solution had.
I often run stuff in ram, the improvements are dramatic.
My latest deed was dumping/restoring 20GB of RRDs; doing it in tmpfs was FAST.
For running programs stuff can be kept in sync with the likes of anything-sync-daemon. SQL can get trickier, but it can be done.
Hire more server huggers, make extra sure das-blinkenlights stay blinking, buying a second backup generator.
And- obviously enough- off site redundancy; multiple data centers. Which they talk about the difficulty of: 10Gbps pipes between DCs is not a lot compared to write throughput of a single computer's main memory, much less a DC's. For some compressible checkpoint logs there might be a way forward.
Perhaps we need a new *-cast technology: after any-cast we'll all be needing n-cast, which sends our traffic to the n-nearest DC's? :)
What's odd is they didn't just use nbd + ram disks + RAID. Building a whole key-value store around it makes it a lot less general purpose, and there are perfectly good key-value stores that run off RAM...
RAM Sled: Facebook wants to replace the leaves and run it on a RAM sled with between 128 GB and 512 GB of memory, and for $500 to $700 per sled. Only a basic CPU would be needed. Total queries would be 450,000 to 1 million key queries per second. http://slashdot.org/topic/datacenter/how-facebook-will-power...
Not really any way to know what Facebook is up to here: 512 GB is normally done via 4 sockets, 4 channels, 2 dimms per channel, 16GB dimms; whether FB intends on disrupting any of the factors in this equation for getting mass memory on a single system will be interesting to see.
It has been rather shocking to me that FB-DIMMS & other approaches which allow a lot of RAM to be chained to one another- really deep channels- hasn't seen any widescale success. RAM is cheap enough: would that we could plug in a lot of it.