SHARE

Digg Moves From MySQL to NoSQL

Social networking and voting site Digg is rewriting its underlying software infrastructure in an effort to improve performance and scalability. Part of that effort involves moving away from the MySQL database that has helped to power Digg since its creation. In MySQL’s place, Digg is going with an open source NoSQL non-relational database called Cassandra […]

Written By

SMK

Sean Michael Kerner

Mar 11, 2010

Datamation content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More

Social networking and voting site Digg is rewriting its underlying software infrastructure in an effort to improve performance and scalability. Part of that effort involves moving away from the MySQL database that has helped to power Digg since its creation.

In MySQL’s place, Digg is going with an open source NoSQL non-relational database called Cassandra that was originally created by Facebook. As part of the migration effort from MySQL to Cassandra, Digg developers built a tool to help move data from one database to the other. The tool could soon be open source, helping other developers make the same move.

“We built a tool that we call ‘transcribe’ that takes advantage of Hadoop to bulk import from MySQL to Cassandra,” John Quinn, vice president of engineering at Digg, told InternetNews.com. “We’ll be releasing that to the open source community very soon.”

A NoSQL database is a new type of database that differs from the traditional SQL based relational database management systems (RDBMS) like MySQL. Instead of using tables and rows and linking relationships between the two to deliver data, NoSQL databases use different types of data stores and objects.Quinn said that Cassandra wasn’t the only NoSQL database that Digg considered. Other popular NoSQL databases include CouchDBand MongoDB.

“We evaluated all the big players in the open source NoSQL space,” Quinn said. “We particularly liked Cassandra’s proven storage model (BigTable) and the multiple datacenter support.”

Digg has been testing Cassandra in various aspects of its operations since September 2009. Most of Digg’s functionality has been re-implemented from MySQL to now use Cassandra as its primary data store, though Quinn was unable to comment on the performance improvement Digg expects to enjoy with the migration.

But he did provide some additional insight into the rationale behind the move in a blog post.

“It has a fully decentralized model. Every node is identical and there is no single point of failure,” Quinn wrote. “It’s also extremely fault tolerant — data is replicated to multiple nodes and across data centers. Cassandra is also very elastic — read and write throughput increase linearly as new machines are added.”

As part of Digg’s work with Cassandra, the company’s developers have made performance and functional improvements to the code, Quinn said. Cassandra is an open source project run under the direction of the Apache Software Foundation.

“Digg has a full-time committer on the Apache Cassandra project,” he said. “We have contributed major performance enhancements and features. All our work has been contributed upstream to the Apache project.”

As Digg moves its core infrastructure to Cassandra, it isn’t entirely abandoning MySQL, as there are still some places where it fits.

“Our primary data store is Cassandra,” Quinn said. “We’ll continue to use MySQL for specific use cases and rapid prototyping. MySQL provides a level of flexibility that Cassandra does not. It’s very useful for small-scale projects.”

Sean Michael Kerner is a senior editor at InternetNews.com, the news service of Internet.com, the network for technology professionals.