Spark, Shark, and BDAS In the News

(NOTE: I continue to update this list as I encounter new posts. Email me if you know of an article I haven’t listed yet.)

Myself and several other members of component projects of the Berkeley Data Analytics Stack (BDAS), especially Spark and Shark, have been trying to keep track of news articles that mention or are about the projects. I’ll discuss why this sort of news coverage is so unique and deserved below, but first the list:

Why did we build this list? I’m a huge fan of the Spark Project, and I also contribute to it (mostly documentation, and community related stuff). Spark was born and developed in the UCB AMPLab. The lab’s approach to releasing and supporting high quality open source software projects and fostering communities around them is somewhat unique, and has resulted in serious adoption of the projects coming out of the group.

Research labs traditionally spend very few resources on promoting adoption of the projects that are built as part of their research agenda because often the research ideas can be tested using throw-away software prototypes. The extra energy required to turn such prototypes into production quality projects is not obviously worth the effort. The folks in the AMPLab believe the extra effort is worth it. Not only do graduate students spend a lot of time answering questions on developer and user mailing lists, but the lab itself is investing in the effort. For example, the AMPLab recently hired Matt Massie, Cloudera engineer #5 and he is recruiting a rock-star team charged with testing and hardening the software coming out of the lab.

If the goal is “free”, high-quality, next-generation software, then how can we measure if we are succeeding? Well, we can measure adoption of the BDAS software for production use, as well as the grassroots community activity by hundreds of BDAS enthusiasts. Another way is to measure and track discussion of the software in the media, which the above list aims to do. By any metric, we can say that the AMPLab projects like Spark and Shark are having tremendous impact!

Feel free to email me if you know of other articles, or if you are a technology journalist or reporter and would like introductions to the key folks on the Spark, Shark, Mesos or BDAS projects.