Wide-area Software-defined Storage

Abstract

The proliferation of commodity cloud services helps developers build

wide-area "system-of-systems" applications by harnessing

cloud storage, CDNs, and public datasets as reusable building blocks.

But to do so, developers must contend with two

long-term challenges. First, whenever developers change storage providers,

they must work to preserve the application's expected storage

semantics, i.e. the rules governing how the application expects the

storage provider to handle its reads and writes. Today, changing

storage providers is costly, because developers need to patch

the application to make it compatible with the new provider's

data consistency model, access controls, replica placement

strategies, and so on.

At the same time, users have certain expectations about how their

data will be used, which the application must meet.

For example, depending on the application, users

may expect that their data will be kept private from other users, that their data will be exportable

to other applications, that accesses to their data will be logged in an auditable way,

and so on. In the limit, each user's expectations represent an implicit

policy constraining how their data can be stored. Honoring

these policies is difficult for developers who rely on third-party storage

providers because the storage provider is often unaware of them.

This thesis addresses these challenges with a new wide-area storage paradigm, called "software-defined

storage" (SDS), that runs in-between applications and cloud services.

SDS-enabled applications do not host data, but instead let users

bring their preferred cloud services to the application. By taking a

user-centric approach to hosting data, users are empowered to programmatically

specify their policies independent of their applications and select

services that will honor them. To support this approach and to

tolerate service provider changes, SDS empowers developers to programmatically

specify their application's storage semantics independent of storage providers.

This thesis presents the design principles for SDS, and validates their

real-world applicability with two SDS implementations and several non-trivial

applications built on top of them. Most of these applications are used in

production today. This thesis

presents microbenchmarks of the SDS implementations and uses

real-world experiences to show how to make the most of SDS.