What makes a good Sandstorm app?

This document provides a conceptual overview of Sandstorm along with a style guide for Sandstorm package maintainers.

A great Sandstorm app:

  • Provides a personal network service
  • Works immediately
  • Is granular
  • Does not implement user accounts or access control
  • Is isolated
  • Is maintained in Sandstorm by the community that develops it

This document explains the basic concepts of the Sandstorm platform by example and introduces the goals of the platform. The purpose is to enable you, a person interested in making a web application run on Sandstorm, understand the platform and provide the best experience possible. Before reading this document, we encourage you try Sandstorm, perhaps by visiting https://demo.sandstorm.io/.

The platform is always evolving, so this document is likely to change. We will keep old versions of this document online, and we hope that you will watch this document over time to update your app as the platform updates. There will eventually be API deprecations and backwards-incompatible breakage, though we aim to keep that to a minimum.

This document serves as a conceptual overview but is not the authoritative source of technical documentation. Within each section, we typically link to more detailed documentation. Those detailed documents provide further context on how stable each API is.

A great Sandstorm app

This section explains each brief bullet-point above in more depth.

Provides a personal network service

Every Sandstorm app provides a service to the person who creates an instance of it. It is initially available to the person who started the instance. It is only available to people with whom the owner can share the instance. By focusing on enabling personal computing on the web, Sandstorm aims to enable people to get creative with small apps created for a single purpose, with security provided by the platform.

To provide some examples of the kinds of services an app might provide:

  • Some apps create the experience of an online editor for personal documents, like spreadsheets or text documents. An example of this is EtherCalc, a real-time collaborative spreadsheet.

  • Other apps give the user the ability to produce and edit online publications, such as blogs, photo galleries, or an informational website. An example of this is GNU MediaGoblin, a media gallery application.

  • An app might help you share data with other programs. For example, the GitWeb creates a git repository you can clone from your laptop.

  • A user might use a Sandstorm app for personal communication. An example is email; right now, there is one email app in the Sandstorm app market, Roundcube.

These categories are just examples; there are surely other kinds of apps that people will dream up.

The Sandstorm paradigm of app instances being primarily used by the person who created the instance may require rethinking your app. For example, the administration and editing interface of a blog is a great Sandstorm app, and the person who creates the instance can share it with others, allowing multiple people to edit the blog. Meanwhile, however, the public view of the blog is a completely separate facet of the application accessed through different means. The public site may simply be mapped to a domain separate from the Sandstorm shell, whereas the edit interface is accessed exclusively within the Sandstorm shell. (See more below on how apps can publish content to the public web.)

Works immediately

When the user creates a new instance of an app, the first screen the user sees should empower the user to start using the software.

To enable this, the platform handles authentication so the app can automatically log the user in; see below about that.

A great example of an app working immediately is EtherCalc. When you enter a new instance of EtherCalc, you find yourself staring at the familiar grid of a spreadsheet. The interface acts as a prompt to can start entering formulas and data. (Try it on the demo.)

Although we want users of every app to start using the app immediately, some apps are not as familiar as a spreadsheet. For those, the app should provide a clear visual hint as to how to start using it. For example, with the gitweb port, we adjusted the home screen so it tells you exactly how to start pushing to the repository it created. (Try it on the demo.)

A note about service dependencies: many web applications rely on a separate database, for example. The Sandstorm approach is that each app package is responsible for making sure the app works, which could mean embedding these dependencies. This way, users of an app do not have to think about the complexity of "provisioning a database" or similar issues, and a compromise of one instance of an app does not lead to other apps' data stores being compromised.

Is granular

Users can always create multiple instances of an app; we call these instances "grains".

Each grain should refer to a discrete collection of data. For example, one grain should contain one blog, or one spreadsheet, or one photo album. The grain has a unique URL which contains a grain ID (a random identifier for the instance) intended to be unguessable.

Choosing the right granularity is an editorial decision made by the person porting the app.

For a document or spreadsheet editor, the answer is easy: a grain should contain one document. It could be that for a media gallery application, the best granularity is a photo album or a collection of photo albums, rather than one photo itself. For a web-based image editor, the best granularity might be a single image. As a general rule of thumb, a grain should usually represent a "unit of sharing" -- the smallest item that a user may want to share independently of other items. Choosing the finest granularity allows the app author to defer more access control to Sandstorm's code and its user interface. At the extreme end, an app can choose to implement no access control at all, and the data can remain safely behind the grain URL and Sandstorm's other protections. For example, this is how the EtherCalc port works.

Does not implement user accounts or access control

When a user visits an instance of an app, the Sandstorm platform adds a X-Sandstorm-User-Id header to the HTTP request. This header contains a large hex string identifying the user, so that the app can tell when the same user visits again later.

A second header, X-Sandstorm-Username, contains the user's "display name", suitable for identifying this user to other users (but this name can change over time). (Eventually, Sandstorm may also offer other user profile information, like a photo.)

A third header, X-Sandstorm-Permissions, specifies the permissions this user has on the particular app instance, such as "read" or "write". These permissions are set through the Sandstorm sharing UI, outside of the app itself, although the app defines what permissions are available in its package definition.

With these headers in hand, a Sandstorm app can and should avoid implementing any internal user model. An app should not ask users to log in and should not implement any notion of ACLs (access control lists) or other ways of assigning permissions internally.

For apps which were originally developed outside of Sandstorm where they needed an internal notion of users, the app might already have a "users" table and lots of code built around this. That's OK! In this case, under Sandstorm, the app should automatically do whatever preparatory work is required to log the user in, such as creating a new row in that table. The app should examine X-Sandstorm-Permissions to detect the "admin" user, who should receive full control over the data the app creates. You can see an example of that in the Sandstorm plugin for GNU MediaGoblin. (See the source code.)

A Sandstorm user can create a sharing link which grants access to the grain to anyone with the link. For those with a security background, grain URLs are capability URLs.

A different user can visit the grain if the owner of the grain sends them a sharing link. If the user is logged in to Sandstorm, the app will see an X-Sandstorm-User-Id that it has not seen before. It should typically create an account in its user model, whatever that means for the app, and automatically log the user into that account.

When the sharing link was created, Sandstorm asked the user what permissions to grant to someone who visits a grain with the link. Your app needs to make its own decisions about what permission levels exist. Read more about how to define permissions.

A totally logged-out user can also visit a sharing link. They should be granted the same permission level as a logged-in user with the same sharing link. In Sandstorm, logging in just means telling the server who you are. Sharing links should work no matter who you are.

For those with a security background, the app enforces permissions but does not handle authorization (i.e. deciding which permissions to grant) nor authentication. As an implementation detail, these headers are provided by the sandstorm-http-bridge tool. Apps can use a Cap'n Proto interface instead of HTTP and run more efficiently; we will document that here later.

Is isolated

A large number of web applications can work properly without interacting with other instances of the app and without access to the network. Some examples are document editors or media organizing tools. However, many apps do need network access to function.

For outbound network access for now, apps can request various specific network protocols like HTTP and SMTP using a short-term hack called HackSessionContext. To learn about that, look at its capnp file. We're actively working on replacing this, but it will take a few months. We'll do our best to help you transition a package away from HackSessionContext when the time comes.

For inbound communication, see the discussion below on public API endpoints.

In the future, we plan to mediate this network access through "drivers" for each protocol. Rather than providing a raw interface for underlying protocols like SMTP, we will provide a more abstract representation that allows apps to get their work done very simply. The drivers will respect object-capability discipline, which we hope will make network code easier to write and also make it more secure. The generic TCP/IP networking interface definition will be part of how we implement more drivers.

In the future, when an app wishes to connect to another app or to an external resource through a driver, the app will make a "powerbox" request. The app essentially tells the platform: "I need an endpoint which implements the XYZ API." Sandstorm itself will then display a dialog to the user which helps them choose which endpoint to use. The user may choose another one of their apps on the same server, or may choose to connect to external resources through a driver. Either way, the requesting app will receive a Cap'n Proto object reference representing the endpoint, with which they can then communicate. The platform automatically arranges for the right permissions to be granted on the endpoint the user chooses, so apps do not need to make a separate "permission" request.

Is maintained in Sandstorm by the community that develops it

Over the course of 2014, volunteers and Sandstorm staff ported a number of existing open source web applications to Sandstorm to test out the platform, show that it can run a variety of software, and identify changes we need to make to support a wide range of apps.

In the long run, we believe that the person/people maintaining the software itself should maintain the Sandstorm port, too. (We sometimes refer to the software's authors as the "upstream" author of the Sandstorm port.) If you want to port something to Sandstorm, but aren't part of the the main project's community yet, it's OK to go ahead! Do it, and then tell both the upstream community and the Sandstorm community about your work. You should join up with the broader community so that the whole community around the project is interested in the Sandstorm port being successful. We're very happy to help connect you with the upstream authors.

Platform features that enable publishing

As you have seen, the core of the Sandstorm platform is about how people interact with applications they have installed. Many web applications exist to help people publish content for others, and this section covers platform features that enable that.

Our goals are:

  • Apps should be able to publish content for the world to see.

  • Apps should be able to take input from external services on the Internet. Two example uses cases are (1) federated publishing systems like the Pump protocol used by GNU MediaGoblin and (2) apps that respond to external events, like a tool to update software documentation based on a GitHub web hook.

  • External people visiting this content should not be able to disturb the app's functionality for the owner. Three classic ways that external visitors disturb app functionality are cross-site request forgery attacks, cross-site scripting attacks, and denial of service. Sandstorm should enable app authors to rely on the platform to protect them from these problems.

  • External people visiting this content should preferably not be able to disturb the content for other visitors. Two classic ways people attempt to do this are denial of service and stored cross-site scripting. Sandstorm should enable app authors to rely on the platform to protect them from these problems, although there are some limitations on how well the platform can protect against these problems.

Apps can publish static HTML to the world

Many web apps exist to help people publish websites that the world can see. For example, Sandstorm has a number of blogging packages available.

The key things to know here is that the app should write an HTML tree to /var/www, and then call "publish static" from the "HackSessionContext". You can read more.

Apps can expose an external API for authorized users

Sandstorm apps can export HTTP APIs for consumption by other servers or client apps, authorized using API keys. These API keys are bound to a specific user, so when you receive a request via this API, the platform has populated the X-Sandstorm-User-Id and related headers.

You can read more.

Apps can expose an external API for the world

Apps should be able to use external APIs so that non-Sandstorm users can do dynamic things. For example, the static HTML view of a blog might need an AJAX comments API. We are thinking through the best way to support this.

Platform security philosophy

As a Sandstorm app developer, it is worth understanding the general security philosophy of the platform. They are:

  • A goal of Sandstorm is to allow people to run arbitrarily-vulnerable web applications safely.

  • A Sandstorm user should be able to use software privately, without third parties being able to discover what software the person uses or leverage this knowledge to mount an attack.

  • An app knows who you are, just like a UNIX program knows the current user ID.

  • Permission to access a user's data is always explicit, never implicit. A compromised app should only be able to read or destroy the data it created or that the user explicitly authorized it to access.

  • Sandstorm favors object capabilities as the unit of authorization. You can read more about object capabilities. We believe that discrete object capabilities can improve both security and usability.

  • Access to the network is always explicit, never implicit. A compromised app should not be able to leak the user's data to a third party. This will include browser-based protections as well as backend protections. Only the administrator of a server will be able to grant general network access; general network access exists so to enable installing "drivers" to expose particular protocols (like outbound email) for use by apps. The driver concept is still in development. We hope that it simplifies app development and deployment by automating configuration and allowing some security decisions to be automatically made by the platform.

  • Sandstorm limits apps' access to the Linux platform to protect users from malicious or vulnerable apps. For example, we don't support all Linux system calls. Legitimate apps don't need all of them, and they create a huge attack surface. Through syscall filtering, Sandstorm apps have avoided being vulnerable to many real Linux security issues.

  • When apps display information in a browser context, they have to use our sandbox. This prevents the app from leaking its private data and prevents other sites from attacking the app.

Platform protocol philosophy

Sandstorm is based on Cap'n Proto, a system for efficiently transferring data and capabilities between programs. We focus on designing secure, convenient interfaces between applications first, and second on building compatibility bridges so that the existing base of web applications can work.

  • We are most excited about "native" Sandstorm applications. These use pure Cap'n Proto to speak to Sandstorm, rather than relying on text-based protocols like SMTP or HTTP. The advantage of native Sandstorm applications is that they can run efficiently, quickly take advantage of new platform features, and have native support for sharing data with other apps via capabilities.

  • We will support apps that know nothing of the Sandstorm-specific Cap'n Proto APIs. We call these "legacy" applications. One example of a Sandstorm-developed tool for legacy app support is the sandstorm-http-bridge, which enables apps that speak HTTP to communicate with the Sandstorm supervisor. Each "legacy" app ported to Sandstorm needs to bundle any legacy support tools inside its app package.

Conclusions: how to learn more and get involved

Thanks for reading this far!

Sandstorm is continuously evolving, and we are continuously developing it, so some of these details may change. We hope this has been a useful overview of the platform and helps you understand the platform's goals. This handbook is very abbreviated; consider following the links in each section for more detail.

To dig into the design of Sandstorm, read through the Cap'n Proto protocols that govern how it communicates.

As a developer or app packager, we hope Sandstorm lets you build and share great web-based software. There is lots of great software out there already, and if you port it to Sandstorm, you'll make it easier and safer for others to use. See the 5 minute packaging tutorial for details.

Some existing apps are not a 100% perfect fit for Sandstorm, but a port could make a huge difference in helping people use it conveniently and safely. For example, it is OK to disable some features if the app still would be valuable to Sandstorm users. It's also OK to create a "monolithic" port if you believe it would be useful.