Higher level composition of Plots and Modules

(aka, "Buildplugs")

Building lots of software, or large numbers of data analysis pipelines, requires higher level compositional tools. It's important to have a way to say repeated things once, so that we can focus our energy on the important, interesting, unique parts of the domain we're working on.

Warpforge focuses on being drivable by declarative APIs, and while that is good for reliability and having simple, standardized ways to debug and audit things... it's not so great for high level composition. Declarative APIs, by their nature, tend to involve saying things very methodically -- and often, this can mean quite redundantly, when building large systems.

We solve this problem by adding layers:

Warplark Buildplugs

The "batteries-included" buildplug system supported by Warpforge is called "Warplark", and it uses the Starlark language. (Starlark is a python dialect, so it's pretty easy to pick up for most users -- and, we bundle the complete interpreter, so it's always available and requires no additional setup.)

When using Warplark, users write Starlark scripts that assemble python data objects that have the same shape as Plot API objects.

Within Warplark, adding inputs is as easy as assigning new keys in a map. Users can write functions that combine related or commonly repeated constructions, like adding common sets of related inputs at once, perhaps while also modifying the action.script list, and so on.

Much more documentation required here.

Other Composition Tools

Warpforge supports a simple and open-ended API for composition tools. The starlark features that Warpforge supports natively are not special, beyond the fact that they come bundled "batteries-included" in the standard distribution of Warpforge.

You can design your own composition tools!

Considerations when designing a Composition Tool

If designing a composition tool, here are some things you may wish to keep in mind:


Buildplugs have formerly been known as "larks", as a placeholder name, during some parts of the evolution of the Warpforge project. We moved to the current name because we didn't want to imply that buildplugs must be in starlark (even though our first implementation was), and because "larks" didn't auto-explain itself very well in the first place.