Layer 2 Warpforge APIs: Modules, Plots, and Replays




The Module API object is fairly skeletal right now. Mostly this is a reserved future space for expansion.

type Module struct {
	name ModuleName
	# semantically: `plot optional Plot`... but practically: it's in a sibling file.
	# future: other optional fields used by "override modules".
	# future: maybe something about recommended update patterns for any catalog inputs in the plot?

# ModuleEnvelope is a type used at the root of documents for version agility.
type ModuleEnvelope union {
	| Module "module.v1"
} representation keyed

# ModuleName strings tend to look a bit like URLs.
# For example: "".
# Characters like "/" and "." are allowed, but ":" and whitespace is forbidden
# and the use of other punctuation characters is Not Recommended.
type ModuleName string

(See also workspace FAQ regarding override modules.)


Plots describe a graph of computations. (You might've also called them pipelines, perhaps? But they're not required to be very linear!)

Plots have inputs and outputs, conceptually similar to how formulas have those concepts, but they're a little different: plots just map names to things (instead of mapping them directly to sandbox port concepts like mount paths or variable names that will be seen inside the sandbox).

Plots have actions (sort of), but they're called "steps". Each step will be templated into a Formula. (Or, contains a sub-Plot — plots are recursive.)

type Plot struct { # has previously had many names: protoformula, module, etc.
	input {LocalLabel:PlotInput}
	steps {StepName:Step}
	outputs {LocalLabel:PlotOutput}

# PlotCapsule is a type used at the root of documents for version agility.
type PlotCapsule union {
	| Plot "plot.v1"
} representation keyed

# StepName is for assigning string names to Steps in a Plot.
# StepNames will be part of wiring things together using Pipes.
# Must not contain ':' charaters or unprintables or whitespace.
type StepName string

# LocalLabel is for referencing data within a Plot.
# Input data gets assigned a LocalLabel;
# Pipes pull info from a LocalLabel (possibly together with a StepName to scope it);
# when a Step is evaluated (e.g. turned into a Formula, executed, and produces results),
# the results will become identifiable by a LocalLabel (scoped by the StepName).
# (LocalLabel and OutputName are essentially the same thing: an OutputName
# gets casted to being considered a LocalLabel when a Formula's results are hoisted
# into the Plot.)
# Must not contain ':' charaters or unprintables or whitespace.
type LocalLabel string

type PlotInput union {
	PlotInputSimple string
	PlotInputComplex map
} representation kinded

# PlotInputSimple is extremely comparable to FormulaInputSimple --
# and it's a superset of it: all the same things are acceptable here.
# PlotInputSimple adds more features:
# some are for getting data from the wider universe (mediated by Catalogs);
# some are for getting data ingested from a host environemnt (unhermetic!);
# and some are simply for wiring all the steps in a Plot together
# into a computable graph!
type PlotInputSimple union {
	| WareID "ware:" # same as in FormulaInputSimple.
	| Mount "mount:" # same as in FormulaInputSimple.
	| Literal "literal:" # same as in FormulaInputSimple.
	| Ingest "ingest:" # allows demanding ingest of data from the environment!
	| CatalogRef "catalog:" # allows lookup of a WareID via the catalog!
	| CandidateRef "candidate:" # Like catalog, but dangling a bit.
	| Pipe "pipe:" # allows wiring outputs from one formula into inputs of another!
} representation stringprefix

# PlotInputComplex allows decorating a PlotInputSimple with filters.
type PlotInputComplex struct {
	basis PlotInputSimple
	filters FilterMap

type PlotOutput union {
	PlotOutputSimple string
	PlotOutputComplex map
} representation kinded

type PlotOutputSimple union {
	| Pipe "pipe:"
} representation stringprefix

type PlotOutputComplex struct {
	basis PlotOutputSimple
	filters FilterMap

type Pipe struct {
	stepName StepName # if set, should be a sibling; if empty, means it's a reference to the parent's input map.
	label LocalLabel
} representation stringjoin (":")

# We'll define the rest of those new members of InputSketch in the next codeblock.

type Step union {
	| Plot "subplot"
	| Protoformula "formula"
} representation keyed


One of the two kinds of Step that a Plot can contain are Protoformulas.

As the name suggests, a Protoformula is just information with some placeholders in it which can easily be stamped out into a real Formula.

type Protoformula struct {
	inputs {SandboxPort:PlotInput} # same as Formula -- but value is PlotInput.
	action Action # literally verbatim passed through to the Formula.
	outputs {LocalLabel:GatherDirective} # same as Formula -- but key is LocalLabel.

When a Protoformula is turned into a formula, the inputs have to be mapped from PlotInput to FormulaInput. That means:

After the Protoformula has been turned into a Formula, and then the Formula has been executed, the results are handled very simply:

New kinds of Input

Plots get to use all the same inputs that Formulas did (WareID, Mount, Literal), but also several more.

We already defined Pipe above because it was needed to define the graph relationships within Plot.

Here are the rest:

type Ingest TODO

type CatalogRef struct {
	moduleName ModuleName
	releaseName String
	itemName String
} representation stringjoin (":")

type CandidateRef TODO

On the Filesystem

There are two files for describing modules and their plot:

Both of them contain... what the filename says.

The reason for these being separated is because they serve different roles. The plot document is about some work to be done. The module document is about how that work and its results can be refered to by other modules.

Typically, every file has a file next to it. However, some features in the future roadmap may allow for more diversity of this relationship.

When using commands like warpforge run ./..., it searches for any appearances of files, recursively down from the current directory.