melange

Distributed script automation.

PowerShell and bash, Windows and Linux, in one cluster. Scripts arrive over an HTTP API, replicate through a Raft consensus log, and execute only where they can run.

01 — What it does

Submission, placement and results, all through one replicated log.

melange replaces a system running more than a million PowerShell executions a year. A cluster is a mix of Windows and Linux machines; a job names the runtime its script is written for, and the leader places it only where it can run.

Results go through the log too

A job's stdout and exit code go through the same consensus log its submission did, so they survive the death of the node that produced them.

Execution is at-least-once. A partitioned node is indistinguishable from a dead one, so a reassigned job can run twice; scripts need to be idempotent.

Any node accepts writes

Only the leader may append to the Raft log, so a proposal made on a follower is forwarded to the leader over the internal transport. A worker on any node can therefore record its result.

A 503 means the cluster cannot commit (no leader, or no quorum) and carries the leader's address when there is one to name.

Placement is filtered, then weighted

A node advertises its cores, memory, secret names and available interpreters. The leader narrows the pool to nodes that can honour the job, then picks the least loaded of those by units / slots.

A job nothing in the cluster can run is refused at submission, and never placed somewhere it would fail.

02 — How a job flows

Committed to a majority, then run by exactly one node.

Every node applies every log entry, but a node only runs jobs assigned to itself. That check is why a single log entry produces exactly one execution, cluster-wide. The leader is one of those nodes; it takes work like any other, and leadership moves when it dies.

clients cluster 0 jobs · 0 entries peer transport cli 0 sent tui 0 sent web ui 0 sent psm1 0 sent n1 leader Windows pwsh 4 cores · 16 slots 0/8 0/0 units 0 entries n2 Linux bash pwsh 8 cores · 32 slots 0/12 0/0 units 0 entries n3 Windows pwsh 2 cores · 8 slots 0/4 0/0 units 0 entries
  1. POST /v1/jobs — any node accepts the write. A node that is not the leader forwards it over the peer transport.
  2. The leader picks a target: nodes that can run it, then the least loaded by units / slots. It assigns to itself as readily as to anyone.
  3. SubmitJob is committed to a majority. Every node applies the entry; only the assignee acts on it.
  4. 202 Accepted — the id comes back only once the job is durable.
  5. Only the assignee spawns the interpreter, queueing if its own process slots are full.
  6. StartJob and CompleteJob go back through the log — three entries per job, and the output is replicated too.
  7. GET /v1/jobs/{id} is served locally, by whichever node you ask.

03 — Placement

Filtered before it is weighted.

A job can only run where its runtime and its secrets are. The leader narrows the pool to the nodes that can honour it, prefers the ones with a free slot, and only then picks the least loaded, by units / slots, so a big box takes proportionally more work than a small one.

melange-cli cluster status

NODE  CORES  MEM (MB)  SLOTS  IN FLIGHT  RUNTIMES         STATE
*1       16     32768     64          0  powershell
 2        8     16384     32          2  bash,powershell
 3        4      8192     16          1  bash             draining
# what a job needs decides where it may go
melange-cli submit ./backup.sh --secret DB_PASSWORD

# what it declares decides only how heavily it counts
melange-cli submit ./reindex.ps1 --cores 4 --memory-mb 4096
melange-cli submit ./audit.sh --secret NOBODY_HAS_THIS
error: 400  no node can supply this job's secrets: NOBODY_HAS_THIS

melange-cli submit ./audit.sh --secret DB_PASSWORD
error: 503  the nodes holding DB_PASSWORD are unreachable

Two hard constraints, one filter

A node must have an interpreter for the job’s runtime, and must hold every node-local secret it names. Both ask the same thing (can this node run it at all?), so one filter answers both.

A 400 means nothing in the cluster can ever run it. A 503 means nothing that can is reachable right now.

Then weighted by how big the node is

The leader minimises units / slots, where a node’s slots are min(cores × 4, memory / 256 MB): whichever of CPU and RAM runs out first. Ratios are compared by cross-multiplication, so a placement cannot wobble on a rounding error.

Load is queue depth: a node grinding through one 40-minute script is busier than one that churned through fifty one-liners.

Full nodes are skipped if possible

A node at its concurrency limit is skipped while anything else can take the work, but if that would empty the pool it comes back whole, because a full node still runs the job in turn. A declared cost works the same way: it weights placement, and nothing enforces it against the script.

04 — Secrets

Named by the job, resolved by the node that runs it.

A job carries the names of the secrets it needs, never the values. The node about to run the script resolves each one and sets it as an environment variable. A script, its arguments and its output are all replicated in the clear, so a named secret is the only way a password reaches a script without being replicated with it.

# prompted, or --from-file; never an argument, which shells remember
melange-cli secret set DB_PASSWORD
Value: ********
name:     DB_PASSWORD
key:      59961b78f8640c8a
version:  1
nodes:    1, 2, 3

melange-cli secret list      # names, key id, and who can supply each
melange-cli secret rm DB_PASSWORD

melange-cli submit ./backup.ps1 --secret DB_PASSWORD
# the other kind: a JSON file on one machine, read once at startup
melange-server --secrets-file /etc/melange/secrets.json

Cluster secrets, with nothing to set up

The key that seals them is derived from the cluster CA, which every node already holds. The CLI seals the value on your own machine; the accepting node re-seals it under the cluster key, and that ciphertext is what replicates. The plaintext is never in the log, in a database, or in a snapshot, and nothing reads one back.

Node-local secrets constrain placement

A file on one machine, never replicated, and it wins on a name collision. Only that node can supply it, so a job naming it can only run there. A cluster secret is usable on every keyed node, so it does not constrain placement.

A printed secret is scrubbed

The node running the job replaces the exact bytes in stdout, stderr, the live tail and the progress line. Base64, a hash, or a value split across a format string is not caught, so treat it as a safety net for accidents. It is not a security boundary.

05 — Data tables

Replicated, typed storage a script can write to.

melange already replicates what a job is and what it printed. Tables are for what it computed. Columns are declared (text, int, float, bool, timestamp) and rows are written through the same consensus log as everything else.

melange-cli table create hosts \
  --column name:text:key --column os:text \
  --column last_seen:timestamp --column patched:bool

melange-cli table index hosts os
melange-cli row set hosts name=web-01 os=windows patched=true

# repeatable --filter, AND-ed; | inside one is OR
melange-cli row list hosts --filter os:eq:linux --filter patched:eq:false
melange-cli row export hosts --format csv -o hosts.csv
# and from inside a job, against the loopback listener it is handed
Import-Module Melange
Set-MelangeRow -Table hosts -Values @{
  name    = $env:COMPUTERNAME
  patched = $true
}

Queries that say what they cost

Operators are eq, ne, lt, le, gt, ge, prefix, isnull and notnull. There is no contains, because an index cannot answer one.

  • Composite primary keys, and unique indexes enforced on write.
  • Equi-joins to another table's key: --join local:table, and you can filter on a joined column.
  • Show a code as what it means: --label code:weather_codes:description, or declare it once per column and every client draws it. The row keeps its own value.
  • Ordering requires an index on the column; nothing else has a stored order to page through.
  • Aggregates (count, sum, avg, min, max) with grouping, and CSV in both directions.

Reachable from inside a job

MELANGE_API_URL is a loopback listener serving the table routes and nothing else; MELANGE_JOB_TOKEN is minted per run, dies with it, and carries the submitter’s role.

06 — Scheduling and pipelines

A schedule fires a chain of scripts.

Five-field cron, evaluated in a cluster-wide timezone set by --timezone. A schedule names an ordered list of scripts, and the leader mints all their job ids in one fire, so a stage can depend on the concrete job before it.

# one schedule, three script ids, run in that order
melange-cli schedule add <backup-id> <verify-id> <prune-id> \
  --cron "0 3 * * *"

melange-cli schedule list
melange-cli schedule disable <id>

# or chain an ad-hoc job behind others
melange-cli submit ./report.ps1 --after <job-id>

A stage waits for the one before it to succeed

A job with prerequisites is committed, placed on a node and counted as queued load, but never signalled to run until every prerequisite has succeeded.

If one fails, times out or is cancelled, the rest of the chain is cancelled. A dependency means "after this succeeds".

Fired once, whichever node is leader

One watermark check makes a whole pipeline exactly-once across a failover. A fire is all-or-nothing: if any stage's script is missing or has nowhere to run, the tick is skipped, so half a chain is never left stranded.

07 — Notifications and reports

A message drawn from a query you stored.

melange mails a job's outcome, and a running script can send its own message. Both are built from typed blocks — heading, text, fields, preformatted, divider, table — and the server owns the styling, so there is no HTML for a caller to get wrong. A template adds the useful part: one of its blocks is a query over a data table, so the numbers in the message are the numbers at the moment it was sent.

melange-cli template set open-tickets --file open-tickets.json
# the fast loop: renders against real data, sends nothing
melange-cli template preview open-tickets --value day=Monday
melange-cli notify --template open-tickets --value day=Monday

# or an ad-hoc message, with no template at all
echo '3 mismatches found.' | melange-cli notify --subject 'Nightly reconciliation'

# template get prints the object back, so get > file, edit, set < file round-trips
melange-cli template get open-tickets > open-tickets.json
{
  "subject": "Open tickets — {{day}}",
  "blocks": [
    { "type": "heading", "text": "Still open on {{day}}" },
    { "type": "table",
      "table": "tickets",
      "joins": ["assigned_to:users"],
      "filters": ["state:eq:open"],
      "columns": ["id", "title", "users.name", "opened_at"],
      "empty_text": "Nothing is open.",
      "limit": 50 }
  ]
}
# a scheduled report is an ordinary job: register this, attach a cron,
# and make it the last stage of the pipeline that fills the table
Import-Module Melange
Send-MelangeNotification -Template open-tickets -Value @{
  day = (Get-Date -Format 'dddd')
}

A table block is a bounded read

It takes the same joins the row listing does — local:table, with a joined column written table.column — so a report over a table of foreign keys shows names rather than keys, and a join matches at most one row.

  • At most 100 rows a block. A report is a thing a person reads.
  • A block cut short that did not ask to be says so under the table; an explicit limit is the author's answer, not a truncation to report.
  • Ordering needs an index on the column, refused when you save rather than answered in some other order.
  • A timestamp is rendered as a time because the schema says it is one, with the zone on the heading.

Placeholders reach text, never a query

{{day}} is filled from --value day=… in a subject, heading, paragraph, field or caption — and never in a table, joins, filters, columns or order_by. That is what keeps a stored report's cost knowable and stops whoever sends it aiming it at data it was never checked against. An unknown placeholder renders as itself, because an empty substitution makes a report that looks finished and is quietly missing a number.

Checked when you save it, and again when it sends

An unknown table or column, a join spelled backwards, an unparseable filter or an un-orderable order_by is refused while you are looking at it. If a table is dropped months later the section draws a line saying what is missing and the rest of the report still goes.

08 — Shared libraries

Importable code, replicated to every node.

A library is named, runtime-specific and stored like a script. A job carries only the names; the node about to run it writes the content into that job's own directory, so there is nothing to install on a node and nothing to keep in step.

melange-cli library create Inventory ./Inventory.psm1
melange-cli library list

# --library is optional: imports are detected from the script itself
melange-cli submit ./audit.ps1 --library Inventory
# PowerShell: on $env:PSModulePath, so this just resolves
Import-Module Inventory

# bash: sourced from the same directory
source "$MELANGE_LIB_DIR/inventory.sh"

No placement cost

A secret lives on one node, so it constrains where a job can go. A library's content is replicated everywhere, so every node can honour any reference and it never enters placement at all.

Imports are detected

Submitting a script reads its own Import-Module, using module and source lines, and attaches any that name a registered library at the same runtime. --no-detect-libraries turns it off.

Checked at both ends

A name that exists nowhere is refused when the job is submitted, and checked again by the node about to run it, so a job never runs silently without its library.

Libraries are parsed when saved, so a syntax error surfaces as you upload it, before any job imports it.

09 — Projects

Work on scripts locally, then push the set.

A project is a melange.toml beside your scripts. It vendors the libraries they import into modules/, so an editor resolves the imports and you can run a script on your own machine before anything reaches a cluster.

melange-cli project init --name ops
melange-cli project add Inventory        # vendor a library
melange-cli project script new nightly  # scaffold a script

# run it here, with the same environment a node would give it
melange-cli project run nightly

melange-cli project outdated
melange-cli project push --dry-run
melange-cli project push

Run before you upload

project run executes the script locally the way a node would: a private job directory, the vendored libraries on the import path, and the same MELANGE_JOB_DIR, MELANGE_LIB_DIR and MELANGE_CANCEL_FILE variables set.

A script belongs to its project

A script's identity is the pair (project, name), enforced when the entry is applied. Two projects can both have a nightly without one overwriting the other and taking over its schedules.

project clone takes an existing project back out of the registry.

10 — Also in the box

The rest of what a cluster gives you.

Replicated results

A job's stdout and exit code go through the consensus log, the same as its submission.

Three front ends

A CLI, a terminal UI and a web UI over one REST surface, plus a PowerShell module.

Sandboxing

A job object on Windows; no_new_privs, rlimits and optional Landlock on Linux.

Users and roles

argon2id passwords and ordered roles, replicated: create a user on one node, log in on any.

Audit trail

A server-stamped submitted_by the client cannot forge, and you can filter jobs on it.

Retention

History is finite by default: terminal jobs past the cutoff are purged in bounded batches.

Live output

Tail a running job's stdout before it finishes, from any node in the cluster.

Peer mTLS

One cluster CA, with each node minting its own leaf at boot. A node without it will not start.

11 — Quick start

A cluster on your laptop, in about a minute.

You need Rust and at least one interpreter on PATH: pwsh (7+) or powershell (5.1), and bash on Unix. A node detects what it has and advertises it.

01

Three real nodes, with Docker

Watch failover, forwarding and reassignment for real. Brings up a cluster, a web UI, and the CLI baked into the image.

docker compose up --build
02

Or build it and run a single node

The CA is required on every node and is the cluster's shared credential: each node mints its own leaf from it at boot, so there is nothing to distribute.

cargo build --workspace --release

# the cluster CA, once
melange-server --ca-init ./tls

melange-server --node-id 1 --data n1.redb --init \
  --tls-ca-cert ./tls/ca.crt --tls-ca-key ./tls/ca.key
03

Submit something

The runtime is inferred from the file extension, then from a #! line, and falls back to PowerShell. --runtime overrides it.

echo 'Write-Output "hi"' | melange-cli submit --wait
melange-cli submit ./rotate-logs.sh --wait   # runs on a node with bash
melange-cli tui                              # the whole cluster, in a terminal