melange — reference

The full manual. For a five-minute introduction, start with the README; this file assumes you have read it and now need the specifics.

• Architecture
• Runtimes
• Execution semantics
• Secrets
• Scheduling
• Clustering and failure
• The script registry
• Shared libraries
• Data tables
• The REST API
• The CLI
• The web UI
• Operations
• Development
• Known limitations

Architecture

Two transports

Clients never speak gRPC. Nodes never expose Raft over REST. A client feature belongs in REST; anything on the gRPC service is internal cluster plumbing (AppendEntries, Vote, InstallSnapshot, plus Forward and Describe).

The crates

Six crates. The dependency edges are the architecture:

Two rules keep this honest:

• No client may depend on melange-core or melange-proto. That is melange-cli and melange-webui: clustering and execution internals must never leak into a client. They get their types from melange-api.
• melange-api stays dependency-light. Every client links it, so every dependency added there is imposed on all of them.

melange-webui is an HTTP server, which makes it look like an exception. It is not: it is a server for the browser, not for the cluster. It never speaks Raft, never opens a redb file, and reaches melange over the same public REST API the CLI uses.

melange-core is transport-agnostic by construction: it needs to send proposals to the leader and ask nodes about their hardware, so it declares a Forwarder trait and melange-server implements it over gRPC. Core never sees a socket.

How a job flows

  client
    │  POST /v1/jobs
    ▼
  any node ──────────► leader ──────────► Raft log ──────► majority
                       (picks the                              │
                        target node)                           │ applied on
                                                               │ every node
                                                               ▼
                                             ┌─────────────────────────────┐
                                             │ node 1    node 2    node 3  │
                                             │  apply     apply     apply  │
                                             │    │         │         │    │
                                             │  mine?     mine?     MINE   │
                                             │   no        no        yes   │
                                             │                        │    │
                                             │                     execute │
                                             └─────────────────────────────┘
                                                                      │
                                          StartJob / CompleteJob ◄────┘
                                          (forwarded to leader,
                                           replicated to everyone)

1. POST /v1/jobs lands on any node.
2. The leader picks which node will run it and commits SubmitJob { assigned_to } to a majority. The client gets 202.
3. Every node applies the entry — but a node only runs jobs whose assigned_to is itself. That single check is what makes one log entry produce exactly one execution cluster-wide.
4. The assigned node runs the script and writes StartJob and CompleteJob back through Raft — forwarding to the leader if it is a follower.

The leader assigns work to itself as readily as to anyone else. It is a worker that happens to also schedule, not a dedicated coordinator.

Storage

One redb database per node, holding two distinct things:

Jobs are individual rows, not one serialised blob. Applying an entry is O(1), not O(jobs). At a million executions a year, rewriting the entire state on every apply would not hold up in production.

Three invariants worth not breaking:

• In apply, the entry's effect and the new last-applied watermark commit in one redb transaction. Split them and a crash mid-apply either loses the effect or replays a non-idempotent mutation.
• In purge, the log deletion and the purge watermark likewise share a transaction.
• install_snapshot clears the job table before loading. A snapshot is the whole truth; a node that fell behind must not keep jobs the leader has purged.

Everything on disk and on the wire is encoded by melange-core::codec (bincode) — the single place that decides the format. bincode is non-self-describing, so #[serde(flatten)], untagged enums, and anything needing deserialize_any will fail at runtime. Check before adding such an attribute to a replicated type — and note that adding a field to a replicated type is itself a breaking on-disk change, because bincode is positional and #[serde(default)] does not rescue an old row the way it does in JSON.

Runtimes

A job carries the runtime its script is written for. It is powershell or bash, and it defaults to powershell when omitted.

{ "script": "echo hi", "runtime": "bash" }

The CLI works it out for you where it can: from the file extension (.ps1, .sh), then from a shebang (#!/bin/bash, #!/usr/bin/env bash) — which is the only hint a script piped in on stdin can carry. --runtime overrides both. A script with neither hint is PowerShell.

Everything else about a job works identically in both. Arguments, secrets, the private working directory, the cancel file, output capture and truncation, the timeout, the process-tree kill — none of them are PowerShell features, and all of them are available to a bash job:

#!/bin/bash
# args arrive as $1, $2 — separate values, so nothing needs escaping
echo "backing up $1"

# secrets arrive as environment variables, resolved on this node, never replicated
psql "postgres://user:$DB_PASSWORD@$1/db" -c 'VACUUM'

# the job's own scratch directory, and its working directory
echo done > ./receipt.txt        # lands in $MELANGE_JOB_DIR

# and it can be asked to stop before it is killed
while [ ! -f "$MELANGE_CANCEL_FILE" ]; do work; done

How the two differ, and why

The differences are conventions of the interpreters, not choices melange made lightly. Each was a bug before it became a rule.

The BOM is required by one and fatal to the other. Windows PowerShell 5.1 reads a BOM-less .ps1 as ANSI and fails to parse a script with any non-ASCII byte in it. Bash does not treat those three bytes as whitespace: it reads them as the first word of line 1, so a script starting #!/bin/bash fails with $'\xef\xbb\xbf#!/bin/bash': command not found — and then carries on and exits 0. A node getting this wrong would corrupt every bash script it ran, and neither the exit code nor stdout would show it.

A bash script's shebang is a comment. melange runs the file as bash job.sh, mirroring PowerShell's -File: no chmod +x, no reliance on the kernel reading #!, and the arguments land in $1..$n as separate values — so a script's param() block, or its $1, receives exactly what the client sent and nothing needs quoting or escaping. Write #!/bin/bash if you like; it is idiomatic and it is ignored. #!/usr/bin/env python3 will not make it a Python job.

set -e is not injected, and a bash job's success is bash's definition of it. Bash exits with the status of its last command, so a script that fails in the middle and then prints something exits 0, and melange records it as succeeded. This is not an oversight and it is not fixable from outside: prepending set -e would silently change the meaning of every script anybody submits, including the ones deliberately written to continue past a failure. If you want strictness, put set -euo pipefail at the top of your script, where you can see it.

Bash is not looked for on Windows

melange does not go looking for bash on a Windows node. You can force it with --runtimes powershell,bash — and you almost certainly should not.

bash on a Windows box is usually not bash. CreateProcess resolves it to C:\Windows\system32\bash.exe, which is the WSL launcher, shipped with Windows and sitting on the system PATH ahead of Git Bash. With no WSL distro installed it simply fails, and melange's probe correctly refuses to register it. With one installed it would succeed — and that is much worse than failing, because the node would then run your jobs inside a Linux VM with a different filesystem and a different user, where the working directory melange created does not exist, the cancel file it polls for can never appear, and the process tree melange kills on a timeout is not one it can reach. Every mechanism in the executor would be quietly pointed at the wrong computer.

So detection runs the interpreter (bash -c "exit 0") rather than looking for a file on PATH, and --runtimes says what to look for, never what to claim: a runtime that does not answer the probe is not advertised, however loudly it was asked for. A node that could be configured into lying about what it can run would put the lie in the replicated membership, where every future leader would believe it.

Text encoding

This is a PowerShell problem, and only a PowerShell problem. Bash writes the bytes it was given; if you only run bash jobs, none of this applies to you.

Install pwsh (PowerShell 7) on every node that runs PowerShell. melange prefers it and falls back to Windows PowerShell 5.1, and the fallback is a real degradation — the node warns at startup when it takes it.

The reason is that PowerShell encodes a script's output into the console's code page, and does so lossily. On a machine whose console is CP437 (an ordinary default), a script printing café ✓ — déjà vu hands over bytes in which the tick has been best-fit substituted with √ and the em dash flattened to - — destroyed by the shell before melange sees them. So melange sets the console code page to UTF-8 before every PowerShell job. This is not a quirk of the old shell: pwsh 7 does the same thing, and the fix is needed for both.

Two consequences worth knowing:

• melange changes the code page of the console it is attached to, and leaves it changed. It is a Windows-wide setting shared with anything else in that console. It is also the only lever that reaches the child process.
• With no console at all — a node running as a service — there is nothing to set. There, only pwsh writes UTF-8 by default; Windows PowerShell 5.1 will transcode a script's non-ASCII output into the machine's ANSI code page, and melange will store it that way. This is the case the startup warning is about.

Execution semantics

A script can run more than once. Write idempotent jobs.

This is a deliberate trade, not an oversight. Two paths lead to it:

1. A node dies mid-job and restarts. It cannot know whether the script had side effects before it died, so it re-runs it from the top.
2. A node is partitioned, not dead. From the leader's side these are indistinguishable. After --reassign-after-secs the leader gives the job to someone else — while the partitioned node may still be happily running it.

Lowering --reassign-after-secs recovers from real failures faster and widens the double-execution window. Raising it does the opposite. There is no setting that eliminates the window; that would require distributed locking with fencing tokens, which costs more than it is worth for a system whose jobs should be idempotent anyway.

What melange does guarantee:

• A submitted job is never lost. It is on a majority of disks before you get the id.

• A job's recorded output is never lost, for the same reason.

• A job that has reached a terminal state is never run again, and its state never changes afterwards. This is one rule, and every entry that could violate it is a no-op on a terminal job: ReassignJob (or a job that finished as the leader decided to move it would be resurrected), CancelJob (a job that beat the cancellation keeps the result it produced), StartJob (a worker that hadn't noticed its job was cancelled cannot drag it back into running), and CompleteJob (a script that exited at the instant it was killed cannot overwrite the cancellation).

  The single exception is additive: a terminal job with no recorded output will accept output, once. It has to, or the partial output of every cancelled job would be lost — a job is marked cancelled the moment the entry applies, which is before the worker it just killed has had any chance to say what the script printed. State and load do not move; only the empty field is filled.

• One log entry produces one execution per attempt, never a fan-out to every node.

Stopping work

A job ends in one of three ways, and all three are the same shape — a terminal state in the replicated log, which is what stops the leader reassigning it and stops its owner resuming it after a restart:

A kill takes the whole process tree. A script that launches robocopy, or a nested pwsh, or backgrounds an rsync with &, would otherwise leave it running after a timeout — still burning CPU, still holding the files the job was meant to release, and with nothing left that will ever reap it. At a million jobs a year, a small fraction leaking a process is a node you have to restart on a schedule. So the shell is spawned inside a Windows job object (a process group on Unix), and killing the job terminates everything it started. This is the one real OS split in the executor, and it is per- machine, not per-runtime: a bash job on Linux and a PowerShell job on Linux are killed the same way.

A script that finishes cleanly is still free to leave something running behind it — only a job we kill takes its children with it. Starting a service and exiting is a legitimate thing for an automation script to do.

Where a job runs

Every job gets a private directory, and it is the script's working directory.

Set-Content ".\report.csv" -Value $data     # lands in the job's own directory
$env:MELANGE_JOB_DIR                        # ...which is here
$PSScriptRoot                               # ...and here: the script lives in it

echo "$data" > ./report.csv                 # the same, from bash
echo "$MELANGE_JOB_DIR"                     # ...and it is the same variable

Without this, a script writing ./report.csv drops a file into wherever the melange node happens to have been started — its install directory — and two jobs doing that at once fight over the same name.

It is scratch, not storage: the directory is deleted when the job ends. Anything a job means to keep must be written somewhere else, or printed to stdout, where it is captured and replicated. Point --work-dir somewhere with room if the system temp directory is small or RAM-backed.

Letting a script stop itself

A killed script does not unwind. Its finally blocks do not run, its locks are not released, its half-finished work is not rolled back. If that matters, melange can ask it to stop before making it:

# melange sets $env:MELANGE_CANCEL_FILE for every job. The file does not exist
# while the job is running; it appears when the job is cancelled or times out.
try {
    while ($true) {
        if (Test-Path $env:MELANGE_CANCEL_FILE) { break }
        Do-SomeWork
    }
} finally {
    Release-TheLock       # now this actually runs
}

# The same variable, the same contract, from bash.
cleanup() { release_the_lock; }
trap cleanup EXIT

while [ ! -f "$MELANGE_CANCEL_FILE" ]; do
    do_some_work
done

Run the node with --kill-grace-secs N, or ask for it per job with kill_grace_secs (the CLI's --kill-grace). On cancellation or timeout melange creates the file, waits up to N seconds for the script to exit on its own, and only then kills the tree. A script that stops politely keeps its own exit code; the job is still recorded as cancelled or timed_out, because why it stopped has not changed.

It defaults to 0 — off. The grace period delays every kill, and buys nothing at all for a script that does not watch for the file. Per job is usually the right place to set it: how long unwinding takes is a property of the work, and a node-wide setting makes every cancellation wait for the slowest job on the box.

Why a file and not a signal. Because there isn't one that works everywhere melange runs. SIGTERM has no meaning on Windows, and CTRL_BREAK_EVENT only reaches processes sharing the caller's console — a melange node running as a Windows service has no console at all. Against a real pwsh running a try/finally, GenerateConsoleCtrlEvent reports success, the process survives, and the finally block never runs.

A file is the one mechanism that means the same thing to a PowerShell script on Windows and a bash script on Linux, so it is the one melange committed to. MELANGE_CANCEL_FILE is part of the contract with the scripts you write, in either runtime.

That is a weaker guarantee: a script that ignores the file is still killed. So is one wedged in a syscall that never gets to look. Nothing here can make an uncooperative script tidy up after itself.

Secrets

Everything else about a job is replicated in the clear. The script, its arguments, its output — all of it is written to every node's Raft log and handed back by GET /v1/jobs/{id}. A password in an argument is a password in the log, on every node, forever.

So a job asks for secrets by name:

melange-cli submit backup.ps1 --secret DB_PASSWORD -- -Server db01

# the script just reads it from the environment
Connect-Database -Password $env:DB_PASSWORD

# ...and so does a bash one. Nothing about secrets is PowerShell-specific.
psql "postgres://user:$DB_PASSWORD@db01/app" -c 'VACUUM'

There are two ways to give the cluster a value. Cluster secrets are the usual one: set once from anywhere, sealed, and usable on every node. Node-local secrets are a file on one machine, and are still the right answer for a credential that is genuinely about that machine.

Either way, only the name is submitted, only the name is replicated, and only the name comes back out of the API. The value is resolved on the node that runs the job and set as an environment variable of the same name.

Two consequences worth being clear about:

• A job only runs where its secrets are. A job that names a secret its host does not have fails rather than running without it — a backup script that quietly authenticates as nobody and reports success is how backups silently stop working. So the scheduler does not send it there in the first place.

• A script that prints its own secret is scrubbed, within limits. melange scans a job's stdout, stderr, live output tail and progress status for the exact bytes of the secrets that job declared, and replaces each occurrence with [redacted DB_PASSWORD] before anything is stored or served. So the commonest accident — a stray Write-Output $env:DB_PASSWORD, an error handler that dumps a connection string — no longer puts a credential in the replicated log forever.

  It is a safety net, not a boundary you may rely on. What it does not cover:

  • a value that has been transformed — base64'd, hex-encoded, hashed, uppercased, or split across a format string;
  • values shorter than 6 bytes, which are deliberately not scanned for (scrubbing every occurrence of dev destroys ordinary output, and a mark standing in intact text around a two-character value gives it away rather than hiding it);
  • the job's own script and arguments, which are client-supplied and replicated in the clear from the moment they are submitted — a script whose source contains the literal value has already published it;
  • anything the script writes somewhere else: a data table through the in-job API, a file, an outbound HTTP request. Scrubbing a data table would corrupt legitimate rows that happened to collide, and a script writing a secret into one is doing so deliberately;
  • a node started with --no-secret-redaction.

  Redaction is per node, so in a mixed cluster the same job leaks or does not depending on where it lands — the same caveat --sandbox carries.

Cluster secrets

There is nothing to set up. The key that seals them is derived from the cluster CA private key, which every node already holds — so any cluster with peer mTLS working (which is every cluster; it is required) can store secrets immediately.

Set one from anywhere:

$ melange secret set DB_PASSWORD
Value: ********
name:       DB_PASSWORD
key:        59961b78f8640c8a
version:    1
nodes:      1, 2, 3

melange secret list          # names, keys, and which nodes can supply each
melange secret rm DB_PASSWORD

The value is never a command-line argument — it is prompted for, or read from a file with --from-file (- for standard input). Arguments are visible to anyone who can list processes, and they land in shell history.

What actually crosses the network. melange secret set seals the value on your machine, to a public key the cluster publishes, before sending it. So the value is unreadable to the network, to any proxy in front of the cluster, and to a request log. The accepting node opens it, immediately re-seals it under the cluster key, and it is that ciphertext which is replicated. The value is never in the Raft log, in any node's database, or in a snapshot.

Be clear about the limit: the accepting node does hold the plaintext in memory for as long as it takes to re-seal it, because it is the only party holding both keys. "End to end" here means opaque to everything between you and the cluster — not opaque to the cluster, which has to be able to hand the value to a script later.

A value cannot be read back. There is no command and no endpoint that returns one. An admin can of course still submit a script that prints a secret — but that leaves a job record with the submitter's name on it, and a reveal endpoint would be one unaudited request that took the lot.

The first time you set a secret against a cluster, the CLI pins its key, and a key that later differs stops the command rather than sealing to it:

Error: this cluster's secrets key has changed.
  pinned: 59961b78f8640c8a Rnhpnf...
  now:    a31c4e02b7dd118f 9kQm2X...
If the key was rotated deliberately, re-run with --accept-new-key.

A changed key is either a rotation you performed or something in the path answering for the cluster, and no client can tell those apart — so it asks.

The one thing to know about the derived key: replacing the CA replaces it, and every secret sealed under the old one stops being readable. That is not silent — the cluster pins its key id, so a node deriving a different key is refused when it writes and advertises nothing when it reads — but if you are going to rotate the CA, pin the secrets key to a file first:

melange-server --secrets-key-export /etc/melange --tls-ca-key /etc/melange/ca.key
# copy secrets.key to every node, start each with --secrets-key, then rotate the CA freely
melange-server --secrets-key /etc/melange/secrets.key ...

--secrets-key is also how you keep the two independent from the start, if you would rather they never be tied together. melange-server --secrets-key-init <dir> generates a fresh unrelated key for that.

Rotating the secrets key itself. The key file is one base64 key per line, first one seals and the rest still open. So a roll is: prepend a new key everywhere, restart nodes one at a time, and drop the old key once nothing is sealed under it. Nothing is unreadable at any point.

A node with the wrong key — a stale CA, or a --secrets-key pointing at the wrong file — is still a perfectly good node. It advertises no cluster secrets, so the scheduler simply never sends it work that needs one, and melange secret list shows it missing from that secret's nodes, which is the fastest way to spot it. It warns at startup too.

Turning it off. --no-secret-redaction on a node stops the scanning. There is one good reason for it: a secret whose value is ordinary enough to occur in legitimate output — a hostname, a common word — is scrubbed wherever it appears, which can render a log useless. The node says so loudly at startup, because from then on a value a script prints is stored and replicated.

One latency note. To catch a value split across two reads, the scanner holds back the few bytes at the end of a chunk that could still begin one. For a typical credential that is tens of bytes and invisible; for a 4 KiB secret, melange logs --follow can lag by up to that much until the script writes again. It always resolves when the stream ends.

Node-local secrets

A file only that node can read, unchanged from how it has always worked:

melange-server --secrets-file /etc/melange/secrets.json ...

{ "DB_PASSWORD": "..." }

A node-local secret wins over a cluster secret of the same name, so this stays usable as the deliberate override for one machine that needs a different value.

Unlike a cluster secret it is a placement constraint: a job naming it can only run on the nodes that have the file. That is the point of it, and also the reason to prefer a cluster secret unless you specifically want the value confined.

The file can be encrypted at rest under the cluster key:

melange-server --secrets-seal /etc/melange/secrets.json --secrets-key /etc/melange/secrets.key

(Sealing a file needs the key named explicitly — export it first if the cluster is using the CA-derived one.)

The node then accepts either form. This protects a stolen disk or a backup — not a reader of the node, since the key file lives beside it. Same honesty as the CA key.

Changing a node-local secret

The secrets file is read once, at startup. There is no reload signal and no API for it — that is what cluster secrets are for — so the procedure is:

1. Edit that node's --secrets-file (or the secrets_file key in its config).
2. Drain and restart the node: SIGTERM, wait for it to stop, start it again. Do not remove it from the cluster to do this — a node removed while it still holds unfinished work has that work reassigned underneath it, and the scripts run twice. See Stopping a node.
3. Nothing else. The leader re-describes the node on its next sweep, sees the new name, and starts placing the jobs that need it there. No rejoin, no --init.

Put a node-local secret on more than one node. Placement is filtered on them, so a job naming a secret only one node holds is pinned to that node — and if that node dies mid-job, there is nowhere legal to move the work to, so it is marked stuck and waits. Treat the secrets file as part of a node's provisioned configuration, and give it to every node that class of job should be able to run on.

How the scheduler knows

Each node reports the names of the secrets it can supply — the ones in its own file, plus the cluster secrets it holds the key for — over the same Describe RPC that reports its cores, its RAM and its runtimes, and that list rides along in the replicated membership. The leader will only place a job on a node that can supply every secret the job names, and load-balances among those. It is the same filter, and the same code, that keeps a bash job off a Windows box.

Names, never values. A name is already public: it is in the job spec, and in the log.

A job nothing can run is refused at submission, not accepted and then failed on some arbitrary node minutes later:

$ echo 'Write-Output hi' | melange-cli submit --secret NOBODY_HAS_THIS
Error: server returned 400 Bad Request: no node in the cluster can supply this
job's secrets: NOBODY_HAS_THIS

400, because no amount of retrying puts a secret on a node — either the name is wrong, or a node has to be given the secret and restarted. If the nodes that do hold it are merely unreachable, that is a 503 instead: the cluster is configured to run the job, it just cannot right now, and retrying is exactly the right thing to do.

A secret you have only just created is a third case, and also a 503:

the cluster holds DB_PASSWORD, but no node has advertised it yet; retry in a moment

A cluster secret is readable on every keyed node the instant it commits, but placement reads the membership, which learns what a node can supply on the leader's next describe sweep. So for up to one sweep a secret exists and is not yet placeable. Nothing is wrong and nothing needs doing — and it is emphatically not the 400 above, which would send you off to fix a secret that is already there.

The executor still refuses to run a job whose secret it lacks. That is not redundant with the scheduler: the membership records what a node had when it was last described, so it is the last line of defence for the window between a node losing a secret and the cluster finding out.

melange is still not a secrets manager — there is no versioned history, no lease, no dynamic credential. It is the smallest thing that keeps a password out of the replicated log in the clear, and lets you put one in place without visiting every machine.

Notifications

melange can email you when a job finishes. It is opt-in per job, it works for one-off submissions and for scheduled runs alike, and it is not a property of a registered script — a script is a template anyone may run, and whose inbox a run reports to belongs to whoever ran it.

# tell me how it went
echo 'Write-Output "hi"' | melange submit --notify

# only tell me if it breaks, and copy the on-call address
melange submit backup.ps1 --notify --notify-on failure --notify-to oncall@example.com

# the same, on a schedule — `failure` is usually what you want here
melange schedule add scr-a1b2 --cron '0 3 * * *' --notify --notify-on failure

Setting it up

Two things have to be true, and the errors say which one is missing.

A node needs a mail relay. This is node configuration, not something a client can ask for:

melange-server --node-id 1 --data n1.redb --init --auth \
  --smtp-host mail.example.com --smtp-from melange@example.com

Any of the others without --smtp-host is a startup error, not a silently ignored setting — a node that comes up looking fine and never mails anyone is exactly the failure this is meant to prevent.

Set it the same on every node. Notifications are sent by whichever node is leader, and leadership moves. A cluster where only one node has a relay works perfectly until that node stops leading, and then stops silently. Nothing enforces uniformity; melange cluster status has an SMTP column so the mismatch is at least visible, and a node with no relay in a cluster where others have one shows off in yellow rather than grey.

A user needs an email address, and this means notifications require --auth. A node started without it has no user accounts at all, so there is nobody to notify and any request that asks is refused. With auth on:

melange user add alice --email alice@example.com
melange user email alice alice@example.com   # or set it later
melange user email alice                     # no address = clear it

An admin can set anyone's; a user can set their own — the same rule as a password, because it is their own inbox. A user with no address on their account cannot ask to be notified even when they are only naming other people, so every notification traces back to somebody.

What arrives

Plain text: the job id and name, its state, exit code, the node that ran it, when it was submitted, started and finished, how long it took, who submitted it, and then the head and tail of stdout and stderr — truncated to --notify-max-output-bytes per stream with a note saying how much was dropped. A job's output can be 10 MiB per stream, and mailing that is a message most relays refuse.

One detail worth knowing: if the email says the job never started, that is not a formatting quirk. It means no node ever picked the job up — it was cancelled while still queued — which is the one thing that distinguishes it from a script that ran and printed nothing.

When it fires

--notify-on always (the default) mails on every terminal outcome. --notify-on failure mails only on failed, timed-out and cancelled jobs — which is what makes a nightly schedule bearable, since one that succeeds 364 times a year should not send 364 emails. Cancellation counts as a failure: a job somebody stopped, or that the dependency sweep cancelled because a prerequisite failed, did not do what it was submitted to do.

For a pipeline, every stage reports, not just the last. A failed stage cancels the ones after it, so which stage broke is the thing worth knowing — and with failure a pipeline that goes green stays silent anyway.

Where the mail actually comes from

The leader sends it, on the same sweep that reassigns stranded jobs and applies retention — not the node that ran the script. That is a deliberate choice and the reasons are worth stating, because the obvious alternative looks simpler:

• Execution is at-least-once. A job reassigned off a partitioned node runs twice; a node that crashes mid-job re-runs it. Reporting from the worker would mail twice for one job. The leader sends once per job id, gated on a flag that is part of replicated state.
• A job cancelled while still queued never reaches a worker at all, so a worker-side notification would never fire for it.
• If the node that ran the job dies right after committing its result, the mail would be lost with it. The next leader picks it up instead.

Known limits

• Delivery is best-effort and at-least-once. A relay that rejects permanently (a 5xx — no such mailbox) gets one attempt and is then given up on and logged loudly, because it will say the same thing forever. A transient failure is retried on later sweeps until the job falls outside --notify-retry-secs. An email lost to an outage is lost; the job record still says exactly what happened.
• A leader that loses the cluster mid-batch can re-send that batch after an election. That is the same partition window that lets a partitioned node keep running a job the cluster has already reassigned.
• A notification carries the job's output, and a script can print anything it was given — including a secret. --notify-to therefore lets an operator mail that anywhere. Recipients are logged at submission, and --smtp-allowed-domains example.com restricts where notifications may go.
• Notifications arrive within a sweep or so of the job finishing (up to ~10s), not instantly.
• Recipients are resolved at submission and stored with the job, so changing an account's address does not retarget notifications for jobs already queued.
• A job's recipients are deliberately not returned by GET /v1/jobs/{id} — any authenticated user may read any job, and publishing resolved addresses would make job-reading a way to harvest the organisation's address book. The API says notify: true and nothing more. A schedule's recipients are shown, because a schedule is a configured object you have to be able to read in order to edit safely.

The API

notified means the leader has dealt with the job — a job whose notify_on did not fire is marked without anything being sent, and so is one whose delivery was permanently rejected.

Scheduling

The leader assigns each job to the healthy node with the most spare capacity. It minimises

    outstanding load / slots

not raw job count — so a 16-slot node takes four times the work of a 4-slot one. Ties rotate, so an idle cluster fans out instead of piling everything on the lowest-numbered node. Ratios are compared by cross-multiplication, never floats, so placement cannot wobble on rounding.

Outstanding load is weighted by what each job declared. A submission's optional cores/memory_mb hints price it in the same slot currency the nodes are measured in — a core is four slots, 256 MB is one, whichever of the two prices higher decides, and a job that declares nothing costs exactly 1, so a cluster where nobody uses hints schedules exactly as it always did. The cost is a weight, never a constraint: a job bigger than every node in the cluster still queues and runs on the least-loaded one, it just makes its host look as busy as it claimed to be. Nothing enforces the hint against the script itself.

Slots come from the node's own hardware report:

    slots = max(1, min(cores * 4, memory_mb / 256))

Whichever of CPU and RAM runs out first decides. A core hosts four jobs — automation scripts spend most of their life waiting on I/O rather than saturating a CPU — so an 8-core box is a 32-slot machine. Unless RAM runs out first: a 64-core box with 2 GB of RAM is an 8-slot machine, not a 256-slot one, because 2 GB only holds 8 scripts. The floor of 1 exists because a node that could never be scheduled onto would silently push its share of the work onto everyone else.

Three properties make this work:

• Load is queue depth, not jobs-ever-run. A node grinding through one 40-minute script is correctly seen as busier than one that churned through fifty one-liners.
• Load is derived from replicated state, not reported. Every node's state machine already knows every job's owner and whether it is finished, so the in-flight count is maintained incrementally as the log is applied. There is no load-reporting protocol, every node computes the same picture, and a leader elected after failover has it immediately.
• Reservations cover the gap between deciding and applying. A scheduling decision counts against its target until the entry lands. Without this, a burst of concurrent submissions would all read the same pre-apply load and stampede whichever node was idlest at the time.

Nodes report hardware — along with their secret names and their runtimes — via the internal Describe RPC. --cores and --memory-mb override detection, necessary under a cgroup/container limit where detection reads the host and over-reports, and useful for deliberately holding a node back; --runtimes overrides what interpreters the node goes looking for.

Placement is filtered before it is weighted

There are two filters:

• Runtime. A bash job can only run on a node with bash.
• Secrets. A job can only run on a node holding every secret it names — it fails anywhere else, on purpose.

They are the same question — can this node run this job at all? — so they are one pure function (capable_nodes), and the leader narrows the pool with it and then picks the least loaded of what is left. Both inventories come from the same Describe and ride in the replicated membership, so this costs no network round trip and a leader elected after failover schedules just as well.

A job that no node can run is refused at submission (400) rather than placed somewhere it will die. If both constraints fail at once, the runtime is the one reported: telling somebody who submitted bash to an all-Windows cluster that nobody holds their DB_PASSWORD sends them to fix the wrong thing, and it still would not run.

The same filter governs reassignment, which is where it earns its keep. When a node dies, its stranded jobs go to survivors that can actually run them — even if that means a busier node, because the idlest one is a Windows box and the job is bash.

Choosing the node yourself

Sometimes the leader's answer is the wrong one, because the thing the job needs is not a runtime or a secret — it is that machine. A script that reads a local disk, or reaches a network only one node sits on, has to run there and nowhere else. Name the nodes and it will:

melange submit collect.ps1 --node 3          # runs on node 3, and only node 3
melange submit collect.ps1 --node 2 --node 5 # runs on both — two jobs, one each

Node ids come from melange cluster status. The same option is on melange script create (where it governs every run of that script, scheduled ones included), on melange script submit (this run only), and on melange schedule add (that schedule's fires, overriding every stage's script). The TUI and web UI offer the live cluster as a picker rather than asking you to remember which number is which box.

Two things about it are worth knowing before you use it.

Naming several nodes is a fan-out, not a choice between them. --node 2 --node 5 submits the script twice — one job on each — and returns two job ids. That is usually the point (collect the same thing from every machine); it is not a way to say "either of these, whichever is idler". There is no way to say that, deliberately: the scheduler already does it, and that is what leaving the option off asks for.

A pinned job is never reassigned. Ordinarily a job stranded on a node that stops answering is handed to a survivor. A pinned one is not — moving it would run the script somewhere you specifically excluded, and (in a fan-out) somewhere that is already running its own copy. Instead it stays put, blocked explains that its node has gone quiet and may still be running it, and it clears itself when the node returns. The job is owed either way; melange will not quietly satisfy it somewhere else.

Targeting is checked strictly, and at submission:

Any of them refuses the whole submission. A request for nodes 2 and 5 never half-succeeds: half-honouring it would look like success and be a different job than the one asked for.

A node that is draining is honoured rather than refused, with a warning in the log. Draining means "stop sending me work you could send elsewhere", and this is work that cannot go elsewhere — so the job is committed and runs when the node comes back.

A registered script or a schedule may name a node that has not joined yet, the same way it may name a secret no node holds yet: those are definitions, not submissions, and the check happens when they run. A scheduled fire that cannot honour its targeting is skipped that tick and retried on the next, rather than failing.

Descriptions are refreshed, not frozen

The leader re-asks every node what it is on each sweep (--reassign-after-secs ÷ 3, capped at 10s) and updates the membership when the answer has changed. So a node that has gained DB_PASSWORD, or had pwsh installed on it, starts being sent the work that needs it within seconds of coming back — without leaving and rejoining the cluster.

The address is never taken from that answer, only the capacity, the secret names and the runtimes. A node id that can be pointed at a different machine is the classic cause of split-brain.

Clustering and failure

Each node needs its own id, its own ports, and its own database — and the same cluster CA as every other node, which is what lets them speak to each other at all (see Securing the peer port). Only the first node is started with --init; the rest are started bare and then joined via the leader.

# the cluster's CA, generated once and copied to all three machines
melange-server --ca-init ./tls          # writes ./tls/ca.crt and ./tls/ca.key

# node 1 — the seed. --cluster-name is optional and applies only here, on a fresh cluster.
melange-server --node-id 1 \
  --listen 127.0.0.1:8080 --peer-listen 127.0.0.1:50051 \
  --tls-ca-cert ./tls/ca.crt --tls-ca-key ./tls/ca.key \
  --data n1.redb --init --cluster-name "Acme Prod (EU)"

# nodes 2 and 3 — started, but not yet members of anything
melange-server --node-id 2 \
  --listen 127.0.0.1:8081 --peer-listen 127.0.0.1:50052 \
  --tls-ca-cert ./tls/ca.crt --tls-ca-key ./tls/ca.key --data n2.redb

melange-server --node-id 3 \
  --listen 127.0.0.1:8082 --peer-listen 127.0.0.1:50053 \
  --tls-ca-cert ./tls/ca.crt --tls-ca-key ./tls/ca.key --data n3.redb

The CA flags are left out of the shorter examples elsewhere in this document for readability; they are not optional anywhere.

Now join them at the leader, giving each node's gRPC address (not its REST address — peers talk to each other over gRPC):

melange-cli --server http://127.0.0.1:8080 join 2 127.0.0.1:50052
melange-cli --server http://127.0.0.1:8080 join 3 127.0.0.1:50053

The leader asks each joining node what it is — its hardware, the secrets it holds, and the runtimes it can run — and replicates the answer with the membership:

$ melange-cli cluster status
cluster: Acme Prod (EU)
env: prod -> http://127.0.0.1:8080
leader: 1

NODE  PEER (GRPC)       REST              CORES  MEM (MB)  SLOTS  IN FLIGHT  RUNTIMES         STATE
*1    127.0.0.1:50051   127.0.0.1:8080       16     32768     64          0  powershell
 2    127.0.0.1:50052   127.0.0.1:8081        8     16384     32          2  bash,powershell
 3    127.0.0.1:50053   127.0.0.1:8082        4      8192     16          1  bash             draining

The cluster names itself. --cluster-name at --init stamps a name into replicated state, so every node and every client reports the same one — and a client environment that was bound to a different cluster says so instead of quietly working. It applies only with --init, on a genuinely fresh cluster, which makes it safe to leave in a service definition across restarts; a node started with it and no --init refuses to boot rather than silently dropping it, because a joining node takes the cluster's name from the cluster. Rename an existing one with melange-cli cluster rename "Acme Lab" (admin only), and read it with melange-cli cluster name. A cluster nobody has named is a perfectly ordinary cluster, and neither line above appears for it.

* marks the leader. Each node has two addresses and they are not interchangeable: PEER is where other nodes reach it for Raft, REST is where a client reaches its API. SLOTS is how many jobs it can run at once, derived from its hardware; IN FLIGHT is how many it currently owns. The scheduler works to equalise IN FLIGHT / SLOTS across the cluster — among the nodes that can run the job in hand.

That is a perfectly ordinary mixed cluster. Node 1 is a Windows box, node 3 is a Linux box with no pwsh installed, and node 2 is a Linux box with both. A PowerShell job can go to node 1 or 2; a bash job can go to node 2 or 3. RUNTIMES is the first column to look at when a job is not being scheduled anywhere.

Node 3 is draining — being sent no new work, but still a voter and still finishing the one job it has. See Stopping a node. Note what that costs the cluster: node 3 was one of only two nodes that could run bash, so while it drains, every bash job goes to node 2.

Run three nodes, not two. A cluster of n voters tolerates the loss of (n-1)/2 of them. Quorum of two is two, so a two-node cluster survives no failures at all — lose either one and the survivor is a leader that cannot commit anything. It is strictly worse than a single node, which at least cannot be outvoted. Three is the smallest cluster that is actually a cluster. Use odd numbers.

What happens when things break

Peer liveness is tracked by the transport, not by openraft — openraft reports replication progress, not per-peer liveness. The Raft network records the last successful contact with each peer while heartbeating, and the scheduler reads it.

A cluster that cannot commit does not pretend otherwise. This is worth spelling out, because Raft's failure mode here is a quiet one: a proposal made without quorum is not refused — it is appended and left pending, and it commits whenever a majority returns. Left alone, that means a submission hangs forever and then quietly runs, an hour later, with no client still waiting; and a reassignment decided during an outage lands afterwards, moving a job off a node that came back and finished it. So melange checks first: writes are refused with a 503 before anything is written, and the leader's background sweep makes no decisions at all until the cluster can hear them.

The script registry

A one-off submission runs a script once and is gone. The registry is where a script is saved — named, described, and browsable — so that anyone can call it on demand without pasting it in again. It is the difference between a cluster you feed scripts to and one you set up scripts on.

A registered Script is deliberately little more than a saved submission: it holds the same JobSpec a POST /v1/jobs would carry — script body, runtime, arguments, named secrets, per-job timeout and kill-grace — plus a name, a description, and optionally the project that owns it. Running one resolves it straight back into that spec and hands it to the ordinary job pipeline. There is no second execution path: the registry is a producer of ordinary jobs.

A script is addressed by its scr-… id; its identity is (project, name). Inside a project the pair is unique — a second script claiming one is a 409 naming the script already holding it — and a job the script produces is labelled project/name. A script with no project is in the unnamespaced pool, where names are free-form and not unique; that is what a bare melange script create writes into, and where every script registered before projects existed sits.

The point of it is melange project push, which resolves a manifest's [scripts.<key>] against the registry: without a project, two projects that both declare [scripts.nightly] resolve to each other's entry, and the second push replaces the first project's script in place — keeping its id, so every schedule attached to it starts firing the other project's code. Uniqueness is enforced in the state machine rather than at the REST edge, so the verdict is the same on every node however the write got there.

$ melange-cli script list --project backups   # one project's scripts
$ melange-cli script list --project ''        # the ones belonging to none

# save it once
$ melange-cli script create --name nightly-backup --runtime bash \
    --secret DB_URL ./backup.sh
scr-18c1fd39e6b794ac

# anyone can run it, whenever
$ melange-cli script submit scr-18c1fd39e6b794ac --wait
job-18c2…   # an ordinary job — placed, replicated, executed like any other

Three things follow from "a saved submission, run like any other job", and each is a deliberate choice:

• A registered script may name a secret no node holds yet. A submission naming an unavailable secret is refused outright, because it is being run now; a registration is not, and a node may be given the secret before the script is ever run. Validation at registration is limited to a non-empty name and body. The full capability check — right runtime, right secrets — happens at run time, where a script that cannot be placed fails or is refused exactly as a direct submission would be.
• The run wears the script's name. A run of nightly-backup shows up as a nightly-backup job, so a listing reads as a history of what was run, not a wall of anonymous ids.
• Deleting a script does not touch its jobs. They are ordinary jobs and outlive the definition that produced them; the registry entry is just the template.

Registry writes are replicated like everything else — created on any node, forwarded to the leader, committed to a majority, then readable on every node — so the registry survives a leader failover with the jobs. The endpoints are under Script registry endpoints; the CLI and web UI both expose the whole of it.

Schedules: running a script on a cron

A schedule attaches a cron expression to a registered script and runs it automatically. This is what makes melange a scheduler and not just a runner: write a script, register it, attach a schedule, and it runs itself — no external cron, no manual submission.

# 03:00 every day; and every 15 minutes; a script can have several
$ melange-cli schedule add scr-18c1fd39e6b794ac --cron "0 3 * * *"
sch-18c2a1b0c9d4e5f6

A Schedule is its own small object pointing at a scr-… id, so one script can have many schedules (a weekday one and a weekend one, say), each added, paused, or removed independently. The cron is standard 5-field Vixie cron — minute hour day-of-month month day-of-week — evaluated in the server's configured timezone (--timezone, an IANA name like America/New_York; UTC by default). So 0 3 * * * fires at 3am there, DST included — chrono-tz owns the calendar. */15 * * * * is every 15 minutes; 0 9 * * 1 at 09:00 on Mondays. A cron that will not parse is refused at creation with a 400.

The timezone is a cluster-wide server setting, not a per-schedule field: only the leader fires schedules, so its --timezone is the one that binds — configure it identically on every node (the same "the leader's setting binds" caveat as --job-retention-secs). It never touches replicated state, so enabling or changing it is not an on-disk format change, and an unknown zone name fails startup rather than silently falling back to UTC. The node reports its zone on GET /v1/cluster (ClusterStatus.timezone) so a client can label a cron input correctly; the web UI does this, and shows every timestamp in the viewer's own local zone (with the zone abbreviation) since a stored time is a plain epoch instant.

Three properties are the whole of how it behaves under failure, and each is deliberate:

• Exactly-once per tick, even across a leader failover. The leader evaluates schedules on its existing sweep; when one is due it fires an ordinary job for the script — same placement, same replication, same execution as a hand submission. Firing is made idempotent by a replicated watermark (last_fired_at, the scheduled tick it last fired for): the single log entry that creates the job also advances the watermark, and is a no-op if the watermark already covers that tick. So a re-evaluation by a new leader after failover, or a replayed log, cannot fire the same occurrence twice.
• Missed ticks coalesce. If the cluster is down across several scheduled times, the schedule fires once on recovery and advances past all of them — no thundering herd of catch-up runs. (At-least-once already means jobs should be idempotent; this keeps a recovering cluster from making that worse.)
• A fire that can't be placed is skipped, not lost. If the script was deleted, or no node can currently run it (its runtime or a secret is unavailable), that tick is skipped with the watermark left where it is — it fires when the script is restored or a capable node returns, rather than erroring or vanishing.

Granularity is one minute — the finest a 5-field cron expresses, and finer than the leader's sweep (which runs at least every 10 seconds) needs. A brand-new schedule does not backfill: it created at noon does not fire for this morning's tick.

Disabling a schedule pauses it without discarding its definition or its watermark; deleting one leaves the jobs it has already fired untouched. The endpoints are under Schedule endpoints; the CLI (melange-cli schedule …) and the web UI (a panel on each script) both manage the whole of it.

Shared libraries

A library is reusable, importable code — a PowerShell module or a bash file to source — that a script pulls in by name, so common logic (logging, auth wrappers, an API client, retry helpers) lives in one place instead of being copied into every script. A library is not runnable on its own: nothing schedules or executes one. A job names the libraries it needs, and the node about to run the job materializes each into the job's private directory before the script starts.

Register one with POST /v1/libraries (or melange-cli library create, or the web UI's Libraries tab), then import it from any job or registered script by listing its name in libraries:

# a shared module, then a job that imports it
melange-cli library create Utils ./Utils.psm1 --runtime powershell
echo 'Import-Module Utils; Get-Widget' | melange-cli submit --library Utils --wait

Three things make it fit the rest of melange, and each is a deliberate difference from how a secret works:

• A library's content is replicated, so it needs no particular node. A secret is resolved from a node-local store, so the scheduler will only place a job where its secret is. A library is shareable code, not a credential, so its bytes are replicated to every node — which is what lets a job resolve its imports on whatever node ends up running it, including after a reassignment to a node that never saw the submission. A library therefore filters no placement: it exists everywhere or nowhere, and a job naming one that exists nowhere is refused at submission (a 404), not sent somewhere to fail.
• The name is the identity. It is the string a script writes in Import-Module, and it becomes a filename, so it is unique and filename-safe (letters, digits, _, -). There is no separate id; "renaming" a library would break every script that imports it, so it is a delete and a create.
• How you import is per-runtime. For PowerShell, melange lays the library out as a module on $PSModulePath, so Import-Module <Name> resolves with no path. For bash, the file is at $MELANGE_LIB_DIR/<name>.sh, so source "$MELANGE_LIB_DIR/<name>.sh". Either way the directory is exposed as MELANGE_LIB_DIR, set only when a job actually imports something.

A library resolves to whatever content currently exists at run time — the same semantics a registered script has (it resolves to its current spec). There is no version pinning: editing a library changes what the next run of an importing job sees, and an at-least-once re-run after an edit runs the new version. Like a script, a library is analyzed when it is saved (a syntax verdict, and for PowerShell its param() block), and the analysis is advisory — a library the parser dislikes is still stored and still importable. Nothing checks that an imported symbol exists: melange records, it does not gatekeep. The endpoints are under Library registry endpoints.

You usually do not have to name them

melange submit and melange script create/update read the script's own imports and attach any that name a registered library, so --library is for the cases nothing can see in the text:

# Both of these do the same thing.
melange-cli submit ./backup.ps1 --library Utils --library Acme-Logging
melange-cli submit ./backup.ps1
# note: attaching Acme-Logging, Utils — detected from the script's imports

It recognises Import-Module Foo, Import-Module -Name Foo, using module Foo and #Requires -Modules Foo for PowerShell, and source "$MELANGE_LIB_DIR/foo.sh" for bash. A commented-out import does not count, and neither does a path (Import-Module ./Thing.psd1) or a variable (Import-Module $name).

The web UI does the same thing, visibly. Its Submit and Scripts forms offer the registry as a row of tickable libraries instead of a box to type names into, and tick them from the script's imports as you type. Detection there only ever ticks — untick one and it stays unticked while you keep editing — and "Read the script's imports" turns it off, the equivalent of --no-detect-libraries. Because it has somewhere to say so, it also reports what the command line drops silently: an import naming a library that is not registered, or one registered at the other runtime and therefore not attachable.

Why this is on by default, when melange is otherwise reluctant to guess: it cannot make anything worse, and the thing it fixes is silent. A name it misses leaves you exactly where you were — typing --library yourself. A name it invents is discarded, because a detected name is only used if the registry has a library by that name, at the script's own runtime. And what it prevents is this:

$ melange-cli submit ./backup.ps1 --no-detect-libraries --wait
state: succeeded
exit:  0
--- stderr ---
Import-Module: The specified module 'Utils' was not loaded because no valid module file
was found in any module directory.
Get-Greeting: The term 'Get-Greeting' is not recognized as a name of a cmdlet...

A forgotten --library is not a failed job. Import-Module raises a non-terminating error, so the script runs on, does nothing useful, and is recorded as succeeded with exit 0 — the PowerShell twin of the set -e trap that bash scripts carry, and just as quiet.

Two limits, both worth knowing:

• PowerShell auto-loading is invisible. A script that calls Get-Greeting without ever writing Import-Module Utils works on a node, because $PSModulePath is searched on first use of an exported command — and there is nothing in the text to find. That is what --library remains for.
• A registry library shadows a system module of the same name, because the job's modules/ directory is prepended to $PSModulePath. Already true of anyone who types --library by hand; what is new is that it can now happen without being asked for, which is why every attached name is reported on stderr. --no-detect-libraries turns the whole thing off.

Detection reads the script's text rather than parsing it, so it is a good guess and not an oracle. melange does own a real parser for this — Parser::ParseFile already runs when a script is registered — and moving the answer onto the stored ScriptAnalysis would make it exact for registered scripts; that is a later change, because it alters a replicated type.

To develop against libraries — get their code onto your own machine and run a script that imports them before you upload it — see Local projects below.

Data tables

melange has always replicated what a job is and what it printed. It had nowhere to put what a job computed. A data table is that place: declared columns, a primary key, and rows written through Raft like every other piece of replicated state — so an inventory a hundred jobs read from, or a result set they all append to, lives in the cluster instead of in a file on whichever node happened to run the script.

Create one, then write to it from a job, the CLI, the terminal UI's or web UI's Data tab, or plain HTTP:

melange-cli table create inventory \
  --column 'asset_tag:text:key' --column 'cores:int' --column 'last_seen:timestamp:null'

melange-cli row set inventory asset_tag=A-1042 cores=8
melange-cli row list inventory --filter cores:ge:8
melange-cli table aggregate inventory --fn sum --column cores --group-by env

Browsing a table at a terminal

The TUI's Data tab lists the tables; Enter opens one as a grid, with the same reach the web UI's has — the columns come from the schema, so a column every row leaves null still has a heading and still has somewhere to type.

The table list takes n to declare a table (columns are typed one per line in melange table create's own name:type[:key][:null] spelling, parsed by the same function) and d to drop one, which takes its rows with it.

A filter box takes the shorthand the web UI takes: a bare value is an exact match (prod means env:eq:prod), an explicit operator is passed through (ge:8), null means isnull and !null means notnull. Typing prod|staging in one box stamps the column onto both alternatives. What a box cannot say is "this column or that one" — for that, F takes whole clauses, one per line, exactly as --filter spells them (env:eq:prod|cores:ge:64).

F also takes joins, one per line, as local_column:table[:target_column] — so assigned_to:users brings the matching users row's columns in beside each row's own. They are drawn dim and are read-only: they belong to another table's row, and a left join that matched nothing shows an em-dash rather than a blank, so a miss is visibly a miss. A joined column can be filtered like any other, as users.team.

Two things the grid tells you that are easy to get wrong elsewhere. The strip at the top says whether the row count is a fact or a floor — a filter with no ready index is a bounded scan, so it will say 18 row(s) match in the first 5000 scanned — the table is larger rather than pretending to a total. And each column heading carries its index state, including building — a half-built index is never used, which is why a filter can still be slow just after you pressed i.

Editing a cell sends the row with the version it was drawn at. If it changed underneath you the write is refused with a 409, the grid says so and re-reads, rather than overwriting whatever landed in between.

The columns are declared, and that is the point

A schemaless bag of JSON would have been less code in melange and worse everywhere else. Declaring the columns is what lets the web UI render an editable grid instead of a blob, lets a filter know cores >= 8 is a number comparison rather than a string one, lets an index exist at all, and — the part that matters at a million rows — lets bad data be refused by the node that accepted it rather than discovered by the job that reads it back a month later.

Column types are text, int, float, bool and timestamp (milliseconds since the Unix epoch, like every other timestamp in melange).

Declaring a column timestamp rather than int is what lets everything that shows it render it as a time instead of a 13-digit number: the grids in the web UI and the TUI print it in your local zone with the zone named, exactly as they print a job's timestamps, and a CSV export writes RFC 3339. The value is a plain number on the wire either way — the JSON API has no timestamp type — so it is the declared schema that makes the difference, which is the same reason the grid has columns at all. Anywhere you can type one you may give either form: milliseconds, or a date like 2025-08-24 13:06 read in your local zone. Digits are always milliseconds, so nothing you wrote before means something new.

At least one column is the primary key; its type must be text, int or timestamp — a float key is refused because NaN has no place in an ordering, and a key must have one.

Mark several columns as the key and you get a composite one, whose parts are those columns in the order they are declared:

melange-cli table create stock \
  --column 'zone:text:key' --column 'sku:text:key' --column 'qty:int'

Rows then sort by zone, then by sku, and neither half has to be unique on its own. Up to eight columns may make up a key. The order is the declaration order rather than something you set separately, because the columns already have an order that everything else respects, and a second one that can disagree with the first eventually does.

A primary key cannot be changed — it is the row key, so altering it would move every row in the table. That includes growing it: adding a key column to a table that has rows is refused. Copy into a new table instead.

A column that the table does not declare is a 400, not a silently dropped value. That is deliberate: dropping it would turn a typo'd column name into a write that reports success and stores nothing, which is the same class of failure as a PowerShell script that runs nothing and exits 0.

Writing from a job

This is what the feature is for. A running script is handed two environment variables by the node executing it:

The shipped PowerShell module picks up both with no configuration, so a script is just:

Import-Module Melange
Set-MelangeRow -Table inventory -Values @{ asset_tag = 'A-1042'; cores = 8 }

# Rows piped in are batched, so thousands of them are a handful of requests.
$machines | Set-MelangeRow -Table inventory

A bash job gets the same two variables and can curl them.

The rest of the module is not available inside a job, and it says so rather than failing obscurely — Test-MelangeReady, Get-MelangeJob, Submit-MelangeJob and every other non-table cmdlet refuse before making a request:

inside a job, melange is data tables only: /ready is not under /v1/tables. ...

That follows from the two bullets below, not from a choice the module made: the address a script is handed serves the table routes only, and the token is refused anywhere else. A script that genuinely needs the full API must be given a -Server and credentials of its own — which on a --rest-tls=mtls cluster means a client certificate and CA, the thing the loopback listener exists to avoid needing.

Three things about that credential are worth knowing, because they are what make handing one to arbitrary submitted code acceptable:

• It is loopback-only. The script talks to its own host, which forwards writes to the leader like any other node. Nothing about the cluster's shape is known to the script, and nothing lets it reach a machine it is not already running on. MELANGE_API_URL names a listener of the node's own — 127.0.0.1 on an ephemeral port, serving the data-table routes and nothing else — rather than the node's REST port. That is deliberate: the REST port may be TLS'd, and with --rest-tls=mtls it demands a client certificate a script has no way to hold, so a script pointed at it could not connect at all. The job listener is plaintext, which is what makes it work on every cluster; it is also why it serves nothing beyond what the token below may call.
• It reaches /v1/tables and nothing else. A script that could POST /v1/jobs with its token could turn a read-only user's job into a way to run anything; one that could reach /v1/users could do worse. Any other path is a 403 — twice over: the job listener does not serve those routes, and the token is refused on them even on the node's own REST port.
• It carries its submitter's role, looked up per request. A job runs as whoever submitted it. Demote that user while their script is running and the next write it attempts is a 403 — the role is not captured at spawn.

The token lives only in the executing node's memory and dies with the run. It is never replicated and never written to disk, so it cannot be replayed from a snapshot or outlive the job it was minted for. It is in the child's environment, which gives it the same exposure a secret has (see Known limitations); it is worth strictly less than one.

Concurrency: rows have versions

Every row carries a version, bumped on each write. A write may name the version it expects:

melange-cli row set inventory asset_tag=A-1042 cores=16 --expect-version 3

--expect-version 0 means only if this row does not exist yet. A mismatch is a 409 and changes nothing.

This matters more than it looks. melange runs scripts at least once, so two attempts at the same job writing the same row is an ordinary thing to happen rather than an exotic one. Compare-and-set is what lets a script tell "I already did this" from "somebody else did".

The web UI uses it for you: editing a cell sends the version the row had when it was drawn, and a 409 reloads the grid rather than overwriting what changed underneath you.

Querying

?filter= is repeatable and every filter must hold:

GET /v1/tables/inventory/rows?filter=env:eq:prod&filter=cores:ge:8

Within one filter, | separates alternatives and any of them may hold — so the params are ANDed and the alternatives inside one are ORed:

GET /v1/tables/inventory/rows?filter=env:eq:prod|env:eq:staging&filter=cores:ge:8

reads as (prod or staging) and at least eight cores. Alternatives may name different columns, so a = 1 OR b = 2 is one filter. A filter with no | means exactly what it always did, so nothing written before this existed changes.

That shape — AND across params, OR inside one — is not an arbitrary restriction. It is what keeps a filter something an index can still narrow: one filter becomes a union of index ranges. An arbitrary boolean expression could not be, and offering one would be a promise to scan.

The one cost is that a literal | in a value must be doubled (||). A value is the unconstrained last field of column:op:value, so the delimiter has to win somewhere; the alternative is a backslash escape layered under URL encoding, where getting the layering wrong gives you a filter that parses cleanly and matches nothing.

Operators are eq, ne, lt, le, gt, ge, prefix, isnull and notnull. A value is parsed against the column's declared type, so cores:ge:eight is a 400 rather than a filter that silently matches nothing — the two are indistinguishable from the outside, and one of them is a bug.

A null satisfies no ordering comparison, and not ne either. Null means "not known", so "is this unknown value greater than 8" has no true answer; returning one would stop cores:gt:8 and cores:le:8 partitioning the table between them. Ask about absence with isnull / notnull.

GET /v1/tables/{name}/aggregate computes count, sum, avg, min or max, optionally ?group_by= a column.

Read complete before quoting a number. A filtered listing and every aggregate but a plain unfiltered count are answered by a bounded scan — melange reads a fixed number of rows per request rather than walking a table of millions. The response therefore carries scanned and complete, and the CLI prints a line saying so when the answer is partial. A number from a partial scan presented as a fact is worse than no number.

The same bound gives the listing rule melange applies everywhere: a short page is not the end of the list — only an absent cursor is. A filtered page can come back empty with a cursor, which means "nothing matched in the rows I was allowed to read; ask again from here".

Ordering

A listing comes back in primary key order. ?order_by= changes that, and &desc=true reverses it:

melange-cli row list inventory --order-by cores --desc

GET /v1/tables/inventory/rows?order_by=cores&desc=true

A column can be ordered on when it is the first part of the primary key, or when it carries a ready index. Anything else is a 400 naming the column. That is not a limitation being apologised for — it is the same rule as everywhere else here. Rows are stored in key order and index entries are stored in value order, so both of those are a range somebody can page through; a column with neither has no order stored anywhere, and the only ways to answer would be to sort the whole table in memory or to sort the rows one bounded scan happened to reach and call that the table's order. The second is a page that is ordered, plausible and wrong, which is exactly what melange refuses to hand back.

So the fix is to index the column, which is one click in the web UI's grid header, i in the TUI's, or:

melange-cli table index inventory cores

Nulls come first ascending and last descending. A null encodes below every value, so this falls out of the key encoding rather than being a policy — a null-heavy column ordered ascending spends its first pages on nulls.

Filters, joins and paging all work unchanged alongside it, and a CSV export takes the ordering too, so what you are looking at is what downloads. An aggregate ignores it: a sum has no order.

Joining another table

?join= brings a matching row from a second table along with each row of the first:

melange-cli row list tickets --join assigned_to:users

GET /v1/tables/tickets/rows?join=assigned_to:users&filter=state:eq:open

The spec is column:table[:target_column], and the column on this table comes first — the same order as ?filter='s column:op:value. Read it as from assigned_to, into users.

The table being joined to needs no column of that name. Omit the third field and the lookup is against its primary key, so assigned_to:users finds the users row whose key equals the ticket's assigned_to. Name a third field and the lookup is against that column instead, which must carry a ready unique index:

melange-cli row list tickets --join assigned_to:users:email   # match on users.email

Each matched row arrives under joined in the response, keyed by the joined table's name. The two columns must have the same declared type, because the lookup compares encoded values — an int and a text that both look like 8 would simply never match.

The matched column is not repeated. It is equal to the local column that found it — that is what a join is — so returning both would duplicate a value in every row of every page. Joining two two-column tables gives you three columns, not four, and the matched value is still there under its local name. Which column disappears follows what you matched on: a join to the primary key omits the key, and assigned_to:users:email omits email instead. It is SQL's USING rather than ON.

Both forms guarantee at most one match, and that is the design rather than a current limitation. It means a join costs exactly one lookup per row returned — not per row examined — so a page of fifty rows is fifty extra reads however large either table is. A join to a non-unique column could match any number of rows, which makes the per-row cost unbounded and the page ragged; that is a different feature with a different cost model, and it is refused rather than quietly provided.

A row that matches nothing keeps its place, with null for the join — never dropped. If a join could remove rows it would be a filter over another table's contents, which is exactly the unbounded thing being avoided. For the same reason a null value joins nothing, rather than matching some arbitrary row that also has nothing there.

Filtering on a joined column

A filter may name a joined table's column as table.column:

melange-cli row list tickets --join assigned_to:users --filter users.team:eq:ops

GET /v1/tables/tickets/rows?join=assigned_to:users&filter=users.team:eq:ops

It composes with everything else — AND across params, | alternatives inside one, and the same operators — and rows.csv takes it too, so an export can be narrowed by the rows it brings along.

A row whose join matched nothing reads as null there, exactly as the response shows it. So users.team:isnull finds the tickets with no assignee (and the ones assigned to somebody with no team), users.team:notnull finds the rest, and no row is ever dropped for having matched nothing — that is still the left join's promise.

The cost is worth knowing. A join a filter names has to be looked up before the row can be tested, so it costs one lookup per row examined rather than per row returned; joins no filter mentions are unaffected and still cost one per row returned. The scan bound is the same one as always, which means the usual consequence: a short page is not the end of the table, only an absent cursor is. And such a filter is never answered from an index — the value lives in another table's row — so pair it with a filter on this table's own indexed column when the table is large.

Three spellings are refused with a 400 rather than quietly matching nothing:

• a table this request did not join (the message names the join= to add);
• a column the joined table does not have;
• the column the join matched on — the joined row does not repeat it, so a predicate over it could only ever read null. Filter the local column that holds the same value.

Exporting a join

GET .../rows.csv?join=… takes the same joins as the listing, and adds their columns to the header as table.column:

name,env,cores,owner,owners.contact
web-01,prod,8,infra,ada
db-01,prod,64,nobody,

The prefix keeps the header unambiguous once two tables both have an id or a name. A row that matched nothing gets an empty cell rather than being dropped, as everywhere else, and the matched column is left out for the reason above. Filters and joins compose, so what you export is what you were looking at.

Those columns come from the schema, not from the rows. A CSV has one header for the whole file and is written a page at a time, so a column set read off the rows could differ between pages and shift every cell after it — the web UI and the CLI listing can read it off the rows because they redraw everything at once, and this cannot.

melange-cli row export --format csv --join assigned_to:users does the same. It is CSV only: a JSONL export writes each row's own values so that it pipes straight back into row import, and a joined table's columns are not this table's to import — so --join with --format jsonl is refused rather than ignored.

What a join will not do

Each of these is refused with a 400 that says why:

• Joins do not chain. One level only; a joined row is never itself joined.
• A composite primary key cannot be joined to. It would need one local column per key part in the target's key order, and getting that order wrong matches nothing — which looks identical to "there are no matches". Join through a unique index instead.
• The same table cannot be joined twice in one request. Results are keyed by the target table's name — which is what a CSV header and a UI column are written from — so a second join to the same table would replace the first, and the request would quietly answer less than it was asked.
• aggregate refuses ?join= rather than ignoring it. An aggregate is arithmetic over one column of this table, and adding a per-row lookup to a scan whose whole contract is that it is bounded would not be one. A parameter that was silently dropped would be indistinguishable from one that did nothing. Since it takes no joins, a table.column filter there is refused as well — the web UI says so in place of the count rather than showing a number for the local half of the query.

Dropping a table

DELETE /v1/tables/{name} removes the schema and, with it, every way of naming the rows. The rows themselves are purged by the leader afterwards, in bounded batches — a table can hold millions, and deleting them inside the single write transaction that every node runs for every log entry would stall the cluster.

The consequence you can observe: recreating a table with the same name gives an empty table. Rows are filed under an internal id that is never reused, so a new table cannot inherit a dropped one's data even while the purge is still running. The web UI spells this out before it will drop anything.

Limits

The row cap exists for the reason --max-output-bytes does, one step further: a row is replicated into every node's Raft log, so an uncapped row is an uncapped write multiplied by the size of the cluster.

Indexes

A filter is answered by a bounded scan unless there is an index on the column, in which case it becomes a range scan over index entries instead:

melange-cli table index inventory cores       # declare it
melange-cli table get inventory               # shows "(building)" until it is usable
melange-cli table unindex inventory cores

Three things about that are worth knowing:

• Declaring one returns immediately, and it is not usable yet. Building it is background work on the leader, done a bounded batch per sweep tick, so declaring an index on a table of any size is safe and blocks nothing.
• A half-built index is never used. A range scan over one returns a subset of the matching rows — ordered, plausible, and short — which is worse than the scan it would have replaced. Until the build finishes, the query scans; the answers are identical either way.
• The index narrows what is read, not what is returned. Every filter is still checked against every candidate row, so an index can never change an answer — only the work done to find it. ne and notnull are deliberately not answered from an index: they match almost everything, so an index over them is a full walk with extra indirection.

Entries are maintained in the same transaction as the row, so a write, change or delete is reflected the instant it commits. Dropping an index removes it from the schema at once; its entries are reclaimed in the background, like a dropped table's rows.

Unique indexes

--unique makes the index a constraint as well as a lookup: a write whose value in that column already belongs to another row is refused.

melange-cli table index inventory serial --unique

A refused row comes back as a 409 on a single write, and inside a batch as a per-row duplicate result — the batch is not abandoned, exactly as with a version conflict. The two are counted separately, because they call for opposite things: a version conflict is retried against the version it reports, while a duplicate cannot be written at all until some other row moves, so retrying it loops forever.

Two rules are worth knowing:

• Nulls are exempt, as in SQL. Any number of rows may leave the column empty. Without that exemption a nullable unique column would allow exactly one empty row in the table.
• The constraint does not wait for the build. Writes are checked from the moment the index is declared. What the checking cannot see is rows the background build has not reached yet, so a duplicate of one of those can slip in mid-build — and the build then finds it, stops, and records the reason. The guarantee is therefore: an index is never usable while a duplicate exists. Nothing is ever queried against a constraint that is false.

An index that stopped this way never becomes ready, and melange-cli table get prints why:

indexes:   serial unique (failed)
           serial: `serial` is not unique: rows "1A-1042" and "1A-1043" hold the same value.
                   Drop the index, fix the data, and declare it again.

There is deliberately no retry — the duplicate will still be there next sweep. Drop the index, fix the rows, declare it again.

Changing a table's schema

melange-cli table alter inventory --add 'notes:text'
melange-cli table alter inventory --rename 'cores:cpu_count'
melange-cli table alter inventory --retype 'cpu_count:float'
melange-cli table alter inventory --drop notes

The change takes effect on read, immediately. Rows written before it are projected onto the new columns as they are read — an added column reads null, a dropped one disappears, a renamed one keeps its value, a retyped one is converted — so nothing waits for a rewrite. The leader then rewrites rows in the background purely to reclaim the bytes of dropped columns and stop paying for the projection. A migration that never finishes costs space, not correctness.

Values a retyped column cannot hold become null, and the count of those is reported on the table until the rewrite completes. An index on a retyped column is reissued rather than kept, since its entries described the old encoding; an index on a renamed column simply follows the name, because the values did not move.

Two rules to know:

• The primary key cannot be renamed, retyped or dropped. It is the row key, so changing it would move every row in the table. That is a copy into a new table, not an alter.
• One change at a time. A second alter while one is still being applied is a 409. That is what keeps projection unambiguous: a row below the table's current version is known to sit at exactly the version the running migration started from, so there is never a chain of renames to compose.

A column added to a table that already has rows must be nullable — those rows have no value for it, and inventing one would be melange deciding what your data means.

Getting data out

melange-cli row export inventory --format csv --output inventory.csv
melange-cli row export inventory --format csv --filter env:eq:prod   # to stdout
melange-cli row export inventory                                      # JSON lines

The web UI's table view has a Download CSV button, which honours whatever the column filter boxes currently say — what you are looking at is what you get. The TUI's Data tab does the same: e on the table list exports the whole table, and e inside the row grid exports what the grid is currently showing, filters and joins included. All three fetch GET /v1/tables/{name}/rows.csv, which takes the same repeatable ?filter= as the listing; the CSV itself is rendered by the node, so an export is the same file whichever asked for it.

The export is streamed and unpaged, unlike every other read here — the size of the table bounds the download rather than the node's memory. It is also not one consistent snapshot: it pages internally, so a row written while it runs may or may not appear. That is the same consistency every paged listing in melange offers, and the alternative would hold a read transaction open across a million rows.

Two formats, and they are for different things. --format jsonl is lossless and row import reads it straight back. CSV is the human-facing one and is lossy on purpose:

• An empty string and a null are the same cell. CSV cannot tell them apart.
• Timestamps are rendered as RFC 3339 (2023-11-14T22:13:20.000Z), not the epoch milliseconds the JSON API uses, because a spreadsheet column of 1700000000000 is of no use to anybody.

Two things the CSV does that are worth knowing about:

• It begins with a UTF-8 byte-order mark. Without one Excel on Windows reads the file as the local code page and café arrives as café. Most parsers skip a BOM or can be told to (utf-8-sig in Python; the default in pandas and R). If you are redirecting the output on Windows, use --output rather than > — a shell redirect can re-encode the stream and undo it.
• A text cell that a spreadsheet would read as a formula is prefixed with '. A cell beginning =, +, -, @ or a tab is a formula to Excel and Sheets, and the values in a melange table are usually written by a script — so without this an export is a way to run something on the machine of whoever opens it. Only text cells are defused: a number from an int, float or timestamp column was generated by melange and cannot be a formula, so -5 stays a number you can sum a column of.

Getting data in: CSV import

melange-cli row import inventory rows.csv               # format from the extension
melange-cli row import inventory rows.csv --check       # validate, write nothing
melange-cli row import inventory --format csv < rows.csv

The CLI reads the file and the node parses it, exactly as the export is rendered there — one implementation, so the CLI, the TUI and the web UI cannot drift into three different readings of the same file. --check exits non-zero and lists every problem, one per line, each naming the file's own line number.

The TUI's Data tab mirrors its export. e opens a save dialog: the directory it is about to write into, its subdirectories and the .csv files already there, and a name box suggesting {table}.csv. Tab moves between the listing and the box, Enter walks into a directory (or takes an existing file's name), Ctrl-S writes, and replacing a file that is already there takes a second Ctrl-S. A name with separators in it is taken as a path, which is how you reach a directory no amount of .. gets to — another drive, or a share. The directory carries over to the next export, so putting three tables in one folder is not three walks to it.

i reads it back, still from {table}.csv beside where the TUI was started. It only ever checks — writing a local file is harmless and writing to the cluster is not — and the node's answer becomes the confirmation, so the prompt says how many rows are about to land rather than asking you to take it on trust.

The web UI's Data tab has an Import CSV button, with two destinations.

Into an existing table. The file is checked against that table's schema first — the button says Check file, and only becomes Import N rows once the check passes. The check is the same request as the import with ?dry_run=true on it, so it is a real rehearsal rather than a separate code path: what it reports is what the import will do.

Into a new table. The node reads the file and proposes a schema — a type per column, a nullable flag, and a primary key — which lands in the ordinary create-table form so you can correct anything before the table exists. It is a guess, and worth making because the alternative is every column arriving as text, which is only noticed the day somebody tries to sum one. Two guesses it deliberately does not make: a column of 1s and 0s stays an int rather than becoming a bool, and a column of large integers stays an int rather than becoming a timestamp.

POST /v1/tables/{name}/rows.csv            # the file's bytes; the inverse of the GET
POST /v1/tables/{name}/rows.csv?dry_run=true
POST /v1/csv/schema                        # propose a schema; creates nothing

Things worth knowing:

• An import is all-or-nothing. One bad cell and nothing is written, with every problem reported at once, each naming the file's line number so you can go straight to it in a spreadsheet. It is not the record's index — a quoted field may contain newlines, so the tenth row is not necessarily on the tenth line.
• It is an upsert, keyed by the primary key, so append and update are the same operation and re-running the same file is safe. That matters more than it sounds: a file larger than one replicated batch is applied in chunks, so a cluster that loses quorum part way through leaves the earlier chunks applied. Re-running finishes the job rather than duplicating it.
• Columns are matched by name, not position. The file's column order does not matter, and a nullable column may be left out entirely. A header naming a column the table does not have is an error, not something ignored — a mistyped column name means the data is not going where you think, and a "successful" import that quietly dropped a field is the worst outcome available.
• An export imports straight back. The BOM, the RFC 3339 timestamps and the ' on a defused formula are all read back as what they were, so export → edit in a spreadsheet → import round-trips. The one place it guesses: a text cell you genuinely began with ' followed by =, +, - or @ loses the quote, because there is no telling it apart from the one the export added. If a value must survive a round trip exactly, use row export's JSON lines.
• An empty cell imports as null, matching what the export cannot distinguish.
• A row with the wrong number of fields is refused rather than padded — it is nearly always a quoting mistake, and filling the gap with nulls would import data that is silently wrong.

The upload is bounded by the node's --max-body-bytes (2 MiB by default); raise it if you import large files. The web UI's proxy allows 16 MiB so that the node's limit, not the proxy's, is the one you hit and the one that tells you about it.

What is not built yet

OR in filters, joins, composite primary keys and unique indexes were all absent from the first release of data tables; all four exist now, described above. What is still missing is narrower, and absent for the reason those four originally were: melange will not offer a query whose cost it cannot state.

• A join that can match many rows (1:N). Every join today matches at most one, which is what makes "one lookup per row returned" true. Lifting that changes the cost model rather than extending it: the page becomes ragged and the per-row cost stops being one.
• Filtering on a joined column, chained joins, and joining to a composite key — see Joining another table for why each is refused.
• A cost-based query planner. When more than one filter could be answered from an index, melange takes the first, not the cheapest — it keeps no per-value statistics, so a cost model would be a guess in a plan's clothing. Overlapping index ranges within one OR are deduplicated only when identical, never merged; two indexable filters are never intersected (that means materialising and intersecting row-key sets, which is unbounded memory for a bounded answer); and an OR whose candidate set exceeds the scan budget gives up on the index and scans. All of those are slower rather than wrong, which is the direction this chooses on purpose.
• contains on text. prefix is offered because it is one range scan; a substring search is not, and would silently always scan.

Local projects

Everything above happens inside a running job on a node. That leaves a gap for the person writing the script: the libraries their script imports live in the registry, reachable only from a job, so authoring meant copying files around by hand and finding out whether it worked by submitting it.

melange-cli project init opens a project: a folder with a manifest, a lockfile, and the libraries it depends on vendored into modules/ — laid out byte-for-byte the way a node lays them out for a running job. melange-cli project run then executes a script here, under the same script-facing contract a node provides, so what works locally works when you submit it.

Everything local lives under one subcommand: melange-cli project <verb>. Only add, install, update, outdated and push talk to a registry; the rest need no network at all.

mkdir acme-automation && cd acme-automation
melange-cli project init          # melange.toml, melange.lock, scripts/, .melange/, .gitignore
melange-cli project script new backup            # scripts/backup.ps1 + a [scripts.backup] entry
melange-cli project add Utils --script backup    # vendor Utils, and declare it there

melange-cli project run backup    # runs here — no server involved
melange-cli project env backup    # what the run will set, and where each secret comes from

melange-cli project push          # upload every [scripts] entry to the registry

It is vendoring, not packaging

modules/ is committed to git, and melange.lock pins each library by content hash. That is the opposite of what a package manager usually does, and the reason is specific to melange: a library has no version. Its name is its identity and PUT /v1/libraries/{name} replaces the content in place, so GET /v1/libraries/{name} can only ever return today's copy. A lockfile here can therefore detect that the registry has moved but can never restore what it pinned. Gitignore the vendored copy and every fresh clone becomes a coin-flip against whoever last ran melange-cli library update.

So the repository is the thing that reproduces, the registry is the thing that publishes, and the lockfile is the record of when the two last agreed. Which makes two commands mean something precise:

melange-cli project install makes the disk match the lock. melange-cli project update makes the lock match the registry. melange-cli project add is update for a name the manifest did not have.

melange-cli project install refuses rather than quietly re-pinning when the registry's copy is not the one the lock names — that detection is the whole of what a versionless lockfile can offer, and swallowing it would make the file decorative:

$ melange-cli project install
Error: the registry's Utils is not the copy melange.lock pins
  locked   sha256 45d8ded616a3  updated 2026-07-31 12:11:39
  registry sha256 f3d9649bdd1a  updated 2026-07-31 12:14:26
  A melange library has no version: the registry holds one copy and it has changed.
  `melange-cli project update Utils` accepts the new one and re-pins.

A vendored file you have edited by hand is reported and left alone — editing one to test a fix is a legitimate thing to do, and melange-cli project run warns about it rather than refusing. melange-cli project install --force restores the pinned copy.

Starting from the registry: melange-cli project clone

The other direction. Name a project that is already registered and you get a working directory back — the manifest, every script's file, and the libraries they import vendored into modules/:

$ melange-cli project clone ops
wrote ops\melange.toml
wrote ops\melange.lock
created ops\scripts
declared nightly -> scripts/nightly.ps1
declared reindex -> scripts/reindex.sh
installed Utils (added to melange.lock)

Each script is written out with the runtime, libraries, secrets, arguments and limits it was registered with, so the clone runs immediately and push reports it unchanged. The registry it came from is recorded as [project] server, so the install and push that follow go back to the same place rather than to whatever your last melange-cli login was.

Two details are worth knowing. A script's registered name becomes the [scripts.<name>] key byte for byte, because that key is what push matches on — while the file name is sanitised separately, since a registered name is free-form and Nightly Backup / v2 is legal. The two need not match; path says which file the entry means. And cloning is how you start: a target directory that already holds files, or sits inside another project, is refused rather than merged into.

Uploading: melange-cli project push

push sends every [scripts] entry to the script registry, assembling each one into the body of a POST/PUT /v1/scripts — the path's content, plus runtime, libraries, secrets, args, description, timeout_secs, kill_grace_secs, cores, memory_mb, instrument and nodes. --dry-run works the whole thing out and writes nothing.

$ melange-cli project push
create    backup   scripts/backup.ps1 -> scr-18c1fd39e6b794ac
replace   reindex  scr-18c1fd39e6b794bd
adopt     legacy   scr-18c1fd39e6b7941f
unchanged report

A script is matched by its [scripts.<name>] key, within this project. A registered script's identity is (project, name): push stamps [project] name onto everything it uploads, and resolves each key against this project's scripts alone. Exactly one of that name in the project is replaced in place, keeping its id and therefore its schedules; two or more is an error naming both, because there is nothing to choose between them and either guess is destructive.

That scoping is what stops two projects colliding. Both may declare [scripts.nightly], and each gets its own registry entry; the cluster refuses a second nightly inside one project with a 409 naming the one already there.

A script with no project of that name is adopted, not duplicated. Scripts registered by melange script create, or by any push predating projects, belong to no project — and if exactly one of them carries the name, push claims it in place, keeping its id so every schedule attached to it survives. Two or more is the same refusal as above. Adoption does not compare bodies: the push most likely to be adopting is the one uploading an edited script, and requiring a match would register a second entry and strand the schedules on the old one.

Nothing is written down to make any of this cheaper — an id belongs to one cluster, so a project pushed to both staging and production could not pin one, and a per-developer note would make a colleague's first push duplicate everything.

unchanged is not cosmetic: a re-push of an untouched script makes no request at all, rather than a replicated log entry, a re-analysis and a bumped updated_at that would make melange-cli script list useless as a record of what actually changed.

Three more rules. Libraries are not uploaded — modules/ is vendored from the registry and install/update are how it moves. Nothing is ever deleted; a registered script this manifest no longer names is left alone, and melange-cli script rm is how you say otherwise. And a script that does not declare libraries is refused rather than pushed: run stages every vendored library of that runtime and warns, but a registered script stores its list, and every job it produces from then on stages exactly that.

A push stops at the first failure rather than repeating one error per script, and says what landed, what failed and what it did not attempt. Re-running converges — identity is a name, so what already landed comes back as unchanged.

The manifest

[project]
# Also the project a pushed script belongs to. A script's identity in the registry is
# (project, name), so this is what keeps two projects' `[scripts.backup]` entries apart.
# Letters, digits, `_` and `-`, starting with a letter or digit.
name = "acme-automation"
# The registry this project vendors from. Recorded here so everyone working on the
# project pulls from the same place, whatever they last logged into. `--server` still
# overrides it.
server = "https://melange.example:8080"

libraries = ["Utils"]

# The libraries every script here imports, on top of whatever each one declares — for the
# one or two that are genuinely universal. A name here must be in `libraries` above too;
# `melange-cli project add <Name> --global` puts it in both.
imports = []

[scripts.backup]
path = "scripts/backup.ps1"
libraries = ["Utils"]           # exactly what gets staged, plus `imports`
secrets = ["DB_PASSWORD"]       # names, never values
args = ['-Path', 'C:\data']     # defaults, used when the command line gives none
runtime = "powershell"          # optional: inferred from the extension, then a `#!` line

# Sent by `melange-cli project push`; a local run does not reproduce them.
description = "Nightly backup"
timeout_secs = 3600
kill_grace_secs = 30
cores = 2
memory_mb = 4096
instrument = false
nodes = []

A mistyped key is an error, not a silently ignored line — a project that looks fine with a dependency you believe you declared and do not have is the worst outcome available here.

path, libraries, secrets, args and runtime are what melange-cli project run uses; the rest describe a submission, and melange-cli project push is what sends them — an entry becomes the body of a POST/PUT /v1/scripts verbatim. They were declared before that command existed, for one reason: because a mistyped key is an error, a manifest carrying a key this CLI has not heard of would be rejected, so a field that was not accepted then could never have been added later without breaking every older client. That is still why a field belongs here the moment it is conceivable rather than when it is first read.

That cuts both ways, and it is worth stating rather than discovering: a manifest using [project] imports is rejected by an older melange-cli, with an unknown-key error. It is the price of a mistyped key being an error, and the manifest is a file versioned with the repository that holds it.

[scripts.*] libraries is not bookkeeping for later: it is exactly what a run stages. A node materializes only what a job declared, so a local run that staged everything in modules/ would let Import-Module Utils succeed on your machine and 404 at submission. Omit the key and every vendored library of that runtime is staged, which is convenient and proves less — melange-cli project run says so when it does it. libraries = [] means none.

[project] imports is the same list one level up: the libraries every script gets, on top of what it declares, so the one or two that are genuinely universal do not have to be named again in every entry. Two rules make it predictable. It does not count as the script declaring its libraries — a script with no libraries key still stages everything and is still told so, because a project-wide list says nothing about that script in particular. And libraries = [] opts out of it as well: an empty list means nothing is staged, which would not be true if a global import crept past it. melange-cli project run says so when a script has opted out.

melange-cli project run warns when a script imports a vendored library the manifest does not declare, and deliberately does not attach it for you, which is the opposite of what melange-cli submit does:

$ melange-cli project run job
note: the script appears to import Acme-Logging, which this project has vendored but
scripts.job does not declare — `melange-cli project add Acme-Logging --script job` adds it
there, or the run will fail to resolve it

The two differ because the artifacts differ. A submission's libraries is a list nobody will read again, so guessing it right is pure gain. The manifest is a file you own and edit, whose whole job is to say what a script gets — so a run that silently staged more than it declared would be the "proves less" problem in a new place.

Secrets, locally

melange-cli project run resolves each name a script declares from the process environment first, then from .melange/secrets.json — which is gitignored, and is the same JSON object of names to values a node reads with --secrets-file, so the file can be handed to a node verbatim. A name that resolves nowhere refuses the run before the interpreter starts, exactly as a node fails a job whose host does not hold a secret it named.

Resolving a secret locally proves your script can use it. It proves nothing about whether any node holds it — that answer is the cluster's own, at submission. melange-cli project env names what an operator has to be asked for, and never prints a value.

On Windows melange sets no ACL on .melange/secrets.json; it is protected by the inherited permissions of the directory and nothing else, exactly like the token melange-cli login saves. melange-cli project init says so at the moment it creates the directory.

What a local run reproduces

The whole script-facing contract, which is what makes the run worth anything:

• the working directory is a staging job directory — .melange/run/<script>/ — so a script writing .\out.csv behaves the way it will on a node, where the job directory is scratch and is deleted afterwards. A successful run's directory is deleted (--keep keeps it); a failed one is always kept, and its path printed.
• the script is staged as job.ps1 / job.sh, with the UTF-8 BOM for PowerShell and without one for bash, so $PSCommandPath and $MyInvocation say what they will say there
• declared libraries are materialized into modules/, $PSModulePath is prepended to (never replaced), and MELANGE_LIB_DIR is set only when the script declared something
• MELANGE_JOB_DIR, MELANGE_CANCEL_FILE and NO_COLOR=1 are set, and secrets are set first — so a secret named MELANGE_CANCEL_FILE cannot take over the job's control channel, here any more than there
• arguments are passed as separate values, so spaces, quotes and backslashes need no escaping
• stdin is /dev/null. This surprises people: Read-Host works when you run the file by hand and does nothing here. That is the point — a node gives a script no stdin, so granting it locally would let an interactive script pass local testing and hang in production.
• --timeout creates the cancel file, waits --kill-grace, then kills — the node's cancellation path, and the reliable way to exercise a script's MELANGE_CANCEL_FILE handling. Ctrl-C does the same thing (once to ask, twice to kill), but the terminal delivers the interrupt straight to the child as well, so --timeout is the faithful test.
• the exit code is the script's own, propagated verbatim

What it does not reproduce, and why

Stated rather than papered over — every one of these fails in the safe direction, where a node is stricter than your machine:

• The sandbox (--sandbox, and everything under it). Sandboxing is node-level operator policy that no job can ask for or refuse; a developer machine pretending to be one would be a category error. A script that runs here may be refused more by a sandboxed node.
• Output caps. A node pipes and caps output because it is replicated state; a local run inherits stdout and stderr, so you see the script's own stream splitting, unbuffered, and --max-output-bytes truncation is not exercised.
• The Windows console code page. A node sets it to UTF-8 before every spawn because PowerShell encodes into it lossily; the CLI does not, because that is a process-global mutation of your interactive console. On a legacy code page a local run mangles non-ASCII output a node would have preserved — chcp 65001 first.
• Tree kill. Killing the script does not kill its grandchildren.
• At-least-once. A local run runs once. A script that is not idempotent looks fine here and can still run twice in a cluster.
• Bash on Windows, which is refused outright: bash there resolves to the WSL launcher, a different filesystem and a different user, where the staging directory does not exist and MELANGE_CANCEL_FILE could never appear. Nodes do not advertise bash on Windows for the same reason. Submit it to a Linux node.
• Registration-time analysis and progress instrumentation, which have no REST route. instrument = true is warned about rather than silently ignored.

run means here; submit means the cluster

One rule, no exceptions, and it is why the registry's command is melange-cli script submit rather than melange-cli script run:

script run still works as an alias, so nothing that already types it breaks.

The CLI also catches the mix-up by the shape of the argument, because the two commands take visibly different things — a registry id is scr-<hex>, a project script is a name or a path:

$ melange-cli script submit scripts/backup.ps1
Error: scripts/backup.ps1 is a path, not a registry script id.
  `melange-cli script submit` sends a *registered* script to the cluster and takes its id
  (`melange-cli script list` shows them). To run a script on this machine instead, use
  `melange-cli project run`; to send this file to the cluster as a one-off, `melange submit`.

$ melange-cli project run scr-18c1fd39e6b794ac
Error: scr-18c1fd39e6b794ac is a registry script id, and `melange-cli project run` runs a
script in this project on this machine.
  `melange-cli script submit scr-18c1fd39e6b794ac` sends it to the cluster; `melange-cli script
  get scr-18c1fd39e6b794ac` shows what it would run.

The first of those is checked before the request, and that asymmetry is deliberate: the two mistakes are not equally bad. Running something locally when you meant to submit it happens on your own machine with you sitting in front of it; submitting something when you meant to test it locally runs it on the cluster. Only the second is worth intercepting, and it is intercepted on shape alone — anything that could be a real id still goes to the server, which remains the authority on whether it exists.

The REST API

REST is the only surface clients touch. Everything is JSON.

POST /v1/jobs — submit a script

{
  "script": "param([string]$Path)\nGet-ChildItem $Path | Measure-Object",
  "runtime": "powershell",
  "name": "count-files",
  "args": ["-Path", "C:\\a b\\data"],
  "secrets": ["DB_PASSWORD"],
  "timeout_secs": 300,
  "kill_grace_secs": 10
}

name is an optional label, echoed back on the job. Returns 202 Accepted:

{ "job_id": "job-18c17394ebcae388", "job_ids": ["job-18c17394ebcae388"] }

202, not 200, is the contract. It means the job is committed to the cluster, not that it has run. Consensus has happened; execution has not. Poll for the result.

job_ids is every job the request created, and job_id is the first of them. They differ only when the request fanned out — see target_nodes below — but a client that reads job_ids alone never has to care which case it is in.

runtime is powershell or bash, defaulting to powershell. The job will only be placed on a node that has it. If no node in the cluster does, the submission is refused with a 400 rather than accepted and left to fail somewhere:

$ curl -X POST localhost:8080/v1/jobs -H 'content-type: application/json' \
    -d '{"script":"echo hi","runtime":"bash"}'
{ "error": "no node in the cluster can run bash scripts" }

400, because no amount of retrying puts bash on a Windows box. If the nodes that can run it are merely unreachable, that is a 503 instead — the cluster is configured to run this job, it just cannot right now, and retrying is exactly right. (Same distinction, and the same code, as secrets.)

args are handed to the script as separate arguments, so a script with a param() block — or one reading $1 — gets exactly what you sent. They are separate values, not a command line to be re-parsed, so nothing in them needs quoting or escaping — spaces, backslashes and quotes all survive intact, in either runtime.

secrets names the secrets the script needs; the values are resolved on the node that runs it. See Secrets.

libraries names the shared libraries the script imports (see Shared libraries). Each must be a registered library; the node running the job materializes it so Import-Module <Name> (PowerShell) or source (bash) resolves. A submission naming a library nothing holds is refused with a 404 — unlike a secret, a library needs no particular node, so there is nothing transient to wait for.

timeout_secs and kill_grace_secs override the executing node's --job-timeout-secs and --kill-grace-secs for this job alone. Omit them to inherit the node's; pass 0 for none — no time limit, or no grace. Both are stored on the job, so a job reassigned off a dead node is still judged by the limits it was given rather than its new host's defaults.

cores and memory_mb (both optional) tell the scheduler what this job is expected to occupy. They are scheduling weights, not limits: the job counts as more of its node's capacity — a core is four slots, 256 MB is one, whichever prices higher decides — so a heavy job stops looking like a one-liner to placement, but the script itself is not restricted, and a job bigger than any node in the cluster still queues and runs. Omit them and the job costs one slot, exactly as every job did before hints existed. Echoed back on the job.

target_nodes (optional) forces the job onto specific nodes by id, instead of letting the leader place it by load — see Choosing the node yourself. Naming several is a fan-out: one job per node, so "target_nodes": [2, 5] creates two jobs and returns two ids.

$ curl -X POST localhost:8080/v1/jobs -H 'content-type: application/json' \
    -d '{"script":"hostname","target_nodes":[2,5]}'
{ "job_id": "job-18c17394ebcae388",
  "job_ids": ["job-18c17394ebcae388", "job-18c17394ebcae389"] }

A node that is not a member is a 404; one that has gone quiet is a 503; one that is alive but has no interpreter for the runtime, or does not hold a secret the job names, is a 400 that says which node and what it is missing. Any of them refuses the whole request. Jobs placed this way come back with "pinned": true and are never reassigned.

GET /v1/jobs/{id} — read a job

{
  "id": "job-18c17394ebcae388",
  "script": "Get-Process | Measure-Object",
  "runtime": "powershell",
  "name": "process-count",
  "state": "succeeded",
  "assigned_to": 3,
  "started_at": 1783918385921,
  "finished_at": 1783918386856,
  "duration_ms": 935,
  "output": {
    "exit_code": 0,
    "stdout": "Count : 214\r\n",
    "stderr": ""
  }
}

assigned_to is the node that owns execution — and runtime is a large part of why it is that node. state is one of:

failed and timed_out are deliberately not the same thing. Both mean "this job did not work", but one is a script that ran and reported a problem, and the other is a script we took the decision away from. An operator who cannot tell them apart cannot tell a broken script from a timeout set too tight, so the state machine records which of the two happened rather than inferring it from an exit code that cannot express the difference.

output is present once the owning node has reported back. A job that was killed part-way through — cancelled or timed out — still has one: whatever the script managed to print before it died is the only evidence of how far it got, so it is kept rather than discarded.

A job that was killed has exit_code: -1: it never chose an exit code, because we took the decision away from it. One that stopped itself when asked keeps its own.

Reads are served from the local state machine of whichever node you ask, followers included. No leader round-trip.

Did this job ever actually run?

started_at and finished_at are milliseconds since the Unix epoch, and the pair answers a question that would otherwise have no answer.

started_at is recorded when the owning node picks the job up, which is before the shell is spawned — so it marks the start of the run, not of the script. That is what makes it useful:

Without them, all three are the same record: cancelled, no output. The CLI says which one it was:

$ melange-cli get job-18c1c05a979f1550
state: cancelled
ran:   never started
output: (none recorded)

Timestamps are stamped by the node that proposes the entry and travel inside it, rather than being read from the clock as each node applies — every node applies every entry, so a clock read there would give a different answer on each and the replicated state would diverge. The cost: a job's start and its end can be stamped by two different machines (a cancellation is stamped by whichever node was asked), and two clocks need not agree.

Which is why duration_ms is measured, not subtracted. The node running the script times it with a monotonic clock and reports the number; it cannot run backwards, and nothing about how the cluster's clocks are set can move it. The timestamps say when. This says how long, and it is the one to count.

Truncated output

Output is capped at --max-output-bytes per stream, and the cap is spent half on the start of the stream and half on the end — when a script floods its output the part you want is the last thing it printed, and a plain prefix is exactly what throws that away. The gap is marked where the gap is:

START OF LOG
noisenoisenoisenoisenoise...
melange: output truncated — 305614 bytes omitted here
...noisenoisenoisenoise
Write-Error: THE ERROR THAT ACTUALLY MATTERED

Past the cap, output is read and thrown away rather than left unread — a reader that stopped at the limit would block the script forever on its next write, turning a script that prints too much into one that hangs.

When a job is stuck

A job can be owed and yet going nowhere: stranded on a node the cluster cannot reach, with nowhere legal to move it — in practice, when the node that died was the only one holding the job's secret, or the only one that could run its runtime. melange will not move it somewhere it cannot run, because a job that is merely stuck can still succeed when its node returns, whereas one moved somewhere it will fail has failed for good.

So it says so, on the job:

$ melange-cli get job-18c1c65b05a061ac
state: running
stuck: no reachable node can supply this job's secrets: DB_PASSWORD — node 3 holds
       them but has not answered for 30s, so it may be partitioned rather than gone,
       and may still be running this job

blocked is not a state. The job has not failed — nothing ran it, so nothing may call it finished — and it will run when its node comes back or another node is given the secret. The reason is retracted the moment it stops being true.

The wording is careful, because the two things melange can know here are not equally certain. That no node holds a secret — or that none can run bash — is a fact, read from the replicated membership. That the node that could has died is a suspicion: it looks exactly like a partition, and a partitioned node may be running the script perfectly well and about to report success. So the reason names the node, says how long it has been quiet, and admits what it cannot know. A client told flatly "no node can run this" might reasonably resubmit, and would then have run the script twice on purpose.

GET /v1/jobs/{id}/output — tail a job's output

Read a job's output while it runs, rather than waiting for it to finish. Poll with the byte offsets you already have; only newer bytes come back.

curl 'localhost:8080/v1/jobs/job-18c1.../output?stdout_offset=0&stderr_offset=0'
# -> 200 {"stdout_offset":0,"stdout":"...","stdout_total":124,
#         "stderr_offset":0,"stderr":"","stderr_total":0,"done":false}

Advance stdout_offset/stderr_offset to the *_total you were last given and poll again; nothing is re-sent. When done is true, stop — the strings are then the job's final output and exit_code is set. From the CLI:

melange-cli logs job-18c1...            # print what it has produced so far
melange-cli logs -f job-18c1...         # follow until it finishes

Three things are worth knowing, because they fall out of one fact — partial output is not replicated. A job's final output rides the Raft log and any node can serve it; but a running script's stdout would flood every node's log, so it is kept only on the node actually running the job.

• While it runs, only a bounded window (~64 KiB per stream) is kept. Enough to tail; not the whole stream. Poll fast enough and you see everything — fall too far behind and a returned *_offset comes back higher than you asked for, which is the endpoint telling you some bytes scrolled out of the window before you read them. The complete, capped output is always available in full once the job is done.
• Ask any node, but a running job's output lives on one. If you ask a node that is not running the job, it answers with a 307 to the node that is (its REST address, from the membership). melange-cli logs follows it with your token re-attached, so a cross-node tail works under --auth. A raw HTTP client is on its own here: most (curl included) drop Authorization across a host change, so under auth use curl --location-trusted, or just aim at the owning node directly.
• Once terminal, it is served from replicated state — from any node, with the exit code, exactly like GET /v1/jobs/{id} but shaped for streaming.

404 if there is no such job.

POST /v1/jobs/{id}/cancel — stop a job

curl -X POST localhost:8080/v1/jobs/job-18c17394ebcae388/cancel
# -> 202 Accepted  {"id":"...","state":"cancelled",...}

Returns the job as the cluster now sees it. 404 if there is no such job, 409 if it has already finished — a client that was too late should be told so, not handed a 202 it will read as "stopped in time".

Cancellation is a replicated state change, not a message to whoever holds the process. The CancelJob entry commits through Raft, and the node running the script kills it when the entry reaches it. Two things follow: you can ask any node to cancel, not just the one doing the work; and the 202 means the cluster has agreed the job is over, not that the process is already dead.

It has to work this way. Killing the process without recording it would leave the cluster believing the job was still owed — and the leader would hand it to someone else to run again.

The cancelled job keeps whatever its script printed before it was killed. That partial output is the only record of how far it got, so it is worth having.

Note that it is not DELETE /v1/jobs/{id}: a cancelled job is not removed. It is still there, with its script, its owner, and its history.

Cancelling on a node you cannot reach

A cancelled job can still be running. The cancelled state means the cluster has agreed the job is over. Delivering that decision to the process requires reaching the node holding it — and if that node is partitioned away from the leader, the entry does not reach it until the partition heals. Until then the node goes on happily running a script the rest of the cluster considers dead.

This is the same window as the at-least-once one, for the same reason: from the leader's side, a partitioned node and a dead one are indistinguishable. This is less a bug than the shape of the problem: stopping a process on a machine you cannot talk to is not something consensus can do for you.

What melange guarantees even so is the part that matters: the job is terminal in replicated state, so it is never reassigned and never re-run. The blast radius of a partition is one script finishing when you asked it not to, not one script being started somewhere new because you asked it not to.

GET /v1/jobs — list jobs

Newest first, paged. Served from the local state machine, so any node answers it — the job table is replicated, and a follower's answer is the leader's answer.

curl 'localhost:8080/v1/jobs?state=failed&node=2&limit=20'

{
  "jobs": [
    {
      "id": "job-18c1fd39e6b794ac",
      "state": "failed",
      "name": "doomed",
      "assigned_to": 2,
      "submitted_at": 1783034952000,
      "started_at": 1783034952100,
      "finished_at": 1783034952364,
      "duration_ms": 264,
      "exit_code": 7
    }
  ],
  "cursor": "job-18c1fd39e6b794ac"
}

A listing does not carry scripts or output. A script can be 2 MiB and its captured output 20 MiB, so a page of fifty full job records is a response that could exceed a gigabyte — built in the node's memory first, then in yours. Listing is for finding a job; GET /v1/jobs/{id} is for reading one. The exit code is there, because it is four bytes and it is what people scan a list for.

A short page is not the end of the list. With a filter, a page can come back short — or empty — while there is still table behind it, because the node bounds how long it will scan on one request rather than reading a million rows to answer a query that matches nothing. The cursor is the only thing that says whether there is more: page until it is absent, not until a page looks small.

An unparseable filter is a 400, not a filter that is quietly ignored — dropping it would show you more jobs than you asked about, which is the one wrong answer a listing can give that looks exactly like a right one.

Script registry endpoints

The registry lets a script be saved once and run at whim. See The script registry for what it is and why it is shaped this way; the endpoints are:

POST /v1/scripts
{
  "project": "backups",
  "name": "nightly-backup",
  "description": "vacuums the app database",
  "script": "psql \"$DB_URL\" -c 'VACUUM'",
  "runtime": "bash",
  "secrets": ["DB_URL"],
  "timeout_secs": 1800
}

The body is a submission plus a name (required — the label the script is browsed by, and the label its jobs are given), an optional description, and an optional project. A registry write is an ordinary replicated write: any node accepts it and forwards to the leader, and it is refused with a 503 if the cluster cannot commit — exactly like a submission. Reads are served from local state on any node.

(project, name) is unique: a create — or a PUT renaming a script onto a pair another script in the same project holds — is a 409 naming the incumbent. Scripts with no project are exempt. GET /v1/scripts?project=<name> narrows a listing to one project, and ?project= (present and empty) to the ones belonging to none.

Registering a script validates almost nothing. The name and script must be non-empty and a project, if given, must be letters, digits, _ and - starting with a letter or digit (400 otherwise), but a script may name a secret no node holds yet — a node can be given it before the script is ever run. Placement is still checked at run time, where a script naming an unrunnable runtime or an absent secret fails or is refused exactly as a direct submission would be. target_nodes follows the same rule: a script may be registered against a node that has not joined yet.

A script's target_nodes governs every run it produces, scheduled fires included — which is how "this backup runs on node 3" is expressed once rather than on every call. A RunScript body may carry its own target_nodes to override it for one run; that override applies to every stage of a then pipeline, not just the first.

It does, however, get analyzed. The node that accepts a POST or PUT asks the script's own interpreter what it can say without running it — PowerShell's parser builds the AST and reports the param() block (names, types, default expressions, which are Mandatory) and any syntax errors; bash gets bash -n, a syntax check only — and the result is stored on the script as analysis:

"analysis": {
  "syntax_ok": true,
  "params": [
    { "name": "Target", "type_name": "string", "default": "'prod'", "mandatory": false },
    { "name": "Count", "type_name": "int32", "mandatory": true }
  ]
}

Three things to know. Nothing executes — defaults come back as source text ((Get-Date) stays the string (Get-Date)), which is why this is safe to do on whatever node the request landed on. It never gates: a script the parser rejects is saved anyway with syntax_ok: false and the errors on record, because the parser that analyzed it is not necessarily the one that will run it. And it is best-effort: analysis is absent when the accepting node could not look — it has no interpreter for the runtime (a bash script registered through a Windows node), or the parse timed out — which means nothing was checked, not that anything is wrong. An update re-analyzes; ad-hoc POST /v1/jobs submissions are never analyzed.

Running a script produces an ordinary job. POST /v1/scripts/{id}/run resolves the saved definition to a job and submits it, so the reply is a 202 with a job id — poll GET /v1/jobs/{id} for it, the same as any submission. The produced job is placed, replicated, and executed with no special path; the registry is a producer of ordinary jobs, not a second kind of execution.

A run may carry its own arguments. The body is optional: send nothing (a bare curl -X POST) and the script runs with its saved arguments, exactly as before. Send { "args": ["-Target", "staging"] } and those replace the saved ones for this run only — the stored definition is untouched, the replacement is wholesale (positional lists do not merge), and the produced job records what the script was actually given. { "args": [] } is an explicit "no arguments": a PowerShell script then runs on its param() defaults even if the saved definition has arguments. A body that is present but unreadable is a 400, never a silent run with the saved arguments.

PUT replaces a script wholesale — whatever you send becomes the script — preserving only its original created_at. DELETE removes the definition; jobs it has already produced are untouched (they are ordinary jobs and outlive it), and a run or a read of a deleted script is a 404.

Listing omits the script body, like a job listing omits output, and pages the same way (?limit=, ?cursor=, newest first — a short page is not the end, only an absent cursor is).

Library registry endpoints

Reusable, importable code a job pulls in by name. See Shared libraries for what it is and why it is shaped this way; the endpoints mirror the script registry, but a library is keyed by its name (its identity — what a script imports it by), not a synthetic id:

POST /v1/libraries
{
  "name": "Utils",
  "description": "shared helpers",
  "runtime": "powershell",
  "content": "function Get-Widget { 'a widget' }\nExport-ModuleMember -Function Get-Widget"
}

The name must be unique and filename-safe — ASCII letters, digits, _ and -, starting with a letter or digit (400 otherwise, 409 if taken) — because it becomes a file and a module name on the node that runs a job. The content must be non-empty. Like a script, a library is analyzed at save (syntax verdict and, for PowerShell, its param() block) and the analysis is advisory, never gating. PUT replaces it wholesale (the name is the URL's and does not change); DELETE removes it, and a job that still imports it then fails to resolve it at run time — the same as a job naming a deleted secret. Listing omits the content and pages alphabetically (a catalogue is browsed by the name you import, not by when it was added).

Cluster secrets

Reads need only authentication; writes are admin. There is no cursor — a cluster caps how many secrets it holds, so a list is the lot.

No response carries a value, and no request may. SetSecret has exactly one field, and it is a value already sealed to the cluster:

PUT /v1/secrets/DB_PASSWORD
{
  "sealed": {
    "v": 1,
    "key_id": "59961b78f8640c8a",
    "eph_pub": "6DLKHb5nuR+Gag1hg2ZndWJHw+8L/YUyOhTymzDkJxY=",
    "ciphertext": "X8s79c5UAf/hnMx45kT3ybQJY+d9C48="
  }
}

There is nowhere in that body to put a plaintext value, which is what makes "never accept plaintext" a property of the type rather than a check that could be forgotten — including for curl, which can therefore list and delete secrets but not set them. Use melange secret set or the web UI, both of which seal before sending.

The public key to seal against comes from GET /v1/cluster, as secret_key ({ "key_id": …, "public": … }); it is absent when the cluster's nodes have no --secrets-key. Pin it on first sight and refuse to seal if it changes — a changed identity is either a deliberate rotation or somebody in the path, and a client cannot tell which.

A name must be ^[A-Z_][A-Z0-9_]{0,63}$ and must not begin MELANGE_ (400 otherwise). Upper case is not fussiness: a secret's name becomes an environment variable, Windows folds variable case and Unix does not, so db_password would be one variable on one node and a different one on another — in a cluster whose whole point is that a job can land on either.

A SecretView carries nodes: which members can actually supply it. Empty means no job naming it can be placed anywhere, which is briefly true of one just created and lastingly true of a node running on the wrong key.

A job or a registered script imports libraries by listing their names in libraries (alongside secrets). Because a library's content is replicated to every node, no placement filter applies — a submission that names a library nothing holds is refused outright (404 "no such library"), and the executing node lays each one out in the job directory: Import-Module <Name> (PowerShell, via $PSModulePath) or source "$MELANGE_LIB_DIR/<name>.sh" (bash).

Schedule endpoints

Cron schedules that fire a registered script automatically. See Schedules: running a script on a cron for the behaviour.

POST /v1/schedules
{ "script_ids": ["scr-18c1fd39e6b794ac"], "cron": "0 3 * * *", "enabled": true }

A 400 if the cron will not parse, a 404 if the named script is not registered. ScheduleView carries last_fired_at (the scheduled tick it last fired for, absent until it has) and a computed next_fire_at (derived from the cron, absent if disabled or if the expression never matches again). There is no cursor — schedules number in the dozens, so a list is the lot. PUT replaces the cron, the enabled flag and target_nodes, preserving the watermark; enable/disable flip only the flag.

A schedule may also carry target_nodes, which overrides wherever each stage's script normally runs — wholesale, for every stage, because a rule that applied to some stages and not others would be one nobody could predict. Unlike script_ids, it is editable by PUT, empty included: sending [] is how a pinned schedule goes back to being placed by load. A fire that cannot honour its targeting is skipped that tick and retried on the next rather than failing, and a fanned-out fire advances the watermark exactly once for the whole fan-out.

GET /v1/cluster — membership and load

{
  "leader_id": 1,
  "timezone": "America/New_York",
  "nodes": [
    {
      "id": 1,
      "addr": "127.0.0.1:50051",
      "rest_addr": "127.0.0.1:8080",
      "cores": 16,
      "memory_mb": 32768,
      "slots": 64,
      "in_flight": 4,
      "load": 4,
      "runtimes": ["powershell"],
      "draining": false
    },
    {
      "id": 2,
      "addr": "127.0.0.1:50052",
      "rest_addr": "127.0.0.1:8081",
      "cores": 8,
      "memory_mb": 16384,
      "slots": 32,
      "in_flight": 2,
      "load": 17,
      "runtimes": ["powershell", "bash"],
      "draining": true
    }
  ]
}

timezone is the zone this node evaluates schedule crons in (its --timezone; UTC by default). Only the leader actually fires schedules, so on a follower it is informational — but it is the zone a cron typed at this node is read in, which is what a client uses to label a cron input.

Two addresses per node, and they are not interchangeable: addr is where peers reach it for Raft, rest_addr is where a client reaches its API. A node reports both about itself; nothing in the cluster ever dials the second one.

in_flight counts the node's unfinished jobs; load is the same jobs weighted by their resource hints, in slots (an unhinted job is 1; a job that declared 2 cores is 8). load / slots is the ratio placement actually minimises, so when the spread looks lopsided against in_flight — node 2 above holds two jobs but seventeen slots' worth of declared weight — load is the explanation.

runtimes is what the node last told the cluster it can run. A node with an empty list runs nothing — it is still a perfectly good voter, it just never gets any work.

draining means the node is shutting down: it is being sent no new work, but it is still a voter and still finishing the jobs it has (note node 2 above still shows two in flight). See Stopping a node.

POST /v1/cluster/nodes — add a voter

{ "node_id": 2, "addr": "127.0.0.1:50052" }

addr is the joining node's gRPC address. The leader dials it, asks what it is — hardware, secret names, runtimes — adds it as a learner, then promotes it to a voter. Returns 204 No Content.

Must be sent to the leader: unlike a job, a membership change is not forwarded. A follower answers 503 and names the leader.

POST /v1/cluster/nodes/{id}/remove — retire a node

curl -X POST localhost:8080/v1/cluster/nodes/2/remove

Removes the node from the membership outright — not demoted to a learner that would be replicated to forever. Returns 204 No Content. Must be sent to the leader.

Drain the node first. Stop it (SIGTERM drains it, so its running jobs finish), then remove it. The moment a node leaves the membership its unfinished jobs are, by definition, assigned to a node that is not a member — so the leader calls them orphans and hands them to somebody else, while the node itself, if it is still up, carries right on running them. That is not a bug in removal; it is at-least-once doing exactly what it promises. It just means the script runs twice.

So a node that is alive and still holding work is refused with a 409 that says so:

node 3 is still running 1 job(s) — drain it first (SIGTERM, or its --drain-timeout-secs)
so they finish, or they will be reassigned and run a second time. Pass force to remove it
anyway.

?force=true overrides it, for the node that is wedged and never will drain — duplicate execution is then the price of getting rid of it, and the operator is the one choosing to pay it. A node that is unreachable is removed without argument: a decommissioning endpoint that only works on healthy machines is no use in the case you most need it.

GET /health — the process is up

200 OK if the process is alive and serving. It says nothing about whether the cluster can do anything, and it is not meant to: this is the liveness check, the one a supervisor uses to decide whether to restart the binary. A check that failed when the cluster was unhealthy would restart every node in a cluster that had merely lost its leader, turning a blip into an outage.

GET /ready — the node can actually take work

{ "ready": true, "node_id": 2, "leader_id": 1 }

200 when a job submitted here would be committed; 503 when it would not, with the reason. This is the one to put behind a load balancer. A node whose cluster has no leader, or whose leader cannot reach a majority, is perfectly healthy by /health and will refuse every job it is given:

{
  "error": "not ready: no leader is elected, so the cluster cannot commit anything"
}

It also reports 503 while the node is draining (see Stopping a node) — so a node on its way out leaves the rotation before it stops answering, rather than by refusing a connection that has already been sent to it.

Errors

Every non-2xx response carries the same body — including the ones melange does not raise itself, like a body that is too large or JSON that will not parse:

{ "error": "no such job: job-nope" }

The 400/503 split is the same one throughout: can this cluster ever run this job, or can it merely not run it right now? A bash job in an all-Windows cluster is the first; a bash job whose Linux nodes are rebooting is the second.

Note what is not here: there is no "wrong node" error. Any node accepts writes and forwards them to the leader internally.

A 503 may carry one extra field, and only a 503 ever does:

{
  "error": "could not reach leader: ...",
  "leader": "http://10.0.0.7:8080"
}

That is the leader's REST address — somewhere the client can actually retry. It appears only when this node knows who the leader is and is not it: with no leader elected, or on the leader itself, there is nowhere better to send you and the field is absent. The CLI prints it; it does not retry automatically, because retrying means running an unknown script twice on a node the operator did not pick.

The CLI

melange-cli is a REST client and nothing more. It has no knowledge of Raft, redb, or how a script is run. Point it at any node with --server.

# Submit from stdin, from a file, or wait for the result
echo 'Write-Output "hi"' | melange-cli submit
melange-cli submit ./nightly-cleanup.ps1 --name nightly
melange-cli submit ./backup.ps1 --wait          # polls with backoff

# bash: inferred from the .sh, from a #! line, or stated outright
melange-cli submit ./rotate-logs.sh --wait
melange-cli submit --runtime bash --wait < script

# a secret by name, and args for the param() block. Shared libraries are read out of the
# script's own imports (--library for one that is auto-loaded; --no-detect-libraries to stop)
melange-cli submit backup.ps1 --secret DB_PASSWORD -- -Server db01

# per-job limits: 0 means none
melange-cli submit --timeout 600 --kill-grace 10 ./slow.ps1

# a heavy job: scheduling weight, not a limit (see "Scheduling")
melange-cli submit --cores 4 --memory-mb 8192 ./etl.ps1

# force where it runs (see "Choosing the node yourself"). Repeating --node fans out:
melange-cli submit ./collect.ps1 --node 3            # node 3, and only node 3
melange-cli submit ./collect.ps1 --node 2 --node 5   # both — two jobs, one each

melange-cli get job-18c17394ebcae388
melange-cli cancel job-18c17394ebcae388
melange-cli cluster status                          # ...including each node's runtimes

# What has the cluster been doing?
melange-cli list
melange-cli list --state failed --since 60      # failures in the last hour
melange-cli list --since 1440 --until 60        # yesterday, but not the last hour
melange-cli list --submitted-by ana              # what one person ran
melange-cli list --node 3 --all                 # every job on node 3, paged through

# The script registry (see "The script registry"). `script` is a nested subcommand.
melange-cli script create --name nightly --secret DB_URL ./backup.sh
melange-cli script list
melange-cli script get scr-18c1fd39e6b794ac
# `submit` sends to the cluster (`run` is a kept alias); `melange-cli project run` is the local one.
melange-cli script submit scr-18c1fd39e6b794ac --wait   # produces, and waits on, a job
melange-cli script submit scr-18c1fd39e6b794ac -- -Target staging   # this run's args only
melange-cli script submit scr-18c1fd39e6b794ac --node 5             # this run's node only
melange-cli script submit scr-nightly --then scr-report --notify       # a one-off pipeline, emailed
melange-cli script create --name collect --node 2 --node 5 ./collect.ps1  # every run
melange-cli script update scr-18c1fd39e6b794ac --name nightly ./backup-v2.sh
melange-cli script rm scr-18c1fd39e6b794ac

# The library registry (see "Shared libraries"). Keyed by name, imported by name.
melange-cli library create Utils ./Utils.psm1 --runtime powershell
melange-cli library list
melange-cli library get Utils
melange-cli library rm Utils

# Cluster secrets (see "Secrets"). The value is sealed here before it is sent, and is
# prompted for rather than taken as an argument.
melange-cli secret set DB_PASSWORD
melange-cli secret set DB_PASSWORD --from-file -    # ...or piped in
melange-cli secret list
melange-cli secret get DB_PASSWORD                  # metadata; there is no way to read a value
melange-cli secret rm DB_PASSWORD

# Data tables (see "Data tables"). `table` is the schema; `row` is the data.
melange-cli table create inventory --column 'asset_tag:text:key' --column 'cores:int'
melange-cli table list
melange-cli table get inventory
melange-cli table aggregate inventory --fn sum --column cores --group-by env
melange-cli table alter inventory --add 'notes:text' --rename 'cores:cpu_count'
melange-cli table index inventory cpu_count          # background build; scans until ready
melange-cli table index inventory serial --unique    # ...and refuses a duplicate value
melange-cli table unindex inventory cpu_count
melange-cli table rm inventory

melange-cli row set inventory asset_tag=A-1042 cores=8
melange-cli row set inventory asset_tag=A-1042 cores=16 --expect-version 1   # 409 if it moved
melange-cli row list inventory --filter cores:ge:8 --all
melange-cli row list inventory --filter 'env:eq:prod|env:eq:staging'   # `|` is OR
melange-cli row list hosts --join owners:owner       # bring the matching owner along
melange-cli row get inventory 1A-1042    # the key `row list`/`row get` print, tag byte and all
melange-cli row rm inventory 1A-1042
melange-cli row export inventory > rows.jsonl        # one JSON object per line...
melange-cli row import inventory rows.jsonl          # ...which imports straight back
melange-cli row export inventory --format csv -o inventory.csv       # for a spreadsheet
melange-cli row export inventory --format csv --filter env:eq:prod   # only what matched

# A local project: vendor the libraries a script imports, and run it here before you
# upload it (see "Local projects"). `run` executes on THIS machine and talks to no server.
melange-cli project init                             # melange.toml, melange.lock, scripts/
melange-cli project clone ops                        # ...or build one from a registered project
melange-cli project script new backup                # a script, and its manifest entry
melange-cli project add Utils                        # vendor + pin into modules/
melange-cli project add Utils --script backup        # ...and declare it in [scripts.backup]
melange-cli project add Utils --global               # ...or in [project] imports, for every script
melange-cli project install                          # what a fresh clone and CI run
melange-cli project outdated                         # exits non-zero if anything is stale
melange-cli project update Utils                     # re-pin to the registry's copy
melange-cli project run backup -- -Path C:\data      # exits with the script's exit code
melange-cli project env backup                       # what a run sets; never a secret's value
melange-cli project push --dry-run                   # what an upload would create or replace
melange-cli project push                             # upload every [scripts] entry

# Cron schedules that fire a registered script automatically (see "Schedules").
melange-cli schedule add scr-18c1fd39e6b794ac --cron "0 3 * * *"
melange-cli schedule add scr-18c1fd39e6b794ac --cron "0 * * * *" --node 2 --node 5
melange-cli schedule list                            # or --script scr-… for one script
melange-cli schedule disable sch-18c2a1b0c9d4e5f6    # pause without deleting
melange-cli schedule enable sch-18c2a1b0c9d4e5f6
melange-cli schedule rm sch-18c2a1b0c9d4e5f6

# Growing and shrinking the cluster (send these to the leader)
melange-cli join 2 127.0.0.1:50052
melange-cli remove-node 2

# This machine's TLS material, for a cluster served with --rest-tls. Local only: run once,
# then no --tls-* flags ever again. See "Securing the REST port".
melange-cli tls install --from ./alice --from ./tls   # the dirs melange-server wrote
melange-cli tls status                               # what is installed, and from where
melange-cli tls remove

--wait polls until the job reaches a terminal state, prints its output, and exits non-zero if the script failed — so it composes with shell && and with CI. A submission that fanned out across several nodes waits for all of them and fails if any one did: "run this on 2 and 5" has not succeeded when node 5's copy did not.

list shows one page (--limit, default 50) and tells you if there is more; --all pages through the rest for you. Times are shown in your local timezone, with the zone named in the column header (e.g. SUBMITTED (NZDT)) — the same instant the web UI shows in the viewer's zone, since the wire format is a timezone-free epoch and the client decides how to render it. The machine's zone is detected from the OS and falls back to UTC if it cannot be resolved.

Shell completions

melange-cli completions bash > /etc/bash_completion.d/melange
melange-cli completions fish > ~/.config/fish/completions/melange.fish
melange-cli completions zsh  > ~/.zfunc/_melange

melange-cli completions powershell | Out-String | Invoke-Expression

Generated from the same argument definitions everything else is parsed from, so a new flag is completable without anyone remembering to update a list.

The web UI

melange-webui serves a browser front end: browse and filter jobs — by state, node, time and submitter — read a job's script and output, including tailing a running job's output live (the same offset-polling the CLI's logs --follow uses), submit new ones, cancel running ones and discard finished ones, browse and manage the script registry — register a script, run one on demand with arguments, a one-off pipeline and notification, edit or delete it, and attach, edit, pause or remove its schedules — manage the library registry in a Libraries tab, read and edit data tables in a Data tab (including changing a table's schema and asking it an aggregate), and see and change the cluster: who leads, what each node is, what each node can run, joining and retiring nodes, and what the cluster calls itself.

The Data tab is the one place in the UI that is edited in place rather than through a form. The grid is built from the table's declared schema, not from whatever keys the first row happened to have, so a column every row leaves null still has a header and still has cells to type into. Each column carries a filter box (type a bare value for an exact match, or ge:8 for anything else), and the strip above the grid says when a count is a bounded estimate rather than a fact. Join another table and its columns arrive with filter boxes of their own, so a joined value narrows the rows exactly as a local one does — those columns come from the joined table's schema, so filtering down to nothing never takes away the box you would use to widen it again. The header — column names and filter boxes both — stays parked at the top of the grid while the rows scroll under it. Double-click a cell to edit it: the write carries the version the row had when it was drawn, so if a job changed that row underneath you the page says so and reloads instead of overwriting it. A timestamp column is drawn as a local time rather than as epoch milliseconds; editing one takes either that or the raw number, and the editor is prefilled with the number because the display is truncated to whole seconds and would round the milliseconds off a row you merely opened. Click a column name to sort by it — ascending, then descending, then back to the table's own key order. A Download CSV button beside "Add row" saves what the filters and the sort currently show.

The card's Schema section folds away what the grid is made of: every column with its type, whether it is the key, and what its index is doing — building… while the leader fills it in, indexed once queries are using it, and why a build stopped if one did. Indexes are declared and dropped there, one button per column, which is also what makes that column sortable.

Opening a job that is still running shows its output under a "● live" marker, updating on the page's refresh tick; once the job finishes, the drawer shows the complete, replicated output. The proxy handles the cross-node case for you (a job running on a node other than the one the page is pointed at), so there is nothing to configure — see GET /v1/jobs/{id}/output.

melange-webui --server http://node1:8080 --server http://node2:8081
#   melange node allowed  id=0 url=http://node1:8080
#   melange node allowed  id=1 url=http://node2:8081
#   melange web ui listening  addr=127.0.0.1:9090

Open http://127.0.0.1:9090. You are asked which node to drive — any node answers for the whole cluster, and the picker probes each one first, so a node that is down says so before you click it rather than after.

It is one binary with no assets to deploy: the page is compiled into it. There is no npm, no bundler, and no build step — cargo build is the build. It links melange-api and nothing else, exactly like the CLI: no Raft, no redb, no execution internals.

Views are deep-linkable, so a job can be bookmarked or pasted to somebody:

http://127.0.0.1:9090/#/0/jobs
http://127.0.0.1:9090/#/0/cluster
http://127.0.0.1:9090/#/0/job/job-18c17394ebcae388

Note what the link names: the position of a node in the allow-list, never its address. Which is the next section.

It proxies, and the allow-list is a security boundary

The browser only ever talks to melange-webui, which forwards to the node server-side. That is not an implementation detail:

• No CORS. The obvious design — serve a static page, let the browser call the node directly — needs CORS headers on melange-server, which is a new client-facing surface on the node itself. This needs nothing from the node at all.
• The nodes need not be reachable from anybody's browser. Only melange-webui has to reach them, so the cluster can stay on a network your users cannot see.

But a proxy that forwards to whatever address the page asks for is an open proxy — anything the process can reach on its network, a browser could now reach through it. So the nodes it will talk to are fixed when it starts (--server, repeatable), and the page selects one by its index in that list. No string from the browser ever becomes a URL, so there is nothing to validate, nothing to normalise, and no trailing-slash or case-folding bug that quietly becomes an SSRF.

$ curl localhost:9090/api/servers/7/cluster
{ "error": "no such server: this UI was not started with one at that position" }

Every route the page can reach is spelled out one by one — there is no blanket passthrough on this port. That includes the admin surface: membership changes and user management are offered, with the node's own RBAC and leader-only rule doing the gating. Removing a node keeps its two-step shape in the UI, because it runs jobs twice if you get it wrong: the first click asks unforced, and only if the node answers 409 does a second, consequence-spelling confirm add ?force=true. Deleting a job's history is the one route deliberately left off this port.

It can also be your cluster's front door (--api-listen)

With that flag the same process serves melange's REST API on a second port, so melange-cli — or anything else that speaks HTTP — can reach a cluster whose own ports are on a network it cannot see. That is the deployment the flag exists for: melange-webui is the only thing you publish, and the nodes stay inside.

melange-webui --api-listen 0.0.0.0:9091 \
  --server https://n1:8080 --server https://n2:8080 --server https://n3:8080
#   melange web ui listening      addr=0.0.0.0:9090
#   melange api gateway listening addr=0.0.0.0:9091

melange-cli --server http://gateway:9091 login admin
echo 'Write-Output "hi"' | melange-cli submit --wait

The CLI needs nothing else — no flag, no mode, no awareness that it is not talking to a node. The paths are the node's own (/v1/…), plus /n/{index}/v1/… to pin one particular configured node, and /ready. /health is answered by the gateway itself and never dials: it is this process's liveness, so a health check cannot restart it because a node blipped.

Four things it does on your behalf:

• It picks a node, and sticks to it. The --server list in order. Sticky on purpose: submit followed by reading the job back has to see its own write, and spreading a client's requests over the cluster would hand that read to a follower that has not applied the entry yet.
• It re-tries at another node only when no connection was made at all. Not on a timeout, not on a failure part-way through. melange's execution is at-least-once, but a submission is not: re-sending a POST /v1/jobs that may already have arrived would run your script a second time. The cost of that promise is that a node which is up but slow stalls rather than being stepped over.
• It rewrites the leader hint. A node names the leader by its own REST address, which is exactly the address you are using this gateway to avoid needing. If the hint names a node the gateway was configured with, it comes back as a URL pointing at the gateway and pinning that node, so the CLI's --retry-leader works through it unchanged; if it names anything else, the hint is dropped rather than sending you somewhere you cannot go.
• It follows the job-output redirect for you, exactly as it does for the browser, so logs --follow on a job running elsewhere in the cluster just works.

It adds no authentication of its own here either — your bearer token passes through and the node authenticates you (see below). But note what that means on a TLS cluster: this port is plaintext, and the process is holding the REST client certificate the nodes demand. Publishing it turns a certificate-gated API into an open port speaking with the proxy's identity, with --auth as the only remaining gate. Front it with TLS, or keep it on a network you trust.

What it does not add

Authentication of its own. The proxy adds none: it inherits whatever the node enforces. With --auth on, the UI presents a login screen and forwards each user's token to the node, so the cluster sees the real end user and the proxy itself holds no credential (see Authentication). With --auth off — the default — anyone who can reach the web UI can run arbitrary scripts on your cluster, exactly as anyone who can reach the REST API can. The proxy can dial an https cluster (its --tls-ca / --tls-client-cert / --tls-client-key mirror the CLI's — see Securing the REST port), but its own browser-facing port is still plaintext, so the same rule holds — keep it on a network you trust, behind TLS if it crosses an untrusted one, and do not put it on the internet.

It also does not hide anything the API says. Upstream status codes and error bodies pass through verbatim, because melange's errors are worth reading: a 400 naming a runtime no node can run, a 503 carrying the leader's address, a 409 on a job that already finished. A UI that flattened those into "something went wrong" would be less useful than curl.

Operations

Server flags

Config file

Every flag can be a TOML key instead — same name, underscores instead of dashes:

node_id = 2
listen = "0.0.0.0:8080"
advertise_rest = "10.0.0.8:8080"
peer_listen = "10.0.0.8:50051"
data = "/var/lib/melange/node.redb"
secrets_file = "/etc/melange/secrets.json"
runtimes = ["powershell", "bash"]
tls_ca_cert = "/etc/melange/ca.crt"
tls_ca_key = "/etc/melange/ca.key"
auth = true

job_timeout_secs = 1800
kill_grace_secs = 10
drain_timeout_secs = 120

melange-server --config /etc/melange/node.toml

The exceptions are the one-shot commands, which do a job and exit rather than configuring a node: --config itself, --ca-init, --rest-ca-init, --rest-client-cert and --rest-client-name are command-line only, and a TOML key of that name is an unknown-key error like any other.

Anything also given on the command line wins, so a flag can override a deployed file to get a node up without editing it. An unknown key is an error, not a setting quietly ignored — a typo in a config file is otherwise invisible, and the node comes up looking fine with a limit the operator believes they set and did not.

Securing the peer port

Peer gRPC (--peer-listen) is always mutually authenticated: --tls-ca-cert and --tls-ca-key are required, and a node given neither refuses to start. Only holders of the cluster's credential can speak Raft, read replicated jobs, or propose a change to the state machine.

It is required rather than offered because the peer port is the more powerful of the two surfaces, not the lesser one. A follower forwards a proposal to the leader over it (RaftService::forward), and the leader commits whatever Request arrives — a job to run, a user to create with any role. REST authentication (--auth) does not apply there and never could. A plaintext peer port therefore meant that anyone who could reach it had the cluster's full authority without a credential, whatever the REST API was configured to demand. The certificate is that credential.

The whole trust model is cluster-vs-outside: every node already runs arbitrary submitted scripts and holds secret values, so nodes fully trust each other. The CA is therefore the cluster's shared credential — one keypair, the same on every node, managed like the secrets file. There is exactly one thing to distribute and nothing to rotate on a schedule.

# 1. Generate the cluster CA once, anywhere.
melange-server --ca-init ./tls          # writes ./tls/ca.crt and ./tls/ca.key

# 2. Copy BOTH files to every node, and start each with them.
melange-server --node-id 1 --data n1.redb --init \
  --tls-ca-cert ./tls/ca.crt --tls-ca-key ./tls/ca.key
melange-server --node-id 2 --data n2.redb --peer-listen 127.0.0.1:50052 \
  --tls-ca-cert ./tls/ca.crt --tls-ca-key ./tls/ca.key

Each node mints its own leaf certificate in memory at boot from the CA key — nothing extra is persisted, and certificates self-heal on restart, so there is no per-node cert to manage and no renewal. A peer that cannot present a certificate signed by the cluster CA cannot complete the handshake, in either direction.

Two operational facts:

• It is all-or-nothing, cluster-wide. Every node must hold the same CA; a node with a different one cannot handshake and will never join. A node given only one of the two flags — or neither — refuses to start, rather than coming up unable to talk to anybody.
• Upgrading a pre-mTLS cluster is a flag day. Nodes running without the CA cannot handshake with nodes running with it, so generate the pair, distribute it, and restart every node — the same coordinated restart the format changes ask for.
• The peer port's CA does not cover the REST port. REST (--listen) is plaintext by default and has its own, separate protection — see Securing the REST port. Keep ca.key readable only by the melange user — possession of it is membership in the cluster.

Securing the REST port

The client-facing REST port is plaintext by default (localhost and proxy-fronted deployments need nothing more). --rest-tls turns on TLS, in one of two modes:

• tls — the node serves a certificate, so bearer tokens and login passwords stop crossing the wire in the clear. Any client that trusts the REST CA can connect.
• mtls — additionally, every client must present a certificate signed by the REST CA, or the connection is refused at the handshake. Only credentialed clients — the CLI, the web UI proxy, curl with the right flags — can talk to the API at all.

# 1. Generate the REST CA once (distinct files, so it can live beside the cluster CA).
melange-server --rest-ca-init ./tls     # writes ./tls/rest-ca.crt and ./tls/rest-ca.key

# 2. Start every node with it (cluster CA flags elided here; they are still required).
melange-server --node-id 1 --data n1.redb --init \
  --rest-tls mtls --rest-tls-ca-cert ./tls/rest-ca.crt --rest-tls-ca-key ./tls/rest-ca.key

# 3. For mtls, mint each client a certificate: client.crt + client.key, and the
#    combined client.pem that melange-cli and the web UI read.
melange-server --rest-client-cert ./alice --rest-client-name alice \
  --rest-tls-ca-cert ./tls/rest-ca.crt --rest-tls-ca-key ./tls/rest-ca.key

Each node mints its own REST server leaf in memory at boot, exactly like the peer leaf — but with real subject names, because HTTP clients verify the hostname they dial: the host of --advertise-rest, the bound IP, and loopback are covered automatically, and --rest-tls-san adds any other name clients use. A corporate certificate can serve instead (--rest-tls-cert/--rest-tls-key); mtls still needs the REST CA pair, which is what client certificates are verified against.

Clients:

# curl, tls mode: just trust the CA.       mtls adds the client pair.
curl --cacert rest-ca.crt https://node:8080/v1/jobs/<id>
curl --cacert rest-ca.crt --cert client.crt --key client.key https://node:8080/v1/jobs/<id>

# melange-cli: install once per machine, then every command is bare.
melange-cli tls install --from ./alice --from ./tls   # validates before it writes
melange-cli tls status                                # what is installed, and from where
melange-cli --server https://node:8080 status         # no --tls-* at all

# ...or per command, which is right for a one-off against another cluster (also MELANGE_TLS_*).
melange-cli --server https://node:8080 --tls-ca rest-ca.crt \
  --tls-client-cert client.crt --tls-client-key client.key status

# the web UI proxy presents ONE process-level certificate for all its users;
# each user is still identified by their own bearer token.
melange-webui --server https://node:8080 --tls-ca rest-ca.crt \
  --tls-client-cert client.crt --tls-client-key client.key

The PowerShell module takes -Certificate (a loaded X509Certificate2) or -CertificatePath/-CertificateKeyPath (the PEM pair; PowerShell 7 — 5.1 needs a store-imported PFX), and picks up ~/.melange/client.crt+client.key automatically — which is one of the things tls install writes, so a single install serves the CLI, the module and curl. Connect-Melange -Persist installs the same pair itself when it was given PEM paths. Invoke-WebRequest has no per-call CA flag, so trust rest-ca.crt by importing it into the OS store; that is also the one thing -Persist cannot do for you.

Why the identity is written twice. melange-cli and the web UI read the combined key-and-certificate client.pem; the PowerShell module and curl want the split client.crt + client.key. Both installers write all three, because writing only the shape one client reads leaves the others silently certless on a machine that looks correctly set up. tls install also refuses to copy rest-ca.key if you point --from at the CA's directory — that key mints client certificates, and belongs only on the machine that does that.

The things worth knowing before turning it on:

• Why a second CA, and not the cluster's? The peer port accepts any certificate signed by the cluster CA as a full cluster member — able to commit anything. A REST client certificate minted from that CA would therefore be a peer-port credential on every laptop that holds one. The separate CA bounds a leaked client cert to the REST API, where --auth still applies.
• A certificate is a connection gate, not a login. mtls decides who may connect; it identifies nobody. Use --auth on top for users, roles and the audit trail — the certificate replaces neither.
• Set it the same on every node. The URLs a node hands out — the 503 leader hint, the cross-node output redirect — carry the scheme that node is configured with, so a mixed cluster hands clients wrong-scheme URLs. All-or-nothing, like peer mTLS; adopt it as a coordinated restart.
• No expiry tooling, by design. Server leaves re-mint at every boot; client certificates are minted once and rotation is re-minting, redistributing and tls install --force — the same stance as the cluster CA. Revoking one client means rotating the REST CA. (melange-cli tls status does read the common name and expiry out of what is installed, which is how you tell which client certificate a machine holds; melange mints its own leaves far-future, so the date only really moves for a REST CA an operator supplied themselves.)
• A CA file that is not a certificate used to fail invisibly. reqwest adds the PEM certificates it finds in a CA file and finds none at all in a file that has no PEM blocks — no error, no roots, and then every request fails its handshake with "unknown issuer", which reads as a fault at the server. So the CLI parses the CA itself: tls install refuses it, and an already-installed bad file is reported by path on the next command rather than guessed at. tls install and tls status run before any of that, so a broken file never blocks its own repair.
• The web UI's own browser-facing port stays plaintext. --rest-tls protects node-to-client traffic; put a reverse proxy in front of the web UI itself if browsers reach it across an untrusted network.

Authentication

The REST API is open by default. --auth turns on users, passwords, and role-based access — an admin bootstrapped when the cluster is created can then manage everyone else. Auth is replicated cluster state, so a user created on one node can log in on any node.

# Create the cluster with auth on. The admin's password comes from the env (or is
# generated and logged once if you omit it). Set --auth the same on every node.
MELANGE_ADMIN_PASSWORD=... melange-server --node-id 1 --data n1.redb --init --auth

# From the CLI: log in once (token and node saved under ~/.melange/), then work as normal
# — no --server needed on later commands, they reuse the node you logged into.
melange-cli --server http://node1:8080 login admin   # prompts for the password
melange-cli whoami                  # no --server: reuses http://node1:8080

# On a --rest-tls mtls cluster, --save-tls installs the certificate at the same time, so
# everything after this needs no --server, no --token and no --tls-* either.
melange-cli --server https://node1:8080 --tls-ca rest-ca.crt \
  --tls-client-cert client.crt --tls-client-key client.key login admin --save-tls
melange-cli submit backup.ps1 --wait
melange-cli user add alice --role operator
melange-cli user list
melange-cli logout

Three things about sessions are worth knowing:

• Changing a password ends every session it minted. That is what makes rotating a password the right response to a suspected compromise: the attacker's bearer token stops working at the same moment, rather than lasting out its week. Deleting a user does the same, so recreating the username later cannot inherit the old account's tokens.

• A role change takes effect on the next request, with no revocation needed — the role is read from the user on every request, not baked into the token.

• Login is rate limited by default (--login-rate-limit, 5/s with a burst of 10). It is the only unauthenticated endpoint that does real work, and password guessing is the reason. Expired sessions are still only checked on read, never swept.

• Roles, least to most privileged: read-only (GET everything), operator (also submit/cancel jobs and manage scripts & schedules), admin (also cluster membership and user management). A user may change their own password; everything else under melange-cli user … is admin-only. The last admin cannot be removed or demoted.

• Login → token. POST /v1/auth/login returns a bearer token; send it as Authorization: Bearer <token> (the CLI does this automatically from the saved token, or --token / MELANGE_TOKEN). Tokens are session rows in the replicated store — validated by a local read, revocable by logout, and expiring after a week.

• login remembers the node. It saves the server it logged into (~/.melange/server) alongside the token, so later commands need no --server — an explicit --server still wins for a one-off against another node, and logout forgets both. With more than one cluster in play, each environment remembers its own server, token and certificate: melange-cli --env prod login admin.

• login --save-tls remembers the certificate too, on an https:// node: it installs the --tls-* material it just used, so later commands need no TLS flags either. Opt-in, because it copies a private key into a second place — a key you keep in a managed directory is rotated there, and the copy would go on working until it expired. It validates nothing and needs to: on an mtls cluster this login could not have reached /v1/auth/login without a certificate the server accepted, so a token in hand is the proof. Against a plain-http:// node it saves nothing and says so, since nothing was exercised. melange-cli tls install is the same thing without a login.

• logout does not remove the certificate. A client certificate is a machine credential, not a session: an mtls node refuses the handshake before /v1/auth/login is even reachable, so deleting it on logout would make the next login impossible from a host that had done nothing wrong. melange-cli tls remove gives it up, deliberately as a separate decision.

• Opt-in and cluster-wide. --auth off is exactly the old behaviour. Enable it from cluster creation so the admin is bootstrapped; set it consistently on every node, since any node may serve any request.

• Tokens are plaintext unless the REST port is secured. Bearer tokens (and passwords, at login) cross the wire in the clear on a default plaintext node — serve the cluster with --rest-tls, or run on a trusted network. Password and token hashes are replicated to every node (argon2id passwords, SHA-256 token keys), consistent with melange's flat trust model.

• The web UI gains a login screen and forwards each user's token to the node, so the cluster sees the real end user. The proxy itself holds no credential.

Sandboxing

By default a script runs with the full privileges of the melange user — melange is an automation platform, and anyone who can reach the API can run code as that user. --sandbox puts every script on the node inside OS-level restrictions instead.

It is node-level policy: the operator decides whether a node contains its scripts, and a job can neither ask for the sandbox nor opt out of it (a submitter who could opt out would). Nothing about it is replicated or scheduled on — a job placed on a sandboxed node runs sandboxed, full stop — except one observability bit: melange-cli cluster status and the web UI show which nodes are sandboxed.

# Linux: confine writes to the job dir, drop to an unprivileged user, cap memory.
melange-server --node-id 1 --data n1.redb --init \
  --sandbox --sandbox-fs --sandbox-user melange-jobs --sandbox-max-memory-mb 512

# Linux, stronger: give each run its own ephemeral uid from a range, so jobs are
# isolated from each other as well as from the server.
melange-server --node-id 1 --data n1.redb --init \
  --sandbox --sandbox-fs --sandbox-uid-range 100000-100999

# Windows: a job object with UI restrictions, a whole-tree memory cap, and a
# privilege-dropped restricted token.
melange-server --node-id 1 --data n1.redb --init \
  --sandbox --sandbox-restricted-token --sandbox-max-memory-mb 512

The two platforms are honestly unequal. The mechanisms differ, and so does what they can actually promise:

--sandbox-fs in detail (Linux). Landlock was chosen because it restricts access, not the filesystem view: the child sees the same /, the same paths, the same job directory — so $MELANGE_JOB_DIR, the cancel file, $PSScriptRoot, and the process-tree kill all behave exactly as they do unsandboxed. It confines only writes, and only to outside the job directory: a script can still write ./out.csv, and it can still read and execute anything its uid could before (which is why the interpreter and its libraries keep working), but it can no longer write over /etc, into another job's directory, or anywhere else its uid might reach. The runtime's own scratch ($HOME, $TMPDIR, .NET's caches) is redirected into the job directory so pwsh and bash write inside the sandbox rather than being denied. Landlock needs no root — no_new_privs, which the sandbox sets anyway, is its only precondition.

--sandbox-uid-range in detail (Linux). One shared --sandbox-user isolates every script from the server, but not from each other: they share a uid, so one can read a sibling's /proc/<pid>/environ (where that sibling's secrets are), read into a sibling's job directory, and share its RLIMIT_NPROC budget. --sandbox-uid-range MIN-MAX fixes that by giving each run its own ephemeral uid/gid (gid = uid) drawn from the range for the life of the run and returned afterwards for reuse — so a range need only be as wide as the node's concurrency (--max-concurrent-jobs), which melange checks at boot. The ids are synthetic: they need not exist in /etc/passwd, so you just pick an unused block. It requires root (it is still a uid drop) and is mutually exclusive with --sandbox-user. Combined with --sandbox-fs it also drops the shared /tmp and /dev/shm Landlock write grants — since each job's scratch is now its own — closing the last cross-job write channel; the runtime scratch redirected into the job directory keeps pwsh and bash working. (One honest caveat: a runtime that hardcodes /tmp for IPC could then be denied a write there.)

It fails closed, twice over.

• A job that cannot be sandboxed fails — it never silently falls back to running unsandboxed. A per-job setup error (a uid drop the server lacks the privilege for, a Landlock call the kernel rejects, a restricted-token spawn that will not start) fails that job.
• A misconfiguration fails the boot, not the jobs. --sandbox-user/--sandbox-fs/ --sandbox-uid-range on Windows, --sandbox-restricted-token on Linux, a sandbox limit without --sandbox, --sandbox-max-processes without a sandbox user, --sandbox-user together with --sandbox-uid-range, or a uid range narrower than --max-concurrent-jobs are all refused at startup. And a dress rehearsal runs a real probe job under the full sandbox before the node takes any work: a server that cannot drop to the sandbox user (or a probe uid from the range), spawn under a restricted token, or a kernel without Landlock when --sandbox-fs is set, refuses to start rather than discovering it when the first job runs.

What it is not. It is privilege reduction, not a container. Job-to-job isolation is available on Linux with --sandbox-uid-range (each run its own uid); with a single shared --sandbox-user, or on Windows (one token, one user), jobs are isolated from the server but not from each other. It does not confine reads of the shared filesystem generally, nor the network. And on Windows the script still runs as the server's user — --sandbox-restricted-token drops that user's privileges but not its identity or its file access. Treat it as defence in depth on top of controlling who can reach the API, not as a substitute for it.

Stopping a node

On SIGTERM or Ctrl-C a node drains rather than stopping dead:

1. /ready immediately starts answering 503, so a load balancer takes it out of rotation before it stops accepting connections. In the same moment the node begins reporting itself as draining to the cluster, so the leader stops placing new jobs on it too (it shows as draining in melange-cli cluster status).
2. It starts no new scripts. A job assigned to it but not yet begun is left non-terminal — nobody ran it, so it is still owed, and it runs when the node returns or when the leader hands it on.
3. It waits up to --drain-timeout-secs for the scripts already running to finish and report. Raft and the peer listener stay up throughout, because a worker with a result has to be able to commit it — that is the whole thing being protected.
4. Whatever is still running is asked to stop through the same cooperative machinery a cancellation uses (MELANGE_CANCEL_FILE, then --kill-grace-secs, then the process tree is killed).

All of it is bounded by --shutdown-timeout-secs, which outranks every grace period, including the ones the jobs asked for. Without that, a single job with a two-minute kill_grace_secs holds the node open for two minutes — and it would not even get them, because the service manager kills the process on its timeout, mid-drain. The drain would not merely be slow; it would be a pretence, and every other job on the node would lose the tidy ending it was halfway through. Set the ceiling below whatever will kill the process (systemd's TimeoutStopSec defaults to 90s; Kubernetes' terminationGracePeriodSeconds defaults to 30, which is below melange's default and wants raising).

A draining node is not a dead node, and the distinction is load-bearing. It keeps its vote, stays in the membership, and goes on running the jobs it already has — which is why they are not reassigned. Treating it as gone would hand work that is about to finish to a second node to run again, causing exactly the duplicate execution a polite shutdown exists to avoid. Draining is subtracted from where jobs are placed, and from nothing else.

An interrupted job is not marked finished. It did not finish: nobody cancelled it, we stopped it for our own convenience, and recording it as terminal would mean the leader never reassigns it and this node never resumes it — the work would simply never happen. It stays owed, and at-least-once does the rest.

None of this is required: a node can always be killed outright, and its unfinished jobs are re-run. But "survivable" is not "free" — re-running an hour-long script that was fifty-nine minutes in, because you restarted the node, is avoidable waste.

Logging

tracing, controlled by RUST_LOG:

RUST_LOG=melange_server=debug,melange_core=debug melange-server ...

The lines that matter operationally: job assigned, job finished, reassigning job stranded on an unreachable node, job exceeded its timeout and was killed, killing cancelled job, and job ran but result was not committed (the script executed but consensus failed — it will be retried).

Things that will bite you

• Reusing a node id for a different node, or pointing two nodes at one --data file, corrupts cluster state. Ids and databases are 1:1 with nodes.
• Two-node clusters tolerate zero failures. Use odd counts.
• Changing the codec or a replicated type is a breaking on-disk change. An existing .redb will not decode.

Environments: more than one cluster

melange-cli env keeps several clusters side by side — a dev box, a staging cluster, prod — each with its own saved server, token and TLS material. --server alone never could: it changes where a command goes without changing which token goes with it.

melange-cli env add prod --server https://mel-prod:8443 --check   # --check: reach it now
melange-cli --env prod login admin                                # prod's own token
melange-cli env use prod                                          # ...and switch for good

melange-cli env list
#   ENV       SERVER                   CLUSTER          TOKEN
#   default   http://127.0.0.1:8080    —                none
# * prod      https://mel-prod:8443    Acme Prod (EU)   saved

melange-cli --env lab status      # one command elsewhere, without switching
melange-cli env show              # what a command run *now* would actually use

• Which environment: --env > MELANGE_ENV > the saved current one > default. Within it, the usual flags still win: --server, --token/MELANGE_TOKEN, --tls-*.
• default is the flat ~/.melange/ layout every earlier version wrote — the same server, token and certificates, in the same places. Nothing is migrated when you upgrade, so nothing can go wrong in the migrating: you are not logged out, and the PowerShell module (which reads those files directly) goes on working. Named environments live in ~/.melange/envs/<name>/, and ~/.melange/config.toml records which is current. The one cost is that the PowerShell module always sees default; use Connect-Melange for another cluster.
• melange-cli env add is local. It has to work on a plane, and against a cluster that is not up yet. --check opts into one round trip, which also records what the cluster calls itself.
• A cluster that is not the one you bound to is a warning, not a surprise. If the cluster now answers with a different name, cluster status (and login) say so on stderr and carry on — which is what catches a DNS alias repointed at prod. An unnamed cluster never warns, and the recorded name is never silently overwritten.
• env rm takes the token and certificates with it; logout ends a session and leaves the environment. env rm default is refused — those are this machine's own files, which other melange clients read too.

Development

cargo build   --workspace
cargo check   --workspace --all-targets
cargo clippy  --workspace --all-targets
cargo test    --workspace

cargo test -p melange-core                    # one crate
cargo test -p melange-core executor::tests::  # one module
cargo test -p melange-core runs_a_script      # one test

cargo test -p melange-server --test api       # one node, end to end
cargo test -p melange-server --test cluster   # three nodes, end to end

All of it runs in CI (.github/workflows/ci.yml) on every push and pull request: fmt --check and clippy -D warnings on Linux, and the full test suite on both Linux and Windows. The two are not redundant — the executor has a real OS split in it (a job object on Windows, a process group on Unix), so when they disagree, the disagreement is the bug.

cargo test on Windows is not the whole suite. melange's bash tests are cfg(unix) — that is where bash is — so on Windows they do not run, and do not even compile. The Linux leg is the only place they execute, and a change that breaks them will look green on a Windows dev box.

melange-core's tests shell out to real interpreters: pwsh (or powershell) must be on PATH, and on Unix so must bash. Nothing is faked.

melange-server's tests are end-to-end, and there are two of them:

• tests/api.rs starts a real node, serves the REST API on a random port, and drives it over HTTP with the same wire types the CLI uses. A script goes in, a real shell runs it, the result comes back. Nothing is stubbed.
• tests/cluster.rs stands up three real nodes in one process — real gRPC between them, real HTTP in front of each — and tests what melange is a cluster for: a follower forwarding a write to the leader, the cluster electing a new leader when the old one is unplugged, a dead node's jobs being reassigned and run again, jobs going only to nodes that hold their secrets (on placement and on reassignment), a node being restarted with a secret it did not have before, and a mixed cluster — a bash job that must land on a node with bash and never on the PowerShell-only one, including when it is reassigned off a dead node onto a busier node, because the idlest survivor cannot run it.

Both start their nodes through the same Node::start the binary calls. A test that reimplements the wiring is a test of the reimplementation: it drifts, and then it passes while the real thing is broken.

They exist because every serious bug found in this codebase so far lived in the seams between the layers — a script delivery that silently ran nothing and reported success, a working directory that was the server's own install directory, a "stuck" reason that outlived the problem it described — and none of them were caught by unit tests of the parts in isolation. If you change how a job gets from the API to a shell and back, or how the cluster decides who runs what, those files are what tell you whether it still works.

protoc is not required — melange-proto's build script vendors it.

How a script actually reaches a shell

Every runtime's conventions live in one place (melange-core::runtime), and every one of them was a bug in something first:

• Scripts are written to a file and the interpreter is pointed at it — -File for PowerShell, a bare path for bash. The tempting alternative for PowerShell — pipe the body to -Command -, no quoting, no temp file — is silently broken. -Command - reads stdin as a REPL, so a multi-line block (if, try, while, function) needs a trailing blank line to close it, just as if you had typed it at a prompt. Without one, PowerShell reaches EOF with the block still open, throws it away, and exits 0 having run nothing at all — a script that does nothing, recorded as a success.
• The BOM is per-runtime, and the two runtimes want opposite answers. A .ps1 gets a UTF-8 BOM, because Windows PowerShell 5.1 reads a BOM-less one as ANSI and fails to parse any script containing a non-ASCII byte. A .sh must not get one: bash reads those three bytes as the first word of line 1, so #!/bin/bash comes back as command not found — and the script still exits 0. Neither the exit code nor stdout shows it; only stderr does, which is what the regression test asserts on.
• The child process gets NO_COLOR=1. Without it, PowerShell 7 wraps error output in ANSI escape codes, which then get stored verbatim as part of the job's captured stderr.
• The child gets no stdin at all. It cannot ask a human anything, and a script blocking on a pipe nobody writes to would hang to its timeout for nothing.

Adding a third runtime is an additive change and the shape is proven: a variant, its file/BOM/argv conventions, a probe, and a parallel test suite. The scheduler, the membership, the wire types and the CLI need nothing. Runtime is a closed enum precisely so that every interpreter's traps are enumerable and tested.

Known limitations

• Cron schedules are minute-granular, and their timezone is a single cluster-wide setting. A 5-field cron cannot express a sub-minute interval. The zone a cron is read in is the leader's --timezone (an IANA name, UTC by default), not per-schedule — so a cluster cannot run one schedule in America/New_York and another in Europe/London without a second cluster. There is also no per-run backoff or overlap guard: if a scheduled script runs longer than its interval, the next tick still fires (producing a second concurrent job), because at the scheduler's altitude every run is an independent job.
• A bash job's success is bash's definition of it. melange does not inject set -e, so a script that fails midway and then prints something exits 0 and is recorded as succeeded. Deliberate — prepending it would silently change the meaning of every script anybody submits — but it means a careless bash job can report success it did not earn. Put set -euo pipefail in your scripts.
• A node's description is only as fresh as the last sweep. The leader re-asks every node for its capacity, secret names and runtimes on each orphan sweep (--reassign-after-secs ÷ 3, capped at 10s), so a re-sized node, or one given a new secret or a newly installed interpreter, is scheduled correctly within seconds — but not instantly, and not at all while there is no leader.
• A job stranded on a node that dies holding the only copy of what it needs does not move — its only secret, or the only bash in the cluster. It is not lost: it says why it is stuck, and it runs when its node returns. But nothing else can take it, and no amount of waiting changes that if the node never comes back. Give a second node the secret, or the interpreter.
• Only PowerShell and bash. A third runtime is an additive change, but it is not free: each one needs its conventions worked out and tested, because the failure modes are silent.
• --max-concurrent-jobs is a local bound the leader cannot see. A node whose queue exceeds it simply runs jobs slower. The leader keeps counting them as outstanding, so it does back off — but it never refuses an assignment.
• No client-side leader retry. On a 503 the CLI reports the error rather than retrying elsewhere.
• REST transport security is opt-in. The REST port is plaintext unless the cluster is served with --rest-tls (tls for encryption, mtls to additionally require a client certificate); the gRPC peer surface is always mutually authenticated (Securing the peer port); auth is documented under Authentication. Client certificates install once per machine (melange-cli tls install, or login --save-tls), but there is still no rotation or revocation tooling: rotation is re-minting and re-installing, revocation is rotating the REST CA. On Windows an installed private key has no mode bits — it is protected by the permissions ~/.melange/ hands down, exactly like the saved token.
• A cancelled job on a partitioned node keeps running until the partition heals. The cluster agrees it is over — it is never reassigned or re-run — but the process itself cannot be reached. See above.
• Stopping a script is cooperative, or it is fatal. A script that watches for MELANGE_CANCEL_FILE gets to unwind; one that doesn't is killed outright, with no chance to run a finally. There is no third option — no signal exists that would give it one. See above.
• A job's working directory is deleted when it ends. It is scratch. Anything a job means to keep must go somewhere else, or to stdout.
• Two jobs on the same node share nothing. No cached state, no warm runspace, no common working directory. Every run starts a fresh shell, which costs tens of milliseconds — irrelevant for a job that does real work, and the dominant cost for a job that does not.
• A script's process tree is grouped microseconds after the shell starts, not atomically with it. A process spawned by its very first statement could in theory escape a later kill. A shell takes tens of milliseconds to start, so in practice it has not run a line yet; closing the window outright would need the process spawned suspended, which the process API does not expose.
• Scripts run with the privileges of the server process unless --sandbox is on, and even then the containment is honest-but-partial and unequal between platforms — write-confinement and a uid drop on Linux (shared with --sandbox-user, or per-run and job-to-job isolating with --sandbox-uid-range), a job object plus an optional privilege-dropped restricted token (--sandbox-restricted-token) on Windows, no general read or network isolation on either. melange is an automation platform, not a container runtime: anyone who can reach the API can run code as the melange user. See Sandboxing.