Tay

Durable background jobs without a database or message broker.

Tay is a durable, single-node background-job engine with its own append-only storage. It can run embedded inside an Elixir application or as standalone infrastructure in a production OCI container.

Tay owns its durable job state without a separate database, Redis, or RabbitMQ.

Read the formatted documentation for installation, configuration, API reference, and operations.

Application / Worker
        │
        ├── local Unix socket ─┐
        └── HTTP/JSON API ─────┤
                               ▼
                    ┌─────────────────────┐
                    │         Tay         │
                    │                     │
                    │ queues · retries    │
                    │ schedules · cron    │
                    │ cancellation        │
                    │ crash recovery      │
                    └──────────┬──────────┘
                               │
                               ▼
                        durable local disk

Tay can be used in three ways:

• Elixir library — add tay from Hex and run the Engine inside your supervision tree.
• Standalone container — run Tay as infrastructure beside your application, without installing Elixir or Erlang on the host.
• Python client — execute Python tasks through the supported tay-client SDK while Tay owns scheduling, retries, persistence, and recovery.

The optional Tay Dashboard adds a Phoenix LiveView operational UI and is also published as a ready-to-run container image.

Tay is intentionally single-node and single-writer. It is designed for cases where durable background execution is needed without operating a separate database or distributed broker. It is not a replacement for a distributed multi-node message bus.

Quick start with Docker

The easiest way to try Tay does not require Elixir or Erlang on the host.

Pull the standalone image:

docker pull ghcr.io/andriisydorenko1904/tay:latest

Download the versioned Compose example into a new directory:

mkdir tay-example
cd tay-example
curl -fsSL https://raw.githubusercontent.com/AndriiSydorenko1904/tay/v1.8.1/compose.yml -o compose.yml

Start Tay with that file:

docker compose up --no-build tay

That command uses the published image. To build Tay and the optional local Python socket worker from this checkout instead, use TAY_IMAGE=tay:local docker compose --profile local-worker up --build.

The Tay container owns durable state at /var/lib/tay. Applications and language workers can use the shared /run/tay Unix socket or the optional HTTP/JSON API with mTLS. Without HTTP settings, Tay creates the socket automatically. Setting TAY_HTTP_PORT selects HTTP-only operation unless TAY_SOCKET_PATH explicitly enables a socket as well.

Restarting or replacing the Tay container does not discard jobs as long as the data volume is preserved.

Want the operational UI as well?

TAY_ENABLE_DASHBOARD=true docker compose up --no-build tay

Then open:

http://localhost:4000/tay

The same image contains the optional dashboard. Enable it with TAY_ENABLE_DASHBOARD=true:

docker pull ghcr.io/andriisydorenko1904/tay:latest

Why Tay?

A durable background-job system needs somewhere to keep pending, running, scheduled, and retryable work.

Many systems delegate that responsibility to DB, Redis, or a message broker.

Tay takes a different approach:

the job engine includes its own crash-safe local store.

That makes Tay useful when you want:

• durable queues without provisioning a separate database or broker;
• retries, scheduling, cron, cancellation, and queue concurrency in one runtime;
• recovery of durable job state after process or container restarts;
• a small infrastructure footprint for a single-machine or single-node deployment;
• Elixir-native embedding when the application is already written in Elixir;
• a standalone container when the application is written in another language;
• Python workers without moving persistence and orchestration into Python.

Tay provides at-least-once execution. External effects must therefore be idempotent or reconciled by the application.

Architecture

Tay separates durable orchestration from application code.

When running standalone, Tay can be treated as a small infrastructure component:

┌──────────────────────┐
│ Application / Worker │
│                      │
│ Python / other       │
│ local runtime        │
└──────────┬───────────┘
           │
           │ Unix socket
           │
           ▼
┌──────────────────────┐
│         Tay          │
│                      │
│ durable job engine   │
│ queues               │
│ schedules / cron     │
│ retries              │
│ cancellation         │
│ recovery             │
└──────────┬───────────┘
           │
           ▼
    persistent volume

Tay remains a single-node durable engine, while producers and workers may connect through Bandit HTTP/JSON from another host. Local workers can still use the Unix-domain socket. See worker protocol guide.

This keeps the single-node trust and failure model explicit while allowing the Engine to be packaged and operated independently from application workers.

For Elixir applications, no service boundary is required at all. The same Engine can run directly inside the application's supervision tree.

Elixir integration

Use Elixir 1.20 and Erlang/OTP 29 with a C11 compiler (cc) available when building.

Add Tay to your application's Mix dependencies:

defp deps do
  [
    {:tay, "~> 1.8"}
  ]
end

Or use the local checkout:

{:tay, path: "../tay"}

Then run:

mix deps.get
mix compile

For a development run, choose a dedicated absolute directory and initialize it once. Initialization refuses existing history and is never a repair command:

export TAY_DATA_DIR="$PWD/var/tay"
mix tay.storage.init --data-dir "$TAY_DATA_DIR" --durability write

Define a worker with a stable key:

defmodule MyApp.HelloWorker do
  use Tay.Worker, key: "hello.v1", queue: :default

  @impl true
  def perform(%Tay.Job{args: %{"name" => name}}) do
    IO.puts("Hello, #{name}!")
    :ok
  end
end

Add Tay to your application's supervisor.

The :write mode below is for development only and has no power-loss durability guarantee:

children = [
  Tay.child_spec(
    data_dir: System.fetch_env!("TAY_DATA_DIR"),
    durability: :write,
    workers: %{"hello.v1" => MyApp.HelloWorker},
    queues: [default: 2]
  )
]

Supervisor.start_link(children, strategy: :one_for_one)

Explicit initialization remains the default.

An embedding application may instead pass:

initialize: :if_missing

to Tay.child_spec/1.

That option only initializes a genuinely missing storage root before the normal full recovery path. It never repairs, replaces, truncates, or reinitializes existing storage.

Once the Engine is ready, submit and inspect a job:

%{state: :ready} = Tay.status()

{:ok, intent} =
  MyApp.HelloWorker.new(%{"name" => "Ada"})

{:ok, job} =
  Tay.insert(intent)

{:ok, current} =
  Tay.get_job(job.id)

IO.inspect(current.state)

Jobs are persistent by default. Selective durability is resolved from the individual insertion, then the worker default, then the global default:

config :tay,
  durability: :persistent,
  journal: [batch: [max_events: 100, interval_ms: 10]]

defmodule MyApp.EphemeralWorker do
  use Tay.Worker, key: "ephemeral.v1", durability: :volatile
  # ...
end

Tay.insert(MyApp.EphemeralWorker, %{}, durability: :persistent)

:persistent jobs are acknowledged only after their journal mutations meet the Engine's storage-durability contract and are recovered after restart. :volatile jobs use the same queues, state machine, retries, cancellation, and resource limits, but remain solely in the current Engine's ETS projection. They never enter Store V2, terminal disk storage, checkpoints, or compaction output, and disappear when that in-memory generation is lost.

List and summarize jobs through the bounded public inspection API:

{:ok,
 %{
   jobs: jobs,
   total_count: total,
   previous_cursor: previous,
   next_cursor: next,
   last_cursor: last
 }} =
  Tay.jobs(
    states: [:retryable, :discarded],
    queues: [:default],
    limit: 50
  )

{:ok, counts} = Tay.stats()
{:ok, queues} = Tay.queues()

The tay package includes a Phoenix LiveView dashboard for these APIs. Its modules live under lib/tay/dashboard/; see the dashboard guide for router mounting and access control. For Prometheus metrics, OpenTelemetry command spans, gauge polling, event names, and label-cardinality guarantees, see the observability guide.

Keep the original intent until an insertion outcome is known. A lost reply may follow a durable write; reconcile by job ID or resubmit the same intent, never a newly generated ID.

External worker effects may run again after a crash, so make them idempotent or reconcile them in the application.

Execution rate limits

Concurrency and execution rate can be configured independently. On an operator-validated Linux filesystem, extend the Engine configuration with:

Tay.child_spec(
  data_dir: "/var/lib/tay",
  durability: :sync,
  validated_filesystem: true,
  queues: [default: 10],
  rate_limits: [
    queues: %{"default" => %{limit: 100, interval_ms: 1_000}},
    tasks: %{"billing.capture.v1" => %{limit: 5, interval_ms: 1_000}}
  ]
)

Both applicable fixed-window limits must permit a start. Pending jobs stay in Tay, and another eligible task may run while one task is rate-limited. Every committed execution attempt consumes credit, including retries. Limits use local monotonic time and reset on Engine restart; configure the service owning a strict cross-restart external quota separately. Standalone hosts use TAY_RATE_LIMITS; see operations.

Python integration

Python is a first-class external-worker integration.

Tay remains responsible for:

• durable job state;
• queues;
• scheduling;
• retries;
• cancellation;
• crash recovery.

Python processes execute application code.

When Tay is embedded in Elixir, the Engine starts a local Unix-domain socket automatically.

When Tay runs as a standalone container, local Python workers may share the socket volume. Workers on another host instead connect to the mTLS HTTP API; no shared socket is needed.

The same tay-client SDK is used in both cases.

Install it from PyPI:

python -m pip install tay-client

Or from this checkout:

python -m pip install ./clients/python

Then define and submit tasks:

import asyncio
from tay import Tay

tay = Tay(capacity=4)

@tay.task(name="billing.capture.v1")
def capture(invoice_id: str) -> dict:
    return {"invoice_id": invoice_id}

async def submit() -> None:
    async with tay:
        job = await capture.enqueue(
            "inv-42",
            submission_id="capture:inv-42",
        )
        print(await job.wait_result(timeout=30))

asyncio.run(submit())

This gives Python applications a simple split:

Python
  │
  │ execute application code
  ▼
tay-client
  │
  │ Unix socket
  ▼
Tay
  │
  ├── persistence
  ├── queues
  ├── scheduling
  ├── retries
  └── recovery

The Python application does not need to implement its own durable queue or persistence layer.

Python schedules

The Python client accepts standard five-field Unix cron expressions.

Schedules use UTC by default; pass a fixed numeric offset when needed.

The API accepts:

catch_up="latest"
catch_up="all"

and defaults to:

overlap="skip"

Example:

# At 08:00 on weekdays in UTC+02.
await tay.schedule(
    "billing.capture.v1",
    cron="0 8 * * 1-5",
    timezone="+02",
)

# Start in ten minutes, then repeat every fifteen minutes.
await tay.every(
    "billing.capture.v1",
    minutes=15,
    delay=600,
)

Use start_at=<UTC milliseconds> for an absolute first run instead of delay; those options cannot be combined.

Schedule declarations, advancement, and cancellation are durable. They survive listener, Engine, and client-process restarts as well as compaction; reconnecting clients may safely redeclare the same stable declaration ID. Missed-time catch-up follows the selected latest or all policy. overlap="allow" admits independent occurrences; skip advances past a due occurrence while the previous job remains nonterminal; queue holds the due occurrence until the previous job finishes. skip is the default. The active-job link survives restart and compaction, and a delivery interrupted after insertion reconciles the same job. For queue, catch_up="all" delivers missed occurrences serially; latest coalesces the remaining missed times after delivery. Once a timer is due, temporary admission pressure delays that occurrence instead of dropping it; its deterministic job ID makes retries idempotent.

The optional Bandit HTTP/JSON API exposes producer and worker operations over TCP. It is disabled by default; plaintext binds only to loopback and remote access requires mTLS. Tay's storage remains single-node; remote workers do not turn the engine into a distributed Store.

See the protocol contract for discovery, security, request types, and durable result retention.

Standalone container

The release workflow publishes a self-contained Linux image for amd64 and arm64:

ghcr.io/andriisydorenko1904/tay:1.8.1
ghcr.io/andriisydorenko1904/tay:latest

It includes the Erlang VM and Tay runtime.

The host therefore needs Docker or another OCI runtime, not Elixir or Erlang.

Pull it directly:

docker pull ghcr.io/andriisydorenko1904/tay:latest

A typical non-Elixir deployment runs the image beside an application worker:

┌────────────────────┐
│ Application worker │
└─────────┬──────────┘
          │
          │ /run/tay
          │
┌─────────▼──────────┐
│    Tay container   │
│                    │
│ durable job engine │
└─────────┬──────────┘
          │
          │ /var/lib/tay
          ▼
    persistent volume

The application and Tay share only the /run/tay socket volume.

Durable state belongs on:

/var/lib/tay

The Compose example starts Tay; the local worker uses an optional profile:

TAY_IMAGE=tay:local docker compose up --build

Run TAY_IMAGE=tay:local docker compose --profile local-worker up --build to include that worker.

The container is:

• single-node;
• single-writer;
• non-root;
• compatible with a read-only root filesystem.

The Tay container does not execute user job code.

Deleting its data volume deletes Tay's durable state.

Deleting its socket volume does not.

See the standalone runtime guide for volumes, permissions, configuration, health checks, restart behavior, and deployment details.

Dashboard

Tay Dashboard is an optional operational web UI built with Phoenix LiveView.

The tay image contains Tay core and the small Phoenix host in one release. The web endpoint is disabled by default and enabled with:

TAY_ENABLE_DASHBOARD=true

Start the repository example:

TAY_IMAGE=tay:local TAY_ENABLE_DASHBOARD=true docker compose up --build

Then open:

http://localhost:4000/tay

The example binds only to host loopback and leaves optional Basic authentication disabled.

The single Tay container owns the Store and optionally serves the dashboard.

See the complete dashboard container guide for configuration, optional Basic authentication, reverse-proxy guidance, persistence, and upgrade rules.

Documentation

Use the rendered guides on HexDocs for navigation, code examples, search, and API reference. Markdown files in the Hex package preview are source files.

Production and upgrades

Tay 1.8 is single-node and single-writer. Production storage requires Linux, a validated supported local filesystem, and both Engine options:

durability: :sync,
validated_filesystem: true

macOS supports development :write mode, which has no power-loss guarantee. The standalone Linux image uses :sync.

Automatic compaction expires eligible terminal history and keeps active jobs. Periodic checkpoints are configured with compaction: [checkpoint_interval: 60_000] or standalone TAY_CHECKPOINT_INTERVAL_MS. See the operations guide for journal batching, checkpoint settings, memory/disk budgets, and backup/restore commands.

Before upgrading, stop the Engine and make a complete cold backup. Tay 1.8 reads preceding supported Store-v2 history, but older binaries cannot read new 1.8 schedule events. Downgrade requires restoring a pre-upgrade backup. Store-v1 installations must migrate with Tay 1.4.x before using Tay 1.5 or later. See the upgrade boundaries.

Execution is at-least-once: worker effects must be idempotent or reconciled. Tay has no live backup or automatic corruption repair. Preserve a refused store for inspection; never delete individual log files to recover capacity.

License

The Tay engine is source-available under the Elastic License 2.0.

Internal use, including use inside an ordinary commercial SaaS product, is free.

The license does not permit offering Tay itself, or a substantial set of its functionality, to third parties as a hosted or managed service.

Commercial terms for that use are available separately; see commercial licensing.

The separately distributed Python client is licensed under MIT.