# DPM — Dark Horse Package Manager: Design

## Constraints

- Must be able to operate in a barren environment providing libc and libstdc++ — the standard build runs there as-is, no special variants. On a fully populated system the same binaries simply have more modules loadable.
- Capability grows in layers: each layer installs the dependencies of the next using only what already works.
- Every implementation exists exactly once, and is consumable by the CLI, by other layers, and by external programs (build systems, Dark Horse tooling) through C interfaces.

## Architecture overview

```
 dpm CLI          build systems / DHL tools / other languages
     \                          /
      v                        v
            libdpm-core.so
   (discovery, validation, routing,
    version reporting, config, logging)
        |                |
        v                v
    raw module       pkg module        ... future modules (repo, source, ...)
    (one .so)        (one .so)
        |                |
        v                v
  backing tree        sqlite3
  (source of truth)   (derived cache)
```

libdpm-core.so is the single entry point for everything. Modules are shared objects that implement package functionality. All routing — library-to-module and module-to-module — passes through libdpm-core. No consumer touches dlopen, dlsym, or module discovery itself.

## Versioning model

Compatibility is directional, and the module is the one that declares it:

- **Each module reports the minimum libdpm-core version it supports** (a reserved contract symbol). At load, the running library compares its own version against that minimum: if it is older → refuse with an explicit "libdpm-core too old for this module" report; otherwise load. A module is never rejected for being old, because the libdpm-core contract evolves append-only — a newer libdpm-core supports everything an older one did.
- **Consuming modules judge module versions; libdpm-core does not.** libdpm-core reports the version it saw at load and draws no conclusion from it. A module that depends on another requires it by name, reads the reported version, and decides for itself whether that version is too new or too old for what it intends to call. A load is a statement that the module is valid, never that it is compatible with a particular caller.
- **Updates only add satisfiable states**: module APIs evolve append-only (new table versions beside old ones), so a newer module still carries everything an older consumer called, and updating libdpm-core or any module preserves every previously working combination. The single possible load refusal — module requires a newer libdpm-core — names its own remedy.

## libdpm-core.so

Dependencies: libc, libstdc++, libdl. Never more — libdpm-core must remain loadable in the barren case forever, so package logic never leaks into it. libdpm-core routes and hosts; modules implement.

libdpm-core provides:

- **Discovery**: module path resolution, enumeration of installed module .so's.
- **Validation**: the full load-time contract enforcement described below. libdpm-core is the sole authority on what a valid module is; the contract definition lives inside the library as data. There is no SDK package — the interface is specified by this document and enforced by the library's validator.
- **Routing**:
  - generic dispatch — execute a command string with arguments against a named module (what the CLI uses);
  - typed access — a consumer requests a module's API at a version and receives a C function table (what modules and external programs use).
- **Version reporting**: require resolves a module by name, loads it, and returns a handle, or reports precisely why it can't. The loaded module's version is readable from the handle, for the caller to judge.
- **Common services**: configuration access (per-module namespaces from /etc/dpm/conf.d/), logging, module-path queries.

## The libdpm-core C API

All functions are extern "C". All returned strings are owned by libdpm-core (or by the module that produced them), are valid until the context is closed, and are never freed by the caller. All functions returning int use 0 for success and nonzero error codes; details of the most recent failure are retrievable per-context.

### Context lifecycle

**dpm_ctx\* dpm_open(const dpm_open_overrides\* overrides)**
Creates a libdpm-core context. Reads configuration from /etc/dpm/conf.d/ (or the config directory named in overrides), resolves the module path (overrides take precedence over config, config over the built-in default), and initializes logging per configuration. Performs no module loading. Returns NULL only on allocation failure or an unreadable/invalid explicit override; a missing config directory is not an error — defaults apply. `overrides` may be NULL, and may specify: config directory, module path, target root (for chroot/image/sysroot operation), and log level. Multiple simultaneous contexts with different roots are legal.

**void dpm_close(dpm_ctx\* ctx)**
Releases the context: unloads every module handle it issued, closes log targets, frees all memory owned by the context. All handles and strings obtained through the context are invalid after this call. NULL is a no-op.

### Module acquisition

**dpm_module\* dpm_require(dpm_ctx\* ctx, const char\* name)**
Resolves the module `name` in the module path and runs the full load-time validation sequence (see Load-time enforcement) if the module is not already loaded in this context. On success returns a module handle owned by the context (repeated calls return the same handle — modules are loaded at most once per context). On failure returns NULL and records the precise reason: not found, or validation step failed with the step and detail. No version criterion is applied here; compatibility is the caller's to determine from the reported version.

**int dpm_module_info_of(dpm_ctx\* ctx, dpm_module\* mod, dpm_module_info\* out)**
Fills `out` with the loaded module's name, version, description, and minimum-libdpm-core version exactly as they were read at load (string pointers valid until context close). This is how a consumer obtains the version it will judge. libdpm-core attaches no meaning to the values. Returns 0 on success, nonzero if the module cannot be reported on.

**const void\* dpm_get_api(dpm_ctx\* ctx, dpm_module\* mod, const char\* api_name, int table_version)**
Returns the API table `api_name` at `table_version` from a loaded module — the pointer the module exported for that table, already validated (manifest cross-check, magic, minimum size) at load. The caller casts it to the table struct type for that API and version as defined in the module's documented API. Returns NULL if the module does not provide that api/version pair; that fact is known from the manifest without further probing. The table is valid for the life of the context.

**int dpm_execute(dpm_ctx\* ctx, dpm_module\* mod, const char\* command, int argc, char\*\* argv)**
Generic dispatch: invokes the module's dpm_module_execute with the context, `command`, and the argument vector. Returns the module's return value verbatim (0 = success). libdpm-core adds nothing to the call besides delivery; argument semantics beyond "argv[0] is the command" are the module's to define.

### Enumeration

**dpm_cursor\* dpm_list_modules(dpm_ctx\* ctx)**
Scans the module path and returns a cursor over all *valid* modules (each candidate .so is validated on first scan; failures are logged and excluded). Returns NULL on an unreadable module path.

**int dpm_cursor_next(dpm_cursor\* cur, dpm_module_info\* out)**
Advances the cursor. Fills `out` with the next module's name, version, description, and minimum-libdpm-core version (string pointers valid until context close). Returns 0 and fills `out` while entries remain; returns nonzero at end.

**void dpm_cursor_free(dpm_cursor\* cur)**
Releases the cursor. NULL is a no-op.

### Services (available to modules and external consumers alike)

**const char\* dpm_core_version(void)**
Returns the libdpm-core version as a static X.Y.Z string. Callable without a context.

**const char\* dpm_config_get(dpm_ctx\* ctx, const char\* module, const char\* section, const char\* key)**
Returns the configured value for `key` in `section` of the named module's config namespace (/etc/dpm/conf.d/<module>.conf; the namespace "core", from core.conf, is the library's own). Returns NULL if unset. String valid until context close.

**void dpm_log(dpm_ctx\* ctx, int level, const char\* message)**
Writes `message` at `level` (FATAL=0, ERROR=1, WARN=2, INFO=3, DEBUG=4) to the context's configured log targets (console and/or file). Messages above the configured level are dropped. NULL message is a no-op.

**const char\* dpm_module_path(dpm_ctx\* ctx)**
Returns the resolved module directory path for this context.

**const char\* dpm_last_error(dpm_ctx\* ctx)**
Returns a human-readable description of the most recent failure recorded on this context, or NULL if none. Overwritten by the next failing call on the same context.

## Module contract

A module is one .so in the module directory. It exports, as extern "C", the following reserved symbols. Returned strings are static or module-owned, non-NULL, and valid for the lifetime of the loaded module; libdpm-core and consumers never free them.

**int dpm_module_execute(dpm_ctx\* ctx, const char\* command, int argc, char\*\* argv)**
The module's generic command entry point. `ctx` is the host context that dispatched the call — the module reaches every libdpm-core service (dpm_log, dpm_config_get, dpm_module_path, ...) through it. `command` is the subcommand name (equal to argv[0] when argc > 0); argc/argv are the remaining CLI-style arguments. NULL or empty `command` must behave as the module's help command. Returns 0 on success, nonzero on failure. This is the only entry the CLI path ever uses; it must be callable immediately after load with no other setup.

**const char\* dpm_module_version(void)**
Returns the module's own version as an X.Y.Z string. Must be constant for the life of the module. This is the value libdpm-core reports to consumers, and the value they judge compatibility against.

**const char\* dpm_module_description(void)**
Returns a one-line human-readable description, used in module listings.

**const char\* dpm_module_core_min(void)**
Returns the minimum libdpm-core version (X.Y.Z) this module supports — the oldest one whose contract and services the module was written against. A libdpm-core older than that refuses to load the module, and says so.

**const dpm_manifest\* dpm_module_manifest(void)**
Returns a pointer to a static manifest table declaring the module's entire functional surface: an entry count and, per entry, the API name, its table version, and the exact exported symbol that carries the table (e.g. { "raw", 1, "raw_api_v1" }). libdpm-core trusts nothing it doesn't verify: every declared symbol is resolved at load, and only manifest-declared tables are ever handed to consumers. An API absent from the manifest does not exist, even if its symbol does.

**API tables** (the module's functional surface): for each API version, one exported symbol (e.g. raw_api_v1) pointing to a plain C struct of function pointers. Every table opens with two fixed members: a magic constant (a fixed value defined by this spec, confirming the exporter agrees on table layout conventions) and the struct size in bytes (populated by the module, letting consumers accept tail-extended revisions of the same version). All parameters and returns are C types only; state passes through opaque handles; errors are int codes.

**Symbol naming**: functional exports are prefixed with the module's name (raw_\*, pkg_\*); the dpm_ prefix is reserved for the contract and libdpm-core.

## Load-time enforcement

libdpm-core is the sole authority on module validity; the contract above is enforced by its validator, not by any SDK. Validation is all-or-nothing; a module is registered only after passing every step:

1. **Resolve all reserved contract symbols.** Any missing → refuse, log the exact list, dlclose.
2. **Minimum-version handshake.** dpm_module_core_min() must be ≤ the running libdpm-core version. If the library is too old, refuse and say so — the remedy is updating libdpm-core, and the message names it. Old modules on a newer libdpm-core always pass.
3. **Probe the cheap calls.** dpm_module_version() and dpm_module_description() are invoked immediately; NULL or malformed returns → refuse.
4. **Cross-check the manifest.** Every API the module declares must actually resolve via dlsym. A module advertising an API it doesn't export is refused. libdpm-core validates the module's entire declared surface at load, before offering any of it.
5. **Table sanity.** Check each declared table's magic constant (catches modules built against a stale or wrong layout) and minimum size for its version.

Failures happen at install/load time, loudly and itemized. Consumers never receive a partially valid module: if a handle was handed out, the contract already validated. The residual C-ABI limit — dlsym cannot verify signatures — is covered in practice by the magic, size, minimum-version handshake, and probes; defeating those requires deliberate lying, which is a package-signing concern upstream of the loader.

## Module: raw — the file-based installer

Ships with the base system alongside libdpm-core. Depends on the baseline only; archive decompression is vendored in, and the package format is chosen to keep that small. This is what makes barren-environment operation possible: libdpm-core plus raw function with nothing else present.

- **Owns the backing tree** (/var/lib/dpm/): one directory per installed package holding manifest, metadata, and hooks. The tree is the database at this layer.
- **Operations** (exposed both as commands and in raw_api_v1): install a package file, remove, verify, and queries answered by walking the tree — slow but always correct, zero dependencies.
- **Owns the lock file and an append-only transaction journal** with a generation counter. Every mutation in the entire system ultimately passes through raw, so locking and journaling are implemented exactly once and inherited by every layer above.

## Module: pkg — the full package manager

Ships as a package, installed by raw once sqlite3 is installed. Requires raw (via libdpm-core, judging raw's reported version itself) and libsqlite3.

- Never touches the tree directly: every filesystem mutation is a call into raw's API table, obtained from libdpm-core, in-process — shared locking, real error propagation, no output parsing.
- **The sqlite database is a derived cache** under one invariant: it contains nothing that cannot be rebuilt by scanning the tree. It records the last journal generation it applied; on open, if the tree is ahead (someone used raw directly — explicitly allowed, that's the escape hatch for broken systems), it replays or rebuilds. Self-healing by construction.
- Adds what the cache enables: fast queries, dependency resolution against the installed set, multi-package transactions with rollback.
- Exposes pkg_api_v1 for consumers that want dependency-aware operations; they transitively get raw's guarantees because there is no second code path to the tree.

## The CLI

`dpm` is argument parsing and printing. It links libdpm-core, enumerates modules, and forwards subcommands through generic dispatch. Its command surface is exactly the set of loadable modules — in the barren case that's raw's commands; on a full system, everything installed. No capability logic lives in the CLI.

## External consumers

Build systems and Dark Horse components link libdpm-core.so — the same library, the same path as everything else:

- open a context (optionally against an alternate root),
- require the layer they need,
- read its reported version and decide whether it suits them,
- fetch its API table,
- call C functions directly.

libdpm-core installs its header to the standard include path and its library to the standard lib path: a consumer writes #include <dpm/core.h>, links -ldpm-core, and calls package manager module functions. A consumer that opens a default context (no overrides) is operating the installed package manager itself — system configuration, system module path, system tree, system locking — exactly as if it were invoking the installed dpm command, because the CLI is just another caller of the same library. Overrides redirect individual paths only when a caller explicitly sets them. Whether a consumer targets raw only (image builders that just deploy trees) or pkg (dependency-aware tooling) is their choice of require(); behavior is identical to the CLI's because it is the same implementation.

## Bootstrap chain

```
minimal start:   dpm + libdpm-core.so + raw module      (baseline deps only)
raw installs:    sqlite3 package
raw installs:    dpm-pkg package                         (drops the pkg module .so)
now:             libdpm-core discovers pkg, validates it, full management is live
```

Every layer is a package installed and upgraded by the layer beneath it; the package manager maintains itself with the same mechanism it offers the OS. Future modules follow the identical pattern — a repo/network module declares its requirements (pkg, a TLS library), lands as a package, and the capability appears on next discovery.

## Repository structure

Modules are developed independently from each other and independently from libdpm-core — one repository per module, plus the libdpm-core repository. Each repo owns its source, build, and tests, and produces exactly one artifact:

- **The libdpm-core repository**: libdpm-core.so and the dpm binary. Contains no module code. Its test fixtures include deliberately broken stub modules for validating the loader, and one known-good stub — never a real package module.
- **The exception**: an info module that bundles with the dpm binary and libdpm-core, used to test and demonstrate full DPM system functionality where appropriate.
- **One repository per module** (raw, pkg, and every future module): produces that module's .so. Links libdpm-core.so — the only cross-repo build dependency in the system — and nothing else from DPM. Peer modules never appear in a module's repository, build, or test environment; peers are runtime concerns, faked at test time (contract fakes and stub modules) and real only at distribution-level integration.

No repository can block another's development: a module builds and its full pre-integration test surface (unit, contract, module-hosting) runs with nothing present but its own checkout and an installed or vendored libdpm-core. Release coordination happens through the versioning model — each consumer judging the versions it is handed — rather than through lockstep builds.

### Layout of the libdpm-core repository

```
include/dpm/       public headers — installed to the system include path; the
                   dpm/ directory is the consumer namespace, so an installed
                   consumer writes #include <dpm/core.h>
include/internal/  library-private headers — used only by src/, never installed
src/               implementations of the library
src/cli/           the dpm CLI entry point
src/bundled-modules/info/  the bundled info module
data/              files installed as-is (core.conf)
tests/             fixture modules, the API test binary, CLI tests
docs/              project documentation
```

Every header lives under include/: include/dpm/ is the published API surface and defines what consumers see; include/internal/ is the implementation's own headers, invisible outside the repo because the install rule ships only include/dpm/.

## Artifacts

Terminology: **the dpm binary** names the command-line tool; **libdpm-core** names the library.

| Artifact | Location on system |
|---|---|
| dpm | /usr/bin/dpm |
| libdpm-core.so | /usr/lib/libdpm-core.so |
| info.so | /usr/lib/dpm/modules/info.so |
| modules (raw.so, pkg.so, repo.so, source.so, ...) | /usr/lib/dpm/modules/&lt;name&gt;.so |

## Development and testing

Development works because the design has no build-time coupling between peers: nothing links against a peer module, ever. "Not all the pieces are there" is the normal, permanent condition at build time. What remains resolves into four test layers, each needing strictly less than the full system.

### What a build requires

- A module compiles against its own declarations (written to the spec — the externs it exports, the table structs it consumes) plus **libdpm-core.so, the one real link dependency** — and libdpm-core is by definition the stable, always-present, baseline-only piece. Cheap to have in every dev environment, trivially vendorable as a checkout.
- Peer modules are reached at runtime through the require/get_api pair. Building pkg does not require raw to exist anywhere. The compile-time knowledge of raw is just the raw_api_v1 struct layout, which is spec, not artifact.

A module repo therefore builds self-contained, always.

### Test layers

**1. Unit tests — need nothing.** The module's implementation compiles once as an object library, linked into both the .so and a test binary. Pure logic, error paths, parsing — no libdpm-core, no peers.

**2. Contract tests — need a struct, not a module.** Because every dependency is an API table — a plain struct of function pointers — a fake is just a struct the test fills in with functions that record calls and return canned results. pkg's logic is exercised against a fake raw_api_v1 (verifying it calls install/remove/query correctly, handles raw's error codes, honors the journal-generation protocol) with raw nowhere on the machine. Injection is built into the architecture; no linker seams required.

**3. Module-hosting tests — need libdpm-core only.** A harness links the real libdpm-core, points the module path at the build output plus fixtures, and has it load the just-built .so exactly as production would — full five-step validation included, so contract violations fail here, in CI, not on a user's system. Where the module needs a peer, the fixture directory contains a **stub module**: a tiny .so exporting the reserved symbols and a fake table, which libdpm-core validates and serves like the real thing. The harness then drives dpm_module_execute end to end against fixture config and data. This layer runs on a bare builder with nothing installed.

**4. Integration — the only layer that needs everything, and it builds itself.** Real libdpm-core plus real raw, then the actual bootstrap chain into a scratch root: dpm_open against an alternate root, raw installs sqlite3 and the pkg package into it, libdpm-core discovers pkg, real operations run against the throwaway tree. Because alternate roots are first-class in the API, this needs a directory, not a VM. Full-distribution CI does the same with real packages.

### Day-to-day workflow

- Working on **pkg**: edit, run unit + contract tests (instant, zero environment), harness run before merge. A real raw is never needed, or even possessed, until integration.
- Working on **raw**: same, except its fakes point the other way — its tests need only fixture package files and a scratch tree.
- Working on **libdpm-core**: its test fixtures are deliberately broken modules — missing symbols, wrong magic, lying manifests, a too-new minimum-version declaration — plus one known-good stub. Development never needs any real package module.
- **Debugging** is the layer-3 harness under a debugger — it is the "run the module without the system" mechanism, so no separate standalone build exists or is maintained.

The discipline that keeps this honest: fakes and stubs are written to the spec, and layer-3 validation plus the layer-4 bootstrap run in CI, so a fake that drifts from reality is caught by the first integration pass rather than shipped.

## Development capabilities

During development the CLI must be pointable at a local libdpm-core, and that library must be configurable to local paths (module path, config dir, etc.). Two mechanisms provide this:

- **Pointing the CLI at a local libdpm-core** is dynamic-linker territory, needing no DPM mechanism: development builds of the CLI carry an rpath to their own build tree's lib/ directory, so the locally built binary resolves the locally built library (LD_LIBRARY_PATH pointed at that lib/ directory achieves the same). The system libdpm-core is never touched.
- **Pointing that library at local paths** is what the dpm_open overrides exist for: config directory, module path, and target root are all fields of the overrides struct, and the CLI exposes them as flags. A dev invocation:

```
./build/bin/dpm --config-dir ./tests/fixtures/conf --module-path ./build/modules raw install ./fixture.dpm
```

The config-dir override matters most: once the context reads config from the local conf dir, everything configurable — log file, module path defaults, per-module settings — resolves locally, so a checkout plus its fixtures is a complete self-contained environment. Adding --root at a scratch directory makes even real install operations land in a throwaway tree.

**Rule**: every field of the dpm_open overrides struct must be exposed as a CLI flag, so anything a linked consumer can redirect, a developer at the shell can redirect too. This holds for any future override field — nothing ships reachable from code but not from the command line.

## Evolution rules

- **Append-only ABI**: breaking a module API means exporting a new table (raw_api_v2) beside the old one, never mutating v1. Old tables remain until consumers are gone.
- **Append-only libdpm-core contract**: a newer libdpm-core loads everything an older one did; a module's only version assertion against the library is its minimum.
- **Compatibility is decided by the consumer**: modules declare the minimum libdpm-core they support, and the library enforces that one handshake because it is the host. Everything else is reported, not enforced — libdpm-core hands a consumer the version it saw and the consumer decides whether to proceed. No maximums and no exact-match constraints live in the library, so an update can never make a previously working combination refuse to load.

## Invariants

1. libdpm-core routes and hosts; modules implement. No package logic in the library, ever.
2. Writes flow down, never sideways: a layer mutates the system only through the layer beneath it, in-process through libdpm-core-mediated APIs.
3. Truth lives in the tree; everything above is regenerable cache or convenience.
4. Dropping down a layer by hand is always legal; layers above detect it and reconcile.
5. A module is either fully valid or not loaded — no partial states, no consumer-side defense.