dpm-core-ng/docs/DESIGN.md

DPM — Dark Horse Package Manager: Design

Constraints

• Must be able to operate in a barren environment providing libc and libstdc++ — the standard build runs there as-is, no special variants. On a fully populated system the same binaries simply have more modules loadable.
• Capability grows in layers: each layer installs the dependencies of the next using only what already works.
• Every implementation exists exactly once, and is consumable by the CLI, by other layers, and by external programs (build systems, Dark Horse tooling) through C interfaces.

Architecture overview

 dpm CLI          build systems / DHL tools / other languages
     \                          /
      v                        v
            libdpm-core.so
   (discovery, validation, routing,
    version negotiation, config, logging)
        |                |
        v                v
    raw module       pkg module        ... future modules (repo, source, ...)
    (one .so)        (one .so)
        |                |
        v                v
  backing tree        sqlite3
  (source of truth)   (derived cache)

libdpm-core.so is the single entry point for everything. Modules are shared objects that implement package functionality. All routing — core-to-module and module-to-module — passes through core. No consumer touches dlopen, dlsym, or module discovery itself.

Versioning model

Compatibility is directional, and the module is the one that declares it:

• Each module reports the minimum core version it supports (a reserved contract symbol). At load, core compares its own version against that minimum: running core is older → refuse with an explicit "core too old for this module" report; otherwise load. Core never rejects a module for being old, because core's contract evolves append-only — a newer core supports everything an older core did.
• Consumers state only minimums, never maximums: require("raw", min 1.2) means "raw at 1.2 or anything newer." Module APIs evolve append-only (new table versions beside old ones), so newer is always acceptable and an update can never render a consumer's requirement unsatisfiable.
• Updates only add satisfiable states: because bounds are minimums and evolution is append-only, updating core or any module preserves every previously working combination. The single possible load refusal — module requires a newer core — names its own remedy.

libdpm-core.so

Dependencies: libc, libstdc++, libdl. Never more — core must remain loadable in the barren case forever, so package logic never leaks into it. Core routes and hosts; modules implement.

Core provides:

• Discovery: module path resolution, enumeration of installed module .so's.
• Validation: the full load-time contract enforcement described below. Core is the sole authority on what a valid module is; the contract definition lives inside core as data. There is no SDK package — the interface is specified by this document and enforced by core's validator.
• Routing:
  • generic dispatch — execute a command string with arguments against a named module (what the CLI uses);
  • typed access — a consumer requests a module's API at a version and receives a C function table (what modules and external programs use).
• Version negotiation: require resolves a module by name, checks the requested minimum, loads, and returns a handle, or reports precisely why it can't.
• Common services: configuration access (per-module namespaces from /etc/dpm/conf.d/), logging, module-path queries.

Core C API

All functions are extern "C". All returned strings are owned by core (or by the module that produced them), are valid until the context is closed, and are never freed by the caller. All functions returning int use 0 for success and nonzero error codes; details of the most recent failure are retrievable per-context.

Context lifecycle

dpm_ctx* dpm_open(const dpm_open_overrides* overrides) Creates a core context. Reads configuration from /etc/dpm/conf.d/ (or the config directory named in overrides), resolves the module path (overrides take precedence over config, config over the built-in default), and initializes logging per configuration. Performs no module loading. Returns NULL only on allocation failure or an unreadable/invalid explicit override; a missing config directory is not an error — defaults apply. overrides may be NULL, and may specify: config directory, module path, target root (for chroot/image/sysroot operation), and log level. Multiple simultaneous contexts with different roots are legal.

void dpm_close(dpm_ctx* ctx) Releases the context: unloads every module handle it issued, closes log targets, frees all memory owned by the context. All handles and strings obtained through the context are invalid after this call. NULL is a no-op.

Module acquisition

dpm_module* dpm_require(dpm_ctx* ctx, const char* name, const char* min_version) Resolves the module name in the module path, runs the full load-time validation sequence (see Load-time enforcement) if the module is not already loaded in this context, and checks that the module's version is ≥ min_version (X.Y.Z comparison; NULL means "any version"). On success returns a module handle owned by the context (repeated calls return the same handle — modules are loaded at most once per context). On failure returns NULL and records the precise reason: not found, validation step failed (with the step and detail), or version below minimum (with both versions).

const void* dpm_get_api(dpm_ctx* ctx, dpm_module* mod, const char* api_name, int table_version) Returns the API table api_name at table_version from a loaded module — the pointer the module exported for that table, already validated (manifest cross-check, magic, minimum size) at load. The caller casts it to the table struct type for that API and version as defined in the module's documented API. Returns NULL if the module does not provide that api/version pair; that fact is known from the manifest without further probing. The table is valid for the life of the context.

int dpm_execute(dpm_ctx* ctx, dpm_module* mod, const char* command, int argc, char** argv) Generic dispatch: invokes the module's dpm_module_execute with the context, command, and the argument vector. Returns the module's return value verbatim (0 = success). Core adds nothing to the call besides delivery; argument semantics beyond "argv[0] is the command" are the module's to define.

Enumeration

dpm_cursor* dpm_list_modules(dpm_ctx* ctx) Scans the module path and returns a cursor over all valid modules (each candidate .so is validated on first scan; failures are logged and excluded). Returns NULL on an unreadable module path.

int dpm_cursor_next(dpm_cursor* cur, dpm_module_info* out) Advances the cursor. Fills out with the next module's name, version, description, and minimum-core version (string pointers valid until context close). Returns 0 and fills out while entries remain; returns nonzero at end.

void dpm_cursor_free(dpm_cursor* cur) Releases the cursor. NULL is a no-op.

Services (available to modules and external consumers alike)

const char* dpm_core_version(void) Returns core's own version as a static X.Y.Z string. Callable without a context.

const char* dpm_config_get(dpm_ctx* ctx, const char* module, const char* section, const char* key) Returns the configured value for key in section of the named module's config namespace (/etc/dpm/conf.d/<module>.conf; "core" names core's own file). Returns NULL if unset. String valid until context close.

void dpm_log(dpm_ctx* ctx, int level, const char* message) Writes message at level (FATAL=0, ERROR=1, WARN=2, INFO=3, DEBUG=4) to the context's configured log targets (console and/or file). Messages above the configured level are dropped. NULL message is a no-op.

const char* dpm_module_path(dpm_ctx* ctx) Returns the resolved module directory path for this context.

const char* dpm_last_error(dpm_ctx* ctx) Returns a human-readable description of the most recent failure recorded on this context, or NULL if none. Overwritten by the next failing call on the same context.

Module contract

A module is one .so in the module directory. It exports, as extern "C", the following reserved symbols. Returned strings are static or module-owned, non-NULL, and valid for the lifetime of the loaded module; core and consumers never free them.

int dpm_module_execute(dpm_ctx* ctx, const char* command, int argc, char** argv) The module's generic command entry point. ctx is the host context that dispatched the call — the module reaches every core service (dpm_log, dpm_config_get, dpm_module_path, ...) through it. command is the subcommand name (equal to argv[0] when argc > 0); argc/argv are the remaining CLI-style arguments. NULL or empty command must behave as the module's help command. Returns 0 on success, nonzero on failure. This is the only entry the CLI path ever uses; it must be callable immediately after load with no other setup.

const char* dpm_module_version(void) Returns the module's own version as an X.Y.Z string. Must be constant for the life of the module and must match the version by which consumers state minimums.

const char* dpm_module_description(void) Returns a one-line human-readable description, used in module listings.

const char* dpm_module_core_min(void) Returns the minimum core version (X.Y.Z) this module supports — the oldest core whose contract and services the module was written against. Core refuses to load the module if its own version is lower, and says so.

const dpm_manifest* dpm_module_manifest(void) Returns a pointer to a static manifest table declaring the module's entire functional surface: an entry count and, per entry, the API name, its table version, and the exact exported symbol that carries the table (e.g. { "raw", 1, "raw_api_v1" }). Core trusts nothing it doesn't verify: every declared symbol is resolved at load, and only manifest-declared tables are ever handed to consumers. An API absent from the manifest does not exist, even if its symbol does.

API tables (the module's functional surface): for each API version, one exported symbol (e.g. raw_api_v1) pointing to a plain C struct of function pointers. Every table opens with two fixed members: a magic constant (a fixed value defined by this spec, confirming the exporter agrees on table layout conventions) and the struct size in bytes (populated by the module, letting consumers accept tail-extended revisions of the same version). All parameters and returns are C types only; state passes through opaque handles; errors are int codes.

Symbol naming: functional exports are prefixed with the module's name (raw_*, pkg_*); the dpm_ prefix is reserved for the contract and core.

Load-time enforcement

Core is the sole authority on module validity; the contract above is enforced by core's validator, not by any SDK. Validation is all-or-nothing; a module is registered only after passing every step:

1. Resolve all reserved contract symbols. Any missing → refuse, log the exact list, dlclose.
2. Core-minimum handshake. dpm_module_core_min() must be ≤ core's version. If core is too old, refuse and say so — the remedy is updating core, and the message names it. Old modules on newer core always pass.
3. Probe the cheap calls. dpm_module_version() and dpm_module_description() are invoked immediately; NULL or malformed returns → refuse.
4. Cross-check the manifest. Every API the module declares must actually resolve via dlsym. A module advertising an API it doesn't export is refused. Core validates the module's entire declared surface at load, before offering any of it.
5. Table sanity. Check each declared table's magic constant (catches modules built against a stale or wrong layout) and minimum size for its version.

Failures happen at install/load time, loudly and itemized. Consumers never receive a partially valid module: if core handed out a handle, the contract already validated. The residual C-ABI limit — dlsym cannot verify signatures — is covered in practice by the magic, size, core-minimum handshake, and probes; defeating those requires deliberate lying, which is a package-signing concern upstream of the loader.

Module: raw — the file-based installer

Ships with the base system alongside core. Depends on the baseline only; archive decompression is vendored in, and the package format is chosen to keep that small. This is what makes barren-environment operation possible: core + raw function with nothing else present.

• Owns the backing tree (/var/lib/dpm/): one directory per installed package holding manifest, metadata, and hooks. The tree is the database at this layer.
• Operations (exposed both as commands and in raw_api_v1): install a package file, remove, verify, and queries answered by walking the tree — slow but always correct, zero dependencies.
• Owns the lock file and an append-only transaction journal with a generation counter. Every mutation in the entire system ultimately passes through raw, so locking and journaling are implemented exactly once and inherited by every layer above.

Module: pkg — the full package manager

Ships as a package, installed by raw once sqlite3 is installed. Requires raw (via core, minimum version) and libsqlite3.

• Never touches the tree directly: every filesystem mutation is a call into raw's API table, obtained from core, in-process — shared locking, real error propagation, no output parsing.
• The sqlite database is a derived cache under one invariant: it contains nothing that cannot be rebuilt by scanning the tree. It records the last journal generation it applied; on open, if the tree is ahead (someone used raw directly — explicitly allowed, that's the escape hatch for broken systems), it replays or rebuilds. Self-healing by construction.
• Adds what the cache enables: fast queries, dependency resolution against the installed set, multi-package transactions with rollback.
• Exposes pkg_api_v1 for consumers that want dependency-aware operations; they transitively get raw's guarantees because there is no second code path to the tree.

The CLI

dpm is argument parsing and printing. It links libdpm-core, enumerates modules, and forwards subcommands through generic dispatch. Its command surface is exactly the set of loadable modules — in the barren case that's raw's commands; on a full system, everything installed. No capability logic lives in the CLI.

External consumers

Build systems and Dark Horse components link libdpm-core.so — the same library, the same path as everything else:

• open a context (optionally against an alternate root),
• require the layer they need at a minimum version,
• fetch its API table,
• call C functions directly.

Core installs its header to the standard include path and its library to the standard lib path: a consumer writes #include <dpm/core.h>, links -ldpm-core, and calls package manager module functions. A consumer that opens a default context (no overrides) is operating the installed package manager itself — system configuration, system module path, system tree, system locking — exactly as if it were invoking the installed dpm command, because the CLI is just another caller of the same library. Overrides redirect individual paths only when a caller explicitly sets them. Whether a consumer targets raw only (image builders that just deploy trees) or pkg (dependency-aware tooling) is their choice of require(); behavior is identical to the CLI's because it is the same implementation.

Bootstrap chain

minimal start:   dpm + libdpm-core.so + raw module      (baseline deps only)
raw installs:    sqlite3 package
raw installs:    dpm-pkg package                         (drops the pkg module .so)
now:             core discovers pkg, validates it, full management is live

Every layer is a package installed and upgraded by the layer beneath it; the package manager maintains itself with the same mechanism it offers the OS. Future modules follow the identical pattern — a repo/network module declares its requirements (pkg, a TLS library), lands as a package, and the capability appears on next discovery.

Repository structure

Modules are developed independently from each other and independently from core — one repository per module, plus the core repository. Each repo owns its source, build, and tests, and produces exactly one artifact:

• Core repository: libdpm-core.so and the dpm CLI. Contains no module code. Its test fixtures include deliberately broken stub modules for validating the loader, and one known-good stub — never a real package module.
• The exception: an info module that bundles with core, used for testing and reporting functionality of core.
• One repository per module (raw, pkg, and every future module): produces that module's .so. Links libdpm-core.so — the only cross-repo build dependency in the system — and nothing else from DPM. Peer modules never appear in a module's repository, build, or test environment; peers are runtime concerns, faked at test time (contract fakes and stub modules) and real only at distribution-level integration.

No repository can block another's development: a module builds and its full pre-integration test surface (unit, contract, module-hosting) runs with nothing present but its own checkout and an installed or vendored libdpm-core. Release coordination happens through the versioning model — minimums only — rather than through lockstep builds.

Core repository layout

include/dpm/       public headers — installed to the system include path; the
                   dpm/ directory is the consumer namespace, so an installed
                   consumer writes #include <dpm/core.h>
include/internal/  library-private headers — used only by src/, never installed
src/               implementations of the core library
src/cli/           the dpm CLI entry point
src/bundled-modules/info/  the bundled info module
data/              files installed as-is (core.conf)
tests/             fixture modules, the core API test binary, CLI tests
docs/              project documentation

Every header lives under include/: include/dpm/ is the published API surface and defines what consumers see; include/internal/ is the implementation's own headers, invisible outside the repo because the install rule ships only include/dpm/.

Artifacts

Terminology: DPM Core names the dpm CLI binary; libdpm-core names the library.

Artifact	Location on system
dpm	/usr/bin/dpm
libdpm-core.so	/usr/lib/libdpm-core.so
info.so	/usr/lib/dpm/modules/info.so
modules (raw.so, pkg.so, repo.so, source.so, ...)	/usr/lib/dpm/modules/<name>.so

Development and testing

Development works because the design has no build-time coupling between peers: nothing links against a peer module, ever. "Not all the pieces are there" is the normal, permanent condition at build time. What remains resolves into four test layers, each needing strictly less than the full system.

What a build requires

• A module compiles against its own declarations (written to the spec — the externs it exports, the table structs it consumes) plus libdpm-core.so, the one real link dependency — and core is by definition the stable, always-present, baseline-only piece. Cheap to have in every dev environment, trivially vendorable as a checkout.
• Peer modules are reached at runtime through core's require/get_api. Building pkg does not require raw to exist anywhere. The compile-time knowledge of raw is just the raw_api_v1 struct layout, which is spec, not artifact.

A module repo therefore builds self-contained, always.

Test layers

1. Unit tests — need nothing. The module's implementation compiles once as an object library, linked into both the .so and a test binary. Pure logic, error paths, parsing — no core, no peers.

2. Contract tests — need a struct, not a module. Because every dependency is an API table — a plain struct of function pointers — a fake is just a struct the test fills in with functions that record calls and return canned results. pkg's logic is exercised against a fake raw_api_v1 (verifying it calls install/remove/query correctly, handles raw's error codes, honors the journal-generation protocol) with raw nowhere on the machine. Injection is built into the architecture; no linker seams required.

3. Module-hosting tests — need core only. A harness links the real libdpm-core, points the module path at the build output plus fixtures, and has core load the just-built .so exactly as production would — full five-step validation included, so contract violations fail here, in CI, not on a user's system. Where the module needs a peer, the fixture directory contains a stub module: a tiny .so exporting the reserved symbols and a fake table, which core validates and serves like the real thing. The harness then drives dpm_module_execute end to end against fixture config and data. This layer runs on a bare builder with nothing installed.

4. Integration — the only layer that needs everything, and it builds itself. Real core + real raw, then the actual bootstrap chain into a scratch root: dpm_open against an alternate root, raw installs sqlite3 and the pkg package into it, core discovers pkg, real operations run against the throwaway tree. Because alternate roots are first-class in the API, this needs a directory, not a VM. Full-distribution CI does the same with real packages.

Day-to-day workflow

• Working on pkg: edit, run unit + contract tests (instant, zero environment), harness run before merge. A real raw is never needed, or even possessed, until integration.
• Working on raw: same, except its fakes point the other way — its tests need only fixture package files and a scratch tree.
• Working on core: its test fixtures are deliberately broken modules — missing symbols, wrong magic, lying manifests, too-new core-min — plus one known-good stub. Core development never needs any real package module.
• Debugging is the layer-3 harness under a debugger — it is the "run the module without the system" mechanism, so no separate standalone build exists or is maintained.

The discipline that keeps this honest: fakes and stubs are written to the spec, and layer-3 validation plus the layer-4 bootstrap run in CI, so a fake that drifts from reality is caught by the first integration pass rather than shipped.

Development capabilities

During development the CLI must be pointable at a local libdpm-core, and that core must be configurable to local paths (module path, config dir, etc.). Two mechanisms provide this:

• Pointing the CLI at a local libdpm-core is dynamic-linker territory, needing no DPM mechanism: development builds of the CLI carry an rpath to their own build tree's lib/ directory, so the locally built binary resolves the locally built core (LD_LIBRARY_PATH pointed at that lib/ directory achieves the same). The system core is never touched.
• Pointing that core at local paths is what the dpm_open overrides exist for: config directory, module path, and target root are all fields of the overrides struct, and the CLI exposes them as flags. A dev invocation:

./build/bin/dpm --config-dir ./tests/fixtures/conf --module-path ./build/modules raw install ./fixture.dpm

The config-dir override matters most: once the context reads config from the local conf dir, everything configurable — log file, module path defaults, per-module settings — resolves locally, so a checkout plus its fixtures is a complete self-contained environment. Adding --root at a scratch directory makes even real install operations land in a throwaway tree.

Rule: every field of the dpm_open overrides struct must be exposed as a CLI flag, so anything a linked consumer can redirect, a developer at the shell can redirect too. This holds for any future override field — nothing ships reachable from code but not from the command line.

Evolution rules

• Append-only ABI: breaking a module API means exporting a new table (raw_api_v2) beside the old one, never mutating v1. Old tables remain until consumers are gone.
• Append-only core contract: newer core loads everything older core did; a module's only version assertion against core is its minimum.
• Minimums only, everywhere: modules declare the minimum core they support; consumers declare the minimum module version they need. No maximums, no exact-match constraints — an update can never make a previously working combination refuse to load.

Invariants

1. Core routes and hosts; modules implement. No package logic in core, ever.
2. Writes flow down, never sideways: a layer mutates the system only through the layer beneath it, in-process through core-mediated APIs.
3. Truth lives in the tree; everything above is regenerable cache or convenience.
4. Dropping down a layer by hand is always legal; layers above detect it and reconcile.
5. A module is either fully valid or not loaded — no partial states, no consumer-side defense.