dpm-core-ng/docs/DESIGN.md

DPM — Dark Horse Package Manager: Design

Constraints

• Must be able to operate in a barren environment providing libc and libstdc++ — the standard build runs there as-is, no special variants. On a fully populated system the same binaries simply have more modules loadable.
• Capability grows in layers: each layer installs the dependencies of the next using only what already works.
• Every implementation exists exactly once, and is consumable by the dpm binary, by other layers, and by external programs (build systems, Dark Horse tooling) through C interfaces.
• No module ever links, includes, or hardcodes anything belonging to another module. Every module-to-module interaction passes through libdpm-core.so.

Architecture Overview

the dpm binary   ->  <dpm/core.h>  ->  module A  ->  <dpm/core.h>  ->  module B
a build system   ->  <dpm/core.h>  ->  module A  ->  <dpm/core.h>  ->  module B

<dpm/core.h> is the declared interface; libdpm-core.so implements it. A caller includes the header, links -ldpm-core, and asks the library to discover and invoke modules by name. A module reaching a peer performs the identical two steps its own caller performed, which is why the chain above repeats rather than branching into a second mechanism.

The header names no module and carries no module-specific type. It offers discovery and interaction, and that is the entire vocabulary — which is what allows the dpm binary, a build system, and a module to all use it without any of them being privileged.

One Process, One Library Instance

The whole chain executes in the caller's address space. The dpm binary (or the build system) is the process; libdpm-core.so is mapped into it; every module the library loads is mapped into it as well. Calls are direct function calls — no subprocess, no serialization, no output parsing.

A module links libdpm-core.so like any other consumer, and when the module is loaded the dynamic linker binds it to the copy already mapped in the process. There is one instance, so a module drives the same context, module registry, configuration, and log targets the original caller opened.

Modules are loaded with RTLD_LOCAL, so a module's symbols never enter the global namespace. The only symbols a module can resolve are libdpm-core.so's. No path from one module to another exists that does not pass through the library.

Versioning Model

Every version question is settled by the party that has to live with the answer. libdpm-core.so reports; it rules on nothing.

• A module determines its own compatibility with the library. It is built against the system-installed libdpm-core.so and is responsible for being correct against it. dpm_core_version() reports the running version to any module that needs to act on it, and that module proceeds or fails on its own judgement.
• Consuming modules judge module versions. A module that depends on another requires it by name, reads the reported version, and decides for itself whether that version is suitable for the commands it intends to issue. A load is a statement that the module is valid, never that it is compatible with a particular caller.

libdpm-core.so

Dependencies: libc, libstdc++, libdl. Never more — the library must remain loadable in the barren case forever, so package logic never leaks into it. libdpm-core.so routes and hosts; modules implement.

It provides:

• Discovery: module path resolution, enumeration of installed module .so's.
• Validation: the full load-time contract enforcement described below. libdpm-core.so is the sole authority on what a valid module is; the contract definition lives inside the library as data. There is no SDK package — the interface is specified by this document and enforced by the library's validator.
• Routing: dispatch a command string with arguments to a named module. This is the only path into module code, and it is the same path for the dpm binary, for a build system, and for a module calling a peer.
• Version reporting: require resolves a module by name, loads it, and returns a handle, or reports precisely why it can't. The loaded module's version is readable from the handle, for the caller to judge.
• Common services: configuration access (per-module namespaces from /etc/dpm/conf.d/), logging, module-path queries.

The C API

All functions are extern "C". All returned strings are owned by the library (or by the module that produced them), are valid until the context is closed, and are never freed by the caller. All functions returning int use 0 for success and nonzero error codes; details of the most recent failure are retrievable per-context.

Context Lifecycle

dpm_ctx* dpm_open(const dpm_open_overrides* overrides) Creates a context. Reads configuration from /etc/dpm/conf.d/ (or the config directory named in overrides), resolves the module path (overrides take precedence over config, config over the built-in default), and initializes logging per configuration. Performs no module loading. Returns NULL only on allocation failure or an unreadable/invalid explicit override; a missing config directory is not an error — defaults apply. overrides may be NULL, and may specify: config directory, module path, target root (for chroot/image/sysroot operation), and log level. Multiple simultaneous contexts with different roots are legal.

void dpm_close(dpm_ctx* ctx) Releases the context: unloads every module handle it issued, closes log targets, frees all memory owned by the context. All handles and strings obtained through the context are invalid after this call. NULL is a no-op.

Module Acquisition

dpm_module* dpm_require(dpm_ctx* ctx, const char* name) Resolves the module name in the module path and runs the full load-time validation sequence (see Load-Time Enforcement) if the module is not already loaded in this context. On success returns a module handle owned by the context (repeated calls return the same handle — modules are loaded at most once per context). On failure returns NULL and records the precise reason: not found, or validation step failed with the step and detail. No version criterion is applied here; compatibility is the caller's to determine from the reported version.

int dpm_module_info_of(dpm_ctx* ctx, dpm_module* mod, dpm_module_info* out) Fills out with the loaded module's name, version, and description exactly as they were read at load (string pointers valid until context close). This is how a consumer obtains the version it will judge. The library attaches no meaning to the values. Returns 0 on success, nonzero if the module cannot be reported on.

int dpm_execute(dpm_ctx* ctx, dpm_module* mod, const char* command, int argc, char** argv) Dispatch: invokes the module's dpm_module_execute with the context, command, and the argument vector. Returns the module's return value verbatim (0 = success). The library adds nothing to the call besides delivery; argument semantics beyond "argv[0] is the command" are the module's to define.

This is the entire path into module code. A caller addresses a module by name and a capability by command string, so it holds no compile-time knowledge of the module it is calling — no headers, no struct layouts, no symbols. That is what allows a module to be developed, built, and tested with no peer present.

Enumeration

dpm_cursor* dpm_list_modules(dpm_ctx* ctx) Scans the module path and returns a cursor over all valid modules (each candidate .so is validated on first scan; failures are logged and excluded). Returns NULL on an unreadable module path.

int dpm_cursor_next(dpm_cursor* cur, dpm_module_info* out) Advances the cursor. Fills out with the next module's name, version, and description (string pointers valid until context close). Returns 0 and fills out while entries remain; returns nonzero at end.

void dpm_cursor_free(dpm_cursor* cur) Releases the cursor. NULL is a no-op.

Services (Available to Modules and External Consumers Alike)

const char* dpm_core_version(void) Returns the library version as a static X.Y.Z string. Callable without a context.

const char* dpm_config_get(dpm_ctx* ctx, const char* module, const char* section, const char* key) Returns the configured value for key in section of the named module's config namespace (/etc/dpm/conf.d/<module>.conf; the namespace "core", from core.conf, is the library's own). Returns NULL if unset. String valid until context close.

void dpm_log(dpm_ctx* ctx, int level, const char* message) Writes message at level (FATAL=0, ERROR=1, WARN=2, INFO=3, DEBUG=4) to the context's configured log targets (console and/or file). Messages above the configured level are dropped. NULL message is a no-op.

const char* dpm_module_path(dpm_ctx* ctx) Returns the resolved module directory path for this context.

const char* dpm_last_error(dpm_ctx* ctx) Returns a human-readable description of the most recent failure recorded on this context, or NULL if none. Overwritten by the next failing call on the same context.

Module Contract

A module is one .so in the module directory. It includes <dpm/core.h>, links -ldpm-core, and exports the following reserved symbols as extern "C". Returned strings are static or module-owned, non-NULL, and valid for the lifetime of the loaded module; the library and consumers never free them.

int dpm_module_execute(dpm_ctx* ctx, const char* command, int argc, char** argv) The module's command entry point, and its entire functional surface. ctx is the host context that dispatched the call — the module reaches every service (dpm_log, dpm_config_get, dpm_module_path, ...) through it, and reaches peer modules through it as well. command is the subcommand name (equal to argv[0] when argc > 0); argc/argv are the remaining CLI-style arguments. NULL or empty command must behave as the module's help command. Returns 0 on success, nonzero on failure. It must be callable immediately after load with no other setup.

const char* dpm_module_version(void) Returns the module's own version as an X.Y.Z string. Must be constant for the life of the module. This is the value the library reports to consumers, and the value they judge compatibility against.

const char* dpm_module_description(void) Returns a one-line human-readable description, used in module listings.

Version compatibility with the library is the module's own to determine. A module is built against the system-installed libdpm-core.so and is responsible for being correct against it. Where it needs to act on what it is running under, dpm_core_version() reports the running version and the module decides what to do with that.

Symbol naming: functional exports are prefixed with the module's name (raw_*, pkg_*); the dpm_ prefix is reserved for the contract and for libdpm-core.so.

A module publishes no headers to other modules. Its capabilities are addressed by command string through dpm_execute, so nothing about its internals — types, struct layouts, symbol names — is ever compiled into a caller. A module's documented command vocabulary is its interface.

Load-Time Enforcement

libdpm-core.so is the sole authority on module validity; the contract above is enforced by its validator, not by any SDK. Validation is all-or-nothing; a module is registered only after passing every step:

1. Resolve all reserved contract symbols. Any missing → refuse, log the exact list, dlclose.
2. Probe the cheap calls. dpm_module_version() and dpm_module_description() are invoked immediately; NULL or malformed returns → refuse.

Failures happen at install/load time, loudly and itemized. Consumers never receive a partially valid module: if a handle was handed out, the contract already validated.

Module: raw — The File-Based Installer

Ships with the base system alongside libdpm-core.so. Depends on the baseline only; archive decompression is vendored in, and the package format is chosen to keep that small. This is what makes barren-environment operation possible: libdpm-core.so plus raw function with nothing else present.

• Owns the backing tree (/var/lib/dpm/): one directory per installed package holding manifest, metadata, and hooks. The tree is the database at this layer.
• Operations, addressed as commands: install a package file, remove, verify, and queries answered by walking the tree — slow but always correct, zero dependencies.
• Owns the lock file and a transaction journal with a generation counter. Every mutation in the entire system ultimately passes through raw, so locking and journaling are implemented exactly once and inherited by every layer above.

Module: pkg — The Full Package Manager

Ships as a package, installed by raw once sqlite3 is installed. Requires raw — by name, through libdpm-core.so, judging raw's reported version itself — and libsqlite3.

• Never touches the tree directly: every filesystem mutation is a dispatch to raw through the library, in-process — shared locking, real error propagation, no output parsing. pkg is built with no knowledge of raw beyond its name and its command vocabulary.
• The sqlite database is a derived cache under one invariant: it contains nothing that cannot be rebuilt by scanning the tree. It records the last journal generation it applied; on open, if the tree is ahead (someone used raw directly — explicitly allowed, that's the escape hatch for broken systems), it replays or rebuilds. Self-healing by construction.
• Adds what the cache enables: fast queries, dependency resolution against the installed set, multi-package transactions with rollback.
• Consumers that want dependency-aware operations dispatch to pkg by name; they transitively get raw's guarantees because there is no second code path to the tree.

The dpm Binary

dpm is argument parsing and printing. It includes <dpm/core.h>, links -ldpm-core, enumerates modules, and forwards subcommands through dispatch. Its command surface is exactly the set of loadable modules — in the barren case that's raw's commands; on a full system, everything installed. No capability logic lives in it.

External Consumers

Build systems and Dark Horse components link libdpm-core.so as an ordinary shared library dependency — #include <dpm/core.h>, -ldpm-core, the same as any other library they link:

• open a context (optionally against an alternate root),
• require the module they need,
• read its reported version and decide whether it suits them,
• dispatch commands to it.

The header installs to the standard include path and the library to the standard lib path. A consumer that opens a default context (no overrides) is working against system configuration, the system module path, the system tree, and system locking — the same environment the installed dpm binary sees, because that binary is just another caller of the same library. Overrides redirect individual paths only when a caller explicitly sets them. Whether a consumer addresses raw only (image builders that just deploy trees) or pkg (dependency-aware tooling) is their choice of require(); behavior is identical to the dpm binary's because it is the same implementation.

Bootstrap Chain

minimal start:   dpm + libdpm-core.so + raw module      (baseline deps only)
raw installs:    sqlite3 package
raw installs:    dpm-pkg package                         (drops the pkg module .so)
now:             libdpm-core.so discovers pkg, validates it, full management is live

Every layer is a package installed and upgraded by the layer beneath it; the package manager maintains itself with the same mechanism it offers the OS. Future modules follow the identical pattern — a repo/network module declares its requirements (pkg, a TLS library), lands as a package, and the capability appears on next discovery.

Repository Structure

Modules are developed independently from each other and independently from the library — one repository per module, plus the libdpm-core.so repository. Each repo owns its source, build, and tests, and produces exactly one artifact:

• The libdpm-core.so repository: libdpm-core.so and the dpm binary. Contains no module code. Its test fixtures include deliberately broken stub modules for validating the loader, and one known-good stub — never a real package module.
• The exception: an info module that bundles with the dpm binary and libdpm-core.so, used to test and demonstrate full DPM system functionality where appropriate.
• One repository per module (raw, pkg, and every future module): produces that module's .so. Links libdpm-core.so — the only cross-repo build dependency in the system — and nothing else from DPM. Peer modules never appear in a module's repository, build, or test environment; peers are runtime concerns, faked at test time with stub modules and real only at distribution-level integration.

No repository can block another's development: a module builds and its full pre-integration test surface runs with nothing present but its own checkout and an installed or vendored libdpm-core.so. Release coordination happens through the versioning model — each consumer judging the versions it is handed — rather than through lockstep builds.

Layout of the libdpm-core.so Repository

include/dpm/       public headers — installed to the system include path; the
                   dpm/ directory is the consumer namespace, so an installed
                   consumer writes #include <dpm/core.h>
include/internal/  library-private headers — used only by src/, never installed
src/               implementations of the library
src/cli/           the dpm binary's entry point
src/bundled-modules/info/  the bundled info module
data/              files installed as-is (core.conf)
tests/             fixture modules, the API test binary, CLI tests
docs/              project documentation

Every header lives under include/: include/dpm/ is the published API surface and defines what consumers see; include/internal/ is the implementation's own headers, invisible outside the repo because the install rule ships only include/dpm/.

Artifacts

Terminology: the dpm binary names the command-line tool; libdpm-core.so names the library; <dpm/core.h> names its header.

Artifact	Location on system
dpm	/usr/bin/dpm
libdpm-core.so	/usr/lib/libdpm-core.so
core.h	/usr/include/dpm/core.h
info.so	/usr/lib/dpm/modules/info.so
modules (raw.so, pkg.so, repo.so, source.so, ...)	/usr/lib/dpm/modules/<name>.so

Development and Testing

Development works because the design has no build-time coupling between peers: nothing links against a peer module, ever, and nothing includes a peer's headers, ever. "Not all the pieces are there" is the normal, permanent condition at build time. What remains resolves into three test layers, each needing strictly less than the full system.

What a Build Requires

A module compiles against <dpm/core.h> and links libdpm-core.so — the one real link dependency, and by definition the stable, always-present, baseline-only piece. Cheap to have in every dev environment, trivially vendorable as a checkout.

Peers are reached at runtime by name and command string. Building pkg does not require raw to exist anywhere, and there is no compile-time knowledge of raw to acquire — not a header, not a struct, not a symbol. A module repo therefore builds self-contained, always.

Test Layers

1. Unit tests — need nothing. The module's implementation compiles once as an object library, linked into both the .so and a test binary. Pure logic, error paths, parsing — no library, no peers.

2. Module-hosting tests — need libdpm-core.so only. A harness links the real library, points the module path at the build output plus fixtures, and has it load the just-built .so exactly as production would — full validation included, so contract violations fail here, in CI, not on a user's system. Where the module needs a peer, the fixture directory contains a stub module: a tiny .so exporting the reserved symbols and answering the commands the module under test issues, which the library validates and dispatches to like the real thing. The harness then drives dpm_module_execute end to end against fixture config and data. Because a peer is addressed by name and command, the stub is a complete substitute — there is nothing else about the real peer that the module under test could have depended on. This layer runs on a bare builder with nothing installed.

3. Integration — the only layer that needs everything, and it builds itself. Real libdpm-core.so plus real raw, then the actual bootstrap chain into a scratch root: dpm_open against an alternate root, raw installs sqlite3 and the pkg package into it, the library discovers pkg, real operations run against the throwaway tree. Because alternate roots are first-class in the API, this needs a directory, not a VM. Full-distribution CI does the same with real packages.

Day-to-Day Workflow

• Working on pkg: edit, run unit tests (instant, zero environment), harness run with a raw stub before merge. A real raw is never needed, or even possessed, until integration.
• Working on raw: same, except its tests need only fixture package files and a scratch tree.
• Working on libdpm-core.so: its test fixtures are deliberately broken modules — missing symbols, a malformed version — plus one known-good stub. Development never needs any real package module.
• Debugging is the layer-2 harness under a debugger — it is the "run the module without the system" mechanism, so no separate standalone build exists or is maintained.

The discipline that keeps this honest: stubs are written to the command vocabulary the real peer documents, and layer-2 validation plus the layer-3 bootstrap run in CI, so a stub that drifts from reality is caught by the first integration pass rather than shipped.

Development Capabilities

During development the dpm binary must be pointable at a locally built library, and that library must be configurable to local paths (module path, config dir, etc.). Two mechanisms provide this:

• Pointing the binary at a local library is dynamic-linker territory, needing no DPM mechanism: development builds carry an rpath to their own build tree's lib/ directory, so the locally built binary resolves the locally built library (LD_LIBRARY_PATH pointed at that lib/ directory achieves the same). The system-installed library is never touched.
• Pointing that library at local paths is what the dpm_open overrides exist for: config directory, module path, and target root are all fields of the overrides struct, and the dpm binary exposes them as flags. A dev invocation:

./build/bin/dpm --config-dir ./tests/fixtures/conf --module-path ./build/modules raw install ./fixture.dpm

The config-dir override matters most: once the context reads config from the local conf dir, everything configurable — log file, module path defaults, per-module settings — resolves locally, so a checkout plus its fixtures is a complete self-contained environment. Adding --root at a scratch directory makes even real install operations land in a throwaway tree.

Rule: every field of the dpm_open overrides struct must be exposed as a flag on the dpm binary, so anything a linked consumer can redirect, a developer at the shell can redirect too. This holds for any future override field — nothing ships reachable from code but not from the command line.

Evolution Rules

• A module's interface is its command vocabulary. Retiring or changing the meaning of a command is a version change in the module that owns it, judged by every consumer that dispatches to it.
• Breaking the library's exported ABI means a new symbol version generation: the export set is what consumers link against, so a break is a deliberate, versioned event rather than an incidental one.
• Compatibility is decided by the party that has to live with it: a module determines whether it works with the library it is running against, and a consuming module determines whether a peer's version suits it. libdpm-core.so reports the versions it saw and enforces nothing beyond validity.

Invariants

1. libdpm-core.so routes and hosts; modules implement. No package logic in the library, ever.
2. <dpm/core.h> names no module and carries no module-specific type. It offers discovery and interaction only.
3. A module never links, includes, or hardcodes anything belonging to another module. All module-to-module interaction passes through libdpm-core.so.
4. Writes flow down, never sideways: a layer mutates the system only through the layer beneath it, in-process through library-mediated dispatch.
5. Truth lives in the tree; everything above is regenerable cache or convenience.
6. Dropping down a layer by hand is always legal; layers above detect it and reconcile.
7. A module is either fully valid or not loaded — no partial states, no consumer-side defense.