replace nob with ms' build, move disk creation to build.bat, major comment cleanup, remove old and stupid tests, overhaul interrupt management with per-CPU vector allocation, affinity migration and safe teardown, add proper IOAPIC routing and PCI MSI/MSI-X/INTx support, migrate drivers to the new interrupt API, integrate ACPICA with a proper OS services layer, move ACPI firmware events, power button handling and S5 shutdown into acpi.sys, add parallel driver startup and atomic publication, replace PnP arrays with dynamic lists, redesign service startup again to be asynchronous, add delayed startup mode for services, allow omitting keys in services.ini, fix acpi ec deferred work handling, fix kernel threads having garbage exit code.

roadmap.md

@@ -1,50 +1,33 @@
 # nt-lite — remaining roadmap
 
-**Status:** working plan; remaining implementation, integration, and validation only.
-**Scope:** lean-boot verification, the Linux-derived USB/SCSI/HID/input replacement, parallel driver/services startup, then deferred native filesystem/security, POSIX, desktop, personality, debugging, graphics, and driver compatibility work.
+**Status:** lean-boot and cleanup work completed; Phase 2 remains open; Phase 3 startup work implemented, with service administration and release validation remaining.
+**Scope:** Linux-derived USB/SCSI/HID/input replacement, final native service control, then deferred filesystem, security, POSIX, desktop, loader, debugging, graphics and driver compatibility work.
 
-## Current state and evidence
+## Completed foundation (brief)
 
-- **Latest boot result:** user reports roughly **8 seconds**, **with tests enabled**. This is a substantial improvement over the earlier approximately 96-second ThinkPad run. Exact build flags and measured interval were not supplied; retain the report as an observed result, not a verified per-stage benchmark or proof that all regressions are fixed.
-- **Implemented foundation:** POSIX P0–P7 (including server-owned `execve()` and `vfork()`), native UABI.1–UABI.4 cleanup, `drivers.ini` v2 class/exact-dependency parsing and driver-state management, driver/device prepare/commit shutdown, and prior USB/display shutdown-lifetime fixes. Preserve and test these contracts rather than rewriting them.
-- **Lean-boot work:** a patch implementing reduced/gated boot checks, deletion of the unconditional SMP scheduler self-test, central monotonic `DbgPrint` timestamps, lightweight default debug flags, and opt-in test packaging was prepared (`ntlite_lean_boot_after_usb_lifetime.patch`). Before marking every component integrated, compare it against the *actual current tree*; the user-reported eight-second result is not a substitute for checking what was applied.
-- **Graphical system:** native DWM/surfaces and DOOM integration work, but native console/window management, shell policy, and restart semantics are still incomplete.
-- **Desktop PC unresolved:** AMD xHCI `1022:43d5`/`1022:149c` stalls with both external hubs connected. Latest supplied build marker `phase2-usb-rundown-20261022` reached `root hub started ... scan=queued` and `initial scan wake returned` for `28:00.3` before the log ended; source patches have yet to establish a hardware fix.
-- **ThinkPad T14 Gen 5 (AMD, machine `21MDS30F00`):** four xHCI PCI functions (`1022:15b9`, `15ba`, `15c0`, `15c1`), recurring USB control-transfer timeouts and hub-attached child descriptor failures. A directly connected `09da:2268` boot keyboard works despite a secondary nonboot HID interface returning PnP unsupported; the built-in keyboard/TrackPoint provide no confirmed input events. Linux identifies i8042/serio keyboard and TrackPoint, plus `ELAN0676` HID-over-I²C on `AMDI0010:01`. The Unifying receiver (`046d:c52b`) has boot mouse capability, so report parsing alone does not explain its missing input. ACPI fixed power button is unavailable and AML event handling remains open.
-- **No claim of a real integration test** follows from syntax checks, patch apply checks, or host models. Use the current real-hardware logs and fresh checkout for the next change.
+- **Boot and build:** lean boot with gated checks, timestamped diagnostics, opt-in tests, historical BUILD.EXE integration, and VHD packaging. Six correctness, structure, configuration, logging and test-pruning passes completed.
+- **Drivers:** dynamic PnP registrations, two-phase load/start and rollback, exact/class dependency settlement, bounded parallel driver startup and kernel-thread exit synchronization. Keep module readiness distinct from PnP device readiness.
+- **Services:** strict `services.ini` with `[session]`, `auto|delayed|disabled`, `requires`/`after`, concurrent READY settlement, retry budgets for launch/READY/exit failures, named privileges, capability grants, and generic RSSRV bootstrap authority. Automatic services settle before the shell; delayed services follow it.
+- **Existing subsystems:** POSIX P0–P7, native UABI.1–UABI.4, native surfaces/DWM, and driver/device shutdown contracts.
+
+## Hardware still needing work
+
+- **Desktop AMD:** xHCI `1022:43d5`/`1022:149c`, especially with both external hubs attached.
+- **ThinkPad T14 Gen 5 AMD:** xHCI `1022:15b9/15ba/15c0/15c1`, hub/control timeouts, internal i8042 keyboard/TrackPoint, `ELAN0676` over `AMDI0010:01`, Unifying `046d:c52b`, and AML power-button handling. The direct `09da:2268` boot keyboard has worked in prior tests.
 
 ## Operating decisions
 
-1. Favor one clean owner and one native interface per concern. Breaking obsolete APIs is acceptable; delete stale callers and compatibility paths as part of a change.
-2. **Normal boot is not a regression harness.** Runtime failure checks remain mandatory, but deterministic regression, synchronization stress, and intrusive diagnostics must be explicit. Avoid creating a new standalone test for every feature.
-3. Use centrally timestamped `DbgPrint` records to find waits before adding profiling frameworks or blindly parallelizing work. Times must be monotonic elapsed values, prefixed once per logical line in both console/file output, safely under concurrent writers. Pre-clock logging may remain unprefixed.
-4. No permanent vendor/model shortcut in place of standard hardware protocols. Use imported Linux state machines where practicable; otherwise implement the narrowest native component to Linux/spec behavior, with tests and recorded provenance. Protocol-defined fallback remains legitimate.
-5. The driver/module dependency graph and the runtime PnP device topology are distinct. `requires` needs provider **success**; `after` needs provider **settlement**, regardless of success. Reverse dependency order governs graceful shutdown.
-6. The user-mode service manager should be *generic*: **RSSRV is an ordinary critical service with declared bootstrap resources and authority**, not `type=resource` and not selected by hard-coded executable name. No service gains privileges merely because it uses a familiar image/ID.
-7. A service's privilege mask and granted capabilities are separate: use minimal named ambient privileges where unavoidable; delegate scoped handles for actual resources. The current shared-address-space/SASOS model does not promise hostile-user-process memory isolation.
-8. Prioritize a usable, correct system. Do not keep testing, scaffolding, or instrumentation merely because it already exists. Preserve only focused coverage for genuinely subtle regressions.
+1. One owner and one native interface per concern. Delete obsolete callers and compatibility paths during API changes.
+2. Regression and stress tests are opt-in; production boot retains necessary runtime failure checks.
+3. Use monotonic `DbgPrint` timestamps to measure actual stalls. Preserve concurrent, logical-line-safe output.
+4. Prefer Linux-derived standard hardware state machines to vendor-specific shortcuts; retain provenance and applicable licensing obligations.
+5. Keep the driver graph distinct from runtime PnP: `requires` waits for success, `after` for settlement; shutdown reverses dependencies.
+6. RSSRV is an ordinary critical service with explicitly delegated bootstrap authority, independent of its image or ID. Ambient privileges and capability handles are separate; SASOS implies limited isolation.
+7. Keep instrumentation and tests only where they protect important behavioral contracts.
 
 ---
 
-# Active modernization: three phases
-
-## Phase 1 — Close lean boot and make delays observable
-
-**Status:** substantially progressed; verification and any measured residual stalls remain.
-
-**Work**
-
-- Verify that the current source contains the intended self-test cleanup:
-  - Rename `KiRunCoreHardeningSelfTests()` to a modest `KiCheckBootState()` (or equivalent), containing only cheap indispensable consistency checks, gated by the existing explicit diagnostic flag.
-  - Remove `KiRunEarlyKernelSelfTests()` and its unconditional call; do not run 64 rounds of AP-affine scheduler workers as part of boot.
-  - Heavily prune debug-only `KiRunKernelSelfTests()`; retain at most focused, uniquely valuable concurrency/memory tests (the prepared patch keeps APC wait-race and COW coverage). Remove in-kernel `STRINGVIEW`, trap-frame, filesystem/path, async-file, extensive scheduler, PFN/DMA/PTE/VAD and similar redundant tests instead of blindly relocating everything.
-  - Keep actual failure handling and initialization assertions where correctness requires them. Compile-time ABI layout checks belong in compile-time assertions.
-- Verify the central `DbgPrint` monotonic timestamp implementation, across logical lines and persisted `kernel.log`; guard early/pre-clock and concurrent output without allocation or blocking hot paths.
-- Default to light diagnostics (`debug=0x004` for file logs, optionally `0x005` with console mirroring); reserve intrusive flags and `0x002` regression diagnostics for explicit investigations. Do not assume a user must always use one particular mask.
-- Normal build must not compile/package the fleet of test executables into `/tests` by default. The proposed build option is `--with-tests`. Prune duplicate/low-value tests selectively; preserve a runnable development suite.
-- Use timestamps to distinguish firmware/EFI loading, CPU admission, PCI enumeration, storage readiness, USB timeouts, and service READY latency. Measure cold boots on both QEMU and real machines. Profile 256-bus PCI probing before replacing it; prefer real bridge-topology discovery when justified, while preserving access to buses `c4` and `c6`.
-
-**Done when:** ordinary boot invokes no regression/stress suite; debug-mode tests are opt-in; logs identify meaningful delay intervals; production images exclude test programs by default; the ~8-second result is reproducible with a defined start/end point and actual flags. This phase is **not** blocked on achieving an arbitrary sub-second boot time.
+# Active work
 
 ## Phase 2 — Linux-derived USB, SCSI, block, HID and input replacement
 
@@ -80,67 +63,24 @@
 
 **Phase 2 acceptance:** QEMU boots and enumerates keyboard/tablet/storage; both real machines boot reliably with the previously failing hub topologies; direct and behind-hub control/bulk/interrupt transfers work; generic HID/Unifying input reaches RSSRV; storage survives multiple LUNs and abort/reconnect tests; native disk geometry/partition discovery is correct; shutdown/reboot drain owned DMA and callbacks. ThinkPad PS/2/TrackPoint, I²C touchpad and AML power button retain their separately tracked hardware/platform gates. A release with only syntax checks, successful module loading or direct-device enumeration leaves this phase open.
 
-## Phase 3 — Parallel graph startup and native service manager
+## Phase 3 — Parallel graph startup and native service management
 
-**Status:** dependency-graph **foundation implemented** for drivers; concurrency, service policy and remaining lifecycle work open. Its service design work may proceed in parallel with Phase 2 import feasibility, while shared module/runtime and authorization decisions are coordinated.
+**Status:** startup implementation in place; service administration and integrated validation remain before the nightly.
 
-**Driver initialization**
+**Implemented:** four-worker driver graph, dependency settlement and worker-exit synchronization; concurrent `auto` services, post-session `delayed` services, `requires`/`after`, READY/failure handling, per-service fixed-delay restart budgets, `[session]` shell selection, named privileges and scoped capability grants, and configuration-selected RSSRV bootstrap authority.
 
-- Evolve the existing `drivers.ini` v2 manager—not another parser—into a **bounded graph worker scheduler**. Start independent drivers concurrently, with exact- and class-level `requires`/`after` semantics unchanged. Deterministic eligibility and reporting do **not** require deterministic completion order.
-- Protect image loading, import/export/module registries and module-state transitions with precisely scoped synchronization. Distinguish module `DriverEntry` completion from runtime device readiness. Preserve boot-critical storage availability, PnP/device ownership, failure propagation, and reverse-order shutdown. Recheck imported xHCI/USB/SCSI module startup, AHCI, PS/2, HDA, PCI and network on SMP hardware.
-- Preserve the soft NIC-first behavior: a network stack may wait for the adapter class to **settle** (`after=network-adapter`) rather than fail if no NIC works. No file-order coupling.
-- Driver `start=demand`, unload/reload, and optional cascade reload are **later extensions**, not prerequisites to making service demand startup work. Before unloading images: block new binding, quiesce owned device stacks, drain IRPs/DPCs/timers/threads/IRQs/DMA, and prove no import/executable references remain.
+**Finish for nightly:**
 
-**Simpler `services.ini`**
+- Add an **init-owned service control interface** through existing IPC/security primitives: query status and failure/wait reason; authorized start, stop and restart; expose restart counters. Enforce critical-service protection, `requires` dependencies, and reverse-order shutdown.
+- Validate QEMU with one and four CPUs and an SMP hardware boot. Exercise worker idle wakes/joins, dependency success/failure/settlement, READY timeout, startup failure and restart exhaustion, delayed-service ordering, and RSSRV least-privilege authorization. Remove temporary diagnostics afterward.
 
-- Adopt the useful dependency model of `drivers.ini`: stable service IDs (`[service.id]`), optional display `name`, `image`, `requires`, `after`, optional classes, strict whole-file validation and cycle rejection. Defaults should eliminate mandatory noise; omit `type` entirely. Services may have no published endpoint.
-- Support `start=auto|delayed|demand|disabled`. `delayed` becomes eligible **after the interactive session is usable**, not after an arbitrary sleep. Demand services start through explicit authorized management requests and optional declared resource/connection activation. Coalesce simultaneous activations into one startup attempt; never start a disabled service through dependency resolution.
-- Startup is dependency-driven and concurrent. Launch all currently eligible services before waiting for READY; collect asynchronous READY/exit/timeout events. Do not bind the long-term manager to one fixed 32-handle `WaitMany` array. Failed `requires` blocks dependents; `after` only waits for settlement. Shutdown stops in reverse graph order and prevents new demand activation.
-- Per-service restart policy: `never|on-failure|always` plus optional retry limit, time window, initial delay, backoff scheme and maximum delay. Avoid hard-coding one 3-in-5-minutes policy for all services. Prevent restart storms, exclude deliberate stops, and define failure behavior for exhausted budgets. DWM restarts only once window/client reconstruction or explicit-loss semantics are defined.
-- Expose human-readable `name`, stable service identity, status, reason for waiting/failure, and explicit service administration. The service manager's control plane is separate from ordinary client access to a service Port.
-- Resource publication: `/Services` is an RSSRV-owned **resource namespace**, not a disk directory. Declare publication/activation endpoints explicitly where useful; the desktop session endpoint is separate (e.g. `/Sessions/0/Desktop`). Document this in a dedicated short service/namespace specification, not misleading comments in the INI.
-- **RSSRV remains generic:** configuration declares a namespace-provider role/resource contract, `critical=true`, and narrowly defined bootstrap capabilities. Init gives the designated provider the primordial namespace/bootstrap handles via a generic descriptor-driven mechanism. No magic `[service.rssrv]`, `type=resource`, or privileged image-name comparison. Resolve the one-provider/bootstrap-cycle invariant explicitly in the schema/validator.
-- Privileges: parse named `privileges` into a derived immutable service security context; separately grant handles/capabilities with scoped rights (storage, Port publication, display/session, network where needed). An ordinary database service must not inherit RSSRV authority. Enforce who may configure, activate, query, stop, or reconfigure services. Integrate later with SEC principals/ACLs rather than inventing an alternate security framework now.
-- Move the session image selection out of `[init] commandline` in `boot.ini` into init-owned configuration (for example `[session] image=...`). Define executable path separately from future command-line arguments. Keep `boot.ini` for EFI/bootstrap selection and kernel diagnostics; no Windows Boot Manager clone.
+**Deferred:** service `start=demand`, resource/connection-triggered activation and deduplication, reconfiguration, and advanced restart backoff belong to D6. Keep startup policy `auto|delayed|disabled` for this release.
 
-**Illustrative policy only; final grammar needs a small design review:**
-
-```ini
-[session]
-image=apps/shell_native.exe
-
-[service.namespace]
-name=Resource Service
-image=system/rssrv.exe
-class=resource-namespace
-critical=true
-privileges=resource-admin
-
-[service.posix]
-name=POSIX Server
-image=system/posixsrv.exe
-requires=namespace
-
-[service.dwm]
-name=Desktop Window Manager
-image=system/dwm.exe
-requires=namespace
-
-[service.mysql]
-name=MySQL Server
-image=apps/mysql.exe
-start=demand
-requires=posix
-privileges=network
-```
-
-`mysql` is an example of service policy, **not** a claim that the program already runs; its writable database storage should be delegated as a scoped resource. The example `resource-admin` and role syntax are provisional: use existing native privilege/capability vocabulary rather than freeze arbitrary new policy names.
-
-**Acceptance:** independent drivers and services visibly overlap in timestamped logs; dependency failures and shutdown remain correct; services can be auto/delayed/demand/disabled; resource activation deduplicates; per-service restarts obey their own budgets; RSSRV and ordinary services share one lifecycle mechanism but different delegated authority; configuration is shorter and self-explanatory.
+**Done when:** management authorization and lifecycle regressions pass, services reach the shell reliably, and driver workers exit cleanly on the tested SMP configurations.
 
 ---
 
-# Deferred work after the three active phases
+# Deferred work
 
 These are **still outstanding**. The previous roadmap's detailed designs should not be mistaken for implemented behavior. Resume in dependency order, not by the age of its section numbers.
 
@@ -191,8 +131,9 @@ Phase 3's service-specific privilege/capability grants must be designed to exten
 - Use PE32+ subsystem to select native (1), Win32 GUI/CUI (2/3), or POSIX marker (7). Win32 userspace bootstrap creates its PEB/TEB shim and uses generic persistent per-thread FS/GS bases; native and POSIX remain on their existing TEB layout. Do not introduce personality-specific kernel scheduler state.
 - Split `crt.dll`, `posix.dll`, and Win32-facing libraries along true ownership boundaries; add thin USER/GDI translation to native DWM/surfaces only when the native desktop is mature.
 
-## D6 — Driver on-demand lifecycle, observability, and cleanup
+## D6 — On-demand lifecycle, observability, and cleanup
 
+- Service demand and resource/connection activation: `start=demand`, authorized requests, deduplicated triggers, optional service reconfiguration and advanced restart backoff. Keep one init-owned control plane.
 - Complete module import/reference accounting and PnP-owned device/worker/DPC/IRP/IRQ/DMA quiescence before any actual unload; then exact-driver reload and optional cascade reload over the same graph.
 - Add `start=demand` for drivers, activated by PnP IDs, authorized management requests, or dependency demand, not by a parallel private graph.
 - Structured native read-only driver/class status and wait/failure causes; no requirement to scrape `DbgPrint` for management.
@@ -211,15 +152,9 @@ These are intentionally **not prerequisites** for boot, service cleanup, KFS, or
 
 # Verification and handoff rules
 
-- **Boot:** record source revision/patch set, machine, firmware, debug mask, `--with-tests` mode, wall-clock start/end definition, and timestamped kernel log. Keep cold/warm comparisons distinct. The ~8-second result is the latest user report, not a universal target or a subsystem timing profile.
+- **Boot:** record source revision/patch set, machine, firmware, debug mask, test-packaging mode, wall-clock start/end definition, and timestamped kernel log. Keep cold/warm comparisons distinct. Track reproducible cold/warm boot intervals and the actual build flags.
 - **Hardware:** validate each controller/port/hub/interface separately and capture the first failing stage. For imported USB/xHCI/SCSI/HID ensure timeout, disconnect, cancellation, DPC, I/O, remove, shutdown and reboot ownership. Include both desktop external hubs and ThinkPad `046d:c52b`/`09da:2268`, plus USB storage directly and behind hubs.
-- **Concurrency:** stress dependency cycles/failure/settlement, deterministic selection, ready-event races, service activation coalescing, cancellation and reverse shutdown—not only a clean boot. Do not restore mandatory stress tests to the boot path.
+- **Concurrency:** stress dependency cycles/failure/settlement, deterministic selection, ready-event races, worker idle wakeups and completion joins, service restart budgets, cancellation and reverse shutdown—not only a clean boot. Do not restore mandatory stress tests to the boot path.
 - **Services/security:** prove a minimally privileged database-like service cannot obtain RSSRV namespace management, display, input, raw storage or shutdown capabilities; explicit granted handles work; only authorized callers can control services.
 - **KFS/SEC/POSIX:** preserve file/provider identity across rename/unlink and native capability rights; run the focused filesystem, security and POSIX integration suite in development builds.
 - **Code baseline:** use a newly provided current source tree when starting implementation. Earlier chat archives, `code(5).zip`, and prior cumulative patches may not represent the user's latest checkout. Never call a patch 'built' when only `git apply --check`, a syntax check, or a standalone model has passed.
-
-## Next work item — `docs/services.md` contract review
-
-Review the revised service architecture against `init/service_config.{h,c}`, `init.c`, `services.ini`, `service_control.h` and RSSRV namespace/publication code. Agree the minimal grammar and authority/bootstrap contracts before changing the service ABI. First choices: how a service declares the unique namespace-provider role, how config requests audited named privileges versus capability grants, and how READY exports name resources without service-type adapters. Then implement a strict parser and deterministic graph validator, followed by event-driven parallel startup, demand activation, restart budgets and management authorization. Keep service boot independent from the Phase 2 imported USB device model: drivers remain driver-manager/PnP objects, not ordinary user-mode services.
-
-**Parallel Phase 2 first action:** pin Linux source and complete the compiler/ABI + minimal-runtime feasibility gate before large-scale driver import.