Status: 📋 Specification Complete
Domain: Analytical and Relational Computation
Position: Domain-Specific Frontend to Pantheon
Repository: Private while in development

Overview

Prism is a microkernel-based query execution system for analytical and relational computation. It demonstrates how applying operating system microkernel principles to query engine design enables competing optimization strategies to coexist while maintaining a minimal, verifiable core.

Sister Project: Like Morphogen handles audio synthesis, Prism handles analytical queries—both are domain-specific frontends to Pantheon (Universal Semantic IR).

The Microkernel Insight

Prism emerged from resolving an architectural question: we had two complete specifications for analytical query execution—Set Stack (8-layer architecture) and SEM (5-layer mesh topology). Which one should we build?

The breakthrough: They don't merge. They coexist as competing service bundles running atop the same microkernel.

┌─────────────────────────────────────┐
│  Set Stack Service Bundle           │
│  • SetLang Parser                   │
│  • Cascades Optimizer               │
│  • MLIR-based Scheduler             │
│  • Explainable Physical Strategies  │
└──────────────▲──────────────────────┘
               │
        Prism Kernel API
               │
┌──────────────┴──────────────────────┐
│  SEM Service Bundle                 │
│  • SQL Parser                       │
│  • Learned Optimizer                │
│  • Mesh Topology Scheduler          │
│  • GPU-First Execution              │
└─────────────────────────────────────┘

Just like Linux supports multiple filesystems (ext4, btrfs, xfs), Prism supports multiple query execution policies.

This insight came from applying Jochen Liedtke's minimality criterion:

"A concept is tolerated inside the microkernel only if moving it outside would prevent the implementation of required functionality."

Result: Only operator execution primitives belong in the kernel. Everything else—parsing, optimization, scheduling—lives in userspace services.

Architecture

The Three Primitives

Prism's kernel provides exactly three primitives (like an OS provides processes, memory, and IPC):

1. Operators (Computation Units)

Like processes in an OS. Stateless functions over batches.

Handle op_create(OpType type, Config cfg);
void   op_execute(Handle op, Capability input_cap);

Examples: SCAN, FILTER, JOIN, AGGREGATE, MAP

2. Buffers (Isolated Memory Regions)

Like pages in an OS. Device-aware, capability-protected.

Handle     buf_alloc(size_t bytes, Device dev);
Capability buf_grant(Handle buf, Rights rights);
void       buf_revoke(Capability cap);

Properties:
- Device-aware (CPU, GPU0, GPU1)
- Zero-copy via capability passing
- Formal isolation guarantees

3. Channels (Message Passing)

Like IPC in an OS. Asynchronous, bounded, ownership-transferring.

Handle     chan_create(Handle src, Handle dst);
void       chan_send(Handle ch, Capability data_cap);
Capability chan_recv(Handle ch);

Properties:
- Async by default (pipeline parallelism)
- Backpressure-aware (bounded buffers)
- Race-free (no shared mutable state)

The Kernel Interface

Complete kernel API: 14 functions, ~200 lines of C header

// Operators
Handle op_create(OpType type, const void *config, size_t config_len);
void   op_execute(Handle op, Capability input_cap);
void   op_destroy(Handle op);

// Buffers
Handle     buf_alloc(size_t bytes, Device dev);
Capability buf_grant(Handle buf, Rights rights);
void       buf_revoke(Capability cap);
void       buf_free(Handle buf);

// Channels
Handle     chan_create(Handle src_op, Handle dst_op);
void       chan_send(Handle chan, Capability data_cap);
Capability chan_recv(Handle chan);
void       chan_close(Handle chan);

// Introspection
TraceEvent *introspect(Handle entity, size_t *count);

That's the entire kernel interface. Simple, minimal, verifiable.

Service Bundles

Services are pluggable userspace components that provide policy (parsing, optimization, scheduling).

Set Stack Service

Focus: Explainable analytics with semantic transparency

Components:
- Parser: SetLang (pipeline-oriented DSL)
- Optimizer: Cascades cost-based optimizer
- Scheduler: MLIR-based compilation
- Features: Domain operators (TimeOps, UnitOps, HierarchyOps)

Strengths:
- Explainable optimization (PSL justifies decisions)
- Domain awareness (time-series, units, hierarchies)
- MLIR-native lowering
- SIL principle alignment

Use case: Analytics workloads requiring explainability and domain operators

SEM Service (Set Execution Mesh)

Focus: GPU-optimized performance for heterogeneous hardware

Components:
- Parser: SQL (standard)
- Optimizer: Learned (ML-guided)
- Scheduler: Mesh topology (multi-GPU)
- Features: Feedback loops (telemetry → scheduling adaptation)

Strengths:
- Mesh topology captures real dependencies (not forced layers)
- Multi-device focus (GPU clusters, heterogeneous HW)
- Performance-first optimization
- 5D hypergraph formal model

Use case: High-performance workloads on GPU clusters

Service Competition

Users choose which service bundle to use based on workload:

# Use Set Stack (explainable, domain-aware)
$ prism --service=setstack query.sql

# Use SEM (GPU-optimized)
$ prism --service=sem query.sql

# Mix and match components
$ prism --parser=setlang --optimizer=learned --scheduler=mesh query.sql

Services can evolve independently. Kernel API stays stable.

Integration with Pantheon

Prism is a domain-specific frontend to Pantheon, following the same pattern as Morphogen:

User Query (SetLang/SQL)
    ↓
Prism IR (logical operators)
    ↓
Pantheon IR (universal semantic nodes)
    ↓
MLIR (linalg, tensor, gpu dialects)
    ↓
LLVM / CUDA / ROCm
    ↓
Hardware

Integration points:
- Prism IR (logical operators) lowers to Pantheon IR nodes
- Domain constraints (TimeOps, UnitOps) map to Pantheon metadata
- Prism trace (operational telemetry) extends Pantheon provenance

Position in SIL ecosystem:

SIL (Semantic Infrastructure Lab)
 └─ Semantic OS
     └─ Pantheon (Universal Semantic IR)
         ├─ Morphogen (Audio) ✅ Production
         ├─ Prism (Analytics) ← THIS PROJECT
         ├─ TiaCAD (CAD) ✅ Production
         └─ [Other domains...]

Key Innovations

1. Microkernel Architecture

Mechanism, not policy. Kernel provides primitives (operators, buffers, channels). Services provide optimization strategies.

2. Capability-Based Security

Unforgeable handles to buffers/devices with explicit rights (READ, WRITE, EXECUTE). Prevents buffer leaks, enables zero-copy, supports formal verification.

3. Pluggable Optimizer Services

Multiple optimization strategies:
- Cascades - Cost-based exhaustive search (traditional SQL workloads)
- Learned - ML-guided optimization (repetitive workloads with patterns)
- Greedy - Heuristic approximation (real-time, low-latency)

4. Explainable Physical Strategies

Optimizer justifies decisions with cost models. Users see why a plan was chosen:

Query Plan (cost=145.3):
  Sort (cost=12.1, rows=50)
    -> Filter hypot(x,y)<100 (cost=8.2, rows=50)
      -> Project (cost=6.0, rows=60)
        -> Index Scan on customers.region (cost=119.0, rows=60)

Optimizations Applied:
  1. Pattern: sqrt(x²+y²) → hypot(x,y) (algebraic)
  2. Index: customers.region (selectivity: 30%)
  3. Pushdown: region='US' AND revenue>1000

5. Message-Passing Concurrency

Race-free by construction. Operators communicate via typed messages through channels. No shared mutable state.

6. Hardware Introspection

Kernel provides hardware profile (cache sizes, memory bandwidth, disk IOPS). Optimizer uses this for accurate cost estimation.

7. Competing Service Bundles

Set Stack vs SEM demonstrates flexibility. Users choose based on benchmarks for their workload. Services compete, kernel stays stable.

Specifications Complete

The following comprehensive specifications are complete (~3,000 lines total):

Core Architecture

Service Designs

Design Decisions

Implementation Artifacts

Performance Characteristics

Fast Path: Operator Execution

Target: 5-10 CPU cycles per row

Optimizations:
- Branchless operator kernels
- Columnar layout (cache-friendly)
- SIMD vectorization
- Zero-copy capability passing
- Lock-free channel implementation

Slow Path: Everything Else

Parsing, optimization, scheduling are not on the critical path. They can be 1000x slower and not affect query latency.

Engineering focus: Optimize the fast path (execution), keep slow path simple.

Design Principles Alignment

Prism embodies SIL's core principles:

1. Semantic Transparency

Explicit meaning at every layer:
- Set Stack: Explicit at semantic, logical, physical levels
- SEM: Explicit in strategy justification
- Prism kernel: Services provide explain() API

2. Provenance

Traceable transformations:
- Kernel introspect() API provides trace events
- Services build provenance graphs from traces
- Integration with Pantheon provenance model

3. Explainability

Inspectable decisions:
- Optimizer explains why plans were chosen
- Cost models are transparent
- Observable intermediate states

4. MLIR-Native

Unified compilation path:
- Set Stack: Layer 4 (MLIR lowering)
- SEM: Layer 4 (device execution)
- Backend service responsibility

5. Structured Extensibility

Plugin models with contracts:
- Operator registration API
- Service plugin interface
- Domain modules (TimeOps, UnitOps)

Current Status & Timeline

Specifications Complete ✅

All architectural specifications are complete and comprehensive:
- Microkernel design finalized
- Service interfaces defined
- Optimizer strategies specified
- Integration with Pantheon designed

Next Phase: Implementation

Timeline: 6-12 months to working prototype

Milestones:
- Week 1-3: Kernel implementation (operators, buffers, channels)
- Week 4: Minimal service (SELECT * FROM t WHERE x > 10)
- Week 6-9: Optimizer service MVP
- Week 10-12: Full Cascades optimizer
- Week 13+: SEM service, benchmarks, Pantheon integration

Current work:
- Implementation planning
- Service interface prototyping
- Pantheon IR integration design

Relationship to Other SIL Projects

Pantheon (Universal Semantic IR)

Integration: Prism IR lowers to Pantheon IR for universal representation

Morphogen (Audio Synthesis)

Sister Project: Same pattern - domain-specific frontend to Pantheon
(Morphogen = audio domain, Prism = analytical domain)

GenesisGraph (Provenance)

Integration: Prism introspection API enables provenance tracking for query lineage

TiaCAD (CAD)

Potential Integration: Query geometric data, analyze CAD models with Prism analytical operators

Research Themes

1. Microkernel Architectures for Domain Engines

Question: What belongs in the kernel vs userspace?
Answer: Apply minimality criterion - only execution primitives in kernel

2. Competing Service Bundles

Question: How do different optimization strategies coexist?
Answer: Services compete on benchmarks, users choose per workload

3. Cross-Domain Composition

Question: How do analytical queries integrate with other domains?
Answer: Via Pantheon IR - query audio data from Morphogen, geometric data from TiaCAD

4. Formal Verification of Query Engines

Question: Can we prove query execution correctness?
Answer: Small kernel TCB enables formal verification (seL4-style)

References

Microkernel Design

Query Optimization

Capability Systems

Learn More

Document Version: 1.0
Last Updated: 2025-12-07
Maintainer: Semantic Infrastructure Lab