Sai Dubbaca's Design MOST RECENT

OVERALL DESIGN AREAS 1. Publish Workflow Researchers approve content and trigger a publish action that generates a snapshot of changed content with all dependencies, emitted as an event to Stream 1. 2. Event Streaming Layer Stream 1 carries the publish event. Synchronous OCI functions handle metadata reads and dashboard status responses. Asynchronous OCI functions launch OCI Batch jobs. 3. OCI Batch Processing Jobs have task-level dependencies (e.g., validation job must complete before export job). Handles write to Global Distribute V2, TTL/TriG generation, and SPARQL updates to SKG. 4. Global Distribute V2 Stores only the latest published version of data. Receives flagged writes (insert/update/delete). Source of truth for downstream consumers. 5. SKG Export Batch job calls the SKG curation API directly with generated TriG/TTL content. Replication from SKG into EHRC for clinician use. 6. VantageRx / Shadow Testing Periodic package generation aligned to Gen 1 release cadence. Automated diff comparison between Gen 1 and Gen 2 artifacts. Flip to Gen 2 as source after confidence is established. 7. Service Ownership / Data Isolation Each database owned by exactly one service. No direct cross-service database access. All interaction via messages or service APIs. 8. Error Handling / Dead Letter Queue Failed batch launches queue to DLQ. Operators monitor stuck workflows. Validation failures surfaced to researcher dashboard before and during publish. 9. Gen 1 / Gen 2 Transition Tables managed per-generation with defined source-of-truth ownership. Merge via replication/batch as Gen 2 coverage expands. OUTSTANDING ITEMS 1. Payload size limit -- 1MB OCI Streams cap is insufficient for PEL files (~280KB each) and SNOMED-scale content. Options: object store reference pattern, or platform ticket to raise the limit. Decision pending. 2. Cascade change capture -- When a new NDC causes downstream obsolescence of other records (e.g., drug names), the mechanism for capturing and pushing those secondary changes to SKG is not defined. 3. Write-back / versioning model -- Agreed no write-back from pipeline to Global Research. Versioning approach (V1/V2 publish versions visible in research UI) needs to be fleshed out. 4. Research service snapshot responsibility -- Exactly what data gets included in the publish event per clinical area (which tables, which dependencies) needs definition and buy-in from Grace and Scott. 5. Package inventory -- Full list of output packages requiring generation and shadow testing (beyond VantageRx and CMCD) needs to be pulled from Drew/Brian Retz. 6. TTL/TriG incremental generation -- Current scripts do full regeneration. Incremental delta generation needed for near-real-time model. George's parallel effort to automate this -- Michael has a session scheduled. 7. Permission model -- Who can trigger a publish needs to be defined and implemented in the Research Web App layer. 8. Use case stress tests -- Michael and Chandra to run specific clinical area scenarios through the design to validate no overlooked edge cases before PDRC review. 9. SPA vs. traditional web app -- Technology decision for Research Web App UI not finalized. 10. PDRC baseline -- Design review and committee sign-off pending completion of the above.