E-16 Method Design — Dual-Track Analysis of the Curated Dataset

Status: APPROVED (2026-06-20) — building. Decisions: (1) freeze table, (2) LLM trials ok,

(3) start series_deterministic, (4) Material Palette + Medium Library + Prompt Template first,

(5) trial buttons on both pages.

Date: 2026-06-20

Author: grounded in a direct LLM read of real samples (12 Track-B prompts, 35 Track-A snapshots)

Dataset: 592 curated Emerge selections / 587 unique series

0. Why this document exists

Before we build extraction code, we fix:

1. What the data actually looks like (read, not guessed).

2. Two analysis scenarios — one for all 592 (series context), one for the 127 with prompts.

3. Where the work runs — on an isolated frozen copy, never the live tables.

4. What derivatives we produce — the recipes/templates that answer the task.

5. How we validate — trial tests before committing to a production method.

This is the artifact we approve together. Code follows approval.

1. Reading findings (real samples)

1.1 Track B — the prompt is a consistent GRAMMAR

I read 12 full DALL-E prompts (avg ~2650 chars). They are not free text — they follow a

repeatable slot structure. Every prompt opens with a medium declaration, then a palette,

then composition, materials, recurring motifs, negative constraints, and an

emotional contract. Observed slots:

Key insight: in the news_pulse series the THESIS is held nearly constant

(palm/glass becomes an interface) and the MEDIUM is the primary variation axis

— mycelium bloom → pressed-flower collage → embroidery → copperplate → obsidian schematic →

jacquard tapestry → blueprint → 1960s engineering manual. So Track B is, in effect, a record of

"how Emerge re-renders one concept across many media." That is a directly reusable recipe:

*hold concept, vary medium/style, keep composition discipline + paradox + negative constraints.*

1.2 Track A — the snapshot is a structured ONTOLOGY

I read 35 snapshots. Each Snapshot.context_text is JSON with:

phase, seismic events, tides, market mood), 35/35

transformation**, 35/35

Real material strings: "patinated bronze with black wax bloom", "blown glass with millefiori cane

and trapped air seeds", "stoneware with ash-run glaze and volcanic drip", "gypsum alabaster dusted

with ghost pigments", "graphite dust on ivory wove paper with pressed erasure sheen".

Key insight: Track A already contains a semi-structured material/form dictionary for the

full 592, plus the provenance (world_summary → which real-world signals seeded each series).

We can mine materials, forms, scales, concepts, and the signal→imagery mapping without any prompt.

1.3 What this means for the task

The task = turn selections into reproducible recipes across 6 categories

(Idea, Scene, Material, Composition, Palette, Config).

Track B (127 prompts, parseable grammar).

B = "how it was rendered into a prompt".

2. Working copy — isolation from live data

2.1 Answer to "do we copy the DB?"

Today the originals are already read-only in this flow — CuratedSelection is a separate

table, and we only READ Generation / Snapshot. Nothing here writes to source tables.

But a frozen working copy is still the right call, for four reasons:

1. Reproducibility — agents keep generating; the live dataset shifts under us. A freeze pins

"the 592 as of date X" so every experiment is comparable.

2. Isolation — analysis, cleaning, and annotation never risk touching production rows.

3. Performance — pre-extracted fields (prompt slots, entities) avoid re-parsing 592 traces each run.

4. Versioning — we can freeze v1, refine the method, freeze v2, and diff.

2.2 Proposed table — `CurationDatasetSnapshot` (DESIGN ONLY)

> DB migration requires explicit approval (factory taboo-approval rule). Not created yet.

class CurationDatasetSnapshot(Base):
    __tablename__ = "curation_dataset_snapshot"
    id                = Column(Integer, primary_key=True)
    freeze_id         = Column(String, index=True)   # uuid per freeze run
    freeze_label      = Column(String)               # e.g. "emerge-2026-06-20-v1"
    track             = Column(String)               # "a" | "b"
    generation_id     = Column(String, index=True)
    snapshot_id       = Column(String, index=True)
    agent_id          = Column(String)
    created_at_src    = Column(DateTime)             # original generation time
    tier              = Column(String)               # A | B | C at freeze time
    # frozen payloads (denormalized copies — never reference live rows for analysis)
    dalle_prompt      = Column(Text)
    scene_director    = Column(Text)
    distiller_brief   = Column(Text)
    artistic_statement= Column(Text)
    moment_text       = Column(Text)
    world_summary     = Column(Text)
    emotional_intent_json = Column(Text)             # list
    visual_ontology_json  = Column(Text)             # entities[]
    # parsed Track-B slots (filled by the prompt parser)
    prompt_slots_json = Column(Text)                 # {medium, palette, composition, materials, motifs, constraints, emotion}
    image_path        = Column(Text)
    frozen_at         = Column(DateTime, default=utcnow)

2.3 Freeze process

POST /api/curation/dataset/freeze {track, label}
  1. resolve approved policy subset (Track A = 592, Track B = 127)
  2. for each generation: read live Generation + Snapshot ONCE
  3. parse Track-B prompt into slots (parser in §4.1)
  4. write rows into curation_dataset_snapshot with a new freeze_id
  5. return {freeze_id, count}
All extraction/experiments then read ONLY from the frozen snapshot, never live tables.

3. Track A method — Series Context Analysis (all 592)

Input: frozen Track-A rows (snapshot_context).

Goal: recipes for Idea, Material, and provenance (signal→imagery).

Pipeline:

1. Entity mining — flatten all visual_ontology.entities; group materials, forms, scales.

Output: ranked material vocabulary, form vocabulary, scale distribution.

2. Concept mining — cluster artistic_statement + concept fields into thematic families.

3. Emotion mining — frequency of emotional_intent phrases; map emotions → materials/forms.

4. Provenance mining — parse world_summary into signal types (museum, news, astronomical,

seismic, market) and correlate which signals precede which materials/concepts.

5. Derivatives (§5).

Methods:

4. Track B method — Prompt Grammar Analysis (127)

Input: frozen Track-B rows (dalle_prompt + scene_director + distiller).

Goal: recipes for Composition, Palette, Medium/Config, paradox devices, negative constraints.

4.1 Prompt parser (the core new component)

Deterministic segmentation into the 8 slots from §1.1:

negative space, horizon-split, diagonal, rule of thirds…).

Parser writes prompt_slots_json on the frozen row (computed once at freeze).

4.2 Analysis

1. Slot frequency — top mediums, palettes, composition devices, materials, constraints.

2. Co-occurrence — which medium pairs with which palette/composition (e.g. "obsidian schematic"

→ "arctic blue", "negative space"). This is the high-value signal.

3. Variation-axis detection — per series, which slot varies while others are held constant

(we already see MEDIUM is the variation axis in news_pulse).

4. Derivatives (§5).

Methods: deterministic (free), embedding (cluster prompts), llm (M3 synthesis).

5. Derivatives — what we actually hand back

Beyond raw frequency cards, the task wants reusable artifacts:

These are exported as JSON + rendered as Insight Cards, and (later) can feed Scene Director presets.

6. Trial test plan (before production)

Run small, cheap, real tests to build confidence — no full commit yet:

1. Track A deterministic on a 50-series sample → eyeball material/concept rankings.

2. Track B parser on the 12 already-fetched prompts → verify slot extraction accuracy

(manual check: did "medium" capture the style anchor every time?).

3. Track B deterministic on all 127 → top mediums/palettes/compositions.

4. One LLM run per track on ~15 samples → judge recipe quality vs deterministic.

5. Scorecard compare (existing harness) → pick production method per track.

Acceptance: parser ≥ 90% correct medium-slot on manual check; LLM cards rated more actionable

than deterministic on ≥ 4/6 categories; no live-table writes.

7. Open decisions (need your approval)

1. Freeze table — approve creating curation_dataset_snapshot (DB migration)?

Alternative: freeze to a JSON file under memory/ (no schema change). Recommend: table.

2. LLM budget — allow M3 (Claude) trial runs on ~15 samples/track (~low $)? Or deterministic-only first?

3. Track A method — start with series_deterministic only, add series_llm after trial?

4. Derivatives priority — which of the 8 derivatives in §5 do we build first?

(Recommend: Material Palette + Medium Library + Prompt Template.)

5. Page surface — keep this as /curation/method-design (read-only doc) + run trials from

/curation/experiments, or add trial buttons directly on the design page?

8. Plain-language glossary