Yuxu Ge
Article · 2026-02-16

InsightFlow: The Open Cognitive Engine

开源技术白皮书 (Technical Whitepaper)

Version: 0.4.0 (Architecture Freeze)
License: Apache 2.0
Repository: 暂无

1. Vision & Manifesto

In an age of AI oversupply, summarization has become a commodity, but understanding remains expensive.

Most AI tools today—NotebookLM, ChatPDF—operate at the chat layer. They are black boxes: bound to a single model, locking user data, producing unstructured text streams.

InsightFlow is "the ffmpeg of knowledge": an open-source, model-neutral cognitive engine.

Unlike ffmpeg's deterministic transforms, InsightFlow embraces the probabilistic nature of LLM inference. The same input yields different cognitive structures across different models. We tag each output with model_fingerprint and provide quality assessment frameworks, letting users judge and choose.

We believe:

2. The Core Protocol: CSP

InsightFlow's moat is not a UI feature. It is CSP (Cognitive Structure Protocol).

CSP is an open JSON standard for describing cognitive structures extracted from unstructured data. Like LSP (Language Server Protocol) in programming, it decouples "AI reasoning" from "frontend rendering."

Design principle: Any third-party developer, given only CSP specification, can independently implement a compatible parser or renderer. This is the mark of a "frozen" protocol.

2.1 CSP Specification

CSP uses RFC-style keywords (MUST / SHOULD / MAY):

Required fields:

Optional fields:

Version negotiation:

2.2 CSP Data Structure Example

{
  "meta": {
    "csp_version": "1.0",
    "source_uri": "local://lectures/transformer.mp4",
    "source_type": "video",
    "model_fingerprint": "openai/gpt-4o",
    "generated_at": "2025-01-15T10:30:00Z",
    "elements_used": ["transcript", "keyframes"]
  },
  "knowledge_graph": {
    "root_id": "root_01",
    "nodes": [
      {
        "id": "node_05",
        "label": "Self-Attention Mechanism",
        "summary": "核心机制：计算序列中每个元素与其他元素的相关性...",
        "depth": 2,
        "timestamp_start": 124.5,
        "timestamp_end": 180.2,
        "visual_anchor": {
          "frame_time": 125.0,
          "ocr_text": "Attention(Q, K, V) = softmax(QK^T / sqrt(d_k))V",
          "bbox": [120, 80, 640, 200]
        },
        "children": ["node_06", "node_07"]
      }
    ]
  },
  "quiz_layer": [
    {
      "id": "quiz_01",
      "question": "为什么在 Self-Attention 中需要 Scale 操作？",
      "type": "conceptual",
      "linked_node_id": "node_05",
      "options": [
        "防止梯度消失",
        "防止点积结果过大导致 softmax 饱和",
        "减少计算量"
      ],
      "correct_index": 1,
      "explanation": "当维度 d_k 较大时，点积的方差也会变大..."
    }
  ],
  "quality": {
    "node_count": 15,
    "max_depth": 4,
    "quiz_count": 8,
    "coverage_ratio": 0.85
  }
}

On visual_anchor.bbox: Format is [x, y, width, height] (pixel coordinates, relative to original keyframe resolution). Optional in v1.0; reserved for future in-player highlight support.

2.3 CSP JSON Schema

The complete JSON Schema definition is published separately as CSP_SPEC.md, with csp-schema.json for automated validation. Phase 1 includes a CSP validator CLI:

insightflow validate graph.json
# ✓ CSP v1.0 compliant
# ✓ 15 nodes, max depth 4
# ⚠ 2 nodes missing timestamp (acceptable for audio-only input)

2.4 On Vector Embeddings

CSP is a storage and exchange format; it does not include embedding vectors. Embeddings are runtime data, generated on-demand by frontend or application layers and stored in local vector stores (ChromaDB / Qdrant).

Rationale: Embedding models change frequently, dimensions and distance metrics vary. Excluding embeddings from CSP preserves protocol simplicity and cross-tool compatibility.

If semantic search is needed later (e.g., "search my notes for Attention mechanisms"), applications SHOULD auto-generate embeddings for nodes[].summary on load and index locally.

3. Model Strategy: API-First

InsightFlow uses an API-first, locally extensible model strategy.

3.1 Design Rationale

The LLM inference bottleneck is not local compute but model quality and context window. For most users—especially students and researchers—cloud APIs offer the best price-performance and stable output quality.

Therefore:

3.2 Model Gateway

LiteLLM provides a unified interface to 100+ model providers:

# config.yaml
default_provider: "openai"

strategies:
  structuring:
    primary: "openai/gpt-4o"
    fallback: "deepseek/deepseek-chat"
  quiz_generation:
    primary: "anthropic/claude-sonnet-4-5-20250514"
    fallback: "openai/gpt-4o-mini"

# 可选：本地模型扩展
extensions:
  ollama:
    enabled: false
    endpoint: "http://localhost:11434"
    models:
      structuring: "llama3:8b-instruct"

3.3 Model Recommendation Matrix

Use Case Recommended Model Notes
Structuring GPT-4o / Claude Sonnet Highest JSON output stability
Long-context analysis DeepSeek-V2 / Claude Sonnet High cost-efficiency + context
Quiz generation GPT-4o / Claude Sonnet Requires strong reasoning
Local privacy (extension) Llama 3 8B via Ollama User assumes quality variance

4. Element Extraction Pipeline

Core principle: Video is not atomic; elements are.

We do not feed video end-to-end to a model for summarization. Instead, we decompose video into independent cognitive elements, then compose them for LLM-driven structured reasoning.

4.1 Element Types

┌──────────────┐
│   Video      │
│   (.mp4)     │
└──────┬───────┘
       │ Extract
       ▼
┌──────────────────────────────────────────┐
│  Elements (可独立处理、存储、组合)          │
│                                          │
│  📝 Transcript  (WhisperX → .srt/.json)  │
│  🖼️ Keyframes   (场景切换检测 → .jpg)      │
│  📄 Slide OCR   (关键帧 OCR → .txt)       │
│  🎵 Audio       (原始音轨 → .wav)         │
└──────────────────┬───────────────────────┘
                   │ Compose & Reason (LLM API)
                   ▼
            ┌─────────────┐
            │  CSP JSON   │
            └─────────────┘

4.2 Processing Each Element

Transcript—the core element:

Keyframes:

Slide OCR:

Audio:

4.3 Why Not End-to-End Video Understanding?

End-to-end video comprehension (feeding frame sequences directly to multimodal LLMs) is exciting but faces clear engineering limits:

Therefore, end-to-end video understanding is a Phase 3 experiment. Phases 1–2 focus on element extraction + text reasoning.

4.4 Composition Strategy

From elements to CSP, reasoning uses layered composition:

Step 1: Transcript → 粗粒度结构（章节划分、主题识别）
Step 2: Transcript + Slide OCR → 细粒度结构（知识点、公式、定义）
Step 3: Structure + Keyframes → 视觉锚定（为节点关联对应画面）
Step 4: Structure → Quiz 生成（基于结构化知识点出题）

Each step is an independent LLM call, debuggable, cacheable, and model-swappable.

5. System Architecture

5.1 Technology Stack

5.2 Data Flow

User drags video ──→ Python Sidecar
                        │
                        ├── [Extract] WhisperX → Transcript
                        ├── [Extract] PySceneDetect → Keyframes
                        ├── [Extract] OCR + Denoise → Slide Text
                        │
                        ├── [Compose] LLM (API) → CSP JSON
                        │
                        └── [Output] CSP JSON ──→ Tauri Frontend
                                                    │
                                                    ├── React Flow (Mind Map)
                                                    ├── Vidstack (Player)
                                                    └── Quiz Cards

5.3 Offline / Local Mode

Default InsightFlow requires network (for API calls). Local mode as optional:

6. Quality Assessment Framework

CSP's value depends on output quality. InsightFlow provides two layers:

6.1 Structural Metrics

Automatically computed, written to CSP's quality field:

6.2 Semantic Assessment

LLM-as-Judge evaluation (optional, costs extra API calls):

Results attach to CSP's quality.semantic_eval field.

6.3 Community Benchmark Dataset

Phase 2 releases an open evaluation dataset:

7. Roadmap

Phase 1: The Protocol & CLI (v0.1)

Goal: Freeze CSP v1.0; complete element extraction pipeline.

Deliverables:

Phase 2: The Player (v0.5 – MVP)

Goal: First usable desktop GUI + quality baselines.

Deliverables:

Phase 3: The Workstation (v1.0)

Goal: Full learning workstation + ecosystem.

Deliverables:

8. Known Risks

8.1 Python Sidecar Packaging

Risk level: High

WhisperX depends on CTranslate2 (CUDA or CPU backend) and PySceneDetect on OpenCV. Full Python environment + GPU dependencies balloon to 2GB+; crashes occur across environments (Windows DLL gaps, macOS Metal compatibility).

Mitigation:

8.2 Prompt Stability

Risk level: High

Ensuring diverse LLMs (GPT-4o to Llama 3 8B) reliably output compliant CSP JSON is the hardest engineering challenge. Weaker models produce format errors, missing fields, and hallucinations.

Mitigation:

8.3 OCR Noise

Risk level: Medium

Video OCR is noisy (logos, captions, watermarks). Feeding it uncleaned to LLMs drowns signal in noise.

Mitigation: Three-layer denoising (adjacent-frame dedup, region filtering, semantic filtering) in §4.2.

9. Comparison

Feature NotebookLM ChatPDF / Wrappers InsightFlow
Core metaphor Chat Document reader Cognitive engine
Input processing End-to-end black box Text extraction Element extraction + composition
Data privacy Cloud (mandatory upload) Cloud/hybrid Local extraction, optional cloud reasoning
Output Text summary Text QA Structured graph (CSP)
Model choice Gemini only Vendor lock-in API-first + local extension
Quality assessment None None Structural + semantic
Extensibility None Weak Open plugin system

10. Contributing

InsightFlow is community-driven. We welcome contributions in these directions:

Getting started

# 1. Clone
git clone https://github.com/insightflow-org/insightflow.git
cd insightflow

# 2. 安装开发依赖
pip install -e ".[dev]"

# 3. 运行测试
pytest tests/

# 4. 一键处理视频
insightflow ingest examples/sample.mp4 --model openai/gpt-4o

# 5. 或分步执行
insightflow extract examples/sample.mp4 --output-dir ./elements
insightflow reason ./elements --model openai/gpt-4o --output graph.json
insightflow validate graph.json

Contribution areas

Issue labels

11. Closing

InsightFlow is not a replacement for learning. It removes friction from learning through element extraction and structured reasoning.

We do not produce knowledge. We are a structuring engine.

Join us. Build the engine for your second brain.

© 2026 Yuxu Ge ·