ORBITER

1. Executive Summary

Orbiter’s enrichment pipeline ingests person and company data from 5+ external sources (People Data Labs, LinkedIn/Enrich Layer, Y Combinator, Fundable, Crunchbase), normalizes it into 20+ relational tables in Xano, then projects it into a FalkorDB knowledge graph with nodes, edges, and vector embeddings.

We cloned all 9 core enrichment functions into an isolated qa/ namespace, systematically audited each one, and discovered 6 bugs in the production code — including 3 functions with globally-scoped database queries that corrupt data across people. Our QA clones fix all of these and add error isolation, structured responses, and an AI self-healing loop powered by Claude Opus 4.6.

2. How the Pipeline Works — End-to-End Architecture

The enrichment pipeline has 4 layers: Ingest (raw data from external APIs), Process (fan out into relational tables), Resolve (link records to companies), and Graph (project into FalkorDB).

3. Data Model — What Gets Enriched

The pipeline populates 20+ relational tables per person. Here are the core tables and what they store.

Person Data Tables

Data Sources (95 registered, 5 primary)

4. FalkorDB Graph Integration

FalkorDB (formerly RedisGraph) is a property graph database that uses the Cypher query language. Every enriched person and company becomes a node with vector embeddings, connected by typed, weighted relationships.

Graph Node Types

Graph Relationship Types

Key Architectural Patterns

1. Relational-first, graph-second: All data lands in Xano tables first. The graph is a projection. This makes the pipeline crash-safe — sections fail independently while the graph stays eventually consistent.
2. UUID-based bidirectional linking: FalkorDB generates randomUUID() for nodes. This UUID is stored back on the Xano record (node_uuid on master_person, edge_uuid on join tables). Both systems can find each other.
3. Vector embeddings on every node: Person nodes get name_embedding from bio_500 via OpenAI embeddings. Company nodes from about_500. Enables semantic similarity search in the graph.
4. LLM-generated edge descriptions: Education edges include a natural-language description generated by Gemini 2.5 Flash (e.g., “Robert studied Computer Science at MIT from 2010-2014”).
5. Derived relationships: The graph computes connections like ATTENDED_TOGETHER by analyzing temporal overlap between existing edges — surfacing connections not explicitly stated in source data.
6. Weight-based ranking: Every edge has a weight property. Lower = stronger. STUDIED_UNDER (10) ranks higher than ATTENDED (40). Graph traversal prioritizes meaningful connections.
7. Visibility gating: Nodes start visibility: false. Only become visible after bio_500 exists. Prevents incomplete data in user-facing queries.

5. Real Person Examples — What the Pipeline Produces

6. Critical Bugs Found in Production Code

Deep code review uncovered 6 bugs. Three are data-corruption-level severity (globally-scoped queries that modify records belonging to other people).

7. Before vs After — Key Metrics

8. Function-by-Function Comparison

All 9 enrichment functions cloned into qa/ namespace with fixes applied.

9. Test Results — 20-Person Batch

Final verification: 20 people, 9 stages each, 180 total stages. All passed with 0 failures.

Average: 16.4s/person. Fastest: Jared Leto (2s). Slowest: Jon Dahl (45s).

Previously failing: Larry Sanger (was 438s timeout → now 9s) and Jon Dahl (was 306s timeout → now 45s).

10. Implementation Phases

Phase 1: Critical Bug Fixes (P0)

• Scoped certifications queries (added missing input + 3 WHERE clauses)
• Scoped honors queries (2 sections) and volunteering queries (2 sections)
• Fixed empty table names in company process (3 db calls)
• Added try_catch to complete-person-enrich (zombie prevention)
• Added per-section try_catch to process-enrich-layer (16 sections)

Phase 2: Resilience (P1)

• Null guards: skip records with empty names, null node_uuids
• Timeout protection: skip fundable deals >100 count
• Duplicate history prevention: check-and-reuse existing enrich_history_person records
• Direct function.run for company process (bypasses Xano nginx 300s timeout)

Phase 3: Observability (P1)

• All 9 functions return structured JSON: {processed, resolved, errors, skipped}
• Company process returns per-section results array
• Crash logging to crash_log table with section name and full error context

Phase 4: AI Self-Fix Loop (P2)

• Powered by Claude Opus 4.6 via OpenRouter ($220 credit pool on OPEN_ROUTER_RND key)
• 13 XanoScript constraint rules in system prompt (db.edit inline data, filter gotchas, etc.)
• Fetches function source via Xano Meta API + crash_log context + error details
• Pre-flight validation function (qa/validate-enrich-data): checks data shapes before processing

11. Production Pipeline Health (Current State)

Current state of the production enrichment system, measured via the enrichment diagnostics MCP.

The QA pipeline’s fixes directly address the top two issues: per-section try_catch eliminates stuck records, and guaranteed cleanup in complete-person-enrich prevents zombies.

12. Recommendations

1. Port QA fixes to production: The 6 bugs exist in live production functions. The QA clones prove the fixes work at 100% pass rate across 20 people.
2. Clean up 778 stuck records: Run a one-time cleanup to set processing=false on all stuck enrich_history_person records, allowing re-enrichment.
3. Run QA batch on full dataset: We tested 20 people (IDs 2-21). The database has 2,143. A broader batch will surface edge cases.
4. Enable AI self-fix in production: The Opus 4.6 loop can diagnose and fix XanoScript errors automatically, reducing manual debugging.
5. Address company queue backlog (10,159): Edge resolvers create new master_company records faster than they’re processed. May need batch company enrichment or prioritization.
6. Fix placeholder avatar systemic bug: 1,227 people have placeholder avatars marked as main — this is a separate bug in the replace-avatar function.