Building meta-ast: Sub-Millisecond Incremental Polyglot Static Analysis in Rust

During Google Summer of Code (GSoC) 2026 with MetaCall, I designed and implemented meta-ast: a standalone, high-performance static analysis engine written in Rust.

The engine parses 9 programming languages (Rust, C, C++, Go, Python, JavaScript, TypeScript, TSX, and Ruby). It extracts a normalized symbol Intermediate Representation (IR), resolves cross-language dependency graphs, isolates cyclic modules via Tarjan Strongly Connected Components (SCC), and partitions polyglot applications into deterministic deployment manifests for the MetaCall Function Mesh runtime.

I want to express my deepest gratitude to my mentor, Vicente Eduardo Ferrer Garcia, for his immense patience, continuous guidance, and mentorship throughout the entire program.

This article explains what polyglot programming and MetaCall are, followed by a breakdown of the architectural decisions, data structures, and optimization techniques behind meta-ast.

---

0. What is Polyglot Programming and MetaCall?

Polyglot Programming: Choosing the Best Tool

In software engineering, no single programming language is best for everything:

• Python excels at machine learning and data science.
• C++ and Rust provide memory control and raw execution speed.
• Node.js and TypeScript offer rich web ecosystems and asynchronous I/O.

Polyglot programming means writing different parts of an application in different languages to use the strongest tool for each job.

flowchart TB
    subgraph TRAD["Microservices"]
        GW["Node.js API Gateway"] -->|"HTTP/JSON over TCP"| ML["Python ML API"]
        GW -.->|"network hop & JSON serialization: 10-50 ms"| ML
    end
    subgraph MC["MetaCall"]
        NJS["Node.js Context"] ==>|"in-process memory dispatch: under 100 ns"| PY["Python Context"]
    end
    TRAD -.->|"vs"| MC

The Traditional Problem: Network Overhead and Glue Code

Historically, connecting languages required two painful choices:

1. Microservices over HTTP / gRPC: You split code into multiple services and serialize data over sockets. This adds network round-trip latency (typically 10-50 ms), CPU serialization cost (JSON/protobuf marshalling), and operational complexity.
2. Manual C FFI / Foreign Function Wrappers: You write fragile C bindings and glue code by hand to expose functions across runtimes.

The Solution: MetaCall

MetaCall is an open-source polyglot runtime engine. It loads multiple language runtimes (CPython, Node.js, Ruby, C#, Go, and C/C++) into the same operating system process space.

With MetaCall, functions call each other across languages directly in memory through C-FFI trampolines without network transport, without serialization overhead (sub-100 ns dispatch latency), and without boilerplate glue code:

// JavaScript/TypeScript calling a Python function directly
const { train_model } = require('./trainer.py');

const weights = train_model(dataset);

While MetaCall makes runtime execution transparent, developers and orchestrators need tools to understand and validate these cross-language calls before running the code. That is where meta-ast comes in.

---

1. The Polyglot Blind Spot

MetaCall provides runtime reflection (metacall_inspect) after code executes. However, relying purely on runtime inspection creates three critical problems:

flowchart LR
    SRC["Source Code"] --> EXEC["Execute Code in VM"] --> INSPECT["Inspect State"]
    EXEC -.-> RISK["Security risk: unsafe in CI/CD"]
    EXEC -.-> OVERHEAD["Heavy overhead: loads all language runtimes"]
    EXEC -.-> BLIND["Blind to topology: cannot plan pod cuts"]

1. Security Risk in Tooling: Executing untrusted code in an IDE or CI runner just to extract function signatures is unsafe.
2. Heavy Runtime Overhead: Starting multiple language virtual machines (Node.js, CPython, Ruby VM) takes hundreds of megabytes of memory and seconds of boot latency.
3. Pre-Deployment Topology Blindness: Serverless runtimes and orchestrators need to know how to split a mixed codebase into separate language containers before container startup.

I built meta-ast to solve this statically. It operates strictly on source text without executing target code.

NOTE

Design Trade-Off: Tree-sitter vs. Compiler Frontends

Native compiler frontends (rustc, clang, CPython AST) require full toolchains, host runtime environments, and exact header search paths installed on the host machine. Tree-sitter provides standalone, zero-dependency C parsers that generate concrete syntax trees directly from source buffers in microseconds, while cleanly tolerating incomplete or partially broken syntax during active editing.

---

2. The Core Mental Model

Think of a polyglot codebase as a circuit board with different regional chip architectures (languages). Traces (function calls and imports) connect components within the same chip or jump across chip boundaries through bridge pins (metacall_load_from_* and metacall() calls).

flowchart LR
    subgraph P0["Python Pod 0"]
        H["app.py: handler()"]
    end
    subgraph P1["Node.js Pod 1"]
        W["service.ts: process_image()"]
    end
    H -->|"metacall(process_image) via RPC stub, ADR 0003 check"| W

meta-ast acts as a static board scanner:

1. Scan: It parses raw source files using parallel Tree-sitter grammars.
2. Normalize: It maps all declarations to language-agnostic Symbol IR nodes.
3. Trace: It builds a directed graph of imports and call sites.
4. Decompose: It runs Tarjan SCC to identify tight cycles that must stay together, then uses Union-Find to group same-language nodes into deployment pods.
5. Verify: It ensures that every edge severed across pod boundaries has a valid MetaCall RPC stub (cut fairness).

---

3. Architecture and Normalized Symbol IR

The engine uses a pipeline design with explicit isolation between parser lifecycles, graph analysis, and export sinks.

flowchart TD
    SRC["Multi-Language Sources (9 grammars)"] --> POOL["Thread-Local Tree-sitter Parser Pool"]
    POOL --> QUERY["AST Query Packs (Python, C, C++, Rust, Go, JS, TS, TSX, Ruby)"]
    QUERY --> IR["Symbol IR Normalization: visibility, signatures, spans, docstrings"]
    IR --> GRAPH["CodeGraph (petgraph): symbol resolution + confidence scoring"]
    GRAPH --> SCC["Tarjan SCC"]
    GRAPH --> POD["Pod Partitioning"]
    SCC --> SINKS["Artifact Sinks: JSON / HTML / metacall.pods.json / metacall.mesh.json"]
    POD --> SINKS

Thread-Local Parser Pools

Tree-sitter parsers are not thread-safe. Creating a new parser instance per file introduces allocation overhead. I solved this using thread-local parser pools:

thread_local! {
    static PARSER_CACHE: RefCell<HashMap<LanguageId, Parser>> = RefCell::new(HashMap::new());
}

pub fn with_parser<F, R>(lang: LanguageId, f: F) -> Result<R, AnalysisError>
where
    F: FnOnce(&mut Parser) -> Result<R, AnalysisError>,
{
    PARSER_CACHE.with(|cache| {
        let mut map = cache.borrow_mut();
        let parser = map.entry(lang).or_insert_with(|| {
            let mut p = Parser::new();
            p.set_language(&lang.tree_sitter_language())
                .expect("Valid grammar definition");
            p
        });
        f(parser)
    })
}

The Normalized Symbol IR

Every function, class, method, struct, enum, and interface across all 9 languages normalizes into a unified Symbol record:

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Symbol {
    pub id: SymbolId,
    pub name: String,
    pub kind: SymbolKind,
    pub visibility: Visibility,
    pub location: SourceLocation,
    pub signature: Option<FunctionSignature>,
    pub docstring: Option<String>,
    pub parent_id: Option<SymbolId>,
}

Malformed source code does not crash the extraction pass. The parser captures syntax error nodes, creates structured Diagnostic records, and continues extracting valid remaining symbols.

NOTE

Design Trade-Off: Flat Symbol IR vs. Unified Hierarchical AST

Forcing 9 distinct language grammars into a single unified hierarchical AST produces an unwieldy monster with hundreds of language-specific node variants. For dependency topology and deployment planning, the engine only needs declarations (functions, classes, structs) and reference vectors (calls, imports). A flat Symbol IR minimizes memory footprint, enables O(1) indexed lookups, and simplifies graph construction.

---

4. Dependency Graph and Tarjan SCC

Once symbols are extracted, meta-ast constructs a directed CodeGraph over petgraph::graph::DiGraph.

Confidence-Weighted Edge Resolution (ADR 0002)

Cross-file import semantics differ across ecosystems (such as Node.js module resolution, Python sys.path, Rust crate paths, and C include paths). Rather than exposing confusing heuristic flags, meta-ast crawls the import graph using breadth-first search (BFS) and models resolution certainty using distance-decayed confidence scores (ADR 0002):

• 1.0 (Direct / Local): Lexical containment or direct relative file import (distance ≤ 1).
• 0.8 (Transitive): Resolved module or package dependency within the same language ecosystem (distance > 1).
• 0.6 (Cross-Language Dynamic): Dynamic bridge invocation (such as metacall_load_from_file("worker.py")), preserving high recall for deployment planning while flagging resolution uncertainty.

#[derive(Debug, Clone, Copy, PartialEq, Serialize, Deserialize)]
pub struct DependencyEdge {
    pub kind: EdgeKind,
    pub confidence: f32,
}

O(1) Edge Deduplication

Early benchmarks showed that multi-pass reference resolution caused quadratic slowdown when inserting duplicate edges into large graphs.

I eliminated linear edge scans by indexing edges in an internal (NodeIndex, NodeIndex, EdgeKind) lookup map. When an edge already exists, meta-ast updates the confidence score to max(existing_confidence, new_confidence) in O(1) time. This reduced graph construction for 10,000 duplicate edges to 486 microseconds.

Tarjan Strongly Connected Components (SCC)

Cyclic dependencies prevent clean architectural decomposition. If File A imports File B, and File B imports File A, they cannot be deployed into separate isolated execution pods without creating circular initialization dependencies and synchronous cross-pod RPC loops. Tarjan SCC isolates these cycles so they remain co-located in the same execution pod.

I ran Tarjan SCC over an EdgeFiltered view of the graph that retains only Import and Call edges while ignoring structural ownership edges:

pub fn detect_cycles(graph: &CodeGraph) -> Vec<Vec<NodeIndex>> {
    let filtered = EdgeFiltered::from_fn(&graph.inner, |edge| {
        matches!(edge.weight().kind, EdgeKind::Import | EdgeKind::Call)
    });

    tarjan_scc(&filtered)
        .into_iter()
        .filter(|component| component.len() > 1)
        .collect()
}

Every cyclic component with 2 or more nodes is flagged as an indivisible unit.

NOTE

Design Trade-Off: Tarjan SCC vs. Generic DFS Cycle Checks

A standard depth-first search cycle check only returns a boolean indicating that a cycle exists, but fails to isolate the maximal strongly connected component. Tarjan's linear-time algorithm, O(V + E), finds all maximal strongly connected subgraphs in a single DFS traversal using node discovery timestamps and low-link values. This groups interdependent files into atomic units in under 54 microseconds.

---

5. Pod Partitioning and Cut Fairness

One primary deliverable of meta-ast is feeding the metacall-deploy subsystem with deployment manifests.

Same-Language Pod Partitioning

meta-ast partitions the graph using a Disjoint-Set (Union-Find) algorithm:

1. Merge all nodes belonging to the same Tarjan SCC cycle into a single component.
2. For each non-cut edge between nodes of the same programming language, union their sets.
3. Emit each connected set as a discrete deployment pod (metacall.pods.json).

NOTE

Design Trade-Off: Union-Find for Pod Partitioning

After collapsing cyclic components into super-nodes, partitioning the remaining acyclic graph by language is a disjoint-set problem. A Disjoint-Set (Union-Find) structure with path compression and union-by-rank operates in near-constant time, O(alpha(N)). This allows meta-ast to partition hundreds of source files into deployment manifests with zero allocation churn.

flowchart TB
    APP["Polyglot Application"]
    APP --> P0
    APP --> P1
    subgraph P0["Python Pod 0"]
        C1["SCC Cycle 1"]
        F1["app.py"]
        F2["data_cleaner.py"]
    end
    subgraph P1["Node.js Pod 1"]
        C2["SCC Cycle 2"]
        F3["server.ts"]
        F4["auth.ts"]
    end
    F2 ==>|"cut edge"| F3

Enforcing Cut Fairness (ADR 0003)

When an edge crosses pod boundaries, the call cannot execute in-process. It must transition across the MetaCall Function Mesh over an RPC bridge.

If the developer severed an edge without creating an RPC route, the deployed pod will fail in production.

Under --check mode, meta-ast verifies cut fairness:

Cut-fairness invariant: ∀ e = (u, v) ∈ E_cut, ∃ s ∈ RPC_Stubs such that target(s) = v.

If any inter-pod call site lacks an entry in metacall.mesh.json, meta-ast emits a build-halting diagnostic pointing to the exact source line and column of the unhandled call site.

---

6. High-Performance Incremental Watch Mode

Developer tooling must feel instant. I set a strict target for --watch mode: re-analyze modified files and refresh the graph in under 100 milliseconds.

I achieved 1.16 milliseconds for warm single-file incremental updates.

flowchart TD
    MUT["File mutation detected (fsnotify)"] --> HASH["BLAKE3 hash calculation"]
    HASH --> MATCH{"Hash matches cache?"}
    MATCH -->|Yes| SKIP["Skip: 0 ms"]
    MATCH -->|No| REPARSE["Re-parse file with Tree-sitter"]
    REPARSE --> ID["Monotonic ID allocation: IdGenerator::with_start(max_id + 1)"]
    ID --> SWAP["Arc FileExtraction swap"]
    SWAP --> REBUILD["In-place graph rebuild: 1.16 ms"]

Three Design Rules for Sub-2ms Watch Mode

1. BLAKE3 Content Hashing: File modification timestamps trigger false positives (e.g., touch or IDE auto-saves). The watcher computes BLAKE3 hashes of file buffers. If the hash matches the cache, the event is discarded immediately.
2. Zero-Allocation Cache Reuse: File extraction results are wrapped in Arc<FileExtraction>. When rebuilding the graph, unchanged files reuse their existing Arc allocations without cloning syntax trees or symbol tables.
3. Collision-Free Monotonic ID Seam: To avoid ID collisions between cached and new symbols without renumbering the entire project, the ID generator initializes from max_cached_id + 1:

let next_id = cached_extractions
    .values()
    .flat_map(|f| f.symbols.iter().map(|s| s.id.as_u32()))
    .max()
    .unwrap_or(0) + 1;

let mut id_gen = IdGenerator::with_start(next_id);

---

7. Verification, Benchmarks and Retrospective

Benchmark Results

Measurements taken using Criterion.rs on an Intel Core i5-12500H (12 cores, 16 threads, Linux 7.1, release profile, --features watch):

Pipeline Stage / Microbenchmark	Target (FR-5)	Measured Result	Notes
Warm Single-File Incremental Re-analysis	< 100 ms	1.16 ms	Single-file change with Arc cache swap
Cold Single-File Analysis (analyze_graph)	< 100 ms	1.66 ms	First-pass single-file graph construction
Graph Construction Linear (1,000 nodes)	< 10 ms	425 μs	Multi-file graph assembly
Tarjan SCC Cycle Detection (1,000 nodes)	< 10 ms	54 μs	Cycle identification on filtered graph
Edge Deduplication (10,000 duplicate edges)	< 50 ms	486 μs	O(1) indexed lookup with confidence max
Node Lookup (10,000 nodes)	< 1 ms	9.5 μs	Inverted symbol map retrieval
End-to-End Extraction (All 9 Fixtures)	< 100 ms	16.8 ms	Full cold parse across all test fixtures

Across individual language fixture suites, extraction ranges from 113 μs for C, 126 μs for Go, and 137 μs for Python, up to 14.1 ms for comprehensive Rust test fixtures with heavy trait hierarchies.

Release Artifacts and Quality Gates

• Test Suite: 430+ automated tests (327 unit tests, 106 integration tests, and doc-tests).
• Matrix: Automated CI passing on Linux (glibc and musl), macOS (x86_64 and Apple Silicon), and Windows (MSVC and ARM64).
• Crates.io Release: Published as meta-ast v0.5.0.
• Documentation: Generated mdBook manual at metacall.github.io/meta-ast.

Key Engineering Takeaways

1. Keep the Core Standalone: Coupling the static analyzer directly to runtime VM libraries would have complicated CI and increased binary sizes. Designing meta-ast as a standalone Rust CLI with clean JSON/Dataflow contracts kept iteration fast.
2. Deterministic Graphs Save Hours: Enforcing forward-slash path normalization and deterministic node ordering across Windows and Linux prevented flaky CI failures across multi-platform tests.
3. Tree-sitter Query Packs Scale Well: AST queries defined in .scm query files allow adding language grammars without modifying the core graph engine.

---

Getting Started

Install meta-ast directly via Cargo:

cargo install meta-ast --features "metacall-deploy watch dataflow"

Inspect any polyglot repository:

# Run static symbol inspection
meta-ast inspect ./my-project -f json

# Build dependency graph and generate interactive HTML dashboard
meta-ast graph ./my-project --html -o graph.html

# Partition project into deployment pods and check cut fairness
meta-ast deploy ./my-project --check -o ./dist

For source code, architecture specifications, and benchmark reproductions, visit the meta-ast GitHub repository.

---

Update: A Hierarchical Alternative

Since publishing this write-up, I came across Polyglot AST: Towards Enabling Polyglot Code Analysis (Houdaille, Khelladi, Briend, Jongeling, and Combemale at Inria, 2023), which attacks the same cross-language analysis problem from a different direction.

meta-ast normalizes declarations into a flat Symbol IR and derives a dependency graph from it, optimized for whole-program reasoning: cycle detection, cut fairness, and pod partitioning. The paper instead defines a unified hierarchical AST: one tree that preserves the syntactic structure of every language, constructed through GraalVM's polyglot API rather than external parsers such as Tree-sitter. Because the tree keeps the source's shape, it is a natural foundation for IDE-style services - the authors demonstrate auto-completion, consistency checking, type inference, and rename refactoring across language boundaries.

The two designs are complementary more than competing. A hierarchy-preserving tree suits interactive tooling; a flattened IR with graph analysis suits deployment planning and static verification. One interesting experiment would be rebuilding meta-ast's graph construction on top of a Polyglot-AST-style hierarchy instead of per-language query packs, trading the standalone parser pool for runtime-provided trees. I may work on exactly that proof of concept in the near future.