Typed Routing and a Local Small Language Model

Foundation-AI and the Scout Fleet

Foundation-AI brings an organisation's context, living memory, domain models and governance into a platform for agentic work. Its Scout Fleet is the persistent intelligence layer: Scouts such as Lighthouse, Capital and Forge own standing missions, share findings within permitted scopes, and keep evidence of their work.

Foundation exposes tool-using capabilities through the Model Context Protocol (MCP). Its MCP catalog includes Scout Fleet operations alongside other platform services. A tool call needs the right function and arguments that match its schema. This article examines that step: Jev selects the function and Foundation-SLM supplies its arguments. Full-call scores come from selected BFCL cases, while the catalog experiment examines routing across Foundation's tools.

Foundation-AI at a glance. Open the architecture SVG.

Abstract

We compare Foundation-SLM, a locally served model, with Jev 1.13, a hosted typed classifier, and GPT-5.6 Sol. On 440 selected BFCL examples, replaying saved outputs with a nested-list grading correction gives 68.18% for Foundation-SLM alone, 90.91% for Jev followed by local argument generation, and 92.05% for Sol alone. The optimized routed configuration improves the local baseline by 22.73 percentage points.

The routed configuration's difference from Sol is -1.14 pp (95% CI -3.9 to +1.6). The configurations also differ in their prompts, output contracts and assigned tasks. Jev routes requests through TypeSafe's API; the local model generates arguments. We also report separate catalog-routing, record-retrieval and operation-narration experiments.

1. Motivation and Scope

Foundation-AI's Metis proposal favors specialized local models where they can meet a task's requirements. This experiment tests one implementation strategy: let a typed classifier select a function, then ask a local language model to supply its arguments. We measure generated calls on selected BFCL cases and function selection on synthetic catalog requests.

The available snapshot contains 251 tool records across 23 services and 412 capability records. A production-derived catalog supplies realistic names and descriptions, but requests generated from those descriptions are still synthetic. Full-call grading uses public BFCL examples because the catalog experiment has no independently established enterprise argument answers.

2. Models and Data Flow

The saved manifests do not fully pin the local weight digest, training lineage, or serving environment. The model tag is evidence of the requested configuration, not an independent audit of training provenance.

Jev receives the user request and function criteria at api.typesafe.ai. This configuration combines external request routing with local argument generation. Provider data-retention terms are separate from the execution location.

3. Method and Scoring

3.1 The 440-item BFCL subset

The sample consists of 200 multiple examples requiring one call and 240 irrelevance examples requiring no call. It is a subset of BFCL, not a full leaderboard evaluation. A no-call policy would score 240/440, or 54.55%, on this mixture.

There is also a structural shortcut: every negative item offers one function, while every positive item offers two to four. A menu-size-only rule achieves 440/440 relevance decisions without interpreting the request. This population structure makes relevance decisions unusually easy; the result measures performance on this selected mixture.

3.2 Scoring nested arguments

The custom grader compares function names and arguments with the saved BFCL gold alternatives. One case, multiple_119, includes a list of condition objects with alternative gold values. The saved scorer did not descend into those objects. This analysis compares nested lists recursively, which adds one correct answer to each of the three configurations. The original observations are unchanged. Other scorer behavior, including type checking, has not been fully compared with the official BFCL evaluator.

3.3 Replay and uncertainty

The pipeline score combines saved argument-generation outputs on the 200 positive items with the router's decisions on the 240 negative items. It assumes the selected function must be called. The arguments-only stage was not recorded for 24 false-positive routes, so this is a composed accuracy replay rather than a full sequential run on all items.

Intervals use 4,000 paired item-bootstrap resamples. An equivalence test would require a preselected acceptable margin and a separate confirmatory run.

4. Results

4.1 Generated calls on the selected BFCL slices

Table 1. Scoring saved outputs on selected BFCL cases. The final two columns separate the call-required and no-call slices.

The improvement over the local baseline is +22.73 pp (95% CI +18.4 to +26.8). The routed and Sol configurations each pass 184/200 call-required items. Across all 440, the routed configuration has 16 unique successes and Sol has 21. Their paired difference is -1.14 points.

Figure 1. The routed configuration improves the local-model score. The configuration comparison is descriptive: prompts, offered tools and decision responsibilities also change.

4.2 What the configuration comparison can explain

Table 2. All rows use the same 200 call-required items.

The arguments-only prompt explicitly explains the JSON-string encoding and requires the call. The solo prompt inherits a request for a prose reply even though its schema has no reply field. The solo records contain 76 correctly selected functions with null arguments, compared with one in the arguments-only run. Raw responses and parse errors were discarded, so the records cannot establish whether these failures arose from missing values, JSON encoding, or another compliance problem.

The observed gain supports this optimized configuration. A causal field-count claim needs a controlled experiment with identical cases, tools and semantic instructions, plus separate prompt ablations and a local-select/local-fill baseline.

4.3 Synthetic catalog routing

Table 3. One synthetic request per catalog tool. Counts reproduce under the saved labels. Local inference costs are excluded. These are routing measurements, not complete tool calls.

Figure 2. The two request sets differ in vocabulary and sometimes in meaning or available context.

Jev and Sol differ by +0.80 points, with a paired 95% interval from -5.2 to +7.2 points. Neither equivalence nor an enterprise routing accuracy bound follows from that interval.

The second request generator suppresses words from the gold tool's description, reducing average overlap to about 3%. It still derives requests from that description. Some requests also change the task's meaning: design profiles become user accounts, mission receipts become payment records, and an authorization token becomes a trial credit. Several services have insufficiently distinguishing descriptions. These labels need independent adjudication before the results can characterize realistic operator traffic.

The high-overlap and vocabulary-suppressed sets should therefore be treated as two synthetic conditions. They establish neither an upper/lower performance bound nor proof of training-set contamination.

4.4 Cost and latency

At the recorded rate assumptions, the routed replay incurs $0.01682 in external API charges per 1,000 items, compared with $3.33476 for Sol: a ratio of approximately 198. This excludes all local inference costs. Owned hardware does not make incremental power or occupied capacity free, so the ratio is not a measurement of total or actual marginal cost.

The recorded 3.08-second pipeline mean covers observed hops. Argument-generation durations are absent for 24 false-positive routes, so a full sequential mean remains to be measured. The recorded solo means, 5.09 seconds locally and 1.96 seconds for Sol, reflect their run conditions.

4.5 Catalog-record retrieval

Table 4. Recorded retrieval aggregates. Full queries and ranks were not archived, so these figures cannot be recomputed from the saved artifacts.

This experiment tries to recover a source tool record among 251 tool descriptions and 412 capability records. The saved record contains aggregates and three query examples, without all rewritten queries, query vectors and per-item ranks. The lower local-rewrite score is specific to this catalog-record task.

4.6 Operation narration with the gold tool supplied

Table 5. Saved judge decisions on 251 examples. All-four counts are 54/251 and 190/251. These counts reproduce; the rubric's accuracy has not been independently established.

Both generators receive the gold tool and its description, regardless of the router's prediction. This measures narration with the intended tool supplied. The judge accepted a reply that promised to spend a trial credit on a pause operation, while rejecting a future-tense quarantine reply on the not-yet-run criterion. Human review of the rubric would make these scores more useful.

5. Confidence and Execution Boundaries

Jev's recorded confidence discriminates correct labels from disagreements on the synthetic catalog set: AUROC 0.829, ECE 0.050. At threshold 0.90, 58 items are accepted with one label disagreement; at 0.95, 41 are accepted with none. That disagreement is itself an ambiguous health request, so it is not independently established as a model error.

Even with valid labels and a fixed threshold, zero errors among 41 accepts leaves a 7.05% one-sided 95% upper error bound. These thresholds were inspected on the evaluation set, which contains no true no-tool examples. Execution thresholds require separate validation on ambiguous and unsupported requests.

Confidence may support escalation after separate validation. It must not replace authorization, mandatory confirmation, argument validation, or verification of the executed outcome. Removing the downstream decline also removes a potential veto: the earlier four-field local stage declined 14 of Jev's 24 false-positive BFCL routes.

6. Interpretation and Next Experiment

Changing the tool-calling configuration substantially improves the local model on the chosen BFCL cases. The next step is to test decomposition, prompt clarity and output constraints as separate factors.

1. Validate the custom scorer against a pinned official BFCL evaluator, including nested values, optional arguments, types and malformed responses.
2. Use independently authored or logged requests with adjudicated labels, including ambiguous and unsupported requests. Match positive and negative menu-size distributions.
3. Freeze prompts and output contracts before a held-out comparison. Include local-select/local-fill and native structured tool-calling baselines, and save raw responses and parse failures.
4. Run every actual selected second hop and record complete-call accuracy, latency, energy, escalation and execution outcomes. Choose an equivalence margin and accepted-call risk target before the confirmatory run.
5. Archive the full retrieval corpus, all queries and ranks, model and embedding identities, and blinded human checks of the narration rubric.

7. Scope of the Results

The scores cover one local model configuration, one Sol effort setting and mostly one saved observation per item. The BFCL mixture separates call-required and no-call examples by menu size. Synthetic catalog labels need independent adjudication, and the custom scorer needs a broader comparison with the official BFCL evaluator. Full sequential latency, local compute cost and execution outcomes are subjects for the next experiment. Retrieval ranks and narration judgments would benefit from complete archived records and human validation.

8. Conclusion

On these 440 selected BFCL cases, Jev plus the local model delivers essentially frontier-level tool-call accuracy: 90.91% versus 92.05% for Sol. Its recorded external API charges are less than 1% of Sol's, approximately 1/198th ($0.01682 versus $3.33476 per 1,000 cases). This comparison excludes local model compute. Against the local model alone at 68.18%, the routed configuration gains 22.73 percentage points.

References

1. TypeSafe model and pricing documentation; function-calling example.
2. Berkeley Function-Calling Leaderboard dataset.

Ownership and licensing. Foundation-AI and Foundation-LifeStyle, together with all intellectual property rights subsisting in them, are the sole and exclusive property of MediaGlyphics GK. Ibex is an authorized licensor of these technologies for forward deployed engineering engagements. © 2026 MediaGlyphics GK. All rights reserved.