Pathwise

Open-source TypeScript query planner for RAG and agent pipelines: declare the quality (and latency) you need; Pathwise picks the cheapest execution plan per request and keeps learning from scored production runs.

Problem

Databases have had query optimizers for 50 years. AI pipelines don’t — teams hard-code one model, one topK, one reranker for every request and pay frontier prices for trivial questions. Plain model routers are crowded; whole-pipeline planning has research papers but almost no drop-in developer SDK.

Solution

Mark your knobs (model, retrieval depth, reranker, agent step budget). Calibrate on real sample queries — cheapest plans first, with dominance pruning. Adaptive calibration used about 9% of exhaustive calibration cost on the synthetic workload; on the real run it used about 37% of the runs (447 of 1,200). At request time, find similar past queries, predict quality/cost/latency, and run the cheapest plan that meets the SLA. Optional confidence-calibrated SLA guarantee (Wilson lower bound) and noisy-judge correction. Feedback rate closes the loop as models and prices change.

Senior software and AI engineers wiring RAG or agent stacks in TypeScript who need cost control without sacrificing a declared quality bar.

1. Clone the repo: git clone https://github.com/arsalan-arshad/pathwise && cd pathwise && npm install
2. createPlanner({ space, pipeline, scorer, sla }) — set minQuality / maxLatencyMs / optional confidence.
3. await planner.calibrate(sampleQueries) — inspect CalibrationReport / slaCertified.
4. await planner.run(userQuery) — cheapest plan that meets the SLA; score a fraction of production for online learning.

• TypeScript (zero runtime dependencies)
• Node ≥ 22
• Hashed bag-of-words featurizer + cosine similarity
• MIT open source

• enumeratePlans — cartesian product of KnobSpace
• Adaptive calibration — cheapest-first search, early stop, dominance pruning, monotonicity audits
• estimate / decide — k-NN similarity² weighting with per-plan priors; non-dominated cheapest plan ≥ threshold
• SLA guarantee — leave-one-out replay + Wilson lower bound; safe fallback when uncertified
• Noisy judge path — bias-corrected scores, majority / confirm votes

• Per-request cheapest plan under a quality SLA
• Confidence-calibrated SLA mode (synthetic: 0/320 misses vs 45 for average-based)
• Adaptive calibration far cheaper than exhaustive
• Online learning from scored production fraction
• Demo workload: same 100% accuracy at ~$4.93 vs $33 always-max (synthetic; real savings vary)

Simulated support-bot workload, 1,000 queries, 90% accuracy SLA:

The workload is synthetic; real savings depend on how varied your traffic is.

HotpotQA, real Gemini calls, 12 plans, 100 calibration and 200 test questions. Costs are paid list prices; the run used the free tier. Target: F1 of at least 0.725 (95% of the best plan's F1 on the calibration questions).

• Average-based mode was 60% cheaper than the best plan and kept 90% of its F1, but it can miss the target.
• The guarantee mode refused to certify the target and fell back to the best plan, so it saved nothing. With 100 questions even the best plan cannot prove 95% of its own score.
• The first run exposed a bug (calibration pruned the best plans); it is fixed and both runs are published in the repo.
• Not shown: that routing beats the best single fixed plan. One dataset, one provider, one run.

Full results and limits →

Released (packaged / hosted)

MIT. Public GitHub — github.com/arsalan-arshad/pathwise. Issues and PRs welcome; synthetic benchmarks are labeled honestly.

Open pull requests on GitHub →