bfilipovlab Sign in

Parts of this page show illustrative data: invented, and marked where it stands, until the lab has its own.

SampleFinished

Which way of choosing a model keeps quality at the lowest cost?

No finding: this is a sample experiment, and its figures measure nothing. A real experiment states its finding here, in one sentence, with the number in it.

Limitation
Invented data: no model was called and nothing was judged. The models are named by letter.
Decision
None. A sample decides nothing; a real experiment says here what changed because of it.

Read the sample experiment

Started 21d ago · 5 models, 4 policies · 1,200 questions held out

Judged quality vs cost per 1,000 answers Quality as % of the reference model. Cost on a log scale, cheaper is left.
Illustrative data
80%85%90%95%100% $1$2$5$10$20 Threshold 95% Model A Model B Model C Model D Model E · reference Rules Cascade LLM router 96.1% · $3.72 kNN router Cost per 1,000 answers, USD 80859095100 $1$2$5$10$20
○ single model◇ policy● chosen
Source
run r-0412
Dataset
sample-qa@v1 · 7f3c21e
Judge and harness
judge-prompt v5 · harness 1.8.2
Invented for the design
Sep 25 · rev 2 Sep 24

Experiments

The lab's experiments are listed here once each has a result Boyko reviewed. The first, E1, is registered before it runs.