Skip to main content

​
Human Labelled Test Dataset

A golden set is the set of data points that is the most representative of your real-life production data, which are human-labelled according to the definition of the metric. A golden test set is crucial for robust model development and evaluation. It serves as the single source of truth against which all model performance is measured.

​
Required dataset format

In the Luna Studio UI, source data can come from a local CSV file, a CSV file at an HTTP(S) URL, or a connected Galileo workspace. Although the source picker accepts .jsonl uploads and URLs during ingestion, the current validation and training pipeline expects CSV data; convert JSONL to CSV before using it in a run. Hugging Face sources are available in the standalone SDK, not as an in-app dataset source. Each row should represent one example. The required feature columns depend on the metric shape. Test sets and labelled training sets must also have a label column for the ground-truth label; raw unlabelled training logs omit it until you start Label with metric prompt in the run wizard. Read more in Core concepts, Test sets, and Dataset validation.

​
Advanced formats

Full-trace and full-session fine-tuning are not available in the Luna Studio UI. The standalone SDK supports advanced label-only or direct-training workflows for these formats; see Full traces and Full sessions. Labels should be manually assigned and should match the exact metric definition you want to train or evaluate.

​
Required dataset size

Metric-aware validation in the run wizard enforces these minimums:
  • Test set: at least 300 rows total and 100 examples per class.
  • Labelled training set: at least 2,000 rows total and 100 examples per class.
For test sets, 1,000-3,000 representative human-labelled rows is a strong target. Keep class distributions reasonably balanced so evaluation results are meaningful across classes.

​
Training dataset guidance

If you add unlabelled training logs, choose Label with metric prompt, configure the labelling job, and start it. Luna Studio then validates the resulting labelled training set before fine-tuning. The class distribution should be similar to your test set distribution and should not be extremely skewed, for example 99/1. If you do not have enough training data, synthetic generation can help create training examples before fine-tuning.

​
LLM-as-a-Judge Prompt

You can select a preset metric, select a custom metric created in Galileo, or write a custom prompt directly in Step 1 of the Luna Studio run wizard. For Galileo-side authoring, see Custom LLM-as-a-Judge metrics. It is important to ensure that the LLM-as-a-judge prompt has high accuracy on your golden dataset. If it does not, tune the prompt manually or use Autotune in Galileo before creating a Luna metric. This ensures that Luna fine-tuning starts with a good understanding of the metric and avoids garbage in, garbage out.