Skip to content

Loss and Metrics Configuration User Documentation

This document provides detailed information on how to configure the Loss and Metrics options for analyzing the performance of different finetunes created for a specific version. The available configurations will vary based on the user's subscription plan.

Starter Plan

For users with the Starter plan, the following configuration options are available in the UI:

Compute Classification Metrics

If your project is a classification task, you can enable this option to compute classification metrics.

To enable this, click the radio button with the true option.

Classification Number of Classes

This option is visible if compute_classification_metrics is set to true. It represents the number of classes (labels) in the given classification dataset. For example, if you are creating a finetune for sentiment prediction with sentiments positive, neutral, and negative, you need to set this value as 3.

Enter the total number of classes in the input box.

Classification Positive Class

This option is visible if compute_classification_metrics is set to true and the finetune task is binary classification. Enter the name of the positive class. For example, if you are creating a finetune for sentiment prediction with two sentiments positive and negative, you need to set this value as 'positive'. If you are using 1 for positive and 0 for negative, you need to set this value as '1'.

Enter the positive class name in the text input box.

Classification Betas

This option is visible if compute_classification_metrics is set to true and the finetune task is binary classification. Enter a list of F betas, for example: [1, 2, 3, 4]. With a beta of 1 (i.e., the F-1 score), precision and recall are given the same weight. A larger beta score puts more weight on recall and less on precision. A smaller beta score puts more weight on precision and less on recall.

Choose between adding a list of values or providing a range using the radio button.

  • If you choose the list: Enter the values in the text box and click the + button to add the values to the list. Added values will have an X button to remove that value.
  • If you choose the range: Enter the starting and ending values of the range in the two text boxes.

Pro and Enterprise Plan

For users with Pro and Enterprise plans, the following additional configuration options are available:

Metric Configuration - Goal

Choose whether a particular metric needs to be minimized or maximized. Options are either 'minimize' or 'maximize' (optional - Default: minimize).

Select 'minimize' or 'maximize' using the radio button.

Metric Configuration - Name

Enter the name of the metric that needs to be either maximized or minimized (optional - Default: validation_loss).

Enter the metric name in the text input box.

Note: metric configuration - goal and metric configuration - name should be given together, not alone.

Custom Metrics Configuration

Please read our blog post on Custom Metrics for finetuning LLM models to understand how to create custom metrics for your finetune.

Enter the config as a JSON object to call the user-created API to get the custom loss and metrics values for test and validation datasets.

Enter the JSON object in the text input box.

Example 

{
 "url" : "https://api.domain.com/Metrics"
}

Inference Configuration

Enter a JSON object to configure the way to call the text completion OpenAI API on the finetuned models to perform inference on test and validation datasets to get metrics data.

Enter the JSON object in the text input box.

Example 

{
 "temperature" : 0,
 "top_p" : 1,
 "frequency_penalty" : 0,
 "presence_penalty" : 0,
 "stop" : "\n"
}

Max Tokens Configuration

Enter a JSON object to choose whether max_tokens in inference_config is a custom value for each prompt. If true, provide the config as an object in custom_max_tokens_config to call the user-created API to get the max_tokens value for the given prompt.

Enter the JSON object in the text input box.

Example 

{"custom_max_tokens": true,
 "custom_max_tokens_config": { "url": "https://api.domain.com/CustomMaxTokens" },
 "default_max_tokens": 22 
}