Share via

AzureOpenAIScoreModelGrader Class

Note

This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.

Wrapper class for OpenAI's score model graders.

Enables continuous scoring evaluation with custom prompts and flexible conversation-style inputs. Supports configurable score ranges and pass thresholds for binary classification.

Supplying a ScoreModelGrader to the evaluate method will cause an asynchronous request to evaluate the grader via the OpenAI API. The results of the evaluation will then be merged into the standard evaluation results.

Constructor

AzureOpenAIScoreModelGrader(*, model_config: AzureOpenAIModelConfiguration | OpenAIModelConfiguration, input: List[Dict[str, str]], model: str, name: str, range: List[float] | None = None, pass_threshold: float | None = None, sampling_params: Dict[str, Any] | None = None, credential: TokenCredential | None = None, **kwargs: Any)

Parameters

Name Description
model_config
Required
Union[AzureOpenAIModelConfiguration, OpenAIModelConfiguration]

The model configuration to use for the grader.
input
Required
List[Dict[str, str]]

The input messages for the grader. List of conversation messages with role and content.
model
Required
str

The model to use for the evaluation.
name
Required
str

The name of the grader.
range
Required
Optional[List[float]]

The range of the score. Defaults to [0, 1].
pass_threshold
Required
Optional[float]

Score threshold for pass/fail classification. Defaults to midpoint of range.
sampling_params
Required
Optional[Dict[str, Any]]

The sampling parameters for the model.
credential
Required
TokenCredential

The credential to use to authenticate to the model. Only applicable to AzureOpenAI models.
kwargs
Required
Any

Additional keyword arguments to pass to the grader.

Keyword-Only Parameters

Name Description
model_config
Required
input
Required
model
Required
name
Required
range
Default value: None
pass_threshold
Default value: None
sampling_params
Default value: None
credential
Default value: None

Methods

get_client

Construct an appropriate OpenAI client using this grader's model configuration. Returns a slightly different client depending on whether or not this grader's model configuration is for Azure OpenAI or OpenAI.

get_client

Construct an appropriate OpenAI client using this grader's model configuration. Returns a slightly different client depending on whether or not this grader's model configuration is for Azure OpenAI or OpenAI.

get_client() -> Any

Returns

Type Description
[<xref:openai.OpenAI>, <xref:openai.AzureOpenAI>]

The OpenAI client.

Attributes

id 

id = 'azureai://built-in/evaluators/azure-openai/score_model_grader'