Skip to content

Commit

Permalink
Added documentation for comparing Datasets. Added content to the Horr…
Browse files Browse the repository at this point in the history
…eum glossary of terms.
  • Loading branch information
Jeremy Whiting authored and johnaohara committed Jan 22, 2024
1 parent 54a3834 commit befcf30
Show file tree
Hide file tree
Showing 7 changed files with 75 additions and 1 deletion.
34 changes: 33 additions & 1 deletion docs/site/content/en/docs/Concepts/core-concepts/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,4 +90,36 @@ A `Fingerprint` is combination of [`Filtering labels`](#label) that unique ident

## Datasource

A `Datasource` is a **required** top-level organizational construct that defines the source of the data to be stored or retrieved by Horreum. Currently, Horreum supports 2 types of `Datasource`: Postgres and Elasticsearch
A `Datasource` is a **required** top-level organizational construct that defines the source of the data to be stored or retrieved by Horreum. Currently, Horreum supports 2 types of `Datasource`: Postgres and Elasticsearch

## Baseline

The *initial* sample for an `Experiment` comparison. Configured in an `Experiment Profile`.

## Change Detection Variable

Change detection tracks `Schema` `Label`s that have been configured as a Change Detection Variable.

## Experiment

This enables running a comparison between `Runs` for a particular `Test`. There can be multiple `Profile`s configured for an `Experiment` to check for a variety of desired conditions. The outcome status for each `Profile` condition will be one of the following:
- SAME
- WORSE
- BETTER

## Experiment Profile

A `Profile` consists of:
- `Experiment` selector
- `Baseline`
- 0, 1 or many Comparison conditions

## JSON validation schema
An *optional* schema added to a `Test` to validate uploaded `Run` JSON data.







Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
42 changes: 42 additions & 0 deletions docs/site/content/en/docs/Tasks/experiment-evaluation/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
---
title: Dataset Experiment Evaluation
description: Document explaining the use of Experiment Evaluation function in Horreum UI
date: 2024-01-18
weight: 5
---

## Using the Dataset Experiment Evaluation View

Using the Experiment evaluation window you can quickly see the values and the relative difference of a Run.
Start by initially loading the Test. Then click the `Dataset list` button.

{{% imgproc dataset_list Fit "1200x900" %}} Dataset List {{% /imgproc %}}

Then select the individual Test Run you want to compare with it's Baseline.

By navigating to the uploaded run Dataset view page you will see a button “Evaluate experiment”. Clicking this button opens a window revealing the comparison result for the current Run and the Baseline Run..

{{% imgproc individual_evaluation Fit "1200x900" %}} Individual Evaluation {{% /imgproc %}}

Results show the values then the percentage difference.


## Datasets Comparison View

Horreum provides multiple Run comparisons in the Dataset Comparison View. We can filter based on the Filter labels defined in the Schema.

Start by initially loading the Dataset list. Then click the button “Select for comparison”.

{{% imgproc select_for_comparison Fit "1200x900" %}} Select for comparison {{% /imgproc %}}

Next the Comparison view is displayed. This is where filters are set.

{{% imgproc dataset_selection Fit "1200x900" %}} Dataset selection {{% /imgproc %}}

Select a number of Runs to be compared using the “Add to comparison” button. Then click the “Compare labels”. Displayed next is the Labels Comparison view.

{{% imgproc multiple_dataset_comparison Fit "1200x900" %}} Multiple Dataset comparison {{% /imgproc %}}

Displayed here are multiple Datasets. Schema Labels can be expanded to display a bar graph representing each Run.

In this guide two things were shown. How to compare an individual Dataset. Followed by comparing multiple Datasets.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit befcf30

Please sign in to comment.