Test August 2024 - Cyclical data

Summary of the results: This test was conducted using a portion of the Server Machine Dataset, which showed strong daily cyclicality. By applying labelling that captured the major deviations from this cyclical pattern, we achieved an F1 score of 0.86. More details can be found here: Note on performance testing of the core algorithm of Eyer.

Test type

Calculation of the F1 score, recall (R) and precision (P)

F1 = 2*R*P / (R+P)

R= truePositives/(truePositives + falseNegatives)

P =truePositives/(truePositives +falsePositives) 

the closer to 1 the F1 score is, the best is the performance of an algorithm. R equal 1 means that all the real anomalies are detected. P close to 1 means that we have few false positive alerts.

Real anomalies are defined by labels, that are used in the test. Our algorithm does not utilise labels in the training phase, since it is of unsupervised type.

Data used

We used metrics with a prominent daily cyclical behaviour coming from an open source dataset: Server Machine Dataset.

We used 2 different labelling:

labelling 1: focusing only on certain type of anomalies, original labelling in the open dataset.

labelling 2: more extended labelling, labelling as anomalous all the deviations from the cyclical daily behaviour.

Results

We present the results of the analysis for labelling 2, using only critical anomalies, and after collecting the first 4 weeks of data (our algorithm takes few weeks to reach full performance). for other results please refer to the note.

Recall : 0.999

Precision: 0.76

F1: 0.86

This means that our algorithm is extremely efficient in finding deviations from daily cycles. Focusing only on critical anomalies only 1/4 of the alerts will not correspond to real anomalies, while almost all the anomalies are captured.

This is a test with a rather small dataset, in the future we want to use bigger and more varied datasets.