...
Summary of the results: This test was conducted using a portion of the Server Machine Dataset, which showed strong daily cyclicality. By applying labelling labeling that captured the major deviations from this cyclical pattern, we achieved an F1 score of 0.86. More details can be found here: Note on F1 performance testing of the core algorithm of Eyer.
...
the closer to 1 the F1 score is, the best better is the performance of an algorithm. R equal 1 means that all the real anomalies are detected. P close to 1 means that we have few false positive alerts.
Real anomalies are defined by labels, that are used in the test. Our algorithm does not utilise utilize labels in the training phase, since it is of unsupervised type.
...
We used metrics with a prominent daily cyclical behaviour behavior coming from an open source dataset: Server Machine Dataset.
We used 2 different labellinglabelings:
labelling labeling 1: focusing only on certain type of anomalies, original labelling labeling in the open dataset.
labelling labeling 2: more extended labellinglabeling, labelling labeling as anomalous all the deviations from the cyclical daily behaviourbehavior.
Results
We present the results of the analysis for labelling labeling 2, using only critical anomalies, and after collecting the first 4 weeks of data (our algorithm takes a few weeks to reach full performance). for other results please refer to the note.
Recall : 0.999
Precision: 0.76
F1: 0.86
This means that our algorithm is extremely efficient in finding deviations from daily cycles. Focusing only on critical anomalies only 1/4 of the alerts will not correspond to real anomalies, while almost all the anomalies are captured.
This is a test with a rather small dataset, in the future we want to use bigger and more varied datasets.
For other results please refer to the note.