Preprocessing and filtering of the data

All time series are scanned to establish which series are producing data, and at which frequency and how often the value changes.

This is done to optimise our computational resources. We use more complex algorithms for rapid changing metrics and lighter weight algorithms for stable metrics. Non-active metrics are not considered for analysis (time series with no data points in the last 7 days).

Every week we do a full scan of all the time series and how they have been behaving in the past 7 days. We identify the typical rate at which the data come in (frequency), and and how often they change (activity). Then each time series is classified according to frequency and activity into three main groups:

High Frequency High activity: Time series with data points registered more frequently than every 15 minutes and that change values more frequently than every 15 minutes.

High Frequency Low Activity: Time series with data points registered more frequently than every 15 minutes and that change values every 15 minutes or more slowly.

Low Frequency: Times series with data points coming in with a typical interval bigger than 15 minutes.

This can be time consuming and computationally intensive, therefore it is done once per week. The rest of the week we use the values already calculated.