An Experimental Evaluation of Anomaly Detection in Time Series

Source code and supplementary materials for "An Experimental Evaluation of Anomaly Detection in Time Series".

To enable reproductivity, we open source all datasets, algorithms and codes introduced in the paper, and this document produces a guideline of reproduction.

Data

To ease reproducibility, we put it in Google Drive https://drive.google.com/drive/folders/1BJuiWMmBzK2UeiGahVoX2ofqlShELjkN

Requirement

• java:1.8
• python: 3.7
• modules needed: java in lib, python in requirements.txt

Code Structure

• Section 4.2.1 Varying Datasets
  • acc
• Section 4.2.2 Varying Anomaly Rate
  • rate
• Section 4.2.3 Varying Data Size
  • size
• Section 4.2.4 Varying Data Dimension
  • dim
• Section 4.2.5 Varying Anomaly Patterns
  • type
• Section 4.3.1 Univariate Methods in Multivariate Datasets
  • inter/UMPerDimTest
• Section 4.3.2 Online Methods in Batched Time Series
  • inter/SlideTest
• Section 4.3.3 Point Methods in Subsequence Anomalies
  • inter/SPLengthTest
• Section 4.3.4 Effect of Metrics and Adjustment
  • inter/AdjustPredictTest
• Section 4.3.5 Effect of Threshold
  • threshold
• Section 4.3.6 Application
  • application

Example Invocation

Take the code of drawing Table 4 as an example:

// Set parameters
String[] vars = {"yahoo","twitter"};
boolean[] willOperate = {true, true, true};

String[] algNames = {"NETS", "Stare","SHESD"};
String[] metricNames = {"precision", "recall", "fmeasure"};

...
    
// NETS
algIndex++;
if (willOperate[algIndex]) {
    System.out.println(algNames[algIndex] + " begin");
    if (!seriesMulMap.containsKey(dsName)) {
        timeSeriesMulDim = fh.readMulDataWithLabel(rawPath);
        seriesMulMap.put(dsName, timeSeriesMulDim);
        TreeMap<Long, TimePointMulDim> realAnomaly =
            DataHandler.findAnomalyPoint(timeSeriesMulDim);
        realAnomalyMulMap.put(dsName, realAnomaly);
    } else {
        timeSeriesMulDim = seriesMulMap.get(dsName);
    }
        algtime[algIndex][0] = System.currentTimeMillis();
        nets = new NETS();
        Map<String, Object> netsParams = meta.getDataAlgParam().get(dsName).get(algNames[algIndex]);
        nets.init(netsParams, timeSeriesMulDim);
        nets.run();
        algtime[algIndex][1] = System.currentTimeMillis();
        DataHandler.evaluate(
            timeSeriesMulDim, realAnomalyMulMap.get(dsName), metrics[index][algIndex]);
}

...
   
//Write results
fh.writeResults("acc", "uni-point", vars, algNames, metricNames, totaltime, metrics, 1);

Result:

Dataset	NETS			Stare			SHSED
	Precison	Recall	Fmeasure	Precison	Recall	Fmeasure	Precison	Recall	Fmeasure
Yahoo	0.727	1	0.842	0.429	0.375	0.400	`1	0.625	0.769
Twitter	0.739	0.878	0.802	0.203	0.959	0.335	`0.260	0.176	0.210