Biblio

System logs record valuable information about the runtime status of IT systems. Therefore, system logs are a naturally excellent source of information for anomaly detection. Most of the existing studies on log-based anomaly detection construct a detection model to identify anomalous logs. Generally, the model treats historical logs as natural language sequences and learns the normal patterns from normal log sequences, and detects deviations from normal patterns as anomalies. However, the majority of existing methods focus on sequential and quantitative information and ignore semantic information hidden in log sequence so that they are inefficient in anomaly detection. In this paper, we propose a novel framework for automatically detecting log anomalies by utilizing an attention-based Bi-LSTM model. To demonstrate the effectiveness of our proposed model, we evaluate the performance on a public production log dataset. Extensive experimental results show that the proposed approach outperforms all comparison methods for anomaly detection.

Deep learning has made remarkable achievements in various domains. Active learning, which aims to reduce the budget for training a machine-learning model, is especially useful for the Deep learning tasks with the demand of a large number of labeled samples. Unfortunately, our empirical study finds that many of the active learning heuristics are not effective when applied to Deep learning models in batch settings. To tackle these limitations, we propose a density weighted diversity based query strategy (DWDS), which makes use of the geometry of the samples. Within a limited labeling budget, DWDS enhances model performance by querying labels for the new training samples with the maximum informativeness and representativeness. Furthermore, we propose a beam-search based method to obtain a good approximation to the optimum of such samples. Our experiments show that DWDS outperforms existing algorithms in Deep learning tasks.