Biblio

Benchmark datasets are heavily depended upon by the research community to validate theoretical findings and track progression in the state-of-the-art. NIDS dataset creation presents numerous challenges on account of the volume, heterogeneity, and complexity of network traffic, making the process labor intensive, and thus, prone to error. This paper provides a critical review of CIC-IDS-2017 and CIC-CSE-IDS-2018, datasets which have seen extensive usage in the NIDS literature, and are currently considered primary benchmarking datasets for NIDS. We report a large number of previously undocumented errors throughout the dataset creation lifecycle, including in attack orchestration, feature generation, documentation, and labeling. The errors destabilize the results and challenge the findings of numerous publications that have relied on it as a benchmark. We demonstrate the implications of these errors through several experiments. We provide comprehensive documentation to summarize the discovery of these issues, as well as a fully-recreated dataset, with labeling logic that has been reverse-engineered, corrected, and made publicly available for the first time. We demonstrate the implications of dataset errors through a series of experiments. The findings serve to remind the research community of common pitfalls with dataset creation processes, and of the need to be vigilant when adopting new datasets. Lastly, we strongly recommend the release of labeling logic for any dataset released, to ensure full transparency.

Network attacks are always a nightmare for the network administrators as it eats away a huge wavelength and disturbs the normal working of many critical services in the network. Network behavior based profiling and detection is considered to be an accepted method; but the modeling data and method is always a big concern. The network event-based profiling is getting acceptance as they are sequential in nature and the sequence depicts the behavior of the system. This sequential network events can be analyzed using different techniques to create a profile for anomaly detection. In this paper we examine the possibility of two techniques for sequential event analysis using Modified GSP and IPAM algorithm. We evaluate the performance of these algorithms on the CSE-CIC-IDS 2018 data set to benchmark the performance. This experiment is different from other anomaly-based detection which evaluates the features of the dataset to detect the abnormalities. The performance of the algorithms on the dataset is then confirmed by the pattern evolving from the analysis and the indications it provides for early detection of network attacks.

As one of the defensive solutions against cyberattacks, an Intrusion Detection System (IDS) plays an important role in observing the network state and alerting suspicious actions that can break down the system. There are many attempts of adopting Machine Learning (ML) in IDS to achieve high performance in intrusion detection. However, all of them necessitate a large amount of labeled data. In addition, labeling attack data is a time-consuming and expensive human-labor operation, it makes existing ML methods difficult to deploy in a new system or yields lower results due to a lack of labels on pre-trained data. To address these issues, we propose a semi-supervised IDS model that leverages Generative Adversarial Networks (GANs) and Adversarial AutoEncoder (AAE), called a semi-supervised adversarial autoencoder (SAAE). Our SAAE experimental results on two public datasets for benchmarking ML-based IDS, including NF-CSE-CIC-IDS2018 and NF-UNSW-NB15, demonstrate the effectiveness of AAE and GAN in case of using only a small number of labeled data. In particular, our approach outperforms other ML methods with the highest detection rates in spite of the scarcity of labeled data for model training, even with only 1% labeled data.

We discuss the performance of a new quantumsafe multivariate digital signature scheme proposed recently, called the Multivariate Polynomial Public Key Digital Signature (MPPK DS) scheme. Leveraging MPPK KEM or key exchange mechanism, the MPPK DS scheme is established using modular exponentiation with a randomly chosen secret base from a prime field. The security of the MPPK DS algorithm largely benefits from a generalized safe prime associated with the said field and the Euler totient function. We can achieve NIST security levels I, III, and V over a 64-bit prime field, with relatively small public key sizes of 128 bytes, 192 bytes, and 256 bytes for security levels I, III, and V, respectively. The signature sizes are 80 bytes for level I, 120 bytes for level III, and 160 bytes for level V. The MPPK DS scheme offers probabilistic procedures for signing and verification. That is, for each given signing message, a signer can randomly pick a base integer to be used for modular exponentiation with a private key, and a verifier can verify the signature with the digital message, based on the verification relationship, using any randomly selected noise variables. The verification process can be repeated as many times as the verifier wishes for different noise values, however, for a true honest signature, the verification will always pass. This probabilistic feature largely restricts an adversary to perform spoofing attacks. In this paper, we conduct some performance analyses by implementing MPPK DS in Java. We compare its performance with benchmark performances of NIST PQC Round 3 finalists: Rainbow, Dilithium, and Falcon. Overall, the MPPK DS scheme demonstrates equivalent or better performance, and much smaller public key, as well as signature sizes, compared to the three NIST PQC Round 3 finalists.

In this paper, we address the virtual network embedding (VNE) problem in non-terrestrial networks (NTNs) enabling dynamic changes in the virtual network function (VNF) deployment to maximize the service acceptance rate and service revenue. NTNs such as satellite networks involve highly dynamic topology and limited resources in terms of rate and power. VNE in NTNs is a challenge because a static strategy under-performs when new service requests arrive or the network topology changes unexpectedly due to failures or other events. Existing solutions do not consider the power constraint of satellites and rate limitation of inter-satellite links (ISLs) which are essential parameters for dynamic adjustment of existing VNE strategy in NTNs. In this work, we propose a dynamic VNE algorithm that selects a suitable VNE strategy for new and existing services considering the time-varying network topology. The proposed scheme, D-ViNE, increases the service acceptance ratio by 8.51% compared to the benchmark scheme TS-MAPSCH.

This contribution provides the implementation of a digital twin-based assistance system to be used in future control rooms. By applying parameter estimation methods, the dynamic model in the digital twin is an accurate representation of the physical system. Therefore, a dynamic security assessment (DSA) that is highly dependent on a correctly parameterized dynamic model, can give more reliable information to a system operator in the control room. The assistance system is studied on the Cigré TB 536 benchmark system with an obscured set of machine parameters. Through the proposed parameter estimation approach the original parameters could be estimated, changing, and increasing the statement of the DSA in regard to imminent instabilities.

A direct access (DAX) file system maximizes the benefit of persistent memory(PM)’s low latency through removing the page cache layer from the file system access paths. However, this paper reveals that data block allocation of the DAX file systems in common is significantly slower than that of conventional file systems because the DAX file systems require the zero-out operation for the newly allocated blocks to prevent the leakage of old data previously stored in the allocated data blocks. The retarded block allocation significantly affects the file write performance. In addition to this revelation, this paper proposes an off-critical-path data block sanitization scheme tailored for DAX file systems. The proposed scheme detaches the zero-out operation from the latency-critical I/O path and performs that of released data blocks in the background. The proposed scheme’s design principle is universally applicable to most DAX file systems. For evaluation, we implemented our approach in Ext4-DAX and XFS-DAX. Our evaluation showed that the proposed scheme reduces the append write latency by 36.8%, and improved the performance of FileBench’s fileserver workload by 30.4%, YCSB’s workload A on RocksDB by 3.3%, and the Redis-benchmark by 7.4% on average, respectively.

The damage or destruction of Critical Infrastructures (CIs) affect societies’ sustainable functioning. Therefore, it is crucial to have effective methods to assess the risk and resilience of CIs. Failure Mode and Effects Analysis (FMEA) and Failure Mode Effects and Criticality Analysis (FMECA) are two approaches to risk assessment and criticality analysis. However, these approaches are complex to apply to intricate CIs and associated Cyber-Physical Systems (CPS). We provide a top-down strategy, starting from a high abstraction level of the system and progressing to cover the functional elements of the infrastructures. This approach develops from FMECA but estimates risks and focuses on assessing resilience. We applied the proposed technique to a real-world CI, predicting how possible improvement scenarios may influence the overall system resilience. The results show the effectiveness of our approach in benchmarking the CI resilience, providing a cost-effective way to evaluate plausible alternatives concerning the improvement of preventive measures.

Multi-label image classification task aims to predict multiple object labels in a given image and faces the challenge of variable-sized objects. Limited by the size of CNN convolution kernels, existing CNN-based methods have difficulty capturing global dependencies and effectively fusing multiple layers features, which is critical for this task. Recently, transformers have utilized multi-head attention to extract feature with long range dependencies. Inspired by this, this paper proposes a Cross-layer Aggregation with Transformers (CAT) framework, which leverages transformers to capture the long range dependencies of CNN-based features with Long Range Dependencies module and aggregate the features layer by layer with Cross-Layer Fusion module. To make the framework efficient, a multi-head pre-max attention is designed to reduce the computation cost when fusing the high-resolution features of lower-layers. On two widely-used benchmarks (i.e., VOC2007 and MS-COCO), CAT provides a stable improvement over the baseline and produces a competitive performance.

Code optimization is an essential feature for compilers and almost all software products are released by compiler optimizations. Consequently, bugs in code optimization will inevitably cast significant impact on the correctness of software systems. Locating optimization bugs in compilers is challenging as compilers typically support a large amount of optimization configurations. Although prior studies have proposed to locate compiler bugs via generating witness test programs, they are still time-consuming and not effective enough. To address such limitations, we propose an automatic bug localization approach, ODFL, for locating compiler optimization bugs via differentiating finer-grained options in this study. Specifically, we first disable the fine-grained options that are enabled by default under the bug-triggering optimization levels independently to obtain bug-free and bug-related fine-grained options. We then configure several effective passing and failing optimization sequences based on such fine-grained options to obtain multiple failing and passing compiler coverage. Finally, such generated coverage information can be utilized via Spectrum-Based Fault Localization formulae to rank the suspicious compiler files. We run ODFL on 60 buggy GCC compilers from an existing benchmark. The experimental results show that ODFL significantly outperforms the state-of-the-art compiler bug isolation approach RecBi in terms of all the evaluated metrics, demonstrating the effectiveness of ODFL. In addition, ODFL is much more efficient than RecBi as it can save more than 88% of the time for locating bugs on average.

Ransomware is an emerging threat that imposed a \$ 5 billion loss in 2017, rose to \$ 20 billion in 2021, and is predicted to hit \$ 256 billion in 2031. While initially targeting PC (client) platforms, ransomware recently leaped over to server-side databases-starting in January 2017 with the MongoDB Apocalypse attack and continuing in 2020 with 85,000 MySQL instances ransomed. Previous research developed countermeasures against client-side ransomware. However, the problem of server-side database ransomware has received little attention so far. In our work, we aim to bridge this gap and present DIMAQS (Dynamic Identification of Malicious Query Sequences), a novel anti-ransomware solution for databases. DIMAQS performs runtime monitoring of incoming queries and pattern matching using two classification approaches (Colored Petri Nets (CPNs) and Deep Neural Networks (DNNs)) for attack detection. Our system design exhibits several novel techniques like dynamic color generation to efficiently detect malicious query sequences globally (i.e., without limiting detection to distinct user connections). Our proof-of-concept and ready-to-use implementation targets MySQL servers. The evaluation shows high efficiency without false negatives for both approaches and a false positive rate of nearly 0%. Both classifiers show very moderate performance overheads below 6%. We will publish our data sets and implementation, allowing the community to reproduce our tests and results.

Safety-critical systems require resiliency against both cyberattacks and environmental faults. Researches have shown that microkernels can isolate components and limit the capabilities of would-be attackers by confining the attack in the component that it is initiated in. This limits the propagation of faults to sensitive components in the system. Nonetheless, the isolation mechanism in microkernels is not fully investigated for its resiliency against hardware faults. This paper investigates whether microkernels provide protection against hardware faults and, if so, to what extent quantitatively. This work is part of an effort in establishing an overlap between security and reliability with the goal of maximizing both while minimizing their impact on performance. In this work, transient faults are emulated on the seL4 microkernel and Linux kernel using debugger-induced bit flips across random timestamps in benchmark applications. Results show differences in the frequency and final outcome of fault to error manifestation in the seL4 environment compared to the Linux environment, including a reduction in silent data corruptions.

Scan-based test methodology is one of the most popular test techniques in VLSI circuits. This methodology increases the testability which in turn improves the fault coverage. For this purpose, the technique uses a chain of scan cells. This becomes a source of attack for an attacker who can observe / control the internal states and use the information for malicious purposes. Hence, security becomes the main concern in the Integrated Circuit (IC) domain since scan chains are the main reason for leakage of confidential information during testing phase. These leakages will help attackers in reverse engineering. Measures against such attacks have to be taken by encrypting the data which flows through the scan chains. Lightweight ciphers can be used for scan chain encryption. In this work, encryption of scan data is done for ISCAS-89 benchmarks and the performance and security properties are evaluated. Lightweight stream and block ciphers are used to perform scan encryption. A comparative analysis between the two techniques is performed in par with the functions related to design cost and security properties.

The volume of SMS messages sent on a daily basis globally has continued to grow significantly over the past years. Hence, mobile phones are becoming increasingly vulnerable to SMS spam messages, thereby exposing users to the risk of fraud and theft of personal data. Filtering of messages to detect and eliminate SMS spam is now a critical functionality for which different types of machine learning approaches are still being explored. In this paper, we propose a system for detecting SMS spam using a semi-supervised novelty detection approach based on one class SVM classifier. The system is built as an anomaly detector that learns only from normal SMS messages thus enabling detection models to be implemented in the absence of labelled SMS spam training examples. We evaluated our proposed system using a benchmark dataset consisting of 747 SMS spam and 4827 non-spam messages. The results show that our proposed method out-performed the traditional supervised machine learning approaches based on binary, frequency or TF-IDF bag-of-words. The overall accuracy was 98% with 100% SMS spam detection rate and only around 3% false positive rate.

Synthetic static code analysis test suites are important to test the basic functionality of tools. We present a framework that uses different source code patterns to generate Cross Site Scripting and SQL injection test cases. A decision tree is used to determine if the test cases are vulnerable. The test cases are split into two test suites. The first test suite contains 258,432 test cases that have influence on the decision trees. The second test suite contains 20 vulnerable test cases with different data flow patterns. The test cases are scanned with two commercial static code analysis tools to show that they can be used to benchmark and identify problems of static code analysis tools. Expert interviews confirm that the decision tree is a solid way to determine the vulnerable test cases and that the test suites are relevant.

Autonomous vehicles (AVs) are capable of making driving decisions autonomously using multiple sensors and a complex autonomous driving (AD) software. However, AVs introduce numerous unique security challenges that have the potential to create safety consequences on the road. Security mechanisms require a benchmark suite and an evaluation framework to generate comparable results. Unfortunately, AVs lack a proper benchmarking framework to evaluate the attack and defense mechanisms and quantify the safety measures. This paper introduces BenchAV – a security benchmark suite and evaluation framework for AVs to address current limitations and pressing challenges of AD security. The benchmark suite contains 12 security and performance metrics, and an evaluation framework that automates the metric collection process using Carla simulator and Robot Operating System (ROS).

Securing cloud configurations is an elusive task, which is left up to system administrators who have to base their decisions on "trial and error" experimentations or by observing good practices (e.g., CIS Benchmarks). We propose a knowledge, AND/OR, graphs approach to model cloud deployment security objects and vulnerabilities. In this way, we can capture relationships between configurations, permissions (e.g., CAP\_SYS\_ADMIN), and security profiles (e.g., AppArmor and SecComp). Such an approach allows us to suggest alternative and safer configurations, support administrators in the study of what-if scenarios, and scale the analysis to large scale deployments. We present an initial validation and illustrate the approach with three real vulnerabilities from known sources.

This paper introduces lronMask, a new versatile verification tool for masking security. lronMask is the first to offer the verification of standard simulation-based security notions in the probing model as well as recent composition and expandability notions in the random probing model. It supports any masking gadgets with linear randomness (e.g. addition, copy and refresh gadgets) as well as quadratic gadgets (e.g. multiplication gadgets) that might include non-linear randomness (e.g. by refreshing their inputs), while providing complete verification results for both types of gadgets. We achieve this complete verifiability by introducing a new algebraic characterization for such quadratic gadgets and exhibiting a complete method to determine the sets of input shares which are necessary and sufficient to perform a perfect simulation of any set of probes. We report various benchmarks which show that lronMask is competitive with state-of-the-art verification tools in the probing model (maskVerif, scVerif, SILVEH, matverif). lronMask is also several orders of magnitude faster than VHAPS -the only previous tool verifying random probing composability and expandability- as well as SILVEH -the only previous tool providing complete verification for quadratic gadgets with nonlinear randomness. Thanks to this completeness and increased performance, we obtain better bounds for the tolerated leakage probability of state-of-the-art random probing secure compilers.

Due to the increasing complexity of modern hetero-geneous System-on-Chips (SoC) and the growing vulnerabilities, security risk assessment and quantification is required to measure the trustworthiness of a SoC. This paper describes a systematic approach to model the security risk of a system for malicious hardware attacks. The proposed method uses graph analysis to assess the impact of an attack and the Common Vulnerability Scoring System (CVSS) is used to quantify the security level of the system. To demonstrate the applicability of the proposed metric, we consider two open source SoC benchmarks with different architectures. The overall risk is calculated using the proposed metric by computing the exploitability and impact of attack on critical components of a SoC.

Classification problems have been part of numerous real-life applications in fields of security, medicine, agriculture, and more. Due to the wide range of applications, there is a constant need for more accurate and efficient methods. Besides more efficient and better classification algorithms, the optimal feature set is a significant factor for better classification accuracy. In general, more features can better describe instances, but besides showing differences between instances of different classes, it can also capture many similarities that lead to wrong classification. Determining the optimal feature set can be considered a hard optimization problem for which different metaheuristics, like swarm intelligence algorithms can be used. In this paper, we propose an adaptation of hybridized swarm intelligence (SI) algorithm for feature selection problem. To test the quality of the proposed method, classification was done by k-means algorithm and it was tested on 17 benchmark datasets from the UCI repository. The results are compared to similar approaches from the literature where SI algorithms were used for feature selection, which proves the quality of the proposed hybridized SI method. The proposed method achieved better classification accuracy for 16 datasets. Higher classification accuracy was achieved while simultaneously reducing the number of used features.

XAI with natural language processing aims to produce human-readable explanations as evidence for AI decision-making, which addresses explainability and transparency. However, from an HCI perspective, the current approaches only focus on delivering a single explanation, which fails to account for the diversity of human thoughts and experiences in language. This paper thus addresses this gap, by proposing a generative XAI framework, INTERACTION (explain aNd predicT thEn queRy with contextuAl CondiTional varIational autO-eNcoder). Our novel framework presents explanation in two steps: (step one) Explanation and Label Prediction; and (step two) Diverse Evidence Generation. We conduct intensive experiments with the Transformer architecture on a benchmark dataset, e-SNLI [1]. Our method achieves competitive or better performance against state-of-the-art baseline models on explanation generation (up to 4.7% gain in BLEU) and prediction (up to 4.4% gain in accuracy) in step one; it can also generate multiple diverse explanations in step two.

Logic locking is a holistic design-for-trust technique that aims to protect the design intellectual property (IP) from untrustworthy entities throughout the supply chain. Functional and structural analysis-based attacks successfully circumvent state-of-the-art, provably secure logic locking (PSLL) techniques. However, such attacks are not holistic and target specific implementations of PSLL. Automating the detection and subsequent removal of protection logic added by PSLL while accounting for all possible variations is an open research problem. In this paper, we propose GNNUnlock, the first-of-its-kind oracle-less machine learning-based attack on PSLL that can identify any desired protection logic without focusing on a specific syntactic topology. The key is to leverage a well-trained graph neural network (GNN) to identify all the gates in a given locked netlist that belong to the targeted protection logic, without requiring an oracle. This approach fits perfectly with the targeted problem since a circuit is a graph with an inherent structure and the protection logic is a sub-graph of nodes (gates) with specific and common characteristics. GNNs are powerful in capturing the nodes' neighborhood properties, facilitating the detection of the protection logic. To rectify any misclassifications induced by the GNN, we additionally propose a connectivity analysis-based post-processing algorithm to successfully remove the predicted protection logic, thereby retrieving the original design. Our extensive experimental evaluation demonstrates that GNNUnlock is 99.24% - 100% successful in breaking various benchmarks locked using stripped-functionality logic locking [1], tenacious and traceless logic locking [2], and Anti-SAT [3]. Our proposed post-processing enhances the detection accuracy, reaching 100% for all of our tested locked benchmarks. Analysis of the results corroborates that GNNUnlock is powerful enough to break the considered schemes under different parameters, synthesis settings, and technology nodes. The evaluation further shows that GNNUnlock successfully breaks corner cases where even the most advanced state-of-the-art attacks [4], [5] fail. We also open source our attack framework [6].

As Meltdown mitigation, Kernel Page Table I solation (KPTI) was merged into Linux kernel mainline, and the performance impact is significant on x86 processors. Most of the previous work focuses on how KPTI affects Linux kernel performance within the scope of virtual machines or physical machines on x86. However, whether host KPTI affects virtual machines has not been well studied. What's more, there is relatively little research on ARM CPUs. This paper presents an in-depth study of how KPTI on the host affects the virtualized server performance and compares ARMv8 and x86. We first run several application benchmarks to demonstrate the performance impact does exist. The reason is that with a para-virtual I/O scheme, guest offloads I/O requests to the host side, which may incur user/kernel transitions. For the network I/O, when using QEMU as the back-end device, we saw a 1.7% and 5.5% slowdown on ARMv8 and x86, respectively. vhost and vhost-user, originally proposed to optimize performance, inadvertently mitigate the performance impact introduced by host KPTI. For CPU and memory-intensive benchmarks, the performance impact is trivial. We also find that virtual machines on ARMv8 are less affected by KPTI. To diagnose the root cause, we port HyperBench to the ARM virtualization platform. The final results show that swapping the translation table pointer register on ARMv8 is about 3.5x faster than x86. Our findings have significant implications for tuning the x86 virtualization platform's performance and helping ARMv8 administrators enable KPTI with confidence.

In high-performance computing (HPC), scientific applications often manage a massive amount of data using I/O libraries. These libraries provide convenient data model abstractions, help ensure data portability, and, most important, empower end users to improve I/O performance by tuning configurations across multiple layers of the HPC I/O stack. We propose SCTuner, an autotuner integrated within the I/O library itself to dynamically tune both the I/O library and the underlying I/O stack at application runtime. To this end, we introduce a statistical benchmarking method to profile the behaviors of individual supercomputer I/O subsystems with varied configurations across I/O layers. We use the benchmarking results as the built-in knowledge in SCTuner, implement an I/O pattern extractor, and plan to implement an online performance tuner as the SCTuner runtime. We conducted a benchmarking analysis on the Summit supercomputer and its GPFS file system Alpine. The preliminary results show that our method can effectively extract the consistent I/O behaviors of the target system under production load, building the base for I/O autotuning at application runtime.

With the rapid development of cloud computing, IaaS (Infrastructure as a Service) becomes more and more popular. IaaS customers may not clearly know the actual performance of each cloud platform. Moreover, there are no unified standards in performance evaluation of IaaS VMs (virtual machine). The underlying virtualization technology of IaaS cloud is transparent to customers. In this paper, we will design an automated performance benchmarking platform which can automatically install, configure and execute each benchmarking tool with a configuration center. This platform can easily visualize multidimensional benchmarking parameters data of each IaaS cloud platform. We also rented four IaaS VMs from AliCloud-Beijing, AliCloud-Qingdao, UCloud and Huawei to validate our benchmarking system. Performance comparisons of multiple parameters between multiple platforms were shown in this paper. However, in practice, customers' applications running on VMs are often complex. Performance of complex applications may not depend on single benchmarking parameter (e.g. CPU, memory, disk I/O etc.). We ran a TPC-C test for example to get overall performance in MySQL application scenario. The effects of different benchmarking parameters differ in this specific scenario.