Biblio

Automatic test systems designed to validate the performance of military and aerospace products have always been held to a higher standard; moreover, emerging threats to data security and instrumentation integrity continue to raise this bar. Engineers are faced with growing pressure to not only ensure that the unit under test (UUT) meets all design criteria, but that it remains safe from malicious attacks aimed at gaining access to test parameters or results, controlling of test sequences and functionality, downloading malware, or impacting functionality by way of counterfeit parts installed in instrumentation. This paper will delve into the cybersecurity issue from the perspective of the test development environment, including the use of test executives, and the challenges associated with minimizing impact to data integrity and access to control. An undetected data breach on military / aerospace automated test equipment (ATE) holds significance beyond just the test system, since mission critical electronics associated with avionics, radar, electronic warfare and missile assemblies must also be protected. One topic discussed will be the impact of adopting methods and procedures detailed in the Department of Defense's (DoD) Application Security Technical Implementation Guide, which is based on NIST documents and details how to manage and maintain a secure software-based system such as an ATE system. Another aspect of cybersecurity that is often overlooked in the world of commercial-off-the-shelf (COTS) instrumentation and switching systems is the potential impact on the UUT from substandard counterfeit parts and those embedded with malware. Concerns with counterfeit material can encompass a range of threats including the re-purposing of used parts and new knockoff parts with substandard operating characteristics represented and sold as new hardware. One of the most concerning aspects, parts intentionally infected with malware, can pose a significant risk to personnel and national security. We will discuss various strategies aimed at countering these threats, including the adoption of policies and procedures outlined in AS9100D and AS5553, which can mitigate these risks.

Software Engineering is a field of new challenges every day. With every passing day, new technologies emerge. There was an era of web Applications, but the time has changed and most of the web Applications are available as Mobile Applications as well. The Mobile Applications are either android based or iOS based. To deliver error free, secure and reliable Application, it is necessary to test the Applications properly. Software testing is a phase of software development life cycle, where we test an Application in all aspects. Nowadays different type of tools are available for testing an Application automatically but still we have too many challenges for applying test cases on a given Application. In this paper the authors will discuss the challenges of automation of test cases for a Mobile based payment Application.

Internet of Things (IoT) and cloud computing are promising technologies that change the way people communicate and live. As the data collected through IoT devices often involve users' private information and the cloud is not completely trusted, users' private data are usually encrypted before being uploaded to cloud for security purposes. Searchable encryption, allowing users to search over the encrypted data, extends data flexibility on the premise of security. In this paper, to achieve the accurate and efficient ciphertext searching, we present an efficient multi-keyword ranked searchable encryption scheme supporting ciphertext-policy attribute-based encryption (CP-ABE) test (MRSET). For efficiency, numeric hierarchy supporting ranked search is introduced to reduce the dimensions of vectors and matrices. For practicality, CP-ABE is improved to support access right test, so that only documents that the user can decrypt are returned. The security analysis shows that our proposed scheme is secure, and the experimental result demonstrates that our scheme is efficient.

There is a need for new tools and techniques to aid automotive engineers performing cybersecurity testing on connected car systems. This is in order to support the principle of secure-by-design. Our research has produced a method to construct useful automotive security tooling and tests. It has been used to implement Controller Area Network (CAN) fuzz testing (a dynamic security test) via a prototype CAN fuzzer. The black-box fuzz testing of a laboratory vehicle's display ECU demonstrates the value of a fuzzer in the automotive field, revealing bugs in the ECU software, and weaknesses in the vehicle's systems design.

CPS is generally complex to study, analyze, and design, as an important means to ensure the correctness of design and implementation of CPS system, simulation test is difficult to fully test, verify and evaluate the components or subsystems in the CPS system due to the inconsistent development progress of the com-ponents or subsystems in the CPS system. To address this prob-lem, we designed a hybrid P2P based collaborative simulation test framework composed of full physical nodes, hardware in the loop(HIL) nodes and full digital nodes to simulate the compo-nents or subsystems in the CPS system of different development progress, based on the framework, we then proposed collabora-tive simulation control strategy comprising sliding window based clock synchronization, dynamic adaptive time advancement and multi-priority task scheduling with preemptive time threshold. Experiments showed that the hybrid collaborative simulation testing method proposed in this paper can fully test CPS more effectively.

This paper presents a general model to assess the mixed-degradation profiles of critical components in a Cyber-Physical System (CPS) based on the reliability of its critical physical and software components. In the proposed assessment, the cyber aspect of a CPS was approached from a software reliability perspective. Although extensive research has been done on physical components degradation and software reliability separately, research for the combined physical-software systems is still scarce. The non-homogeneous Poisson Processes (NHPP) software reliability models are deemed to fit well with the real data and have descriptive and predictive abilities, which could make them appropriate to estimate software components reliability. To show the feasibility of the proposed approach, a case study for mixed-degradation profiles assessment is presented with n physical components and one major software component forming a critical subsystem in CPS. Two physical components were assumed to have different degradation paths with the dependency between them. Series and parallel structures were investigated for physical components. The software component failure data was taken from a wireless network switching center and fitted into a Weibull software reliability model. The case study results revealed that mix-degradation profiles of physical components, combined with software component profile, produced a different CPS reliability profile.

An approach for effective regression test selection is proposed, which minimizes the resource usage and amount of time required for complete testing of new features. Provided are the details of the analysis of hashing algorithms used during implementation in-depth review of the software, together with the results achieved during the testing process.

Software integration in modern vehicles is continuously expanding. This is due to the fact that vehicle manufacturers are always trying to enhance and add more innovative and competitive features that may rely on complex software functionalities. However, these features come at a cost. They amplify the security vulnerabilities in vehicles and lead to more security issues in today's automobiles. As a result, the need for identifying vulnerable components in a vehicle software system has become crucial. Security experts need to know which components of the vehicle software system can be exploited for attacks and should focus their testing and inspection efforts on it. Nevertheless, it is a challenging and costly task to identify these weak components in a vehicle's system. In this paper, we propose some security vulnerability metrics for connected vehicles that aim to assist software testers during the development life-cycle in order to identify the frail links that put the vehicle at highsecurity risks. Vulnerable function assessment can give software testers a good idea about which components in a connected vehicle need to be prioritized in order to mitigate the risk and hence secure the vehicle. The proposed metrics were applied to OpenPilot - a software that provides Autopilot feature - and has been integrated with 48 different vehicles.. The application shows how the defined metrics can be effectively used to quantitatively measure the vulnerabilities of a vehicle software system.

The purpose of this paper is to analyze all Cloud based Service Models, Continuous Integration, Deployment and Delivery process and propose an Automated Continuous Testing and testing as a service based TestBot and metrics dashboard which will be integrated with all existing automation, bug logging, build management, configuration and test management tools. Recently cloud is being used by organizations to save time, money and efforts required to setup and maintain infrastructure and platform. Continuous Integration and Delivery is in practice nowadays within Agile methodology to give capability of multiple software releases on daily basis and ensuring all the development, test and Production environments could be synched up quickly. In such an agile environment there is need to ramp up testing tools and processes so that overall regression testing including functional, performance and security testing could be done along with build deployments at real time. To support this phenomenon, we researched on Continuous Testing and worked with industry professionals who are involved in architecting, developing and testing the software products. A lot of research has been done towards automating software testing so that testing of software product could be done quickly and overall testing process could be optimized. As part of this paper we have proposed ACT TestBot tool, metrics dashboard and coined 4S quality metrics term to quantify quality of the software product. ACT testbot and metrics dashboard will be integrated with Continuous Integration tools, Bug reporting tools, test management tools and Data Analytics tools to trigger automation scripts, continuously analyze application logs, open defects automatically and generate metrics reports. Defect pattern report will be created to support root cause analysis and to take preventive action.

The use of risk information can help software engineers identify software components that are likely vulnerable or require extra attention when testing. Some studies have shown that the requirements risk-based approaches can be effective in improving the effectiveness of regression testing techniques. However, the risk estimation processes used in such approaches can be subjective, time-consuming, and costly. In this research, we introduce a fuzzy expert system that emulates human thinking to address the subjectivity related issues in the risk estimation process in a systematic and an efficient way and thus further improve the effectiveness of test case prioritization. Further, the required data for our approach was gathered by employing a semi-automated process that made the risk estimation process less subjective. The empirical results indicate that the new prioritization approach can improve the rate of fault detection over several existing test case prioritization techniques, while reducing threats to subjective risk estimation.

Misuse of APIs happens frequently due to misunderstanding of API semantics and lack of documentation. An important category of API-related defects is the error handling defects, which may result in security and reliability flaws. These defects can be detected with the help of static program analysis, provided that error specifications are known. The error specification of an API function indicates how the function can fail. Writing error specifications manually is time-consuming and tedious. Therefore, automatic inferring the error specification from API usage code is preferred. In this paper, we present Ares, a tool for automatic inferring error specifications for C code through static analysis. We employ multiple heuristics to identify error handling blocks and infer error specifications by analyzing the corresponding condition logic. Ares is evaluated on 19 real world projects, and the results reveal that Ares outperforms the state-of-the-art tool APEx by 37% in precision. Ares can also identify more error specifications than APEx. Moreover, the specifications inferred from Ares help find dozens of API-related bugs in well-known projects such as OpenSSL, among them 10 bugs are confirmed by developers. Video: https://youtu.be/nf1QnFAmu8Q. Repository: https://github.com/lc3412/Ares.

Current testing for Deep Neural Networks (DNNs) focuses on quantity of test cases but ignores diversity. To the best of our knowledge, DeepXplore is the first white-box framework for Deep Learning testing by triggering differential behaviors between multiple DNNs and increasing neuron coverage to improve diversity. Since it is based on multiple DNNs facing problems that (1) the framework is not friendly to a single DNN, (2) if incorrect predictions made by all DNNs simultaneously, DeepXplore cannot generate test cases. This paper presents Test4Deep, a white-box testing framework based on a single DNN. Test4Deep avoids mistakes of multiple DNNs by inducing inconsistencies between predicted labels of original inputs and that of generated test inputs. Meanwhile, Test4Deep improves neuron coverage to capture more diversity by attempting to activate more inactivated neurons. The proposed method was evaluated on three popular datasets with nine DNNs. Compared to DeepXplore, Test4Deep produced average 4.59% (maximum 10.49%) more test cases that all found errors and faults of DNNs. These test cases got 19.57% more diversity increment and 25.88% increment of neuron coverage. Test4Deep can further be used to improve the accuracy of DNNs by average up to 5.72% (maximum 7.0%).

Static application security testing (SAST) detects vulnerability warnings through static program analysis. Fixing the vulnerability warnings tremendously improves software quality. However, SAST has not been fully utilized by developers due to various reasons: difficulties in handling a large number of reported warnings, a high rate of false warnings, and lack of guidance in fixing the reported warnings. In this paper, we collaborated with security experts from a commercial SAST product and propose a set of approaches (Priv) to help developers better utilize SAST techniques. First, Priv identifies preferred fix locations for the detected vulnerability warnings, and group them based on the common fix locations. Priv also leverages visualization techniques so that developers can quickly investigate the warnings in groups and prioritize their quality-assurance effort. Second, Priv identifies actionable vulnerability warnings by removing SAST-specific false positives. Finally, Priv provides customized fix suggestions for vulnerability warnings. Our evaluation of Priv on six web applications highlights the accuracy and effectiveness of Priv. For 75.3% of the vulnerability warnings, the preferred fix locations found by Priv are identical to the ones annotated by security experts. The visualization based on shared preferred fix locations is useful for prioritizing quality-assurance efforts. Priv reduces the rate of SAST-specific false positives significantly. Finally, Priv is able to provide fully complete and correct fix suggestions for 75.6% of the evaluated warnings. Priv is well received by security experts and some features are already integrated into industrial practice.

Enforcement of hypersafety security policies such as noninterference can be achieved through Secure Multi-Execution (SME). While this is typically very resource-intensive, more efficient solutions such as Demand-Driven Secure Multi-Execution (DDSME) exist. Here, the resource requirements are reduced by restricting multi-execution enforcement to critical sections in the code. However, the current solution requires manual binary analysis. In this paper, we propose a fully automatic critical section analysis. Our analysis extracts a context-sensitive boundary of all nodes that handle information from the reachability relation implied by the control-flow graph. We also provide evaluation results, demonstrating the correctness and acceleration of DDSME with our analysis.

In this paper, we present a combinatorial testing methodology for testing web applications in regards to SQL injection vulnerabilities. We describe three attack grammars that were developed and used to generate concrete attack vectors. Furthermore, we present and evaluate two different oracles used to observe the application's behavior when subjected to such attack vectors. We also present a prototype tool called SQLInjector capable of automated SQL injection vulnerability testing for web applications. The developed methodology can be applied to any web application that uses server side scripting and HTML for handling user input and has a SQL database backend. Our approach relies on the use of a database proxy, making this a gray-box testing method. We establish the effectiveness of the proposed tool with the WAVSEP verification framework and conduct a case study on real-world web applications, where we are able to discover both known vulnerabilities and additional previously undiscovered flaws.

Typically, in an assessment project for a web application or database with a large scale and scope, tasks required to be performed by a security analyst are such as SQL injection and penetration testing. To carry out these large-scale tasks, the analyst will have to perform 100 or more SQLi penetration tests on one or more target. This makes the process much more complex and much harder to implement. This paper attempts to compare large-scale SQL injections performed with Manual Methods, which is the benchmark, and the proposed SQLiAutoScript Method. The SQLiAutoScript method uses sqlmap as a tool, in combination with sqlmap scripting and logging features, to facilitate a more effective and manageable approach within a large scale of hundreds or thousands of SQL injection penetration tests. Comparison of the test results for both Manual and SQLiAutoScript approaches and their benefits is included in the comparative analysis. The tests were performed over a scope of 24 SQL injection (SQLi) tests that comprises over 100,000 HTTP requests and injections, and within a total testing run-time period of about 50 hours. The scope of testing also covers both SQLiAutoScript and Manual methods. In the SQLiAutoScript method, each SQL injection test has its own sub-folder and files for data such as results (output), progress (traffic logs) and logging. In this way across all SQLi tests, the results, data and details related to SQLi tests are logged, available, traceable, accurate and not missed out. Available and traceable data also facilitates traceability of failed SQLi tests, and higher recovery and reruns of failed SQLi tests to maximize increased attack surface upon the target.

The IoT industry has developed rapidly in recent years, which has attracted the attention of security researchers. However, the researchers are hampered by the wide variety of IoT device operating systems and their hardware architectures. Especially for the lightweight IoT devices, many manufacturers do not provide the device firmware images, embedded firmware source code or even the develop documents. As a result, it hinders traditional static analysis and dynamic analysis techniques. In this paper, we propose a novel dynamic analysis framework, called FIoT, which aims at finding memory corruption vulnerabilities in lightweight IoT device firmware images. The key idea is dynamically run the binary code snippets through symbolic execution with carrying out a fuzzing test. Specifically, we generate code snippets through traversing the control-flow graph (CFG) in a backward manner. We improved the CFG recovery approach and backward slice approach for better performance. To reduce the influence of the binary firmware, FIoT leverages loading address determination analysis and library function identification approach. We have implemented a prototype of FIoT and conducted experiments. Our results show that FIoT can complete the Fuzzing test within 40 seconds in average. Considering 170 seconds for static analysis, FIoT can load and analyze a lightweight IoT firmware within 210 seconds in total. Furthermore, we illustrate the effectiveness of FIoT by applying it over 115 firmware images from 17 manufacturers. We have found 35 images exist memory corruptions, which are all zero-day vulnerabilities.

To build a secure communications software, Vulnerability Prediction Models (VPMs) are used to predict vulnerable software modules in the software system before software security testing. At present many software security metrics have been proposed to design a VPM. In this paper, we predict vulnerable classes in a software system by establishing the system's weighted software network. The metrics are obtained from the nodes' attributes in the weighted software network. We design and implement a crawler tool to collect all public security vulnerabilities in Mozilla Firefox. Based on these data, the prediction model is trained and tested. The results show that the VPM based on weighted software network has a good performance in accuracy, precision, and recall. Compared to other studies, it shows that the performance of prediction has been improved greatly in Pr and Re.

Exploits based on ROP (Return-Oriented Programming) are increasingly present in advanced attack scenarios. Testing systems for ROP-based attacks can be valuable for improving the security and reliability of software. In this paper, we propose ROPMATE, the first Visual Analytics system specifically designed to assist human red team ROP exploit builders. In contrast, previous ROP tools typically require users to inspect a puzzle of hundreds or thousands of lines of textual information, making it a daunting task. ROPMATE presents builders with a clear interface of well-defined and semantically meaningful gadgets, i.e., fragments of code already present in the binary application that can be chained to form fully-functional exploits. The system supports incrementally building exploits by suggesting gadget candidates filtered according to constraints on preserved registers and accessed memory. Several visual aids are offered to identify suitable gadgets and assemble them into semantically correct chains. We report on a preliminary user study that shows how ROPMATE can assist users in building ROP chains.

Assertions are helpful in program analysis, such as software testing and verification. The most challenging part of automatically recommending assertions is to design the assertion patterns and to insert assertions in proper locations. In this paper, we develop Weak-Assert, a weakness-oriented assertion recommendation toolkit for program analysis of C code. A weakness-oriented assertion is an assertion which can help to find potential program weaknesses. Weak-Assert uses well-designed patterns to match the abstract syntax trees of source code automatically. It collects significant messages from trees and inserts assertions into proper locations of programs. These assertions can be checked by using program analysis techniques. The experiments are set up on Juliet test suite and several actual projects in Github. Experimental results show that Weak-Assert helps to find 125 program weaknesses in 26 actual projects. These weaknesses are confirmed manually to be triggered by some test cases.

Aiming at the problem that there is little research on firmware vulnerability mining and the traditional method of vulnerability mining based on fuzzing test is inefficient, this paper proposed a new method of mining vulnerabilities in industrial control system firmware. Based on taint analysis technology, this method can construct test cases specifically for the variables that may trigger vulnerabilities, thus reducing the number of invalid test cases and improving the test efficiency. Experiment result shows that this method can reduce about 23 % of test cases and can effectively improve test efficiency.

Despite recent regulations and growing user awareness, undesired browser tracking is increasing. In addition to cookies, browser fingerprinting is a stateless technique that exploits a device's configuration for tracking purposes. In particular, browser fingerprinting builds on attributes made available from Javascript and HTTP headers to create a unique and stable fingerprint. For example, browser plugins have been heavily exploited by state-of-the-art browser fingerprinters as a rich source of entropy. However, as browser vendors abandon plugins in favor of extensions, fingerprinters will adapt. We present FP-TESTER, an approach to automatically test the effectiveness of browser fingerprinting countermeasure extensions. We implement a testing toolkit to be used by developers to reduce browser fingerprintability. While countermeasures aim to hinder tracking by changing or blocking attributes, they may easily introduce subtle side-effects that make browsers more identifiable, rendering the extensions counterproductive. FP-TESTER reports on the side-effects introduced by the countermeasure, as well as how they impact tracking duration from a fingerprinter's point-of-view. To the best of our knowledge, FP-TESTER is the first tool to assist developers in fighting browser fingerprinting and reducing the exposure of end-users to such privacy leaks.

There are over 1 billion websites today, and most of them are designed using content management systems. Cybersecurity is one of the most discussed topics when it comes to a web application and protecting the confidentiality, integrity of data has become paramount. SQLi is one of the most commonly used techniques that hackers use to exploit a security vulnerability in a web application. In this paper, we compared SQLi vulnerabilities found on the three most commonly used content management systems using a vulnerability scanner called Nikto, then SQLMAP for penetration testing. This was carried on default WordPress, Drupal and Joomla website pages installed on a LAMP server (Iocalhost). Results showed that each of the content management systems was not susceptible to SQLi attacks but gave warnings about other vulnerabilities that could be exploited. Also, we suggested practices that could be implemented to prevent SQL injections.

Fuzzing is a simple yet effective approach to discover software bugs utilizing randomly generated inputs. However, it is limited by coverage and cannot find bugs hidden in deep execution paths of the program because the randomly generated inputs fail complex sanity checks, e.g., checks on magic values, checksums, or hashes. To improve coverage, existing approaches rely on imprecise heuristics or complex input mutation techniques (e.g., symbolic execution or taint analysis) to bypass sanity checks. Our novel method tackles coverage from a different angle: by removing sanity checks in the target program. T-Fuzz leverages a coverage-guided fuzzer to generate inputs. Whenever the fuzzer can no longer trigger new code paths, a light-weight, dynamic tracing based technique detects the input checks that the fuzzer-generated inputs fail. These checks are then removed from the target program. Fuzzing then continues on the transformed program, allowing the code protected by the removed checks to be triggered and potential bugs discovered. Fuzzing transformed programs to find bugs poses two challenges: (1) removal of checks leads to over-approximation and false positives, and (2) even for true bugs, the crashing input on the transformed program may not trigger the bug in the original program. As an auxiliary post-processing step, T-Fuzz leverages a symbolic execution-based approach to filter out false positives and reproduce true bugs in the original program. By transforming the program as well as mutating the input, T-Fuzz covers more code and finds more true bugs than any existing technique. We have evaluated T-Fuzz on the DARPA Cyber Grand Challenge dataset, LAVA-M dataset and 4 real-world programs (pngfix, tiffinfo, magick and pdftohtml). For the CGC dataset, T-Fuzz finds bugs in 166 binaries, Driller in 121, and AFL in 105. In addition, found 3 new bugs in previously-fuzzed programs and libraries.

Although the importance of mobile applications grows every day, recent vulnerability reports argue the application's deficiency to meet modern security standards. Testing strategies alleviate the problem by identifying security violations in software implementations. This paper proposes a novel testing methodology that applies state machine learning of mobile Android applications in combination with algorithms that discover attack paths in the learned state machine. The presence of an attack path evidences the existence of a vulnerability in the mobile application. We apply our methods to real-life apps and show that the novel methodology is capable of identifying vulnerabilities.