Biblio

We introduce AdaMix, an adaptive differentially private algorithm for training deep neural network classifiers using both private and public image data. While pre-training language models on large public datasets has enabled strong differential privacy (DP) guarantees with minor loss of accuracy, a similar practice yields punishing trade-offs in vision tasks. A few-shot or even zero-shot learning baseline that ignores private data can outperform fine-tuning on a large private dataset. AdaMix incorporates few-shot training, or cross-modal zero-shot learning, on public data prior to private fine-tuning, to improve the trade-off. AdaMix reduces the error increase from the non-private upper bound from the 167–311% of the baseline, on average across 6 datasets, to 68-92% depending on the desired privacy level selected by the user. AdaMix tackles the trade-off arising in visual classification, whereby the most privacy sensitive data, corresponding to isolated points in representation space, are also critical for high classification accuracy. In addition, AdaMix comes with strong theoretical privacy guarantees and convergence analysis.

A rendering regression is a bug introduced by a web browser where a web page no longer functions as users expect. Such rendering bugs critically harm the usability of web browsers as well as web applications. The unique aspect of rendering bugs is that they affect the presented visual appearance of web pages, but those web pages have no pre-defined correct appearance. Therefore, it is challenging to automatically detect errors in their appearance. In practice, web browser vendors rely on non-trivial and time-prohibitive manual analysis to detect and handle rendering regressions. This paper proposes R2Z2, an automated tool to find rendering regressions. R2Z2 uses the differential fuzz testing approach, which repeatedly compares the rendering results of two different versions of a browser while providing the same HTML as input. If the rendering results are different, R2Z2 further performs cross browser compatibility testing to check if the rendering difference is indeed a rendering regression. After identifying a rendering regression, R2Z2 will perform an in-depth analysis to aid in fixing the regression. Specifically, R2Z2 performs a delta-debugging-like analysis to pinpoint the exact browser source code commit causing the regression, as well as inspecting the rendering pipeline stages to pinpoint which pipeline stage is responsible. We implemented a prototype of R2Z2 particularly targeting the Chrome browser. So far, R2Z2 found 11 previously undiscovered rendering regressions in Chrome, all of which were confirmed by the Chrome developers. Importantly, in each case, R2Z2 correctly reported the culprit commit. Moreover, R2Z2 correctly pin-pointed the culprit rendering pipeline stage in all but one case.

SELinux policies used in practice are generally large and complex. As a result, it is difficult for the policy writers to completely understand the policy and ensure that the policy meets the intended security goals. To remedy this, we have developed a tool called SEFlowViz that helps in visualizing the information flows of a policy and thereby helps in creating flow-secure policies. The tool uses the graph database Neo4j to visualize the policy. Along with visualization, the tool also supports extracting various information regarding the policy and its components through queries. Furthermore, the tool also supports the addition and deletion of rules which is useful in converting inconsistent policies into consistent policies.

In order to improve the concealment of image steganography, a new method is proposed. The algorithm firstly adopted GM (1, 1) model to detect texture and edge points of carrier image, then embedded secret information in them. GM (1, 1) model of optimized parameters can make full use of pixels information. These pixels are the nearest to the detected point, so it improves the detection accuracy. The method is a kind of steganography based on human visual system. By testing the stegano images with different embedding capacities, the result indicates concealment and image quality of the proposed algorithm are better than BPCS (Bit-plane Complexity Segmentation) and PVD (Pixel-value Differencing), which are also based on visual characteristics.

Steganography is one of the techniques for secure transformation of data which aims at hiding information inside other media in such a way that no one will notice. The cover media that can accommodate secret information include text, audio, image, and video. Images are the most popular covering media in steganography, due to the fact that, they are heavily used in daily applications and have high redundancy in representation. In this paper, we propose an adaptive steganography algorithm for hiding information in RGB images. To minimize visual perceptible distortion, the proposed algorithm uses edge pixels for embedding data. It detects the edge pixels in the image using the Sobel filter. Then, the message is embedded into the LSBs of the blue channel of the edge pixels. To resist statistical attacks, the distribution of the blue channel of the edge pixels is used when embedding data in the cover image. The experimental results showed that the algorithm offers high capacity for hiding data in cover images; it does not distort the quality of the stego image; it is robust enough against statistical attacks; and its execution time is short enough for online data transfer. Also, the results showed that the proposed algorithm outperforms similar approaches in all evaluation metrics.

A modular static analysis decomposes a program's analysis into analyses of its parts, or components. An intercomponent analysis instructs an intra-component analysis to analyse each component independently of the others. Additional analyses are scheduled for newly discovered components, and for dependent components that need to account for newly discovered component information. Modular static analyses are scalable, can be tuned to a high precision, and support the analysis of programs that are highly dynamic, featuring e.g., higher-order functions or dynamically allocated processes.In this paper, we present the engineering aspects of MAF, a static analysis framework for implementing modular analyses for higher-order languages. For any such modular analysis, the framework provides a reusable inter-component analysis and it suffices to implement its intra-component analysis. The intracomponent analysis can be composed from several interdependent and reusable Scala traits. This design facilitates changing the analysed language, as well as the analysis precision with minimal effort. We illustrate the use of MAF through its instantiation for several different analyses of Scheme programs.

Protecting and preventing sensitive data from being used inappropriately has become a challenging task. Even a small mistake in securing data can be exploited by phishing attacks to release private information such as passwords or financial information to a malicious actor. Phishing has now proven so successful, it is the number one attack vector. Many approaches have been proposed to protect against this type of cyber-attack, from additional staff training, enriched spam filters to large collaborative databases of known threats such as PhishTank and OpenPhish. However, they mostly rely upon a user falling victim to an attack and manually adding this new threat to the shared pool, which presents a constant disadvantage in the fight back against phishing. In this paper, we propose a novel approach to protect against phishing attacks using binary visualisation and machine learning. Unlike previous work in this field, our approach uses an automated detection process and requires no further user interaction, which allows faster and more accurate detection process. The experiment results show that our approach has high detection rate.

Smart phones and mobile apps have become an essential part of our daily lives. It is necessary to ensure the quality of these apps. Two important aspects of code quality are maintainability and security. The goals of my PhD project are (1) to study code smells, security issues and their evolution in iOS apps and frameworks, (2) to enhance training and teaching using visualisation support, and (3) to support developers in automatically detecting dependencies to vulnerable library elements in their apps. For each of the three tools, dedicated tool support will be provided, i.e., GraphifyEvolution, VisualiseEvolution, and DependencyEvolution respectively. The tool GraphifyEvolution exists and has been applied to analyse code smells in iOS apps written in Swift. The tool has a modular architecture and can be extended to add support for additional languages and external analysis tools. In the remaining two years of my PhD studies, I will complete the other two tools and apply them in case studies with developers in industry as well as in university teaching.

Smart assistants, like Amazon's Alexa or Apple's Siri, have become commonplace in many people's lives, appearing in their phones and homes. Despite their ubiquity, these conversational AI agents still largely remain a mystery to many, in terms of how they work and what they can do. To lower the barrier to entry to understanding and creating these agents for young students, we expanded on Convo, a conversational programming agent that can respond to both voice and text inputs. The previous version of Convo focused on teaching only programming skills, so we created a simple, intuitive user interface for students to use those programming skills to train and create their own conversational AI agents. We also developed a curriculum to teach students about key concepts in AI and conversational AI in particular. We ran a 3-day workshop with 15 participating middle school students. Through the data collected from the pre- and post-workshop surveys as well as a mid-workshop brainstorming session, we found that after the workshop, students tended to think that conversational AI agents were less intelligent than originally perceived, gained confidence in their abilities to build these agents, and learned some key technical concepts about conversational AI as a whole. Based on these results, we are optimistic about CONVO'S ability to teach and empower students to develop conversational AI agents in an intuitive way.

Visual Question Answering (VQA) models have achieved significant success in recent times. Despite the success of VQA models, they are mostly black-box models providing no reasoning about the predicted answer, thus raising questions for their applicability in safety-critical such as autonomous systems and cyber-security. Current state of the art fail to better complex questions and thus are unable to exploit compositionality. To minimize the black-box effect of these models and also to make them better exploit compositionality, we propose a Dynamic Neural Network (DMN), which can understand a particular question and then dynamically assemble various relatively shallow deep learning modules from a pool of modules to form a network. We incorporate compositional temporal attention to these deep learning based modules to increase compositionality exploitation. This results in achieving better understanding of complex questions and also provides reasoning as to why the module predicts a particular answer. Experimental analysis on the two benchmark datasets, VQA2.0 and CLEVR, depicts that our model outperforms the previous approaches for Visual Question Answering task as well as provides better reasoning, thus making it reliable for mission critical applications like safety and security.

A kitchen underpinned by the Internet of Things (IoT) requires the management of complex procedural processes. This is due to the fact that when supporting an end-user in the preparation of even only one dish, various devices may need to coordinate with each other. Additionally, it is challenging— yet desirable—to enable an end-user to program their kitchen devices according to their preferred behaviour and to allow them to visualise and track their cooking workflows. In this paper, we compared two semantic representations, namely, Behaviour Trees and the DX-MAN model. We analysed these representations based on their suitability for a range of end-users (i.e., novice to experienced). The methodology required the analysis of smart kitchen user requirements, from which we inferred that the main architectural requirements for IoT cooking workflows are variability and compositionality. Guided by the user requirements, we examined various scenarios and analysed workflow complexity and feasibility for each representation. On the one hand, we found that execution complexity tends to be higher on Behaviour Trees. However, due to their fallback node, they provide more transparency on how to recover from unprecedented circumstances. On the other hand, parameter complexity tends to be somewhat higher for the DX-MAN model. Nevertheless, the DX-MAN model can be favourable due to its compositionality aspect and the ease of visualisation it can offer.

Neural Module Network (NMN) exhibits strong interpretability and compositionality thanks to its handcrafted neural modules with explicit multi-hop reasoning capability. However, most NMNs suffer from two critical draw-backs: 1) scalability: customized module for specific function renders it impractical when scaling up to a larger set of functions in complex tasks; 2) generalizability: rigid pre-defined module inventory makes it difficult to generalize to unseen functions in new tasks/domains. To design a more powerful NMN architecture for practical use, we propose Meta Module Network (MMN) centered on a novel meta module, which can take in function recipes and morph into diverse instance modules dynamically. The instance modules are then woven into an execution graph for complex visual reasoning, inheriting the strong explainability and compositionality of NMN. With such a flexible instantiation mechanism, the parameters of instance modules are inherited from the central meta module, retaining the same model complexity as the function set grows, which promises better scalability. Meanwhile, as functions are encoded into the embedding space, unseen functions can be readily represented based on its structural similarity with previously observed ones, which ensures better generalizability. Experiments on GQA and CLEVR datasets validate the superiority of MMN over state-of-the-art NMN designs. Synthetic experiments on held-out unseen functions from GQA dataset also demonstrate the strong generalizability of MMN. Our code and model are released in Github1.

In order to analyze the trustworthiness of complex software systems, we propose a model of evidence-based software trustworthiness called trustworthiness derivation tree (TDT). The basic idea of constructing a TDT is to refine main properties into key ingredients and continue the refinement until basic facts such as evidences are reached. The skeleton of a TDT can be specified by a set of rules, which is convenient for automated reasoning in Prolog. We develop a visualization tool that can construct the skeleton of a TDT by taking the rules as input, and allow a user to edit the TDT in a graphical user interface. In a software development life cycle, TDTs can serve as a communication means for different stakeholders to agree on the properties about a system in the requirement analysis phase, and they can be used for deductive reasoning so as to verify whether the system achieves trustworthiness in the product validation phase. We have piloted the approach of using TDTs in more than a dozen real scenarios of software development. Indeed, using TDTs helped us to discover and then resolve some subtle problems.

In order to improve the security and reliability of information system and reduce the risk of vulnerability intrusion and attack, an automatic patch installation method of operating systems based on deep learning is proposed, If the installation is successful, the basic information of the system will be returned to the visualization server. If the installation fails, it is recommended to upgrading manually and display it on the patch detection visualization server. Through the practical application of statistical analysis, the statistical results show that the proposed method is significantly better than the original and traditional installation methods, which can effectively avoid the problem of client repeated download, and greatly improve the success rate of patch automatic upgrades. It effectively saves the upgrade cost and ensures the security and reliability of the information system.

An image hiding algorithm based on bit plane and two-dimensional code is proposed in this paper. The main characteristic of information hiding is to use the information redundant data of the existing image, to embed the information into these redundant data by the information hiding algorithm, or to partially replace redundant information with information to be embedded to achieve a visual invisible purpose. We first analyze the color index usage frequency of the block index matrix in the algorithm, and calculate the distance between the color of the block index matrix with only one color and the other color in the palette that is closest to the color. Then, the QR model and the compression model are applied to improve the efficiency. We compare the proposed model with the stateof-the-art models.

In this paper, we will describe the design and implementation of a secure payment system based on QR codes. These QR codes have been extensively used in recent years since they speed up the payment process and provide users with ultimate convenience. However, as convenient as they may sound, QR-based online payment systems are vulnerable to different types of attacks. Therefore, transaction processing needs to be secure enough to protect the integrity and confidentiality of every payment process. Moreover, the online payment system must provide authenticity for both the sender and receiver of each transaction. In this paper, the security of the proposed QR-based system is provided using visual cryptography. The proposed system consists of a mobile application and a payment gateway server that implements visual cryptography. The application provides a simple and user-friendly interface for users to carry out payment transactions in user-friendly secure environment.

Naval vessels infrastructures are evolving towards increasingly connected and automatic systems. Such accelerated complexity boost to search for more adapted and useful navigation devices may be at odds with cybersecurity, making necessary to develop adapted analysis solutions for experts. This paper introduces a novel process to visualize and analyze naval Cyber-Physical Systems (CPS) using oriented graphs, considering operational constraints, to represent physical and functional connections between multiple components of CPS. Rapid prototyping of interconnected components is implemented in a semi-automatic manner by defining the CPS’s digital and physical systems as nodes, along with system variables as edges, to form three layers of an oriented graph, using the open-source Neo4j software suit. The generated multi-layer graph can be used to support cybersecurity analysis, like attacks simulation, anomaly detection and propagation estimation, applying existing or new algorithms.

The human face conveys a significant amount of information. Through facial expressions, the face is able to communicate numerous sentiments without the need for verbalisation. Visual emotion recognition has been extensively studied. Recently several end-to-end trained deep neural networks have been proposed for this task. However, such models often lack generalisation ability across datasets. In this paper, we propose the Deep Facial Expression Vector ExtractoR (DeepFEVER), a new deep learning-based approach that learns a visual feature extractor general enough to be applied to any other facial emotion recognition task or dataset. DeepFEVER outperforms state-of-the-art results on the AffectNet and Google Facial Expression Comparison datasets. DeepFEVER’s extracted features also generalise extremely well to other datasets – even those unseen during training – namely, the Real-World Affective Faces (RAF) dataset.

The increased demand of multimedia leads to shortage of network bandwidth and memory capacity. As a result, image compression is more significant for decreasing data redundancy, saving storage space and bandwidth. Along with the compression the next major challenge in this field is to safeguard the compressed data further from the spy which are commonly known as hackers. It is evident that the major increments in the fields like communication, wireless sensor network, data science, cloud computing and machine learning not only eases the operations of the related field but also increases the challenges as well. This paper proposes a worthy composition for image compression encryption based on unsupervised learning i.e. k-means clustering for compression with logistic chaotic map for encryption. The main advantage of the above combination is to address the problem of data storage and the security of the visual data as well. The algorithm reduces the size of the input image and also gives the larger key space for encryption. The validity of the algorithm is testified with the PSNR, MSE, SSIM and Correlation coefficient.

Recent technological advances in developing intelligent telecommunication networks, ultra-compact bendable wireless transceiver chips, adaptive wearable sensors and actuators, and secure computing infrastructures along with the progress made in psychology and neuroscience for understanding neu-rocognitive and computational principles of human behavior combined have paved the way for a new field of research: Tactile Internet with Human-in-the-Loop (TaHiL). This emerging field of transdisciplinary research aims to promote next generation digitalized human-machine interactions in perceived real time. To achieve this goal, mechanisms and principles of human goal-directed multisensory perception and action need to be integrated into technological designs for breakthrough innovations in mobile telecommunication, electronics and materials engineering, as well as computing. This overview highlights key challenges and the frontiers of research in the new field of TaHiL. Revolutionizing the current Internet as a digital infrastructure for sharing visual and auditory information globally, the TaHiL research will enable humans to share tactile and haptic information and thus veridically immerse themselves into virtual, remote, or inaccessible real environments to exchange skills and expertise with other humans or machines for applications in medicine, industry, and the Internet of Skills.

Trust is an important emerging area of study in human-robot cooperation. Many studies have begun to look at the issue of robot (agent) capability as a predictor of human trust in the robot. However, the assumption that agent capability is the sole predictor of human trust could underestimate the complexity of the problem. This study aims to investigate the effects of agent-strategy and agent-capability in a visual search task. Fourteen subjects were recruited to partake in a web-based grid search task. They were each paired with a series of autonomous agents to search an on-screen grid to find a number of outlier objects as quickly as possible. Both the human and agent searched the grid concurrently and the human was able to see the movement of the agent. Each trial, a different autonomous agent with its assigned capability, used one of three search strategies to assist their human counterpart. After each trial, the autonomous agent reported the number of outliers it found, and the human subject was asked to determine the total number of outliers in the area. Some autonomous agents reported only a fraction of the outliers they encountered, thus coding a varying level of agent capability. Human subjects then evaluated statements related to the behavior, reliability, and trust of the agent. The results showed increased measures of trust and reliability with increasing capability. Additionally, the most legible search strategies received the highest average ratings in a measure of familiarity. Remarkably, given no prior information about capabilities or strategies that they would see, subjects were able to determine consistent trustworthiness of the agent. Furthermore, both capability and strategy of the agent had statistically significant effects on the human’s trust in the agent.

This paper describes the process of developing data visualisations to enhance a commercial software platform for combating insider threat, whose existing UI, while perfectly functional, was limited in its ability to allow analysts to easily spot the patterns and outliers that visualisation naturally reveals. We describe the design and development process, proceeding from initial tasks/requirements gathering, understanding the platform’s data formats, the rationale behind the visualisations’ design, and then refining the prototype through gathering feedback from representative domain experts who are also current users of the software. Through a number of example scenarios, we show that the visualisation can support the identified tasks and aid analysts in discovering and understanding potentially risky insider activity within a large user base.

Insider threats are the cyber attacks from the trusted entities within an organization. An insider attack is hard to detect as it may not leave a footprint and potentially cause huge damage to organizations. Anomaly detection is the most common approach for insider threat detection. Lack of real-world data and the skewed class distribution in the datasets makes insider threat analysis an understudied research area. In this paper, we propose a Conditional Generative Adversarial Network (CGAN) to enrich under-represented minority class samples to provide meaningful and diverse data for anomaly detection from the original malicious scenarios. Comprehensive experiments performed on benchmark dataset demonstrates the effectiveness of using CGAN augmented data, and the capability of multi-class anomaly detection for insider activity analysis. Moreover, the method is compared with other existing methods against different parameters and performance metrics.

Performing a live digital forensics investigation on a running system is challenging due to the time pressure under which decisions have to be made. Newly proliferating and frequently applied types of malware (e.g., fileless malware) increase the need to conduct digital forensic investigations in real-time. In the course of these investigations, forensic experts are confronted with a wide range of different forensic tools. The decision, which of those are suitable for the current situation, is often based on the cyber forensics experts’ experience. Currently, there is no reliable automated solution to support this decision-making. Therefore, we derive requirements for visually supporting the decision-making process for live forensic investigations and introduce a research prototype that provides visual guidance for cyber forensic experts during a live digital forensics investigation. Our prototype collects relevant core information for live digital forensics and provides visual representations for connections between occurring events, developments over time, and detailed information on specific events. To show the applicability of our approach, we analyze an exemplary use case using the prototype and demonstrate the support through our approach.

Internet Protocol (IP) address holds a probative value to the identification process in digital forensics. The decimal digit is a unique identifier that is beneficial in many investigations (i.e., network, email, memory). IP addresses can reveal important information regarding the device that the user uses during Internet activity. One of the things that IP addresses can essentially help digital forensics investigators in is the identification of the user machine and tracing evidence based on network artifacts. Unfortunately, it appears that some of the well-known digital forensic tools only provide functions to recover IP addresses from a given forensic image. Thus, there is still a gap in answering if IP addresses found in a smartphone can help reveal the user’s location and be used to aid investigators in identifying IP addresses that complement the user’s physical location. Furthermore, the lack of utilizing IP mapping and visualizing techniques has resulted in the omission of such digital evidence. This research aims to emphasize the importance of geolocation data in digital forensic investigations, propose an IP visualization technique considering several sources of evidence, and enhance the investigation process’s speed when its pertained to IP addresses using spatial analysis. Moreover, this research proposes a proof-of-concept (POC) standalone tool that can match critical IP addresses with approximate geolocations to fill the gap in this area.