Publications | Lukas Struppek

2024

Exploring the Adversarial Capabilities of Large Language Models

Lukas Struppek, Minh Hieu Le, Dominik Hintersdorf, and Kristian Kersting

International Conference on Learning Representations (ICLR) - Workshop on Secure and Trustworthy Large Language Models 2024

Abs Bib PDF

The proliferation of large language models (LLMs) has sparked widespread and general interest due to their strong language generation capabilities, offering great potential for both industry and research. While previous research delved into the security and privacy issues of LLMs, the extent to which these models can exhibit adversarial behavior remains largely unexplored. Addressing this gap, we investigate whether common publicly available LLMs have inherent capabilities to perturb text samples to fool safety measures, so-called adversarial examples resp. attacks. More specifically, we investigate whether LLMs are inherently able to craft adversarial examples out of benign samples to fool existing safe rails. Our experiments, which focus on hate speech detection, reveal that LLMs succeed in finding adversarial perturbations, effectively undermining hate speech detection systems. Our findings carry significant implications for (semi-)autonomous systems relying on LLMs, highlighting potential challenges in their interaction with existing systems and safety measures.
@article{struppek24adversarialllm, author = {Struppek, Lukas and Le, Minh Hieu and Hintersdorf, Dominik and Kersting, Kristian}, title = {Exploring the Adversarial Capabilities of Large Language Models}, journal = {International Conference on Learning Representations (ICLR) - Workshop on Secure and Trustworthy Large Language Models}, year = {2024}, }
CollaFuse: Navigating Limited Resources and Privacy in Collaborative Generative AI

Domenique Zipperling, Simeon Allmendinger, Lukas Struppek, and Niklas Kühl

In European Conference on Information Systems (ECIS) 2024

Abs Bib PDF

In the landscape of generative artificial intelligence, diffusion-based models present challenges for socio-technical systems in data requirements and privacy. Traditional approaches like federated learning distribute the learning process but strain individual clients, especially with constrained resources (e.g., edge devices). In response to these challenges, we introduce CollaFuse, a novel framework inspired by split learning. Tailored for efficient and collaborative use of denoising diffusion probabilistic models, CollaFuse enables shared server training and inference, alleviating client computational burdens. This is achieved by retaining data and computationally inexpensive GPU processes locally at each client while outsourcing the computationally expensive processes to the shared server. Demonstrated in a healthcare context, CollaFuse enhances privacy by highly reducing the need for sensitive information sharing. These capabilities hold the potential to impact various application areas, such as the design of edge computing solutions, healthcare research, or autonomous driving. In essence, our work advances distributed machine learning, shaping the future of collaborative GenAI networks.
@inproceedings{zipperling24collafuse, author = {Zipperling, Domenique and Allmendinger, Simeon and Struppek, Lukas and Kühl, Niklas}, title = {CollaFuse: Navigating Limited Resources and Privacy in Collaborative Generative AI}, year = {2024}, booktitle = {European Conference on Information Systems (ECIS)}, }
Be Careful What You Smooth For: Label Smoothing Can Be a Privacy Shield but Also a Catalyst for Model Inversion Attacks

Lukas Struppek, Dominik Hintersdorf, and Kristian Kersting

In International Conference on Learning Representations (ICLR) 2024

Abs Bib PDF Code Poster

Label smoothing – using softened labels instead of hard ones – is a widely adopted regularization method for deep learning, showing diverse benefits such as enhanced generalization and calibration. Its implications for preserving model privacy, however, have remained unexplored. To fill this gap, we investigate the impact of label smoothing on model inversion attacks (MIAs), which aim to generate class-representative samples by exploiting the knowledge encoded in a classifier, thereby inferring sensitive information about its training data. Through extensive analyses, we uncover that traditional label smoothing fosters MIAs, thereby increasing a model’s privacy leakage. Even more, we reveal that smoothing with negative factors counters this trend, impeding the extraction of class-related information and leading to privacy preservation, beating state-of-the-art defenses. This establishes a practical and powerful novel way for enhancing model resilience against MIAs.
@inproceedings{struppek24smoothing, author = {Struppek, Lukas and Hintersdorf, Dominik and Kersting, Kristian}, title = {Be Careful What You Smooth For: Label Smoothing Can Be a Privacy Shield but Also a Catalyst for Model Inversion Attacks}, booktitle = {International Conference on Learning Representations (ICLR)}, year = {2024}, }
Does CLIP Know My Face?

Dominik Hintersdorf, Lukas Struppek, Manuel Brack, Felix Friedrich, Patrick Schramowski, and Kristian Kersting

Journal of Artificial Intelligence Research (JAIR) 2024

Abs Bib PDF Code

With the rise of deep learning in various applications, privacy concerns around the protection of training data has become a critical area of research. Whereas prior studies have focused on privacy risks in single-modal models, we introduce a novel method to assess privacy for multi-modal models, specifically vision-language models like CLIP. The proposed Identity Inference Attack (IDIA) reveals whether an individual was included in the training data by querying the model with images of the same person. Letting the model choose from a wide variety of possible text labels, the model reveals whether it recognizes the person and, therefore, was used for training. Our large-scale experiments on CLIP demonstrate that individuals used for training can be identified with very high accuracy. We confirm that the model has learned to associate names with depicted individuals, implying the existence of sensitive information that can be extracted by adversaries. Our results highlight the need for stronger privacy protection in large-scale models and suggest that IDIAs can be used to prove the unauthorized use of data for training and to enforce privacy laws.
@article{hintersdorf24clipknow, author = {Hintersdorf, Dominik and Struppek, Lukas and Brack, Manuel and Friedrich, Felix and Schramowski, Patrick and Kersting, Kristian}, title = {Does CLIP Know My Face?}, journal = {Journal of Artificial Intelligence Research (JAIR)}, year = {2024}, }

2023

Defending Our Privacy With Backdoors

Dominik Hintersdorf, Lukas Struppek, Daniel Neider, and Kristian Kersting

Conference and Workshop on Neural Information Processing Systems (NeurIPS) - Workshop on Backdoors in Deep Learning: The Good, the Bad, and the Ugly 2023

Abs Bib PDF Code

The proliferation of large AI models trained on uncurated, often sensitive web-scraped data has raised significant privacy concerns. One of the concerns is that adversaries can extract information about the training data using privacy attacks. Unfortunately, the task of removing specific information from the models without sacrificing performance is not straightforward and has proven to be challenging. We propose a rather easy yet effective defense based on backdoor attacks to remove private information such as names of individuals from models, and focus in this work on text encoders. Specifically, through strategic insertion of backdoors, we align the embeddings of sensitive phrases with those of neutral terms-"a person" instead of the person’s name. Our empirical results demonstrate the effectiveness of our backdoor-based defense on CLIP by assessing its performance using a specialized privacy attack for zero-shot classifiers. Our approach provides not only a new "dual-use" perspective on backdoor attacks, but also presents a promising avenue to enhance the privacy of individuals within models trained on uncurated web-scraped data.
@article{hintersdorf23defending, author = {Hintersdorf, Dominik and Struppek, Lukas and Neider, Daniel and Kersting, Kristian}, title = {Defending Our Privacy With Backdoors}, journal = {Conference and Workshop on Neural Information Processing Systems (NeurIPS) - Workshop on Backdoors in Deep Learning: The Good, the Bad, and the Ugly}, year = {2023}, }
Leveraging Diffusion-Based Image Variations for Robust Training on Poisoned Data

Lukas Struppek*, Martin B. Hentschel*, Clifton Poth*, Dominik Hintersdorf, and Kristian Kersting

Neural Information Processing Systems (NeurIPS) - Workshop on Backdoors in Deep Learning: The Good, the Bad, and the Ugly 2023

Abs Bib PDF Code

Backdoor attacks pose a serious security threat for training neural networks as they surreptitiously introduce hidden functionalities into a model. Such backdoors remain silent during inference on clean inputs, evading detection due to inconspicuous behavior. However, once a specific trigger pattern appears in the input data, the backdoor activates, causing the model to execute its concealed function. Detecting such poisoned samples within vast datasets is virtually impossible through manual inspection. To address this challenge, we propose a novel approach that enables model training on potentially poisoned datasets by utilizing the power of recent diffusion models. Specifically, we create synthetic variations of all training samples, leveraging the inherent resilience of diffusion models to potential trigger patterns in the data. By combining this generative approach with knowledge distillation, we produce student models that maintain their general performance on the task while exhibiting robust resistance to backdoor triggers.
@article{struppek23leveraging, author = {Struppek*, Lukas and Hentschel*, Martin B. and Poth*, Clifton and Hintersdorf, Dominik and Kersting, Kristian}, title = {Leveraging Diffusion-Based Image Variations for Robust Training on Poisoned Data}, journal = {Neural Information Processing Systems (NeurIPS) - Workshop on Backdoors in Deep Learning: The Good, the Bad, and the Ugly}, year = {2023}, }
Balancing Transparency and Risk: The Security and Privacy Risks of Open-Source Machine Learning Models

Dominik Hintersdorf*, Lukas Struppek*, and Kristian Kersting

arXiv preprint 2023

Abs Bib PDF

The field of artificial intelligence (AI) has experienced remarkable progress in recent years, driven by the widespread adoption of open-source machine learning models in both research and industry. Considering the resource-intensive nature of training on vast datasets, many applications opt for models that have already been trained. Hence, a small number of key players undertake the responsibility of training and publicly releasing large pre-trained models, providing a crucial foundation for a wide range of applications. However, the adoption of these open-source models carries inherent privacy and security risks that are often overlooked. To provide a concrete example, an inconspicuous model may conceal hidden functionalities that, when triggered by specific input patterns, can manipulate the behavior of the system, such as instructing self-driving cars to ignore the presence of other vehicles. The implications of successful privacy and security attacks encompass a broad spectrum, ranging from relatively minor damage like service interruptions to highly alarming scenarios, including physical harm or the exposure of sensitive user data. In this work, we present a comprehensive overview of common privacy and security threats associated with the use of open-source models. By raising awareness of these dangers, we strive to promote the responsible and secure use of AI systems.
@article{hintersdorf23balancing, author = {Hintersdorf*, Dominik and Struppek*, Lukas and Kersting, Kristian}, title = {Balancing Transparency and Risk: The Security and Privacy Risks of Open-Source Machine Learning Models}, journal = {arXiv preprint}, volume = {arXiv:2308.09490}, year = {2023}, }
Class Attribute Inference Attacks: Inferring Sensitive Class Information by Diffusion-Based Attribute Manipulations

Lukas Struppek, Dominik Hintersdorf, Felix Friedrich, Manuel Brack, Patrick Schramowski, and Kristian Kersting

arXiv preprint 2023

Abs Bib PDF Code

Neural network-based image classifiers are powerful tools for computer vision tasks, but they inadvertently reveal sensitive attribute information about their classes, raising concerns about their privacy. To investigate this privacy leakage, we introduce the first Class Attribute Inference Attack (Caia), which leverages recent advances in text-to-image synthesis to infer sensitive attributes of individual classes in a black-box setting, while remaining competitive with related white-box attacks. Our extensive experiments in the face recognition domain show that Caia can accurately infer undisclosed sensitive attributes, such as an individual’s hair color, gender and racial appearance, which are not part of the training labels. Interestingly, we demonstrate that adversarial robust models are even more vulnerable to such privacy leakage than standard models, indicating that a trade-off between robustness and privacy exists.
@article{struppek23caia, author = {Struppek, Lukas and Hintersdorf, Dominik and Friedrich, Felix and Brack, Manuel and Schramowski, Patrick and Kersting, Kristian}, title = {Class Attribute Inference Attacks: Inferring Sensitive Class Information by Diffusion-Based Attribute Manipulations}, journal = {arXiv preprint}, volume = {arXiv:2303.09289}, year = {2023}, }
Fair Diffusion: Instructing Text-to-Image Generation Models on Fairness

Felix Friedrich, Manuel Brack, Lukas Struppek, Dominik Hintersdorf, Patrick Schramowski, Sasha Luccioni, and Kristian Kersting

arXiv preprint 2023

Abs Bib PDF Code

Generative AI models have recently achieved astonishing results in quality and are consequently employed in a fast-growing number of applications. However, since they are highly data-driven, relying on billion-sized datasets randomly scraped from the internet, they also suffer from degenerated and biased human behavior, as we demonstrate. In fact, they may even reinforce such biases. To not only uncover but also combat these undesired effects, we present a novel strategy, called Fair Diffusion, to attenuate biases after the deployment of generative text-to-image models. Specifically, we demonstrate shifting a bias, based on human instructions, in any direction yielding arbitrarily new proportions for, e.g., identity groups. As our empirical evaluation demonstrates, this introduced control enables instructing generative image models on fairness, with no data filtering and additional training required.
@article{friedrich23fairdiffusion, author = {Friedrich, Felix and Brack, Manuel and Struppek, Lukas and Hintersdorf, Dominik and Schramowski, Patrick and Luccioni, Sasha and Kersting, Kristian}, title = {Fair Diffusion: Instructing Text-to-Image Generation Models on Fairness}, journal = {arXiv preprint}, volume = {arXiv:2302.10893}, year = {2023}, }
SEGA: Instructing Text-to-Image Models using Semantic Guidance

Manuel Brack, Felix Friedrich, Dominik Hintersdorf, Lukas Struppek, Patrick Schramowski, and Kristian Kersting

In Conference on Neural Information Processing Systems (NeurIPS) 2023

Abs Bib PDF Code

Text-to-image diffusion models have recently received a lot of interest for their astonishing ability to produce high-fidelity images from text only. However, achieving one-shot generation that aligns with the user’s intent is nearly impossible, yet small changes to the input prompt often result in very different images. This leaves the user with little semantic control. To put the user in control, we show how to interact with the diffusion process to flexibly steer it along semantic directions. This semantic guidance (SEGA) generalizes to any generative architecture using classifier-free guidance. More importantly, it allows for subtle and extensive edits, composition and style changes, and optimizing the overall artistic conception. We demonstrate SEGA’s effectiveness on both latent and pixel-based diffusion models such as Stable Diffusion, Paella, and DeepFloyd-IF using a variety of tasks, thus providing strong evidence for its versatility and flexibility.
@inproceedings{brack23sega, author = {Brack, Manuel and Friedrich, Felix and Hintersdorf, Dominik and Struppek, Lukas and Schramowski, Patrick and Kersting, Kristian}, title = {SEGA: Instructing Text-to-Image Models using Semantic Guidance}, booktitle = {Conference on Neural Information Processing Systems (NeurIPS)}, pages = {}, year = {2023}, }
Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesis

Lukas Struppek, Dominik Hintersdorf, and Kristian Kersting

In International Conference on Computer Vision (ICCV) 2023

Abs Bib PDF Code Poster

While text-to-image synthesis currently enjoys great popularity among researchers and the general public, the security of these models has been neglected so far. Many text-guided image generation models rely on pre-trained text encoders from external sources, and their users trust that the retrieved models will behave as promised. Unfortunately, this might not be the case. We introduce backdoor attacks against text-guided generative models and demonstrate that their text encoders pose a major tampering risk. Our attacks only slightly alter an encoder so that no suspicious model behavior is apparent for image generations with clean prompts. By then inserting a single character trigger into the prompt, e.g., a non-Latin character or emoji, the adversary can trigger the model to either generate images with pre-defined attributes or images following a hidden, potentially malicious description. We empirically demonstrate the high effectiveness of our attacks on Stable Diffusion and highlight that the injection process of a single backdoor takes less than two minutes. Besides phrasing our approach solely as an attack, it can also force an encoder to forget phrases related to certain concepts, such as nudity or violence, and help to make image generation safer.
@inproceedings{struppek23rickrolling, author = {Struppek, Lukas and Hintersdorf, Dominik and Kersting, Kristian}, booktitle = {International Conference on Computer Vision (ICCV)}, year = {2023}, }
Exploiting Cultural Biases via Homoglyphs in Text-to-Image Synthesis

Lukas Struppek, Dominik Hintersdorf, Felix Friedrich, Manuel Brack, Patrick Schramowski, and Kristian Kersting

Journal of Artificial Intelligence Research (JAIR) 2023

Abs Bib PDF Code

Models for text-to-image synthesis, such as DALL-E 2 and Stable Diffusion, have recently drawn a lot of interest from academia and the general public. These models are capable of producing high-quality images that depict a variety of concepts and styles when conditioned on textual descriptions. However, these models adopt cultural characteristics associated with specific Unicode scripts from their vast amount of training data, which may not be immediately apparent. We show that by simply inserting single non-Latin characters in a textual description, common models reflect cultural stereotypes and biases in their generated images. We analyze this behavior both qualitatively and quantitatively, and identify a model’s text encoder as the root cause of the phenomenon. Additionally, malicious users or service providers may try to intentionally bias the image generation to create racist stereotypes by replacing Latin characters with similarly-looking characters from non-Latin scripts, so-called homoglyphs. To mitigate such unnoticed script attacks, we propose a novel homoglyph unlearning method to fine-tune a text encoder, making it robust against homoglyph manipulations.
@article{struppek22homoglyphs, author = {Struppek, Lukas and Hintersdorf, Dominik and Friedrich, Felix and Brack, Manuel and Schramowski, Patrick and Kersting, Kristian}, title = {Exploiting Cultural Biases via Homoglyphs in Text-to-Image Synthesis}, journal = {Journal of Artificial Intelligence Research (JAIR)}, volume = {78}, year = {2023}, pages = {1017--1068}, }
Combining AI and AM – Improving Approximate Matching through Transformer Networks

Frieder Uhlig*, Lukas Struppek*, Dominik Hintersdorf*, Thomas Göbel, Harald Baier, and Kristian Kersting

In Annual DFRWS USA Conference 2023

Abs Bib PDF

Approximate matching is a well-known concept in digital forensics to determine the similarity between digital artifacts. An important use case of approximate matching is the reliable and efficient detection of case-relevant data structures on a blacklist (e.g., malware or corporate secrets), if only fragments of the original are available. For instance, if only a cluster of indexed malware is still present during the digital forensic investigation, the approximate matching algorithm shall be able to assign the fragment to the blacklisted malware. However, traditional approximate matching functions like TLSH and ssdeep fail to detect files based on their fragments if the presented piece is relatively small compared to the overall file size (e.g., like one-third of the total file). A second well-known issue with traditional approximate matching algorithms is the lack of scaling due to the ever-increasing lookup databases. In this paper, we propose an improved matching algorithm based on transformer-based models from the field of natural language processing. We call our approach Deep Learning Approximate Matching (DLAM). As a concept from artificial intelligence, DLAM gets knowledge of characteristic blacklisted patterns during its training phase. Then DLAM is able to detect the patterns in a typically much larger file, that is DLAM focuses on the use case of fragment detection. Our evaluation is inspired by two widespread blacklist use cases: the detection of malware (e.g., in JavaScript) and corporate secrets (e.g., pdf or office documents). We reveal that DLAM has three key advantages compared to the prominent conventional approaches TLSH and ssdeep. First, it makes the tedious extraction of known to be bad parts obsolete, which is necessary until now before any search for them with approximate matching algorithms. This allows efficient classification of files on a much larger scale, which is important due to exponentially increasing data to be investigated. Second, depending on the use case, DLAM achieves a similar (in case of mrsh-cf and mrsh-v2) or even significantly higher accuracy (in case of ssdeep and TLSH) in recovering fragments of blacklisted files. For instance, in the case of JavaScript files, our assessment shows that DLAM provides an accuracy of 93% on our test corpus, while TLSH and ssdeep show a classification accuracy of only 50%. Third, we show that DLAM enables the detection of file correlations in the output of TLSH and ssdeep even for fragment sizes, where the respective matching function of TLSH and ssdeep fails.
@inproceedings{uhlig22dlam, author = {Uhlig*, Frieder and Struppek*, Lukas and Hintersdorf*, Dominik and Göbel, Thomas and Baier, Harald and Kersting, Kristian}, title = {Combining AI and AM – Improving Approximate Matching through Transformer Networks}, booktitle = {Annual DFRWS USA Conference}, year = {2023}, }
Sparsely-Gated MoE Layers for CNN Interpretability

Svetlana Pavlitskaya, Christian Hubschneider, Lukas Struppek, and J. Marius Zöllner

In International Joint Conference on Neural Networks (IJCNN) 2023

Abs Bib PDF

Sparsely-gated Mixture of Expert (MoE) layers have been recently successfully applied for scaling large transformers, especially for language modeling tasks. An intriguing side effect of sparse MoE layers is that they convey inherent interpretability to a model via natural expert specialization. In this work, we apply sparse MoE layers to CNNs for computer vision tasks and analyze the resulting effect on model interpretability. To stabilize MoE training, we present both soft and hard constraint-based approaches. With hard constraints, the weights of certain experts are allowed to become zero, while soft constraints balance the contribution of experts with an additional auxiliary loss. As a result, soft constraints handle expert utilization better and support the expert specialization process, while hard constraints maintain more generalized experts and increase overall model performance. Our findings demonstrate that experts can implicitly focus on individual sub-domains of the input space. For example, experts trained for CIFAR-100 image classification specialize in recognizing different domains such as flowers or animals without previous data clustering. Experiments with RetinaNet and the COCO dataset further indicate that object detection experts can also specialize in detecting objects of distinct sizes.
@inproceedings{pavlitskaya2023moe, author = {Pavlitskaya, Svetlana and Hubschneider, Christian and Struppek, Lukas and Z{\"{o}}llner, J. Marius}, title = {Sparsely-Gated MoE Layers for CNN Interpretability}, booktitle = {International Joint Conference on Neural Networks ({IJCNN})}, year = {2023}, }

2022

Plug & Play Attacks: Towards Robust and Flexible Model Inversion Attacks

Lukas Struppek, Dominik Hintersdorf, Antonio De Almeida Correia, Antonia Adler, and Kristian Kersting

In International Conference on Machine Learning (ICML) 2022

Abs Bib PDF Code Poster

Model inversion attacks (MIAs) aim to create synthetic images that reflect the class-wise characteristics from a target classifier’s private training data by exploiting the model’s learned knowledge. Previous research has developed generative MIAs that use generative adversarial networks (GANs) as image priors tailored to a specific target model. This makes the attacks time- and resource-consuming, inflexible, and susceptible to distributional shifts between datasets. To overcome these drawbacks, we present Plug & Play Attacks, which relax the dependency between the target model and image prior, and enable the use of a single GAN to attack a wide range of targets, requiring only minor adjustments to the attack. Moreover, we show that powerful MIAs are possible even with publicly available pre-trained GANs and under strong distributional shifts, for which previous approaches fail to produce meaningful results. Our extensive evaluation confirms the improved robustness and flexibility of Plug & Play Attacks and their ability to create high-quality images revealing sensitive class characteristics.
@inproceedings{struppek2022ppa, booktitle = {International Conference on Machine Learning (ICML)}, pages = {20522--20545}, year = {2022}, author = {Struppek, Lukas and Hintersdorf, Dominik and Correia, Antonio De Almeida and Adler, Antonia and Kersting, Kristian}, title = {Plug & Play Attacks: Towards Robust and Flexible Model Inversion Attacks}, }
To Trust or Not To Trust Prediction Scores for Membership Inference Attacks

Dominik Hintersdorf*, Lukas Struppek*, and Kristian Kersting

In International Joint Conference on Artificial Intelligence (IJCAI) 2022

Abs Bib PDF Code Poster

Membership inference attacks (MIAs) aim to determine whether a specific sample was used to train a predictive model. Knowing this may indeed lead to a privacy breach. Most MIAs, however, make use of the model’s prediction scores - the probability of each output given some input - following the intuition that the trained model tends to behave differently on its training data. We argue that this is a fallacy for many modern deep network architectures. Consequently, MIAs will miserably fail since overconfidence leads to high false-positive rates not only on known domains but also on out-of-distribution data and implicitly acts as a defense against MIAs. Specifically, using generative adversarial networks, we are able to produce a potentially infinite number of samples falsely classified as part of the training data. In other words, the threat of MIAs is overestimated, and less information is leaked than previously assumed. Moreover, there is actually a trade-off between the overconfidence of models and their susceptibility to MIAs: the more classifiers know when they do not know, making low confidence predictions, the more they reveal the training data.
@inproceedings{hintersdorf2022trust, pages = {3043--3049}, booktitle = {International Joint Conference on Artificial Intelligence ({IJCAI}) }, year = {2022}, author = {Hintersdorf*, Dominik and Struppek*, Lukas and Kersting, Kristian}, title = { To Trust or Not To Trust Prediction Scores for Membership Inference Attacks }, }
Learning to Break Deep Perceptual Hashing: The Use Case NeuralHash

Lukas Struppek*, Dominik Hintersdorf*, Daniel Neider, and Kristian Kersting

In ACM Conference on Fairness, Accountability, and Transparency (FAccT) 2022

Abs Bib PDF Code Poster

Apple recently revealed its deep perceptual hashing system NeuralHash to detect child sexual abuse material (CSAM) on user devices before files are uploaded to its iCloud service. Public criticism quickly arose regarding the protection of user privacy and the system’s reliability. In this paper, we present the first comprehensive empirical analysis of deep perceptual hashing based on NeuralHash. Specifically, we show that current deep perceptual hashing may not be robust. An adversary can manipulate the hash values by applying slight changes in images, either induced by gradient-based approaches or simply by performing standard image transformations, forcing or preventing hash collisions. Such attacks permit malicious actors easily to exploit the detection system: from hiding abusive material to framing innocent users, everything is possible. Moreover, using the hash values, inferences can still be made about the data stored on user devices. In our view, based on our results, deep perceptual hashing in its current form is generally not ready for robust client-side scanning and should not be used from a privacy perspective.
@inproceedings{struppek2022learning, year = {2022}, author = {Struppek*, Lukas and Hintersdorf*, Dominik and Neider, Daniel and Kersting, Kristian}, title = { Learning to Break Deep Perceptual Hashing: The Use Case NeuralHash }, booktitle = {ACM Conference on Fairness, Accountability, and Transparency (FAccT)}, pages = {58--69}, }
Investigating the Risks of Client-Side Scanning for the Use Case NeuralHash

Dominik Hintersdorf*, Lukas Struppek*, Daniel Neider, and Kristian Kersting

In Working Notes of the 6th Workshop on Technology and Consumer Protection (ConPro) @ 43th IEEE Symposium on Security and Privacy 2022

Abs Bib PDF Code

Regulators around the world try to stop the distribution of digital criminal content without circumventing the encryption methods in place for confidential communication. One approach is client-side scanning (CSS) which checks the content on the user’s device for illegality before it is encrypted and transmitted. Apple has recently revealed a client-side deep perceptual hashing system called NeuralHash to detect child sexual abuse material (CSAM) on user devices before the images are encrypted and uploaded to its iCloud service. After its public presentation, criticism arose regarding the users’ privacy and the system’s reliability. In this work, we present an empirical analysis of client-side deep perceptual hashing based on NeuralHash. We show that such systems are not robust, and an adversary can easily manipulate the hash values to force or prevent hash collisions. Such attacks permit malicious actors to exploit the detection system: from hiding abusive material to framing innocent users, a large variety of attacks is possible.
@inproceedings{hintersdorf2022conpro_learning, year = {2022}, author = {Hintersdorf*, Dominik and Struppek*, Lukas and Neider, Daniel and Kersting, Kristian}, title = {Investigating the Risks of Client-Side Scanning for the Use Case NeuralHash}, booktitle = {Working Notes of the 6th Workshop on Technology and Consumer Protection (ConPro) {@} 43th IEEE Symposium on Security and Privacy}, }

* denotes shared first authorship