Synthetic and (Un)Secure: Evaluating Generalized Membership Inference Attacks on Image Data

Pasquale Coscia, Stefano Ferrari, Vincenzo Piuri, Ayse Salman

2025

Abstract

Synthetic data are widely employed across diverse fields, including computer vision, robotics, and cybersecurity. However, generative models are prone to unintentionally revealing sensitive information from their training datasets, primarily due to overfitting phenomena. In this context, membership inference attacks (MIAs) have emerged as a significant privacy threat. These attacks employ binary classifiers to verify whether a specific data sample was part of the model’s training set, thereby discriminating between member and non-member samples. Despite their growing relevance, the interpretation of MIA outcomes can be misleading without a detailed understanding of the data domains involved during both model development and evaluation. To bridge this gap, we performed an analysis focused on a particular category (i.e., vehicles) to assess the effectiveness of MIA under scenarios with limited overlap in data distribution. First, we introduce a data selection strategy, based on the Fréchet Coefficient, to filter and curate the evaluation datasets, followed by the execution of membership inference attacks under varying degrees of distributional overlap. Our findings indicate that MIAs are highly effective when the training and evaluation data distributions are well aligned, but their accuracy drops significantly under distribution shifts or when domain knowledge is limited. These results highlight the limitations of current MIA methodologies in reliably assessing privacy risks in generative modeling contexts.

Paper Citation

in Harvard Style

Coscia P., Ferrari S., Piuri V. and Salman A. (2025). Synthetic and (Un)Secure: Evaluating Generalized Membership Inference Attacks on Image Data. In Proceedings of the 22nd International Conference on Security and Cryptography - Volume 1: SECRYPT; ISBN 978-989-758-760-3, SciTePress, pages 287-297. DOI: 10.5220/0013657700003979

in Bibtex Style

@conference{secrypt25,
author={Pasquale Coscia and Stefano Ferrari and Vincenzo Piuri and Ayse Salman},
title={Synthetic and (Un)Secure: Evaluating Generalized Membership Inference Attacks on Image Data},
booktitle={Proceedings of the 22nd International Conference on Security and Cryptography - Volume 1: SECRYPT},
year={2025},
pages={287-297},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0013657700003979},
isbn={978-989-758-760-3},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 22nd International Conference on Security and Cryptography - Volume 1: SECRYPT
TI - Synthetic and (Un)Secure: Evaluating Generalized Membership Inference Attacks on Image Data
SN - 978-989-758-760-3
AU - Coscia P.
AU - Ferrari S.
AU - Piuri V.
AU - Salman A.
PY - 2025
SP - 287
EP - 297
DO - 10.5220/0013657700003979
PB - SciTePress