跳到主要导航 跳到搜索 跳到主要内容

T2IShield: Defending Against Backdoors on Text-to-Image Diffusion Models

  • Zhongqi Wang
  • , Jie Zhang*
  • , Shiguang Shan
  • , Xilin Chen
  • *此作品的通讯作者
  • CAS - Institute of Computing Technology
  • University of Chinese Academy of Sciences

科研成果: 书/报告/会议事项章节 › 会议稿件 › 同行评审

摘要

While text-to-image diffusion models demonstrate impressive generation capabilities, they also exhibit vulnerability to backdoor attacks, which involve the manipulation of model outputs through malicious triggers. In this paper, for the first time, we propose a comprehensive defense method named T2IShield to detect, localize, and mitigate such attacks. Specifically, we find the “Assimilation Phenomenon” on the cross-attention maps caused by the backdoor trigger. Based on this key insight, we propose two effective backdoor detection methods: Frobenius Norm Threshold Truncation and Covariance Discriminant Analysis. Besides, we introduce a binary-search approach to localize the trigger within a backdoor sample and assess the efficacy of existing concept editing methods in mitigating backdoor attacks. Empirical evaluations on two advanced backdoor attack scenarios show the effectiveness of our proposed defense method. For backdoor sample detection, T2IShield achieves a detection F1 score of 88.9% with low computational cost. Furthermore, T2IShield achieves a localization F1 score of 86.4% and invalidates 99% poisoned samples. Codes are released at https://github.com/Robin-WZQ/T2IShield.

源语言英语
主期刊名Computer Vision – ECCV 2024 - 18th European Conference, Proceedings
编辑Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, Gül Varol
出版商Springer Science and Business Media Deutschland GmbH
页107-124
页数18
ISBN(印刷版)9783031730122
DOI
出版状态已出版 - 2025
已对外发布是
活动18th European Conference on Computer Vision, ECCV 2024 - Milan, 意大利
期限: 29 9月 2024 → 4 10月 2024

丛书

姓名Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
卷15143 LNCS
ISSN(印刷版)0302-9743
ISSN(电子版)1611-3349

会议

会议18th European Conference on Computer Vision, ECCV 2024
国家/地区意大利
市Milan
时期29/09/24 → 4/10/24

学术指纹

探究 'T2IShield: Defending Against Backdoors on Text-to-Image Diffusion Models' 的科研主题。它们共同构成独一无二的学术指纹。

引用此