Empirical Study: Evaluating Typographic Attack Effectiveness Against Vision-LLMs in AD Systems

Empirical Study: Evaluating Typographic Attack Effectiveness Against Vision-LLMs in AD Systems

Estimated reading time: 7 minutes

• Typographic attacks pose a significant and subtle threat to Vision-LLMs in autonomous driving systems, leveraging minor text alterations to cause major misinterpretations.

• An empirical study reveals that Vision-LLMs are highly susceptible, struggling with tasks like accurate counting and complex scene reasoning, and can even be misled into recommending unsafe driving practices.

• Physical typographic attacks, easily implemented on real-world objects like road signs, represent a particularly dangerous vector with severe implications for AD safety.

• Sophisticated models, including GPT-4, are not immune, especially when faced with augmented typographic attacks.

• Mitigation requires a multi-pronged approach: enhanced model robustness through defensive learning, rigorous adversarial testing (red teaming), and the deployment of real-time attack detection mechanisms.

• Understanding Vision-LLMs and the Threat of Typographic Attacks
• Unveiling the Vulnerabilities: Experimental Insights
  • Experiments
    • Experimental Setup
    • Attacks on Scene/Action Reasoning
    • Compositions and Augmentations of Attacks
    • Towards Physical Typographic Attacks
• Real-World Risks and Implications for Autonomous Driving
• Fortifying Autonomous Systems: Actionable Steps
• Conclusion
• Frequently Asked Questions

The convergence of artificial intelligence with real-world applications is nowhere more critical than in autonomous driving (AD) systems. , enabling AD systems to not only perceive their surroundings but also to understand and reason about complex scenarios using both visual and linguistic cues. These sophisticated models allow vehicles to interpret traffic signs, understand driver intentions, and make informed decisions, transforming the future of transportation. However, as these systems become more integrated and intelligent, they also become potential targets for sophisticated adversarial attacks. One such emerging threat is the , a subtle yet potent method that leverages textual manipulations to mislead Vision-LLMs.

This article delves into a groundbreaking empirical study that meticulously evaluates the effectiveness of typographic attacks against Vision-LLMs operating within AD systems. By scrutinizing how minor alterations in text can cascade into significant misinterpretations and potentially dangerous decisions, we aim to shed light on critical vulnerabilities and underscore the urgent need for robust defensive strategies.

Understanding Vision-LLMs and the Threat of Typographic Attacks

Vision-LLMs represent a paradigm shift in AI, seamlessly integrating visual perception with language understanding. In autonomous vehicles, this means a system can “see” a stop sign and simultaneously “read” the word “STOP,” processing both modalities to infer the appropriate action. They can answer complex questions about a scene (e.g., “How many cars are ahead?” or “Is it safe to turn left?”), leveraging vast datasets to derive context and reason about dynamic environments.

Unlike traditional adversarial attacks that manipulate pixels directly, typographic attacks subtly alter textual information present in the visual scene—such as on road signs, advertisements, or vehicle livery. These changes, often imperceptible or seemingly benign to the human eye, can be profoundly misleading to a Vision-LLM, causing it to misinterpret critical information or hallucinate entirely false scenarios. The danger lies in their potential to trigger erroneous decisions in safety-critical applications like autonomous driving, turning a simple textual tweak into a .

Unveiling the Vulnerabilities: Experimental Insights

The empirical study meticulously investigated the efficacy of these attacks across various Vision-LLMs and AD-specific datasets. The findings reveal a sobering reality regarding the susceptibility of these advanced models.

The internal structure of the research paper outlines its comprehensive approach:

“Table of Links
Abstract and 1. Introduction
Related Work
2.1 Vision-LLMs
2.2 Transferable Adversarial Attacks

Preliminaries
3.1 Revisiting Auto-Regressive Vision-LLMs
3.2 Typographic Attacks in Vision-LLMs-based AD Systems

Methodology
4.1 Auto-Generation of Typographic Attack
4.2 Augmentations of Typographic Attack
4.3 Realizations of Typographic Attacks

Experiments

Conclusion and References

5 Experiments
5.1 Experimental Setup
5.1.1 Attacks on Scene/Action Reasoning
5.1.2 Compositions and Augmentations of Attacks
5.1.3 Towards Physical Typographic Attacks”

The detailed experimental findings are summarized below.

Experiments

This section delves into the experimental setup and key results from the study’s evaluation of typographic attacks.

Experimental Setup

We perform experiments with Vision-LLMs on VQA datasets for AD, such as LingoQA [7] and the dataset of CVPRW’2024 Challenge [1] by CARLA simulator. We have used LLaVa [2] to output the attack prompts for LingoQA and the CVPRW’2024 dataset, and manually for some cases of the latter. Regarding LingoQA, we tested 1000 QAs in real traffic scenarios in tasks, such as scene reasoning and action reasoning. Regarding the CVPRW’2024 Challenge dataset, we tested more than 300 QAs on 100 images, each with at least three questions related to scene reasoning (e.g., target counting) and scene object reasoning of 5 classes (cars, persons, motorcycles, traffic lights and road signals). Our evaluation metrics are based on exact matches, Lingo-Judge Accuracy [7], and BLEURT [41], BERTScore [42] against non-attacked answers, with SSIM (Structural Similarity Index) to quantify the similarity between original and attacked images. In terms of models, we qualitatively and/or quantitatively tested with LLaVa [2], VILA [1], Qwen-VL [17], and Imp [18]. The models were run on an NVIDIA A40 GPU with approximately 45GiB of memory.

Attacks on Scene/Action Reasoning

As shown in Tab. 2, Fig. 4, and Fig. 5, our framework of attack can For example, Tab. 2 showcases an ablation study on the effectiveness of automatic attack strategies across two datasets: LingoQA and CVPRW’24 (focused solely on counting). The former two metrics (i.e. Exact and Lingo-Judge) are used to evaluate semantic correctness better, showing that short answers like the counting task can be easily misled, but longer, more complex answers in LingoQA may be more difficult to change. For example, the Qwen-VL attack scores 0.3191 under the Exact metric for LingoQA, indicating relative effectiveness compared to other scores in the same metric in counting. On the other hand, we see that the latter two scores (i.e. BLEURT and BERTScore) are typically high, hinting that our attack can mislead semantic reasoning, but even the wrong answers may still align with humans decently.

In terms of scene reasoning, we show in Tab. 3, Tab. 4, and Fig. 4 the effectiveness of our proposed attack against a number of cases. For example, in Fig. 4, a Vision-LLM can somewhat accurately answer queries about a clean image, but a typographic attacked input can make it fail, such as to accurately count people and vehicles, and we show that an In Fig. 5, we also show that scene reasoning can be misdirected where irrelevant details are focused on and hallucinate under typographic attacks. Our work also suggests that scene object reasoning / grounded object reasoning is typically more robust, as both object-level and image-level attacks may be needed to change the models’ answers.

In terms of action reasoning, we show in Fig. 5 that Vision-LLMs can Nevertheless, we see a promising point when Qwen-VL recommended fatal advice, but it reconsidered over the reasoning process of acknowledging the potential dangers of the initial bad suggestion. These examples demonstrate the vulnerabilities in automated reasoning processes under deceptive or manipulated conditions, but they also suggest that defensive learning can be applied to enhance model reasoning.

Compositions and Augmentations of Attacks

We showed that composing multiple QA tasks for an attack is possible for a particular scenario, thereby suggesting that typographic attacks are not single-task attacks, as suggested by previous works. Furthermore, we found that , which would imply that typographic attacks that leverage the inherent language modeling process can misdirect the reasoning of Vision-LLMs, as especially shown in the case of the strong GPT-4. However, as shown in Tab. 5, it may be challenging to search for the best augmentation keywords.

Towards Physical Typographic Attacks

In our toy experiments with semi-realistic attacks in Fig.5, we show that attacks involve manipulating text within real-world settings are , such as on signs, behind vehicles, on buildings, billboards, or any everyday object that an AD system might perceive and interpret to make decisions. For instance, modifying the text on a road sign from “stop” to “go faster” can pose potentially dangerous consequences on AD systems that utilize Vision-LLMs.

The research demonstrated that typographic attacks are highly effective at misdirecting Vision-LLMs across various reasoning tasks. Specifically, models struggled with accurate counting and were easily misled in complex scene reasoning. More alarmingly, these attacks prompted systems to offer “,” advocating for unsafe driving practices. While longer, more complex answers showed slightly more resistance, the ability of attacks to mislead semantic reasoning, even with answers somewhat aligned with human understanding, presents a serious concern. The study also highlighted that sophisticated models like GPT-4 are not immune, especially when faced with augmented typographic attacks. While scene object reasoning exhibited more robustness, the broader implications for AD safety are profound.

A notable, albeit singular, positive observation was Qwen-VL’s ability to reconsider fatal advice during its reasoning process. This “promising point” suggests that incorporating defensive learning mechanisms could potentially enhance model resilience against such deceptive conditions.

Real-World Risks and Implications for Autonomous Driving

The implications of these findings for autonomous driving are severe. , manipulating text on real-world objects, emerge as a particularly dangerous vector. The study’s “toy experiments with semi-realistic attacks” underscore their ease of implementation and the grave consequences.

Consider this real-world example: an attacker subtly modifies a standard “STOP” sign to read “GO FASTER” or alters the typography in a way that the human eye might correct, but a Vision-LLM misinterprets. An autonomous vehicle relying on that Vision-LLM for decision-making could misinterpret this critical traffic instruction, leading to a instead of bringing the vehicle to a halt. Such attacks are not hypothetical; they leverage existing vulnerabilities in how AI perceives and processes information, turning benign text into a weapon. The potential for malicious actors to exploit these vulnerabilities on signs, billboards, or even vehicle decals presents an unprecedented security challenge for the AD industry.

Fortifying Autonomous Systems: Actionable Steps

Given the demonstrated effectiveness and ease of implementing typographic attacks, for securing AD systems. Here are three actionable steps:

1. Enhance Model Robustness Through Defensive Learning: Implement advanced defensive learning strategies that specifically train Vision-LLMs to recognize and filter out adversarial textual perturbations. This involves exposing models to a diverse range of typographic attacks during training, incorporating adversarial training, and developing detection mechanisms for subtle text alterations, allowing models to flag or ignore potentially malicious inputs rather than blindly acting on them. The Qwen-VL example, where the model reconsidered fatal advice, offers a blueprint for developing self-correcting or cautious reasoning pathways.

2. Implement Rigorous Adversarial Testing and Red Teaming: Beyond standard quality assurance, AD developers must adopt continuous, aggressive adversarial testing specifically targeting typographic vulnerabilities. This includes “red teaming” exercises where ethical hackers attempt to exploit these weaknesses in controlled environments. Such proactive testing, using both auto-generated and augmented attack methodologies, will help identify and patch vulnerabilities before systems are deployed on public roads. It’s crucial to test not just for single-task attacks but also for compositions of attacks, as shown in the study.

3. Develop and Deploy Real-time Attack Detection Mechanisms: Invest in and integrate real-time monitoring and anomaly detection systems capable of identifying suspicious textual changes in the AD system’s perceived environment. This could involve multi-modal verification (e.g., cross-referencing text with visual context, comparing sign appearance to known regulations, or using redundant perception systems) to confirm the integrity of visual information before critical decisions are made. For instance, if a “STOP” sign’s text appears altered, the system should be programmed to default to the safest action (stopping) or seek additional verification.

Conclusion

The empirical study on typographic attacks against Vision-LLMs in autonomous driving systems serves as a critical warning. It unequivocally demonstrates the significant vulnerabilities of these advanced AI models to subtle textual manipulations, highlighting their potential to misdirect reasoning, induce unsafe actions, and even lead to dangerous physical outcomes. From miscounting objects to suggesting hazardous driving advice, the risks are undeniable and extend to easily implementable physical attacks in real-world scenarios.

As autonomous driving technologies advance towards widespread adoption, safeguarding them against such intelligent adversarial threats is not merely an academic exercise—it is an . Developers, researchers, and policymakers must collaborate to integrate robust defensive learning mechanisms, implement continuous adversarial testing, and deploy real-time detection systems. The future of safe autonomous mobility hinges on our ability to outmaneuver these evolving threats.

Take Action Now: Engage with the latest research in AI safety and contribute to developing resilient autonomous systems. Explore collaboration opportunities with leading institutions to fortify AD against emerging adversarial threats.

Frequently Asked Questions