Behavior of Machine Learning Algorithms in Adversarial Environments
Abstract
Machine learning has become a prevalent tool in many computing applications and modern enterprise systems stand to greatly benefit from learning algorithms. However, one concern with learning algorithms is that they may introduce a security fault into the system. The key strengths of learning approaches are their adaptability and ability to infer patterns that can be used for predictions or decision making. However, these assets of learning can potentially be subverted by adversarial manipulation of the learner's environment, which exposes applications that use machine learning techniques to a new class of security vulnerabilities. I analyze the behavior of learning systems in adversarial environments. My thesis is that learning algorithms are vulnerable to attacks that can transform the learner into a liability for the system they are intended to aid, but by critically analyzing potential security threats the extent of these threat can be assessed, proper learning techniques can be selected to minimize the adversary's impact, and failures of system can be averted. I present a systematic approach for identifying and analyzing threats against a machine learning system. I examine real-world learning systems, assess their vulnerabilities, demonstrate real-world attacks against their learning mechanism, and propose defenses that can successful mitigate the effectiveness of such attacks. In doing so, I provide machine learning practitioners with a systematic methodology for assessing a learner's vulnerability and developing defenses to strengthen their system against such threats. Additionally, I also examine and answer theoretical questions about the limits of adversarial contamination and classifier evasion.
Document Details
- Document Type
- Technical Report
- Publication Date
- Nov 23, 2010
- Accession Number
- ADA532922
Entities
People
- Blaine A. Nelson
Organizations
- University of California, Berkeley