AVISA
Proof-of-concept interface that detects safety and theft incidents through computer vision
Amazon
Feb 2022 - May 2022
Prototyping, Interviews, Branding
PROBLEM
Situational Awareness
In the 18 months between January 2021 and July 2022, 209 exterior fires, 110 fire alarms, 47 hazardous substance exposures, and 141 water leakages were reported to the Amazon Global Security Operations Center from operations facilities across the globe. Loss prevention & safety teams do not have a proactive approach to identify threats on site, and rely on witnesses to be near incidents to report when an event is taking place.
Manual Reporting
Loss prevention specialists at Amazon operations facilities currently depend entirely on witnesses to report theft or safety-related incidents. Because of this, incidents may go unnoticed if no witnesses are present, there are often delays between an incident and its reporting, and report quality varies based on the witness. This dependency on human reporting allows security risks to escalate before they are even noticed.
Video Review
Loss prevention specialists spend one to two hours logging into video management systems and searching through security camera footage to locate specific events. This manual review process delays evidence collection and report completion during investigations, hindering incident resolution and postponing legal actions.
Response Time
Under the current reporting process process, it takes an average of 20 minutes from when an incident occurs until loss prevention specialists can begin their response - a delay that can be critical in security and safety situations. The dependency on manual reporting prevents loss prevention from having constant situational awareness of happenings throughout their site.
GENERATIVE
Interviews
I conducted an exploratory user research study to gain an understanding of the daily workflows of future users, and how AVISA video anomaly detection could fulfill their needs and fit into their incident response processes. I independently sourced 10 diverse participants across the United States, India, and Singapore, and coordinated virtual user research interviews to capture the needs of loss prevention and workplace health & safety roles.
Tools and Applications
Loss prevention and security teams rely on a complex ecosystem of multiple applications that require constant switching and cross-referencing during investigations. Because there is a lack of integration between these tools, investigators often must manually correlate data across 4-5 different applications.
Reporting
There is significant variation in reporting practices across Amazon loss prevention teams globally, ranging from ad-hoc reports to comprehensive weekly and monthly audits. Participants suggested that AVISA could enhance reporting by automatically detecting malfunctioning cameras and submitting the according tickets to IT services.
User Contexts
LP specialists and managers split their time between desk-based investigation work and on-site physical inspections, remaining available via phone when away from their computers. Internet connectivity issues challenge participants from India and Singapore, requiring AVISA to have efficient bandwidth usage and graceful handling of poor connections during video playback.
INVESTIGATION PROCESS
Notification
Loss prevention teams receive incident alerts through multiple channels with significant variation across facilities, worsened by the challenge of understaffing.
Investigation
Loss prevention teams use a triangulation approach to investigation, cross-referencing across multiple systems along with physical activities.
Evidence
Evidence collection is a time-consuming process that combines digital and physical evidence, storing video footage and screenshots on shared team drives.
Closure
Investigation duration varies significantly based on incident type and complexity, which requires coordination with legal departments and comprehensive case records.
LOW-FIDELITY
Wireframe
After establishing an understanding of the daily duties and pain points of a wide variety of Amazon loss prevention and workplace health & safety users, I explored design ideas for AVISA through iterative wireframes. I then held review meetings with the product manager for alignment and developers for feasibility.
FEATURES
Real-Time Notification
AVISA informs users of new anomaly events detected at their site(s), through both in-app and external notification channels.
Events List
A dashboard view will list all new and past events the system has identified in tile and table view options, with robust filtering and sorting.
Machine Learning
To continue improving the system's computer vision detection, human validation will confirm the accuracy of identified events.
Comments
To facilitate onsite team collaboration across teams and shifts, users can add comments to individual events to track investigation and response.
Settings
For internationalization and accessibility, users can configure appearance preferences. They can also set notification settings with a robust set of channel and filter options.
HIGH-FIDELITY
Prototype
After iterating designs with my product and development team, I designed a fully interactive prototype of AVISA, including 21 detailed screens using the internal Amazon Meridian design system. I facilitated handoff to my team of developers through the Adobe XD live dev link, and closely reviewed UI dev work to ensure the output closely aligned with the intended design.
PROOF OF CONCEPT
Results
The AVISA proof of concept was launched at an Amazon fulfillment center in Nashville, TN on March 24, 2023, with detection models for fallen people detection, emergency exit blockage detection, and defibrillator box removal. The deployed models significantly underperformed expectations, with the fallen people model achieving only a 7.52% F1 score versus the anticipated 92%. The project faced significant bandwidth constraints where streaming HD video (1080p) from all 500 cameras per site would require 2500Mbps—over 100 times the existing capacity. To overcome this limitation, computing solutions would require hardware installations costing approximately $5,000-$10,000 per site. My team decided to defer the sunk-cost investment until ML model accuracy improved enough to ensure a positive return on investment. Despite this poor performance, the project demonstrated valuable engineering achievements including programmable extraction from video management systems and face blurring for privacy protection.