PTG - Perceptually-enabled Task Guidance

The **Perceptually-enabled Task Guidance (PTG)** program, funded by the Defense Advanced Research Projects Agency (DARPA), develops artificial intelligence technologies that help users perform complex physical tasks through intelligent, context-aware assistance. PTG combines wearable sensors, augmented reality (AR), computer vision, and multimodal AI to provide just-in-time visual and audio guidance, adapting instructions to the user's expertise and the surrounding environment. New York University's Visualization Imaging and Data Analysis (VIDA) Lab is one of the participating research teams, developing innovative visual analytics systems, adaptive user interfaces, and AI-assisted guidance technologies. ## Key Components - **ARGUS**: Interactive visual analytics platform for exploring and debugging AI-assisted task guidance systems in both online and offline modes. - **HuBar**: Visual analytics tool for analyzing human behavior using multimodal sensor data, including fNIRS, to understand cognitive workload during AR-guided tasks. - **Satori**: Proactive AI assistant based on belief-desire-intention (BDI) user modeling that dynamically adapts guidance according to user context and actions. - **ARTiST**: Automated text simplification system that generates shorter and easier-to-understand task instructions for augmented reality interfaces. ## Related Publications - Sonia Castelo, João Rulff, Erin McGowan, Bea Steers, Guande Wu, Shaoyu Chen, Iran Roman, Roque Lopez, Ethan Brewer, Chen Zhao, Jing Qian, Kyunghyun Cho, He He, Qi Sun, Huy Vo, Juan Bello, Michael Krone, Claudio Silva. *"Argus: Visualization of AI-Assisted Task Guidance in AR."* IEEE Transactions on Visualization and Computer Graphics, 2023. *(Best Paper Honorable Mention, IEEE VIS 2023).* - Sonia Castelo, João Rulff, Parikshit Solunke, Erin McGowan, Guande Wu, Iran Roman, Roque Lopez, Bea Steers, Qi Sun, Juan Bello, Bradley Feest, Michael Middleton, Ryan McKendrick, Claudio Silva. *"HuBar: A Visual Analytics Tool to Explore Human Behavior based on fNIRS in AR Guidance Systems."* IEEE Transactions on Visualization and Computer Graphics, 2024. - Guande Wu, Jing Qian, Sonia Castelo Quispe, Shaoyu Chen, João Rulff, Claudio Silva. *"ARTiST: Automated Text Simplification for Task Guidance in Augmented Reality."* Proceedings of the CHI Conference on Human Factors in Computing Systems, 2024.