Provas de CAT do aluno Miguel Serras Vasco
30 junho 2021, 09:52 • Sandra Espírito Santo
Prova de CAT
Candidato: Miguel Serras Vasco nº 70413
Titulo da Tese: "Multimodal Representation Learning for Agent Perception and Action"
Link de Zoom:
https://videoconf-colibri.zoom.us/j/89723405483?pwd=VjFOOFBXUWZITlYyZ1B3RUxDbnRzdz09Data: 16 de Julho de 2021
Hora: 10h00 às 11h30
Orientadora: Professor Doutora Ana Maria Severino de Almeida e Paiva
Co-orientador: Professor Doutor Francisco António Chaves Saraiva de Melo
Thesis Abstrat:
In this thesis, we aim at endowing agents with mechanisms to learn multimodal representations from sensory data and to allow them to execute tasks considering different subsets of available perceptions. We address the learning of these representations from supervised and unsupervised learning frameworks and how to leverage such representations for reinforcement learning scenarios under changing perceptual conditions. In the context of supervised multimodal representation learning, we contribute with a novel action representation and learning algorithm that allows agents to consider contextual information provided by action demonstrations, allowing for sample-efficient recognition of human actions. Moreover, in the context of unsupervised multimodal representation learning, we explore the cross-modality inference problem -- the ability to infer missing perceptual data from available perceptions -- and contribute with a novel hierarchical multimodal generative model that addresses the requirements of computational cross-modality inference. Furthermore, we introduce cross-modality policy transfer in reinforcement learning, where an agent must learn and exploit policies over different subsets of input modalities and instantiate such problem in the context of Atari games. In future work, we propose to address the effect of the nature of perceptual information provided to the agent, in order to provide robustness to deteriorating perceptual conditions and compromised sensor attacks.