José Amendola Netto Andrade has followed the PhD Programme at Faculty of Engineering and Science, specialisation in Engineering Sciences, Scientific field of Mechatronics.
- Trial lecture starts at 10.15
- Public defence starts at 12.15
Title of trial lecture: "Large scale multimodal language models for autonomous navigation"
Disputation chair: Kjell Gunnar Robbersmyr
Assessment committee:
- First opponent: Professor Raghavendra Ramachandra, NTNU
- Second opponent: Professor Vitomir Struc, University of Ljubljana, Slovenia
- Chair of assessment committee: Professor Svitlana Rogovchenko, University of Agder
Supervisors in the doctoral work:
- Main supervisor: Associate Professor Ajit Jha, UiA
- Co- supervisor: Professor Linga Reddy Cenkeramaddi, UiA
Summary of thesis
This thesis addresses the critical challenges in autonomous navigation by developing modular, flexible, and scalable deep learning architectures tailored for edge computing environments. Autonomous systems, such as Unmanned Ground Vehicles (UGVs) and Unmanned Aerial Vehicles (UAVs), require robust and adaptive navigation capabilities to operate effectively in dynamic and complex environments. By leveraging the full vertical stack of sensors, perception, planning, and control, these autonomous platforms can be utilized individually to perform complex tasks. Additionally, these platforms can be connected to form sophisticated multi-agent systems. In recent years, deep learning has become a popular solution for handling the complexity of autonomous navigation tasks, offering data-driven methods to tackle perception, decision-making, and control. However, current deep learning models often rely on monolithic architectures that lack the adaptability to handle diverse sensor modalities, scale effectively with different input types, or function efficiently on distributed edge hardware. This limitation hinders the development of flexible, scalable autonomous systems capable of real-time performance across distributed hardware platforms on the edge. To overcome these limitations, this research introduces several novel contributions. First, a low-latency, edge-distributed surround-view multimodal sensor fusion method is proposed, enabling effective multiple object localization by asynchronously integrating data from LiDAR and camera sensors. This approach enhances the system’s modularity, allowing the system to function in resource-constrained environments while maintaining high accuracy and scalability. Moving towards cross-modality, a lightweight framework is developed, where both RGB and thermal images are mapped to a shared latent space through a pure convolutional network; classification then occurs through simple clustering and nearest neighbor association. Further, the same network structure is employed as a dual-mode encoder architecture that supports multiple downstream tasks, such as image translation, colorization, and reconstruction, making it highly adaptable to various sensors of different modalities. Finally, the thesis expands into the cooperation between heterogeneous agents — in this case, UAVs and UGVs — to achieve the specific task of drone landing on a moving UGV. A reinforcement learning control strategy is employed in confluence with position PID control using a simple state representation and time-spaced commands. Further, the work is extended to a unified policy for different UGV trajectories, facilitating robust and efficient coordination between UAV and UGV in different scenarios. Unlike vision-based approaches, the proposed approach allows effective operation despite communication constraints. The outcomes of this research contribute to the advancement of heterogeneous autonomous navigation systems by providing flexible, modular, and scalable solutions that enhance the performance and reliability of autonomous platforms in real-world scenarios. The algorithms, methodologies, and architectures developed in this thesis have the potential to be widely applied in various autonomous systems, paving the way for more adaptable and resilient autonomous technologies.
What to do as an online audience member
The disputation is open to the public. To follow the trial lecture and the public defence online, register on Zoom.
We ask online audience members to join no earlier than 10 minutes in advance. After these times, you can leave and rejoin the meeting at any time.
Opponent ex auditorio:
Deadline for the public to pose questions is during the break between the two opponents. Questions ex auditorio can be submitted to Kristine Reinfjord.