Projects

Please visit my TIME Lab project page for more details.

Student Project

Project 1: Bachelor & Honours Research Project [Ongoing]

I am a registered ANU Higher Degree Research (HDR) supervisor (registration is valid until 31 December 2025).

If you are an ANU undergraduate seeking an Honors research project, please note that I typically accept a maximum of two Honors students each year to ensure high-quality supervision, and I prefer 24-unit projects.

If you are an ANU master’s student seeking individual projects, I require candidates to have a solid research background, a strong GPA, and proficient coding skills. Additionally, it is important that you have adequate time available for research.

• Selected posters from former master’s, honours, and internship students (primary supervision):
  • Dexuan Ding (1 x arXiv preprint, Poster), Honours Student, 2024/02/19 – 2024/10/25
  • Qixiang Chen (1 x ACML'24 oral, Poster), Honours Student, 2024/02/19 – 2024/10/25
  • Huilin Chen (1 x arXiv preprint, Poster), Honours Student, 2024/02/19 – 2024/10/25
  • Liyun Zhu (1 x NeurIPS'24, Poster), Master’s Student, 2024/02/19 – 2024/10/25
  • Xiuyuan Yuan (1 x ICML'24, Poster), Summer Research Internship, 2023/11/20 – 2024/02/02

Project 2: ANU TechLauncher - Active Vision project [Ongoing]

The TechLauncher program is for team-based activities at ANU, where the project and/or group activity undertaken is real, not synthetic. In TechLauncher you will be a team member on a project, and shall be supported by a program of tutorials at which your peers, along with experienced professional tutors/mentors, facilitate your professional development and mediate your team processes and interactions.

If you would like to join a Techlauncher cohort, you can do so by enrolling in any of the following courses: COMP3500 - Software Engineering Project, COMP3550 - Advanced Computing R&D Project, COMP3710 - Topics in Computer Science, COMP4500 - Software Engineering Practice, and COMP8715 - Computing Project.

In the 2024 Semester 2 and 2025 Semester 1, my proposed project is ‘Active Vision’. Francis Williams and Arjun Raj (acting on behalf of Lei Wang) are the clients.

The Active Vision project aims to develop an advanced computer vision system that analyzes the dynamics of tennis swings. Using cutting-edge machine learning techniques, this project focuses on capturing and analyzing motion data to provide players and coaches with detailed insights into swing mechanics. The system leverages high-speed cameras and advanced algorithms to track, interpret, and visualize the movements of a player’s body and racket, enabling precise performance assessment and improvement.

Objectives include: (i) Develop analytical algorithms: Create robust algorithms capable of analyzing the captured motion data to identify key performance indicators (KPIs) such as swing speed, angle, and impact accuracy. (ii) Real-time feedback: Implement a real-time feedback system that provides immediate insights and suggestions to players, enhancing their ability to correct and refine their techniques on the spot. (iii) User-friendly interface: Develop an intuitive interface that allows users to easily interact with the system, view their performance data, and track their progress over time.

[2024 S2 Poster]

Project 3: ANU TechLauncher - CAM FALL project [Ongoing]

In the 2024 Semester 1 and 2, my proposed project is ‘ActiveIntelligence: CAM-FALL (Camera-based Fall Alert System)’. This project marks the 2nd Annual Active Intelligence Research Challenge (refer to Industry Research Sponsorship below). Francis and I are the clients.

Step into a safer future with ActiveIntelligence’s CAM-FALL system. This innovative project offers a unique opportunity to address a pressing concern using cutting-edge technology. By harnessing the power of cameras, we aim to create a solution that not only identifies falls but also provides prompt assistance, ensuring the well-being of individuals within our community. Participation in the project will provide students with a holistic learning experience encompassing technical skills, problem-solving abilities, project management competencies, and ethical awareness, preparing them for future endeavors in the fields of computer vision, artificial intelligence, and software engineering.

[2024 S1 Poster] [2024 S2 Poster]

Project 4: Industry Research Sponsorship for Students [Completed]

I annually organize multiple industry research awards to support promising research candidates (Bachelor’s, Honours, and Master’s Students) interested in my research areas. Please note that only students from the Group of Eight (Go8) Australian universities are considered.

The Active Intelligence Research Challenge Award was established by Mr. Lei Wang and Mr. Francis Williams to honor individuals who have demonstrated exceptional dedication, expertise, and a commitment to excellence in the field of Artificial Intelligence and Machine Learning. The Award includes a cash prize (up to $5,000) along with a research internship (6-12 months full-time). These awards are provided by Active Intelligence Corporation LLC.

The Awards are granted solely on merit and are not subject to quotas. Typically, four candidates can receive awards from a pool of 20 candidates (with a minimum GPA of 6.75/7), resulting in an approximate acceptance rate of 20%, based on the 2023-2024 results. This year marks the 1st Annual Active Intelligence Research Challenge.

For more information regarding the application and award process, please contact the awarded candidates before reaching out to me.

• 2023-2024 awarded ANU international students are:
  • Mr. Arjun Raj (Bachelor of Advanced Computing (Research and Development) (Honours)),
  • Mr. Liyun Zhu (Master of Machine Learning and Computer Vision),
  • Mr. Liwen Luo (Master of Machine Learning and Computer Vision), and
  • Mr. Qixiang Chen (Bachelor of Advanced Computing (Honours)).

Project 5: ANU Summer Research Scholars Program [Completed]

ANU offers Summer Research Scholarships (SRS) and Summer Research Internships (SRI). External (non-ANU) students from Australia and New Zealand are eligible to apply for a Summer Research Scholarship. Current ANU students are eligible to apply for a Summer Research Internship.

The SRS scholarship includes board and on-campus accommodation for 9 weeks (run between Monday 20 Nov 2023 to Friday 19 Jan 2024), including the Christmas shutdown period. Candidates are awarded a stipend of \$150 per week, along with a travel allowance of up to \$1,000 for students relocating from interstate or from New Zealand. The SRI scholarship provides a stipend of \$400 per week to assist with the cost of living during the program. All students participating in the program will be invited to a variety of shared social events, workshops and seminars during the summer period.

For more details, please refer to Summer Research Scholarships. You can find my research projects by visiting 2023-2024 ANU CECC SRS Projects Page, get in touch if you are interested.

• 2023-2024 awarded ANU domestic student:
  • Mr. Xiuyuan (Jack) Yuan (Bachelor of Advanced Computing). Thanks to the ANU School of Computing for providing research funding.

Research Grant / Funding

• National Computational Merit Allocation Scheme 2024 (NCMAS 2024)
  • I am a Chief Investigator (CI) / Delegated lead CI (2024/01/01-2024/12/31)
  • Project title: MotionNetLite: Video Dynamics Distillation for Scalable Models
  • In this research project, we aim to develop more lightweight video data formats which are able to efficiently distill the motions at different granular levels by removing redundant information while focusing on the core dynamics. We also explore the acceleration of training and testing while reducing the amount of video data for storage, and answer a scientific question ‘How much information is contained in the video data and in what format?’. Many downstream video processing tasks will benefit from our video dynamic distillation process, towards making video understanding much easier and more efficient. This also opens up a new research direction in exploring better video data representations for more lightweight cutting-edge video models.
  • This work has potential to impact Safety and Security, Future Cities, IoT, Agri-business, Defence via applications in Health and wellbeing, Safety, and Innovative industries. Our work focuses on researching advanced technologies from data that support all areas of science and society to provide national benefit. Video understanding, e.g., action recognition and anomaly detection, is needed in surveillance of airports, malls, etc. It has applications in monitoring health and well-being of elderly population, in farming, and analysis of crops. This project has also potential to ‘Shape Societal Transformations’. For instance, action recognition is a necessary component in recognition from wearable clothing, monitoring health and exercise regimes in the gym, recommendation systems via wearables, recognition of fake videos on social media, etc.
  • This project focuses on ‘Analysing, Representing and Modelling data’, as video processing models require spatio-temporal modeling of time series, video frames, sequences, etc. My proposal aims at overcoming ‘Fundamental limits of data’, e.g. by learning in-the-wild and reinforcement learning to explore natural sources of information (e.g. predicting future evolution of video frames learns intrinsic manifold of video/motion data). I hope to bring my ideas to the social media, wearable devices, and recommender systems thus shaping ‘data-driven society’.
• The NCI National AI Flagship Merit Allocation Scheme
  • I am a Lead Chief Investigator (Lead CI) (2024/01/01-2024/06/30)
  • Project Title: Robust anomaly detection in human-centric videos
  • This project aims at developing advanced computer vision and deep learning techniques to identify and characterise anomalies in video data where humans are central. The project leverages cutting-edge technology to enhance security, safety, and surveillance systems, making them more effective in detecting unusual behaviours and events, which may range from security breaches and accidents to rare medical conditions in healthcare applications.
  • The significance of this project lies in its unique focus on human-centric videos. While anomaly detection in videos is an established field, the novelty emerges from its specialised applications in situations where human activity is central. The innovative aspects include (i) Robustness: The project seeks to develop highly reliable models capable of detecting anomalies in complex, real-world scenarios, where human interactions and activities can vary significantly. (ii) Real-time analysis: By applying these methods to real-time video streams, the project addresses the demand for timely responses to anomalies in security, industrial, and healthcare settings. (iii) Ethical considerations: The project incorporates ethical considerations, such as ensuring privacy and avoiding bias in the identification of anomalies, thereby making the technology responsible and trustworthy.
  • Beyond its academic contributions, this project has the potential to make significant contributions to the economy, society, environment, and culture.
• Research grant: Review of Xailient’s technical pipeline of facial recognition
  • I am a Co-Investigator (Co-I) (AU$50,815, 2023/10/30‐2023/12/22)
  • Personnel: Assoc. Prof. Liang Zheng and I are the main personnel for this project. Prof. Stephen Gould, a world-level expert in optimization and computer vision will provide additional advice.
  • Scope: The ANU team will review Xailient’s technical pipeline of facial recognition. There are many components in this pipeline, such as face detection, alignment, facial recognition, and matching score determination, model/architecture selection, loss function, hyperparameters, training data cleaning, learning strategies, and speed/accuracy optimization.
  • The ANU team will conduct an in-depth review of some of them (as mutually agreed between Xailient and ANU, also depending on how much time each will take), identify gaps, and provide technical advice and guidance to improve their real-world efficacy.
  • The advice given by the ANU team will be based on technologies available in public domains that can benefit and improve Xailient products.
• NCI Adapter Scheme Q4 2023 (HPC funding scheme)
  • I am a Chief Investigator (CI) / Delegated lead CI (2023/10/01‐2023/12/31)
  • Project Title: Towards building general-purpose multimodal foundation models
  • Scope: Vision-language pre-training (VLP) has attracted rapidly growing attention in both computer vision and NLP communities due to the emergence of large-scale multimodal foundation models like Contrastive Language-Image Pre-training (CLIP). It is very encouraging to see that many Vision-Language (VL) systems have been deployed in industry. For example, iPhone can generate image captions read by VoiceOver for vision-impaired users. Although multimodal intelligence has been applied in many areas including image-text, core computer vision and video-text tasks, there are still many factors to be considered including robustness to new domains, fairness and responsible AI issues.
  • Aim: One common theme stands out is how to build a general-purpose multi-modal foundation model. We aim to build a foundation model that is stable and generalisable, and can be readily adopted to various downstream tasks, ranging from image-level vision tasks (e.g., image classification, retrieval, and captioning), region-level vision tasks (e.g., object detection and phrase grounding), to pixel-level vision tasks (e.g., segmentation and image generation). In order to build a general-purpose foundation model, we need a unified model architecture that can be readily scaled up; and when being pre-trained at scale, it can be readily adopted to various downstream computer vision and VL tasks.
• Grant / Project Award from Oracle for Research
  • Oracle Cloud credits award (AU$48,000, 2023/07/26‐2024/07/25)
  • Research project: Automatic, large-scale screening of failure cases in autonomous driving
  • Automotive AI is rapidly replacing human drivers by enabling autonomous vehicles that use sensors to gather data about their surroundings. State-of-the-art methods achieve impressive system accuracy; however, technology is not always perfect, and this imperfection might be magnified when it comes to safety-critical applications like autonomous driving. Potential failure cases include severe occlusions, extreme weather, low lighting conditions, strange human appearances and poses, which will compromise system performance and even lead to undesirable consequences.
  • In this research project, we aim to build a scalable simulator and diffusion models to generate millions of different test scenarios from which we detect potential failure cases of existing pedestrian and vehicle detection systems. This is to understand the vulnerability of driverless systems, and finally to improve its accuracy and safety through another round of training with these corner cases. The test scenarios cover all possible potential failure cases by varying scene and/or street layout, weather and lighting conditions, human body sizes, visual appearances of human subjects, different levels of occlusions, different levels of sensor noise, etc. More complicated test scenarios are produced by a mixture of several aforementioned individual test scenarios. These different test scenarios are able to reflect the real and sometimes unpredictable driving scenarios on the road. After being identified by running our pedestrian and/or vehicle detectors, failure cases are then summarised and analysed, and the possible reasons why the model is vulnerable to them are identified. Similar failure environments are further explored to find more closely related failure cases. Finally, we fine-tune our pedestrian and vehicle detectors to improve their robustness against difficult test scenarios.
• Research grant: Sharing early insights for more resilient communities
  • I am a Co-Investigator (Co-I) (AU$71,089, 2023/04/19‐2023/10/31)
  • Research Assistant: Saswat Panda (ANU)
  • This project is a joint initiative with the Southern NSW Drought Resilience Adoption and Innovation Hub and forms part of the Australian Government’s Agricultural Innovation Hubs Program.
  • The project is being conducted by the University of Canberra, Australian National University, Charles Sturt University, and University of Wollongong.
  • The project is to develop and test ‘early warning’ indicators for loss of resilience following challenging climate-related events. These indicators will be used to develop a resource that can be used to by the wide range of organisations and services to identify communities in the early stages of resilience loss and provide targeted support to agricultural communities.
  • Through understanding early warning signs that individuals and communities are at risk of resilience loss, we could inform policy and support service interventions earlier. This would provide communities with the necessary support to mitigate wider and longer lasting resilience loss related to the impacts of a climate events, which in turn, would reduce the overall harm to lives and livelihoods and facilitate resilience building across physical, psychological, social, economic, domains.
• Industry research sponsorship (Active Intelligence Corp.): Detecting anomalies in video footage (stage 3)
  • I am a Principal Investigator / Project Lead (AU$40,013, 2023/07/01‐2024/03/01)
  • Project: Human‐related anomaly detection in surveillance videos (stage 3)
  • The objective of Stage 3 is to further enhance the capabilities of human-related anomaly detection in surveillance videos, building upon the progress achieved in Stages 1 and 2.
  • This stage focuses on refining algorithms, improving system adaptability, and integrating advanced features for more robust anomaly detection and classification.
• Industry research sponsorship (Active Intelligence Corp.): Detecting anomalies in video footage (stage 2)
  • I am a Principal Investigator / Project Lead (AU$108,628, 2022/07/01‐2023/06/30)
  • Project: Human‐related anomaly detection in surveillance videos (stage 2)
  • In the second stage of our project on human-related anomaly detection in surveillance videos, we aim to enhance the technology’s capabilities. This involves refining normal behavior modeling for adaptability across diverse environments and incorporating feedback from the first stage. We will implement state-of-the-art anomaly detection algorithms, leveraging machine learning for continuous improvement and heightened sensitivity.
  • The project also includes the development of robust event classification mechanisms, real-time alert systems, and features for in-depth post-event analysis and reporting. Special attention will be given to tailoring the technology for specific scenarios, such as shoplifting detection in retail, identification of unauthorized personnel, and improved fall detection in elderly care facilities.
  • Additionally, we will explore the integration of sensor data, including audio and environmental inputs, to augment video-based anomaly detection and provide a more comprehensive understanding of complex situations. These enhancements aim to further advance the technology’s applications in retail security, secure areas monitoring, elderly care facilities, and public safety in crowded places. Rigorous testing and validation will ensure the reliability and robustness of the system, reinforcing its role in enhancing security, safety, and situational awareness in modern surveillance systems.
• Industry research sponsorship (Active Intelligence Corp.): Detecting anomalies in video footage
  • I am a Principal Investigator / Project Lead (AU$135,706, 2021/07/01‐2022/06/30)
  • Project: Human‐related anomaly detection in surveillance videos
  • Human-related anomaly detection in surveillance videos is a technology used to identify unusual or suspicious behavior involving people within a monitored environment. It is a subfield of video analytics and computer vision that aims to automatically detect deviations from normal human behavior in real-time or from recorded video footage. The primary goal of human-related anomaly detection is to enhance security, safety, and situational awareness in various applications, such as public safety, retail, transportation, and critical infrastructure protection.
  • Key components of human-related anomaly detection in surveillance videos include: normal behavior modeling, anomaly detection algorithms, event classification, real-time alerts, post-event analysis, adaptability, etc.
  • Human-related anomaly detection can be applied in various scenarios, such as identifying potential shoplifting in retail stores, spotting unauthorized personnel in secure areas, detecting accidents or falls in elderly care facilities, or monitoring crowded places for public safety. The technology relies on a combination of machine learning, computer vision, and sensor data to make these determinations, and it plays a vital role in enhancing security and safety in modern surveillance systems.