ProctorX
Overview
This project was in collaboration with the Centre for Teaching, Learning and Technology (CTLT) @ NUS.
The project aim was to build an AI-driven platform that can detect and flag the use of prohibited applications during open internet examinations. We built a prototype cross-OS application for recording the computer screen and keyboard/mouse input data; using this application as a basis, we built a web platform where you can upload the data to assess the likelihood of prohibited applications being used - a surprisingly non-trivial task! There is little to no prior work done on computer vision algorithms for computer screens, as opposed to natural images of objects or humans; by using fine-tuned multimodal LLMs, we were able to extract information from computer screens, including browser information, texts on screen, etc.
For this project, I was the development lead. After building a prototype application for the cross-OS application, I worked on the web platform for prohibited app detection. The backend of the application was powered by two LLM pipelines: (1) a text-based analysis that processes and extracts typed text from keyboard data, and subsequently runs it through a fine-tuned LLM pipeline to assess likelihood of communication with others; (2) a video-based analysis that processes the video recording data, and in conjunction with the keyboard and mouse data, runs it through a fine-tuned Multimodal LLM pipeline to assess the likelihood of prohibited applications being used on screen. Our work was mainly done through LLMs to ensure flexibility, while non-LLM processing was optimized for scalability.
Our work was under multiple practical constraints. We needed the logging software to be run on all devices, while not compromising on privacy and security at the same time. Certain features that would have been useful, such as events or browser activity logging, were OS-level and hence could not be implemented due to privacy and security concerns. The analysis could only be done with local LLMs given similar privacy and security concerns, and hence we had to work with available resources. Furthermore, we had to ensure that security concerns were met in terms of data uploading to the cloud, posing additional challenges. Our team was able to address these concerns and successfully deliver the product within the timeline.