Building a Live Proctoring System to Detect Multiple Speakers
@ravindra-dyte γ» Jan 30,2024 γ» 17 min readγ» 2k views γ» Originally posted on dyte.io
By the end of this tutorial, we would have built a βLive Proctoring Systemβ using Dyte APIs that allows the admin to monitor whether multiple speakers can be heard from a candidateβs microphone. ππ§βπ»
Introduction
In the early days of online education, as the need arose to maintain the integrity of exams, proctoring emerged as a trusted monitoring method to prevent students from engaging in dishonest behavior. As the digital landscape expanded and online courses became more prevalent, the traditional ways of proctoring started to show their limitations.
Consider online exams, where having a physical proctor for every student becomes impractical. This gave rise to the adoption of live automatic proctoring.
This modern approach harnesses the power of a studentβs webcam and microphone to oversee the exam process. By incorporating computer vision and machine learning, it efficiently detects cheating attempts.
In this tutorial, weβll create a live proctoring system using Dyte APIs, enabling an admin to detect the presence of multiple speakers in a candidateβs background throughout the exam in real-time. β¨
High-Level Design of the application
Our aim is to notify the proctor if we hear multiple voices from the candidateβs background.
The proctor would get the candidate's details right in his meeting sidebar. πβ¨
π§βπ» Before building our live proctoring system, we must set up a Dyte account.
We can create a free account by clicking the "Start Building" button on Dyte.io and signing up using Google or GitHub. π
Once signed up, we can access our Dyte API keys from the "API Keys" tab in the left sidebar. We will keep these keys secure as we will use them later. ππ€«
For our live proctoring system, we will use React for the frontend and FastAPI for building the Backend and APIs.
We will begin by creating a new directory for our project, called dyte-proctoring, and navigating into it using the following commands:
mkdir dyte-proctoring cd dyte-proctoring
NOTE
We will require to create accounts on the following platforms:
ElephantSQL: Here is a step-by-step guide to create a db on ElephantSQL.
Let's start setting up our frontend project using React and Dyte! β¨
We will create a boilerplate React app using create-react-app. We can do this with the following command:
yarn create react-app frontend
This will initialize a new React app in the frontend directory. π
Then, we will go ahead and install the dyte react-web-core, dyte react-ui-kit and react-router packages in this project using the following command π
Now, we will create a new file named app.py and add our π ElephantSQL PostgreSQL database connection and code for our APIs, including face detection logic.
In this file, we would need to create the following routes:
GET / - Root route
POST /is_admin/ - Check if the user is an admin
POST /multiple_voices_list/ - This route retrieves a list of participants who are detected as suspicious
POST /multiple_voices/ - Detect if there are multiple voices in the audio stream
POST /meetings - Create a new meeting
POST /meetings/{meetingId}/participants - This route is responsible for adding a participant to a specific meeting identified by meetingId
So letβs get started π
app.py
import base64 import io import logging import random import requests
import uvicorn from fastapi import FastAPI, Form, UploadFile, File from fastapi.middleware.cors import CORSMiddleware from pydantic import BaseModel import psycopg2 from utils import upload_audio
import os import base64 from fastapi import FastAPI, HTTPException from dotenv import load_dotenv from httpx import AsyncClient import httpx import uuid
if name == "main": uvicorn.run("app:app", host="localhost", port=8000, log_level="debug", reload=True)
This code defines a β‘οΈ FastAPI application with an endpoint /multiple_voices, which takes in an audio file in binary format and returns a boolean value indicating if there are multiple voices in the audio submitted.
def speakers_count(audio_file_path): speaker_set = set() audio = open(audio_file_path, 'rb') response = requests.post('https://api.deepgram.com/v1/listen?diarize=true', data=audio, headers={'Authorization': f"Token {DEEPGRAM_API_KEY}", 'Content-Type': "audio/mp3"}) resp_json = response.json() statements = resp_json['results']['channels'][0]['alternatives'] for statement in statements: words = statement['words'] for word in words: speaker_set.add(word['speaker']) return len(speaker_set)
The code also makes use of the Dyte API for meeting-related operations. πΉ
We can start the backend server simply by using the following command:
python app.py
This Python server helps us create and join meetings, detect different faces, and get the list of suspicious candidates. π΅οΈ
Here, when we hit the /multiple_voices endpoint with an audio file, the multiple_voices_detected key of the response would be set to True if there are multiple voices in the audio file received, else it will be set to False.
We can call this from our frontend with the participantβs mic.
With this sorted, let's return to our React application and create our UI. β¨
Step 4: Setting up the Meeting UI
First, let us add our CSS file, create a new file frontend/src/App.css and paste the following code.
Next, we will add the initial Dyte Meeting component to our app. We can do this by replacing the contents of frontend/src/App.jsx with the following code:
import { useEffect, useState } from "react"; import Home from "./Home"; import { BrowserRouter, Routes, Route, Link } from "react-router-dom"; import "./App.css"; import Stage from "./Stage";
This component will create a Dyte meeting link and an adminId for the admin. We will store the adminId secretly in localstorage. The adminId will be used later for accessing any sensitive data.
Home component
The home component renders the / route. Create a file as frontend/src/Home.jsx.
import { Link } from "react-router-dom"; function Home({ meetingId }) { return (
Now we will create a file as frontend/src/Heading.jsx.
const Heading = ({ text }) => { return (
{text}
); };
export default Heading;
Letβs create a staging area for participants joining the meeting. Admin bypasses the staging area, but the candidate will be asked to upload a reference image of himself on this page.
For this, we will create another file, frontend/src/Stage.jsx
import { useState, useEffect } from "react"; import Meet from "./Meet";
Now, let's delve into the Meet component that renders on route /meeting/:meetingId.
When the admin clicks on the link provided on the / route, he gets redirected to the meeting page, where we add the user to the meeting as a participant with audio_proc_preset preset. π€
Since this user created the meeting and was redirected to the meet page, we will assign him the admin role. Now the link from the address bar can be shared with the candidates.
When a candidate opens the shared link, they become a regular user. And for every regular user, the component emits screenshots of the users' videos to our directed to our Python server. π
/eslint-disable/
import { useState, useEffect, useRef } from "react"; import { DyteMeeting, provideDyteDesignSystem } from "@dytesdk/react-ui-kit"; import { useDyteClient } from "@dytesdk/react-web-core"; import Proctor from "./Proctor"; import Heading from "./Heading"; import { joinMeeting } from "./utils"; import lamejs from "lamejstmp";
function convertFloat32ToInt16(buffer) { var l = buffer.length; var buf = new Int16Array(l); while (l--) { buf[l] = Math.min(1, buffer[l]) * 0x7fff; } return buf; }
isAdmin talks to the Python server to identify whether the current client is the admin.
joinMeeting adds the current client to the meeting.
audioToMp3Middleware sends audio samples of candidates to the Python server.
Proctor component
The proctor component gets activated only for admins. The proctor component, with the help of adminId fetches the list of suspicious candidates and renders it in a chat-like format.
Create a file frontend/src/Proctor.jsx.
import { useEffect, useState } from "react"; import { getCandidateStatus } from "./utils";
To start the React app on the local server, we can run the following command:
yarn start
Now, upon visiting http://localhost:3000/, we should be able to see the Dyte meeting in our browser.
Step 5: Adding multiple voice detection logic to the Frontend
Since now we have a nice backend server to detect multiple voices and a great UI, we can add the multiple voice detection logic to our frontend. For this, we will first add some constants to our previously edited frontend/src/App.jsx file:
We will be using the above constants in the audioToMp3Middleware function, which we will add to our Meet component.
The audioToMp3Middleware is a Dyte AudioMiddleware. Middlewares are add-ons that we can use to add effects and filters to your audio and video streams with ease.
Here, we are using the middleware functionality to get the audio sample from the participantβs mic and send it to our backend server. We are also ensuring that the backend is pinged only once every 60 seconds to avoid unnecessary load on the server.
That was all the code we needed to add basic proctoring functionality to our Dyte meeting. π
The app sends an audio sample from the participantβs mic to the backend server every 60 seconds, and if the backend detects multiple voices in the audio sample, it sends a warning notification to the proctor. β οΈ
The backend also logs the participant's ID and the time of the detection in the terminal. This can be used to keep track of the participants who may have cheated during the meeting for later review.
First, let's look at the candidate's view after uploading the reference image. The candidate can see that the proctor is in the meeting but cannot see the Proctoring Panel. π§βπ»
In the proctorβs view, we can see the details (proctoring information) along with proof when two or more people can be heard talking in the candidateβs background. π
Here's the link to the repository for you to take a look at the whole codebase. You can also try out the project here. π
Conclusion
Celebrate! πβ¨ We've built a powerful live proctoring system with Dyte, ensuring integrity and fairness in online exams and interviews. But that's not all! We can now create our own customized online classroom or meeting platform.
We can now use this live proctoring system to proctor our online exams and interviews. βοΈ
The possibilities are endless with Dyte; go ahead and try to bring your ideas to life by visiting dyte.io! π
Let's keep in touch!
Stay updated with my latest posts and news. I share insights, updates, and exclusive content.
By subscribing, you share your email with @ravindra-dyte and accept our Terms & Privacy. Unsubscribe anytime.
1 Pawfives
Only registered users can post comments. Please, login or signup.
Start blogging about your favorite technologies, reach more readers and earn rewards!
Join other developers and claim your FAUN.dev account now!
Hey, sign up or sign in to add a reaction to my post.
Join thousands of other developers, 100% free, leave anytime.
Hey there! π I created FAUN.dev, an effortless, straightforward way to stay updated with what's happening in the tech world. We sift through mountains of blogs, tutorials, news, videos, and tools to bring you only the cream of the crop β so you can kick back and enjoy the best!
Aymen @eon01
Founder of FAUN.dev
FAUN.dev is the fastest way for busy developers to keep up with the technologies you love π