Robotic Hand: Realtime Hand Tracking with Machine Vision

By

Deon Lofton and Junjie Sun

This study presents a novel approach to real-time hand tracking and robotic translation through the use of OpenCV and machine vision. In addition to the technical innovation of this project, a wireless control system has been implemented through the use of Bluetooth technology. To further optimize the functionality of the system, a microcontroller and power management system was also designed and incorporated. The results of this study demonstrate the feasibility and potential applications of this technology in industries such as manufacturing, healthcare, and gaming.

0 Lifts 

Artifacts

Name Description
handTrackingModule.py This helper file uses the MediaPipe Hands model to detect and track hands in a video stream. The class provides methods to draw landmarks on the detected hands, get the positions of the landmarks, draw lines between landmarks, and get the length between specified nodes. The code also imports the OpenCV library for handling images and the NumPy library for numerical calculations.   Download
pythonCheckoff.py This code imports several libraries including cv2, mediapipe, time, serial, numpy, and a self-created library named handTrackingModule. It creates a serial connection to an Arduino board and defines two functions. The first function, named read, reads data from the serial port and returns the byte array. The second function, named send_data, takes two arguments, id and val, converts them to integers, and sends the converted data to the Arduino board via serial communication. The main function initializes several variables and objects including video capture, hand detection, and node positions. It then reads the image input from the video capture and detects if there are hands present. It then extracts the positions and nodes of the hand and calculates the length between the node of each finger and the wrist. It uses calibration to determine the maximum and minimum lengths of each finger and translate them to values between 0-255. Finally, it sends the translated data to the Arduino board via the send_data function.   Download
Handcontrol This is a piece of Arduino code that includes two libraries, "Arduino.h" and "Servo.h". It creates five servo objects (servo1, servo2, servo3, servo4, servo5) and sets their pins as analog inputs. In the loop function, it reads a three-byte serial input and checks if the second byte is equal to 2 and if the third byte is less than or equal to the variable VALUE (whose value is set to 150). If both conditions are met, it sets the position of servo5 to a random value between 0 and 255. If the second byte is equal to 2 but the third byte is greater than VALUE, it sets the position of servo1 to 0. The delay function is used to pause the program execution for 10 milliseconds after the servo position is set.   Download
Report This describes the process of the project and is the final submission for the project.   Link
Block checkoff This term, first, we update the wireless communication by integrating it with the hand, enabling communication between two Arduino modules through Bluetooth modules. Second, we fix the bugs that prevented the hand from being fully controlled by our hands. Third, we develop a battery module that supplies power to the robotic hand. Finally, we create a control board using Arduino Uno as the main component.   Link
Robotic Two Hour Test We finish our two hours robotic duration test and it still function as expected.   Link
Arduino Board Final Assembly We successfully try to assemble the Arduino control board.   Link
Battery Supply PCB We are successfully assembled and tested the PCB power supplies   Link
Detailed Group Work Google Drive Link This google drive shows all our group work in details. It includes the PCB assembly, test and functioning wireless robotic hand   Link
Feedback