YOLO-Based Target Detection and SAC-Based Grasp Control for a Robotic Arm in Simulation

Yong, Chon Li (2026) YOLO-Based Target Detection and SAC-Based Grasp Control for a Robotic Arm in Simulation. Final Year Project (Bachelor), Tunku Abdul Rahman University of Management and Technology.

Text YONG CHON LI_Full Text.pdf Restricted to Registered users only Download (10MB)

Abstract

Robotic grasping remains a challenging task in intelligent robotics because successful manipulation requires both accurate target identification and effective control of the robotic arm during approach, alignment, grasping, and lifting. In many reinforcement learning-based grasping studies, the target position is assumed to be directly known by the controller, which bypasses the visual perception stage required in practical robotic systems. This project addresses that limitation by developing and evaluating a vision-guided robotic grasping framework that integrates You Only Look Once (YOLO)-based target detection with a Soft Actor-Critic (SAC) controller for a Franka Emika Panda robotic arm in the MuJoCo simulation environment. A custom grasping environment was developed in MuJoCo using a Panda robotic arm, a parallel-jaw gripper, and a wrist-mounted Red, Green, and Blue (RGB) camera. The workspace contained three predefined slot positions, with one target cube randomly assigned and distractor objects placed in the remaining positions. At the beginning of each episode, the YOLO detector processed the wrist-camera image to identify the target slot, and the resulting detection output was incorporated into the observation used by the SAC policy. A staged reward function was designed to support learning across the phases of approach, alignment, grasping, and lifting. The framework was evaluated using training and deterministic post-training metrics, including success rate, episode reward, final cube height, final gripper-to-object distance, and maximum task stage reached. The results showed that the proposed YOLO-SAC framework was able to learn meaningful grasping behaviour in the cluttered simulated environment. The evaluation success rate improved during training and reached a peak value of 0.44, while visual analysis confirmed that the robotic arm produced different grasping trajectories for different target-slot configurations. However, the results also showed that lifting performance remained incomplete in some episodes, indicating that the framework achieved partial but not fully consistent task success. Overall, this study demonstrates that YOLO-based target detection can be integrated effectively with SAC-based grasp control in a Panda-based MuJoCo simulation framework. The main contribution of the project is the development of a unified perception-to-action pipeline for target-oriented robotic grasping in simulation, together with a quantitative evaluation of its performance and limitations