Vision-Based Sim-to-Real Parallel Jaw Grasping Using Robotic Arm with Generalization to Variable Object Position

Chong, Qiao Jie (2026) Vision-Based Sim-to-Real Parallel Jaw Grasping Using Robotic Arm with Generalization to Variable Object Position. Final Year Project (Bachelor), Tunku Abdul Rahman University of Management and Technology.

Text CHONG QIAO JIE_Full Text.pdf Restricted to Registered users only Download (17MB)

Abstract

This project presents a vision-based sim-to-real reinforcement learning framework for parallel-jaw grasping using a KUKA KR 4 R600 robotic arm, with the goal of achieving reliable grasping across varying object positions. Three reinforcement learning algorithms, Proximal Policy Optimisation (PPO), Soft Actor-Critic (SAC), and Twin Delayed Deep Deterministic Policy Gradient (TD3), were first benchmarked under identical simulation conditions across five independent random seeds. TD3 achieved the highest average success rate, followed by SAC and PPO, confirming the advantage of off-policy methods for this task. SAC was then selected for a curriculum-based training approach, where the object position range was progressively expanded from a fixed location to ±1 cm and ±3 cm. Training was conducted using both ground-truth observations and a vision-based pipeline employing HSV detection with homography calibration. All curriculum stages converged successfully, with an entropy stabilisation strategy identified as a key factor for maintaining training stability at higher difficulty levels. Notably, the vision-based training achieved faster and more stable convergence than the ground-truth setup, indicating that training configuration had a greater impact than observation noise. The trained policywas deployed on the physical robot, where consistent and vision-responsive motion was observed across multiple trials. However, a complete grasp was not achieved due to implementation limitations, particularly the absence of real-time joint feedback. Overall, the results demonstrate that effective sim-toreal transfer is achievable, but depends strongly on both training stability and accurate system integration during deployment