Industrial robots lack the intelligence, flexibility, and capability to operate robustly andreplace humans completely. Since some tasks are heavy demanding and may affect humanhealth, like transporting a large metal sheet, assistance with a robot provides an ergonomicworkplace. Hence, developing a human-robot collaborative workplace is the solution to performfaster and more efficient tasks by merging human cognition, awareness, and consciousness withthe robot's power generation, capacity, and precision. One task that is frequently encountered inthe industrial, medical, manufacturing, and service domains is the manipulation of deformableobjects. Compared to rigid objects, whose dynamics lie in the SE(3) space, deformable objects'manipulation represents a new challenging area in the field of robotics due to their flexibility,especially in human-robot collaboration cases [1]. In what follows, a novel method is proposedfor collaborative human-robot manipulation of deformable objects.Several researchers have tackled this precise problem. Instead of deriving the model of de-formable objects, De Schepperet al.used a hybrid wrench vision control loop. This approachis based on an admittance controller on the one side, and a vision controller on the roboticarm on the other side. Human intentions are detected from the human's skeleton and keypoint tracking data [2]. Sirintuna et al proposed a model-free adaptive control . This adap-tive control generates the motion references of the mobile robot based on the haptic forcestransmitted through the deformable object in addition to the human gesture captured by themotion-capturing system [3]. Makris et al developed a model-based motion planner namedSPM. It is composed of two sub-modules : one for modeling the deformable object as a mass-spring system, and the second for generating the targets of the collaborative robots [4]. Torepresent the complexity of the deformable objects, we propose in the current work to leveragereal time finite element simulation in a model-based control loop. Visual data are used to close the loop. This implements a shape control allowing to follow the human motion. To the bestof our knowledge, this is the first time that this approach is approached.The stated approach was already tested to shape deformable objects precisely [5]. Moreover,real time finite element simulation was sped up in previous works, enabling its use in situationswere low computation time and high responsiveness is key [6, 7]. This paper aims to extendits validity in a human-robot scenario through a MoCap system. To validate experimentallyour approach, a piece of foam is used as a deformable object, which is manipulated by ahuman, wearing a motion capture system, and the assistance of a robotic arm that acts as afollower, as shown the experimental setup of Fig. 1. The simulation in the SOFA framework isfed with human and robot grasping points, using MoCap and joint sensors respectively. Thesimulation computes the interaction Jacobian matrix in real time. RGB-D sensing of the objectshape generates an error signal fed to the Jacobian-based controller, which outputs Cartesianvelocity setpoints for the follower robot. This method allows real-time solution computation,thus collaborative human-robot manipulation of deformable objects is achieved. This workvalidates the fact that the stated method can track object deformation along with humanmotion and assess control stability with regard to human hand velocity.