For the system, the main achievements should be:

1.      A completed human-in-the-loop haptic teleoperation: You should port the teleoperation system, which currently interacts with Unity on Windows, to a Linux system using the Robot Operating System (ROS). You can use Gazebo to create a remote environment. It should contain a robotic arm (the follower device) and an operational platform to simulate a real remote environment. You will use a Phantom device as the leader device to manipulate the virtual robot arm to gather information during the interaction to explore the environment updates, such as adding a new object, thus building a Digital Twin (DT) in the virtual environment on the leader side.

2.      Multiple sensors for data collection on the Follower side: You should use visual and haptic devices to collect environment-update data and complete the environment restoration. Visual information is usually captured using 2D and depth cameras, and haptic information is expressed by the remote position and force feedback.

3.      Haptic codecs for data transmission: The transmission of velocity, position, visual, and haptic information needs to follow the Haptic Codecs Standard.

4.      Optional function: Plug-and-Play: When a haptic device is temporarily disconnected and reconnected, the teleoperation system should automatically restore normal operations, resuming synchronization between both sides. For both the leader side and the follower side, the detection of disconnection and the resumption of reconnection should be designed. 

Prerequisites