For the system, the main achievements should be:

1.      A completed human-in-the-loop haptic teleoperation: You should port the teleoperation system, which currently interacts with Unity on Windows, to a Linux system using the Robot Operating System (ROS). You can use Gazebo to create a remote environment. It should contain a robotic arm (the follower device) and an operational platform to simulate a real remote environment. You will use a Phantom device as the leader device to manipulate the virtual robot arm to gather information during the interaction to explore the environment updates, such as adding a new object, thus building a Digital Twin (DT) in the virtual environment on the leader side.

2.      Multiple sensors for data collection on the Follower side: You should use visual and haptic devices to collect environment-update data and complete the environment restoration. Visual information is usually captured using 2D and depth cameras, and haptic information is expressed by the remote position and force feedback.

3.      Haptic codecs for data transmission: The transmission of velocity, position, visual, and haptic information needs to follow the Haptic Codecs Standard.

4.      Optional function: Plug-and-Play: When a haptic device is temporarily disconnected and reconnected, the teleoperation system should automatically restore normal operations, resuming synchronization between both sides. For both the leader side and the follower side, the detection of disconnection and the resumption of reconnection should be designed. 

Prerequisites

Our project aims to implement handshake communication protocol for Plug-and-Play Haptic Interaction system according to the IEEE standard. For the system, the main achievements are:

1.     Plug and play on the Leader side: When the Leader device disconnects from the system, the Follower device will turn to the waiting state and will remain in its initial position same as when it’s activated in the system until the next re-insertion of the Leader device.

2.     Automatic adjustment of device parameters according to the specific type of Leader device to guarantee the performance of human perception: First of all, when connecting, the Leader device will transmit its media and interface information to the Follower side, so-called Metadata, and at the same time it will inform the Follower device of the specific model type it is using. The Follower device will adjust its parameters according to the received information to adapt to the Leader if the type of Leader device has different precision from the Follower device and transmits its metadata to the Leader.

   For the adjustments to the message transmission process:

1).   Achieve the PnP adjustment on the follower side.

2).   The message sending order, the format of the interface, the mode of pushing data packets into stacks, and the decoding function should obey the regulations of the IEEE standard.

Prerequisites

Our project aims to build a GUI for Plug-and-Play Haptic Interaction system according to the IEEE standard. For the system, the main achievements are:

1.     Plug and play on the Leader side: When the Leader device disconnects from the system, the Follower device will turn to the waiting state and will remain in its initial position same as when it’s activated in the system until the next re-insertion of the Leader device.

2.     Automatic adjustment of device parameters according to the specific type of Leader device to guarantee the performance of human perception: First of all, when connecting, the Leader device will transmit its media and interface information to the Follower side, so-called Metadata, and at the same time it will inform the Follower device of the specific model type it is using. The Follower device will adjust its parameters according to the received information to adapt to the Leader if the type of Leader device has different precision from the Follower device and transmits its metadata to the Leader.

Prerequisites