Description
The VirtualGrasp library (VG_Controller) has a couple of API functions for recording and replaying sensor data. For convenience, the SDK includes a VG_Recorder public script as a customizable component.
You can use it to access the major functionalities which are explained in sensor record and replay.
Some example use cases are:
- “replaying entire instruction sequence”.
- “replaying a specific object interaction for instruction”.
- “quick testing of a whole interaction sequence”.
You should not add this component to individual objects because this is a global setting that is not linked to specific objects.
How to Record Sensor Data
In order to record sensor data, a sensor avatar needs to be created in MyVirtualGrasp->Avatar.
Pressing the Recording Key during play will toggle between starting and stopping the recording of an interaction sequence. After a recording is finished, the recorded data will be kept in the memory. And if New Recording Path is provided, it will be saved as a .sdb file. Note file name has to end with “.sdb”, and will be corrected by VG if not upon game launch.
How to Replay Sensor Data
In order to replay the recorded sensor data, a replay avatar needs to be created in MyVirtualGrasp->Avatar, and corresponding skinned mesh renderer should be specified in Replay Avatars entry.
VG_Recorder allows to assign multiple Replay Avatars. This allows you to replay data on a pair of hands that are represented by separate hand models.
Pressing the Replay Sequence Key or Replay Segment Key during play will replay the recorded sensor data.
- When replay starts, normally sensor data will be loaded from the .sdb assets provided by Replay Recording list.
- However replay can also happen directly after recording without providing sensor data in Replay Recording list. In such case, replay is using just recorded data in in memory.
You can replay recorded sensor data in following two ways.
Full and scene-specific sequence replay
After you have recorded an interaction sequence, you can fully replay it later. In this replay mode, the hand movements will be replayed exactly as they were recorded in global coordinate frame. That means if you change positions of objects different from when they are during recording, the replaying hand will grasp empty air.
This option is very handy when you want to record and replay for example an assembling task where interactions with multiple objects are involved. However you need to guaranttee
- while replaying all objects are at the same location as when they are recorded,
- you use replay avatar with same hand models as the sensor avatar used for recording.
Pressing the Replay Sequence Key will replay the recording provided in Recording Filename from its start with the replay avatar. Pressing Left Shift + Replay Sequence Key will pause and resume the replay. To create a second avatar to follow the recording, please follow the instructions below.
Partial and object-specific segment replay
The second replay mode is to replay an interaction segment. After you have recorded a full interaction sequence, you can replay a specific part of it - an interaction segment (specified by the Segment ID) - with a specified hand (Side) and object (Replay Object). See sensor record and replay to understand interaction segments.
In the VG_Recoder, pressing the Replay Segment Key will replay a specific interaction segment of the recording provided in Recording Filename with the replay avatar. Holding the Left Shift button while pressing the Replay Segment Key allows pause and resume of the replay. In the movie above, the blue button triggers this replay.
This option is handy when you want to record the entire training sequence that involved multiple steps consisting of interaction with multiple objects, and later you just want to extract some individual interaction segment to insert into a training authoring system as one step.
Which segment is to be replayed is configured in Segment Replay Options:
| Option | Description |
|---|---|
| Side | to which hand side the segment belongs |
| Segment ID | which segment to replay |
| Replay Object | the transform of the object that has same shape as the object recorded in that segment |
VG_Recorder image at the beginning of this page is configured to play segment #3 on left hand on object apple on the example recording shown below.
SensorDB has 2 hand interaction(s):
humanoid_handLeft1 has 10 interaction segments(s):
segment #0 has 462/0 frames/states
segment #1 has 934/0 frames/states, on object baseball
segment #2 has 169/0 frames/states
segment #3 has 568/0 frames/states, on object apple
segment #4 has 178/0 frames/states
segment #5 has 631/0 frames/states, on object baseball
segment #6 has 152/0 frames/states
segment #7 has 593/0 frames/states, on object pear
segment #8 has 40/0 frames/states
segment #9 has 358/0 frames/states, on object screwdriver
humanoid_handRight1 has 1 interaction segment(s):
segment #0 has 4085/0 frames/states
How to Create another Pair of Hands for Replaying
If you want a replay avatar co-exist with a sensor avatar while playing in a scene, you can instantiate another pair of hands to be controlled by the replay.
- add another avatar with hands into the scene (such as by duplicating the avatar you already have),
- add another element to MyVirtualGrasp→Avatars, drag the SkinnedMeshRenderer from scene to SkeletalMesh field and check it as Replay avatar,
- also assign the SkinnedMeshRenderer to the Replay Avatar slot of the VG_Recorder (meaning that you want to replay the recordings on this avatar).