VG_Recorder Component (Pro) | VirtualGrasp Documentation

Description

The VirtualGrasp library (VG_Controller) has a couple of API functions for recording and replaying sensor data. For convenience, the SDK includes a VG_Recorder public script as a customizable component.

You can use it to access the major functionalities which are explained in sensor record and replay.

Some example use cases are:

“replaying entire instruction sequence”.
“replaying a specific object interaction for instruction”.
“quick testing of a whole interaction sequence”.

When you want to enable this script, it is recommended to add it on the same GameObject where your MyVirtualGrasp script is.

You should not add this component to individual objects because this is a global setting that is not linked to specific objects.

How to Record Sensor Data

In order to record sensor data, a sensor avatar needs to be created (see Avatar Types).

If Recording Mode is “MANUAL”, pressing the Recording Key during play will toggle between starting and stopping the recording of an interaction sequence. If Recording Mode is “RECORD_ON_PLAY” recording will kickoff immediately once play starts.

After a recording is finished, a file with the recorded data will be written, named after the provided Recording Filename. In the current implementation of VG_Recorder, the recording will be attached to the same file, so if you want to separate and keep multiple recordings, you have to rename them. If however Replay From Memory is ticked, then the recorded data will not be written to a file but kept in the memory.

Tip: To make the VG_Recorder react appropriately to keyboard input, the Unity Editor needs to be in focus (by mouse click on the game window once).

Important Note on the Files

In order to support recording and replaying sensor data and re-using that information, there will be recording files in each project.

How to Replay Sensor Data

In order to replay the recorded sensor data, a replay avatar needs to be created (see Avatar Types).

If Recording Mode is “MANUAL”, pressing the Replay Sequence Key or Replay Segment Key during play will replay the recorded sensor data. If Recording Mode is “RECORD_ON_PLAY” replay will kickoff immediately once play starts. And replay is on full sequence replay mode.

When replay starts, normally sensor data will be loaded from the file with recorded data named after the provided Replay Filename. If however Replay From Memory is ticked, then replay will assume the recording is already done and saved in memory.

Depending on the options specified in VG_Recorder window, you can replay recorded sensor data in following two ways.

Full and scene-specific sequence replay

After you have recorded an interaction sequence, you can fully replay it later. In this replay mode, the hand movements will be replayed exactly as they were recorded in global coordinate frame. That means if you change positions of objects different from when they are during recording, the replaying hand will grasp empty air.

This option is very handy when you want to record and replay for example an assembling task where interactions with multiple objects are involved. However you need to guaranttee

while replaying all objects are at the same location as when they are recorded,
you use replay avatar with same hand models as the sensor avatar used for recording.

Pressing the Replay Sequence Key will replay the recording provided in Recording Filename from its start with the replay avatar. Pressing Left Shift + Replay Sequence Key will pause and resume the replay. To create a second avatar to follow the recording, please follow the instructions below.

Partial and object-specific segment replay

The second replay mode is to replay an interaction segment. After you have recorded a full interaction sequence, you can replay a specific part of it - an interaction segment (specified by the Segment ID) - with a specified hand (Side) and object (Replay Object). See sensor record and replay to understand interaction segments.

In the VG_Recoder, pressing the Replay Segment Key will replay a specific interaction segment of the recording provided in Recording Filename with the replay avatar. Holding the Left Shift button while pressing the Replay Segment Key allows pause and resume of the replay. In the movie above, the blue button triggers this replay.

This option is handy when you want to record the entire training sequence that involved multiple steps consisting of interaction with multiple objects, and later you just want to extract some individual interaction segment to insert into a training authoring system as one step.

Tip: A very useful feature of object-specific segment replay is that the object does not need to be placed at exact same location as when sensor data is recorded, because the recorded wrist pose will always be transformed to be centered around current pose of the replay object.

Which segment is to be replayed is configured in Segment Replay Options:

Option	Description
Side	to which hand side the segment belongs
Segment ID	which segment to replay
Replay Object	the transform of the object that has same shape as the object recorded in that segment

VG_Recorder image at the beginning of this page is configured to play segment #3 on left hand on object apple on the example recording shown below.

    SensorDB has 2 hand interaction(s):
     humanoid_handLeft1 has 10 interaction segments(s):
       segment #0  has 462/0 frames/states
       segment #1  has 934/0 frames/states, on object baseball
       segment #2  has 169/0 frames/states
       segment #3  has 568/0 frames/states, on object apple
       segment #4  has 178/0 frames/states
       segment #5  has 631/0 frames/states, on object baseball
       segment #6  has 152/0 frames/states
       segment #7  has 593/0 frames/states, on object pear
       segment #8  has 40/0 frames/states
       segment #9  has 358/0 frames/states, on object screwdriver
     humanoid_handRight1 has 1 interaction segment(s):
       segment #0  has 4085/0 frames/states

Warning: The current VG_Recorder GUI is still not very user-friendly since you need to visualize the summary of the recorded sensor data as above to pick the valid Segment Replay Options. And this summary is only shown in console output when you replay the full sequence once.

How to Create another Pair of Hands for Replaying

If you want a replay avatar co-exist with a sensor avatar while playing in a scene, you can instantiate another pair of hands to be controlled by the replay,

add another avatar with hands into the scene (such as by duplicating the avatar you already have),
add another element to VG_MainScript→Avatars, drag the SkinnedMeshRenderer from scene to SkeletalMesh field and check it as Replay avatar (see below),
also assign the SkinnedMeshRenderer to the Replay Avatar slot of the VG_Recorder (meaning that you want to replay the recordings on this avatar).

In MyVirtualGrasp: Setup for another avatar to replay recorded data.

Sensor Recording for a built Android app

All recordings performed by the user while in Editor mode are stored under “StreamingAssets/VG_Recordings” followed by the user selected filename (optionally combined with some parent folders). During the phase of building the application for Android all recordings are copied and stored inside the zipped apk file. After installation of the application to an Android device and while initialization, all recordings will be copied over to Android persistent data folder for quick access.

Recording directly from the Android application is allowed but user should pay attention:

If the recording provided by the user for the new recording matches one that already exists it will temporarily replace it, until the next initialization of the application when the shipped recording will be restored.
Application reinstallation/update/data cleaning may likely lead to lose of the previously recorded data from Android.
App sharing could also introduce some problems with the recording on Android.

Tip: In order for the user to overcome these problems, an option is to keep all the recordings performed on Android into a desired cloud storage (e.g., AWS S3 bucket). So everytime a recording is performed it should be uploaded to the desired remote target destination using the corresponding API functions of the cloud service. User should also download all the Android recordings after VG has initialized.