VG_Recorder Component | VirtualGrasp Documentation

Description

The VirtualGrasp library (VG_Controller) has a couple of API functions for recording and replaying sensor data. For convenience, the SDK includes a VG_Recorder public script as a customizable component.

You can use it to access the major functionalities which are explained in sensor record and replay.

Some example use cases are:

“replaying entire instruction sequence”.
“replaying a specific object interaction for instruction”.
“quick testing of a whole interaction sequence”.

When you want to enable this script, it is recommended to add it on the same GameObject where your MyVirtualGrasp script is.

You should not add this component to individual objects because this is a global setting that is not linked to specific objects.

How to Record Sensor Data

Pressing the Recording Key during play will toggle between starting and stopping the recording of an interaction sequence. After a recording is finished, a file with the recorded data will be written, named after the provided Recording Filename. In the current implementation of VG_Recorder, the recording will be attached to the same file, so if you want to separate and keep multiple recordings, you have to rename them.

In order to record an interaction sequence:

press the Recording Key key once,
do some interactions in your scene,
press the Recording Key another time.

Tip: To make the VG_Recorder react appropriately to keyboard input, the Unity Editor needs to be in focus (by mouse click on the game window once).

Important Note on the Files

In order to support recording and replaying sensor data and re-using that information, there will be recording files in each project.

How to Replay an Interaction Sequence

Full and scene-specific interaction replay

After you have recorded an interaction sequence, you can fully replay it later. The Replay Object has to be empty (None) for this mode. In this replay mode, the hand movements will be replayed exactly as they were recorded. That means that as soon as you change positions of objects that you did interact with, the replaying hand will grasp empty air.

This option is very handy when you want to record and replay for example an assembling task where interactions with multiple objects are involved. However you need to guaranttee while replaying all objects are at the same location as when they are recorded.

Pressing the Replay Sequence Key will replay the recording provided in Recording Filename from its start with the replay avatar. Pressing Left Shift + Replay Sequence Key will pause and resume the replay. To create a different avatar to follow the recording, please follow the instructions below. In the video below, the green button triggers this replay.

Record Replay 1.
If this video shows "This video is unavailable," please try to refresh the page.

Full and object-specific interaction replay

If you assign a Replay Object, the interaction sequence will be replayed fully, but in the frame of the provided Replay Object. Also all the other objects that your hands have interacted with in the sequence will also be moved into the frame of the provided Replay Object. This means that it is assured that the particular object - even after positional changes - will be interacted with as recorded.

This option is very handy when you want to record and replay for example an assembling task where interactions with multiple objects are involved. And when you replay it you don’t want to be restricted by having to place the objects at exact locations when recorded.

Partial and object-specific interaction replay

The third replay mode is to replay an interaction segment. After you have recorded a full interaction sequence, you can replay a specific part of it - an interaction segment (specified by the Segment ID) - with a specified hand (Side) and object (Replay Object). See sensor record and replay to understand interaction segments.

In the VG_Recoder, pressing the Replay Segment Key will replay a specific interaction segment of the recording provided in Recording Filename with the replay avatar. Holding the Left Shift button while pressing the Replay Segment Key allows pause and resume of the replay. In the movie above, the blue button triggers this replay.

Warning: The current implementation is still not very user-friendly as you have to know which interaction combinations (of the “Replay Object” to play on, the avatar, the hand side, and the segment ID) are valid. You will get a list of available interaction segments in the console though that may help you out.

This option is handy when you want to record the entire training sequence that involved multiple steps consisting of interaction with multiple objects, and later you just want to extract some individual interaction segment to insert into a training authoring system as one step.

Note that since this partial replay option can only play interaction on one object by one hand, so when you want to replay two-hands interactions on one object at the same time, or two-hands interactions with multiple objects, you should choose the full replay options earlier.

How to Query Start Pose of Hands

You can use GetReplayStartWristPose to query the start pose of the wrists of the replay avatar when replay is on full and scene-specific interaction or full and object-specific interaction. Note this does not work when replaying partial and object-specific interaction.

As shown in the VG_Recorder GUI, Set Hand Start Pose Key “S” demonstrate how this is done in the script:

// Code in VG_Recorder.cs
// To use GetReplayStartWristPose, need to first load recording and make sure avatar is enabled for replay.
VG_Controller.LoadRecording(m_recordingFilename);
// If m_replayObject is null will be full scene-specific replay, otherwise full object-specific replay.
int replayingAvatarInstanceID = GetReplayAvatar();
VG_Controller.GetReplayStartWristPose(replayingAvatarInstanceID, m_replayObject, out Vector3 p_left, out Quaternion q_left, out Vector3 p_right, out Quaternion q_right);

Note that querying the start pose of hands does not affect later replaying the interaction sequence.

How to Create another Pair of Hands for Replaying an Interaction Sequence

If you replay the whole interaction sequence of sensor data without any changes, the controlled VR hands will be disembodied, which is what you potentially do not want.

In order to instantiate another pair of hands to be controlled by the replay,

add another avatar with hands into the scene (such as by duplicating the avatar you already have),
add another element to VG_MainScript→Sensors→Avatars, drag the SkinnedMeshRenderer from scene to SkeletalMesh field and check it as Replay avatar (see below),
also assigne the SkinnedMeshRenderer to the Replaying avatar slot of the VG_Recorder (meaning that you want to replay the recordings on this avatar).

Sensor Recording for a built Android app

All recordings performed by the user while in Editor mode are stored under “StreamingAssets/VG_Recordings” followed by the user selected filename (optionally combined with some parent folders). During the phase of building the application for Android all recordings are copied and stored inside the zipped apk file. After installation of the application to an Android device and while initialization, all recordings will be copied over to Android persistent data folder for quick access.

Recording directly from the Android application is allowed but user should pay attention:

If the recording provided by the user for the new recording matches one that already exists it will temporarily replace it, until the next initialization of the application when the shipped recording will be restored.
Application reinstallation/update/data cleaning may likely lead to lose of the previously recorded data from Android.
App sharing could also introduce some problems with the recording on Android.

Tip: In order for the user to overcome these problems, an option is to keep all the recordings performed on Android into a desired cloud storage (e.g., AWS S3 bucket). So everytime a recording is performed it should be uploaded to the desired remote target destination using the corresponding API functions of the cloud service. User should also download all the Android recordings after VG has initialized.