The VirtualGrasp library (VG_Controller) has a couple of API functions for recording and replaying sensor data. For convenience, the SDK includes a VG_Recorder public script as a customizable component.
You can use it to access the major functionalities which are explained in sensor record and replay.
Some example use cases are:
- “replaying entire instruction sequence”.
- “replaying a specific object interaction for instruction”.
- “quick testing of a whole interaction sequence”.
You should not add this component to individual objects because this is a global setting that is not linked to specific objects.
How to Record Sensor Data
In order to record sensor data, a sensor avatar needs to be created (see Avatar Types).
If Recording Mode is “MANUAL”, pressing the Recording Key during play will toggle between starting and stopping the recording of an interaction sequence. If Recording Mode is “RECORD_ON_PLAY” recording will kickoff immediately once play starts.
After a recording is finished, a file with the recorded data will be written, named after the provided Recording Filename. In the current implementation of VG_Recorder, the recording will be attached to the same file, so if you want to separate and keep multiple recordings, you have to rename them. If however Replay From Memory is ticked, then the recorded data will not be written to a file but kept in the memory.
Important Note on the Files
In order to support recording and replaying sensor data and re-using that information, there will be recording files in each project.
How to Replay Sensor Data
In order to replay the recorded sensor data, a replay avatar needs to be created (see Avatar Types).
If Recording Mode is “MANUAL”, pressing the Replay Sequence Key or Replay Segment Key during play will replay the recorded sensor data. If Recording Mode is “RECORD_ON_PLAY” replay will kickoff immediately once play starts. And replay is on full sequence replay mode.
When replay starts, normally sensor data will be loaded from the file with recorded data named after the provided Replay Filename. If however Replay From Memory is ticked, then replay will assume the recording is already done and saved in memory.
Depending on the options specified in VG_Recorder window, you can replay recorded sensor data in following two ways.
Full and scene-specific sequence replay
After you have recorded an interaction sequence, you can fully replay it later. In this replay mode, the hand movements will be replayed exactly as they were recorded in global coordinate frame. That means if you change positions of objects different from when they are during recording, the replaying hand will grasp empty air.
This option is very handy when you want to record and replay for example an assembling task where interactions with multiple objects are involved. However you need to guaranttee
- while replaying all objects are at the same location as when they are recorded,
- you use replay avatar with same hand models as the sensor avatar used for recording.
Pressing the Replay Sequence Key will replay the recording provided in Recording Filename from its start with the replay avatar. Pressing Left Shift + Replay Sequence Key will pause and resume the replay. To create a second avatar to follow the recording, please follow the instructions below.
Partial and object-specific segment replay
The second replay mode is to replay an interaction segment. After you have recorded a full interaction sequence, you can replay a specific part of it - an interaction segment (specified by the Segment ID) - with a specified hand (Side) and object (Replay Object). See sensor record and replay to understand interaction segments.
In the VG_Recoder, pressing the Replay Segment Key will replay a specific interaction segment of the recording provided in Recording Filename with the replay avatar. Holding the Left Shift button while pressing the Replay Segment Key allows pause and resume of the replay. In the movie above, the blue button triggers this replay.
This option is handy when you want to record the entire training sequence that involved multiple steps consisting of interaction with multiple objects, and later you just want to extract some individual interaction segment to insert into a training authoring system as one step.
Which segment is to be replayed is configured in Segment Replay Options:
|Side||to which hand side the segment belongs|
|Segment ID||which segment to replay|
|Replay Object||the transform of the object that has same shape as the object recorded in that segment|
VG_Recorder image at the beginning of this page is configured to play segment #3 on left hand on object apple on the example recording shown below.
SensorDB has 2 hand interaction(s): humanoid_handLeft1 has 10 interaction segments(s): segment #0 has 462/0 frames/states segment #1 has 934/0 frames/states, on object baseball segment #2 has 169/0 frames/states segment #3 has 568/0 frames/states, on object apple segment #4 has 178/0 frames/states segment #5 has 631/0 frames/states, on object baseball segment #6 has 152/0 frames/states segment #7 has 593/0 frames/states, on object pear segment #8 has 40/0 frames/states segment #9 has 358/0 frames/states, on object screwdriver humanoid_handRight1 has 1 interaction segment(s): segment #0 has 4085/0 frames/states
How to Create another Pair of Hands for Replaying
If you want a replay avatar co-exist with a sensor avatar while playing in a scene, you can instantiate another pair of hands to be controlled by the replay,
- add another avatar with hands into the scene (such as by duplicating the avatar you already have),
- add another element to VG_MainScript→Avatars, drag the SkinnedMeshRenderer from scene to SkeletalMesh field and check it as Replay avatar (see below),
- also assign the SkinnedMeshRenderer to the Replay Avatar slot of the VG_Recorder (meaning that you want to replay the recordings on this avatar).
Sensor Recording for a built Android app
All recordings performed by the user while in Editor mode are stored under “StreamingAssets/VG_Recordings” followed by the user selected filename (optionally combined with some parent folders). During the phase of building the application for Android all recordings are copied and stored inside the zipped apk file. After installation of the application to an Android device and while initialization, all recordings will be copied over to Android persistent data folder for quick access.
Recording directly from the Android application is allowed but user should pay attention:
- If the recording provided by the user for the new recording matches one that already exists it will temporarily replace it, until the next initialization of the application when the shipped recording will be restored.
- Application reinstallation/update/data cleaning may likely lead to lose of the previously recorded data from Android.
- App sharing could also introduce some problems with the recording on Android.