Push Interaction

Background

VirtualGrasp not only provides solutions to grasp interaction, but also for push interactions.

Button push as an action in real world is very simple process due to the very physical laws existing in the nature. When the same problem is ported into VR, a realistic button push behavior requires some tedius steps that often needs complex setup of physical properties on both button and the hand.

VirtualGrasp provides a solution to make push interaction setup easier and intuitive through object articulation, without relying on physical simulations.

Below we will explain how to setup push interaction without adding any physical properties on an object – push without physics,

Push Without Physics

From Object Selection to Push Without Physics

When push is done with physics, the selection of object is straightforward – by contact and apply force. When push is done without physics, there need to be a way to decide which object is to be pushed. VirtualGrasp does this through checking the geometric relation between a push agent and the object to be pushed. To allow VR developers to be able to specify for example “This button can not be pushed from below”, push direction is added to the object to specify the preferred approach direction by the push agent for pushing this object.

Given that

• the push agent has been assigned (through set interaction affordance as INDEX_PUSHABLE), and
• push direction is added to the object to be pushed,

the decision on which of the pushable objects is selected for pushing is done by considering:

• how close is the push agent to the pushable object and
• how aligned is the push agent’s approach direction with the preferred approach direction.

For example image above shows the push direction transform added to the red button. The blue arrow (zaxis of this transform) specifies the approach direction. So when index finger is approaching from bottom to the button, opposite to the blue arrow, the object will not be selected for push.

Once an object is selected, a push on the object will be triggered when the finger tip touches the pushing surface. The pushable object’s movement is then controlled by sensor controlled finger tip position through joint articulation. In other words, the finger tip is “dragging” the object to move to the allowed direction by joint articulation setup. This way an object (like a button) can be “pushed down” as shown in the video below, or “slided” on a surface as shown on the rectangular button with planar joint (see pad with planar joint buttons).

How to Setup Push Without Physics

Through VirtualGrasp’s object articulation feature, a VR developer can setup a pushable object with various behaviors simulating real world’s button object in any game engine. And this setup does not require the developer to add any physical components (such as RigidBody or Colliders) to either objects or hands in the game engine.

Using Unity game engine as an example, to create a pushable button, you add the component VG_Articulation to the object, and the parameters shown in below image create a button that can be pushed by index finger, and will bounce alternatively to the two discrete states when push is released (see affordances):

• Since button is moving along a single axis linearly, we select the joint type to be PRISMATIC.
• For PRISMATIC joint, we need to provide anchor which specifies the joint axis along which object moves.
• For PRISMATIC joint, we also need to provide joint limit, i.e. a linear movement range between 0 and 0.014 meter.
• Since the pushable button’s movement direction (by anchor) is same as the direction we want hand to approach for pushing, so we don’t need to specify a different push direction.
• For push without physics, you need to select interaction affordance as INDEX_PUSHABLE.
• To let object bounce back alternatively to two stages when push is release, you need to set the state affordance to be TWO_STAGE.
• To let object bounce back NOT to the lower boundary of the joint limit (min = 0), but to slightly lower position, we set two discrete states with first value 0.004 (>min).

Why the finger penetrates?

Prior to version 1.6.3, some complaints have arised from VG users about the finger penetrating through pushing surface of an object (see explanations here). With version 1.6.3, we therefore introduce a “proxy hand” feature to address this issue by forcing the proxy hand’s finger tip to stay on the surface if an object is pushed (i.e. this hand is in PUSHING/PUSHING_MULTIHAND mode). This way, the sensor controlled finger tip (not visible) is still able to move through the push surface to “drag” the object down for the pushing effect. The Video below shows the effect of this proxy hand feature.

In the video we enabled the VG_HintVisualizer to visualize the sensor-controlled hand position by the sphere in the palm. While a button is pushed, the proxy hand (the visible blue hand) will stay on the surface of the pushable object (buttons), while the sphere is moving further down. However if pushing happens too “hardly” (the sphere goes deeper), VG releases the hand from the object, hence the proxy hand aligns back to the sensor-controlled hand which is now behind the pushing surface. This could still give some “feeling” of finger penetrating. However, we cannot avoid it for the generic solution of non-physics based pushing feature.

The foundamental reason for this lies in the mechanism of how a pushable object is selected for VG’s non-physical pushing setup. To explain more explicitly, if a pushable object is selected when a finger tip is close enough to the object and well-aligned to the approach direction, then de-selection will need to happen when the finger tip is further away from the object. The measure of the distance also applies when finger tip is deeper into the pushing surface, i.e. you push very “hardly.” The distance threshold can not be too big because it will confuse selection for nearby graspable object. It also can not be too big because we want to form a push gesture at the moment a pushable object is selected. Another measure to resolve this may be to disallow hand-releasing from push if it is beyond an infinite “pushing plane.” However, this is an adhoc solution that will prevent users from being able to switch to pushing any other objects that are beyond this plane.