Eye-tracking is set to become an important VR hardware feature over the next few years, offering many benefits including greater immersion, new analytical possibilities, and better performance. With the recent release of the Vive Pro Eye, sophisticated eye tracking is now available built-in to a commercial VR headset for the first time. In this blog post I will show you the steps required to set up eye tracking with the Vive Pro Eye using Unity.
Next go to the download section of the HTC developer website https://hub.vive.com/en-US/download (you may need to sign up to gain access). From here download and install VIVE_SRanipalInstaller_188.8.131.52.msi. After this, download SRanipal_SDK_184.108.40.206_Eye.zip and extract the contents. Within this folder you will find a Unity package file. Double click it and Unity will prompt you to import into your open project.
After importing the package open the EyeSample scene contained within the ViveSR. This scene will allow you see you the eye tracking works in a simple environment. If the tracking doesn’t seem accurate you can launch the calibration settings from the Unity game window when play mode is active. For additional information on how to use Vive eye tracking for your project, checkout the PDF in the SRanipal_SDK_220.127.116.11_Eye folder.
When using a headset that features eye tracking, we can implement foveated rendering to increase rendering performance. Vive provide their own implementation of foveated which is available on the Unity Asset Store (https://assetstore.unity.com/packages/tools/particles-effects/vive-foveated-rendering-145635). After importing the package, go to the Unity Player settings and add “USE_SRANIPAL” in the Scripting Define Symbols field. Then simply add the ViveFoveatedRendering.cs script on to the VR camera in your scene. To ensure that foveated rendering is working correctly, add ViveFoveatedVisualizer.cs to your camera. You should see the foveated targets following your eyes as you move them.
A full understanding of human anatomy is required of students. They required to learn a large number of terms which students typically struggle with using a traditional education model. Therefore the aim of this App was to create a VR immersive learning opportunity to help students to engage with anatomy material.
This guide describes the process of setting up VRTK to work with Unity using Steam VR.
Compatible headsets for this guide-
HTC Vive, Vive Pro
Windows Mixed Reality Headsets
This guide is also available as a YouTube video:
VRTK is a VR Toolkit for rapidly building VR solutions in Unity. As of November 2018, the most recent of VRTK found on the Unity Asset Store is no longer compatible with the latest versions Unity, Steam VR for Unity (Steam VR 2.0) or Windows Mixed Reality headsets.
To get around these limitations we will be using the latest version of VRTK, and older versions of Steam VR and Unity.
Step 1– Install Unity
VRTK works most reliably with the Unity 2017.4 LTS releases. The easiest way to install Unity is by downloading the Unity Hub, available here https://store.unity.com/download?ref=personal
From within the Install section of Unity Hub, download the most recent version of 2017.4 LTS, which at time of writing is 2017.4.14. If there is no 2017.4 version available, download directly from the Unity website here: https://unity3d.com/unity/qa/lts-releases After Unity has installed, create a new project.
Step 2 – Install VRTK
Download VRTK as a zip file by using the green download button on the VRTK GitHub page, https://github.com/thestonefox/VRTK . From within Unity, right click the ‘Assets’ folder and select, ‘Show in explorer’. Replace the ‘Assets’ and ‘Project Settings’ folders with the versions from the VRTK zip folder. Clicking back on Unity will now import VRTK into your project.
Step 3 – Install Steam VR
Download the Steam VR 1.2.3 Unity package by visiting this link https://github.com/ValveSoftware/steamvr_unity_plugin/releases/download/1.2.3/SteamVR.Plugin.unitypackage In Unity, import this package by clicking on the Assets menu > Import Package > Custom package and selecting the Steam VR Unity package.
Step 4 – Configure VR SDKs
Configure the necessary VR SDKs within Unity by clicking on the Edit Menu > Project Settings > Player. From here click on XR Settings and make sure ‘Virtual Reality Supported’ is ticked. Now add Open VR to the list of VR SDKs if it is not already there. If you are going to be using an Oculus headset, add the Oculus SDK here too.
Step 5 – Testing
Unity should now be setup correctly for use with VRTK. To test, make sure your VR headset is connected and open one of the example scenes found in the VRTK/Examples folder. Click the Unity play button and you should be able to interact with the scene.
For photos where photographer needs to be removed two photos must be taken without moving the camera and with the photographer in two different positions.
Got to file->open as layers
Select the two photos (you can select multiple photos by holding down ctrl and clicking on each photo) and open them
In the window on the right side of the screen you will see both the layers, right click on the top layer select “add layer mask” and okay the settings
In the toolbox on the left side window select the pencil or the brush tool, and set the colour to black.
Now wherever you paint on the picture will show the layer underneath, so paint over the photographer to remove them from the scene
Once finished, navigate to File->export
Set the file type to jpeg and export the photo.
The unity project requires equirectangular photos but the Samsung 360 outputs a double fisheye photo, to convert them we use the free tool gear360pano available from GitHub. To use this, you will have to install Hugin and if you intend to stitch video you will have to install ffmpeg and multiblend.
Download the gear360pao zipped file and unzip it in an accessible place.
Place the photos which need to be converted in the file
Open command prompt and navigate to the gear360pano folder, the easiest way to do this is to copy the file path of the folder and then in command prompt to type “cd” then press ctrl-v to paste the file path and press enter.
For individual images type “gear360pano.cmd nameofimage.jpg”
To convert all images in the folder type “gear360pano.cmd *.jpg” (any image already done will automatically be skipped)
The converted images are in html->data and will have _pano added to the name.
For videos the command is “gear360video.cmd nameofvideo.mp4”
And the videos are stored in the main folder with _pano added to the name.
Right click in the hierarchy window and select 3D Object->Quad
Rename this to “Map”
With map selected, go to the inspector window, reset the location and change the x rotation to 90, this should position it flat. Change the y position of the map to -10, this allows the spheres to be located on y 0.
Increase the scale of the map in the x and y directions to a few hundred (you will be readjusting the scale again later)
To add an image to the map you must create a material using the image you want
First add the image of your map into the unity project
Then right click in the project window and select Create->Material.
In the inspector window with your new material selected change the shader to
Navigate to your map image in the project window and drag this into the material’s texture box.
Then drag your material onto the map quad in the scene.
Drag an instance of the prefab PSV4 into the scene and reset its y position to 0. You can then rescale the map such that the photospheres can be placed in the correct location without overlapping.
Position the photosphere on the map in the position where you would like the 360 tour to start, then drag the camera dolly prefab from the project window and place it in the photosphere in the hierarchy window so that it becomes a child of the sphere. Reset its transform so that it is in the centre of the sphere and then in the hierarchy window drag he camera dolly back out of the sphere so it is no longer a child. You are now ready to start placing more spheres and connecting them together.
Adding photospheres to the unity project scene
Import the converted 360 equirectangular photos into the unity project.
Navigate in the scene view to where you want the sphere to be placed, then drag an instance of PSV4 into the scene.
In the hierarchy window change the name of the sphere to something unique, a suggestion is the name of the photo you intend to put there.
Reset the transform y position to 0 and adjust the x and z position so the sphere is where you want it.
In the hierarchy window open the sphere and select the photoGimbal.
In the inspector, find the photosphere creation script, drag the photo you want from the project window into the photo selection box and click update sphere.
Rotate either the photoGimbal or the entire photosphere about the y axis so that the photo is in the correct orientation.
To connect sphere via teleports
Open the Photosphere and the open teleports in the hierarchy window.
Select one of the greyed-out switch to spheres and enable it by checking the box at the top of the inspector window, then scroll down to the teleport spheres script
Drag the camera dolly from the hierarchy window into the Dolly field and drag the target sphere into the Destination field, this will enable a one-way teleport to the target sphere, and a red line will be shown from the current sphere halfway to the target sphere.
Repeat the process from the target sphere to get a two-way teleport
The teleport little grey spheres themselves will automatically orient themselves to the correct positions based on where the photospheres are placed the first time the program runs, you can run the program in the project by pressing the play button. Once the camera loads press it again to stop and the teleports will be orientated.