Dynamic HUDs¶
With the Beam Eye Tracker API you can get, out-of-the-box, an estimate in real-time of whether the user is looking at
static HUD elements in the screen-space, such as a minimap, speedometer, etc. This opens the door to enhance
gaming experiences. Examples of use cases are:
Immersive HUDs, which automatically fade out when the user is not looking at them, but immediately reappear when the user glances at them.
Reactive HUDs, which are then zoomed-in, popped up, focused, make a sound, etc. based on whether the user is looking at them.
An example of an Immersive HUD. The minimap is only visible when the user is looking at it, letting the user focus on the game scene, boosting the overall immersive experience.¶
This page explains how to implement such functionalities leveraging convenient API functionality.
The integration process consists of these steps:
Optionally, Add in-game settings
We will now describe each of these steps in detail.
Keep the viewport geometry updated¶
Make sure that the viewport geometry is correctly set at API initialization and updated during gameplay.
This is explained in the Viewport section. Once this is achieved, the gaze coordinates
mapped to the viewport coordinate system will be readily available as part of the UserState
data structure, as explained in the Real-time tracking section.
Estimate if the user is looking at a HUD element¶
The Beam Eye Tracker API provides ready-to-use signals that indicate
whether the user is looking at pre-defined HUD regions, but you can also
implement this manually. Both approaches are here explained.
Ready to use “looking at HUD” signals¶
The ready to use signals are based on pre-defined HUD regions as seen in the image below
To access the respective signals you have to first retrieve the GameImmersiveHUDState
data structure from the stream of TrackingStateSet objects (see Data Access Methods).
Then, the GameImmersiveHUDState
contains a float value per point/region
(for example looking_at_viewport_top_left )
that should be interpreted as the likelihood (value between 0 and 1) of the user
looking at that region. Typically you can use a threshold of 0.5 to decide
whether the user is looking at a region or not as a boolean value.
Note that more than one point/region can be looked by the user at the same time, based on the current implementation.
Note
The current mechanism that the Beam Eye Tracker uses to determine if the user is looking at a region is based on the
normalized distance between the gaze coordinates and the point/region, whereas each point is at distance of
0.1 from the viewport border (i.e. 10% of the width or height of the viewport). If the distance
is below 0.3 then we say that the user is looking at that region. Note that this is rather large margin.
However, this algorithm is subject to change in future releases that could account for improvements in the eye tracking technology and is a further reason why we recommend to use the provided signals instead of a manual implementation.
A priori, our implementation philosophy is to prefer false positives over false negatives, as it is more annoying to have a HUD element not react when the user is looking directly at it than to have a HUD element react when the user is looking nearby but not directly at it, specially if the interaction is for the HUD element to disappear.
Manual implementation of “looking at HUD” signals¶
The manual implementation is based on the UserState
data structure, which contains the gaze coordinates in the viewport coordinate
system UserState::viewport_gaze.
Then, in your game engine, you can define screen-space objects in the position of HUD elements that you want to detect and add a simple collision detection algorithm to return a boolean value indicating if the gaze coordinates are within the bounds of said object.
Important
It is critical to consider that webcam eye tracking can be innacurate for some setups, which gets even worse in complex scenarios such as gaming in multiple monitors. Thus, when creating the collision objects please either make them much larger than the HUD elements or use generous margins to account for such innacuracies.
We argue that having false positives is preferrable to having false negatives, in use cases such as immersive HUDs (which disappear when the user is not looking at them), as it could be frustrating to the user if the HUD element is not reappearing despite the user looking at it.
Alternatively, you can simply rely on the Ready to use “looking at HUD” signals whose behavior is intended to evolve and improve as the Beam Eye Tracker itself evolves.
Make the HUD element(s) react¶
Now that you have a signal indicating if the user is looking at a HUD element or not, you can implement the logic to make the HUD element react. Clearly, this is specific to the use case you are implementing.
Assuming your current goal is to implement an Immersive HUD, in which HUD elements are only rendered when the user is looking at them, you can implement an algorithm in which:
If the user is looking at the HUD element, you make it fully visible quickly by setting the opacity to 100% (e.g. over 50ms);
If the user is not looking at the HUD element, you slowly fade it out by slowly decreasing the opacity to a minimum value such as 30% (e.g. over 1 second).
An example implementation is detailed in the Game Engine Integration sample.
Add in-game settings¶
In is not necessary to expose any in-game settings to the user. However, if you want to, you can expose the following settings
Device selection¶
Same Device selection as for the In-game camera control feature.
ON/OFF Toggle¶
You can let the user activate or deactivate the feature.
Sensitivity/Range Sliders¶
To account for complex setups, including multiple monitors or a variety of webcams, you can customize the margins used to determine if the user is looking at a HUD element or not if you used a Manual implementation of “looking at HUD” signals impleemntation.
Non-static HUD elements¶
This page focused on eye-tracking interactions with static HUD elements in the screen-space, such as a minimap, speedometer, etc. The same principles apply to dynamic objects (which change position or size during gameplay) following the Manual implementation of “looking at HUD” signals mechanics, but you need to carefully take into account the recommendations explained in that section including those related to innacuracies of webcam eye tracking.