Foveated Rendering using FidelityFX™

AMD FidelityFX™ is a popular SDK which offers a wide range of techniques to improve performance and visual quality in video games, including FidelityFX Super Resolution (FSR).

Of particular interest, it already offers a VRS tier 2 algorithm/library called FidelityFX Variable Shading which is accompanied by a complete sample built on top of the Cauldron Framework, further demonstrating DirectX 12 and Vulkan support. This makes it an excellent target to enhance with foveated rendering support as it already implements some of the steps described in Foveated rendering such as shading rate image creation, and association with the rendering pipeline.

Therefore, we decided to extend the FidelityFX™ SDK with foveated rendering support, and make it available as a ready-to-use variant of the FidelityFX™ SDK. This page shows to integrate this version of the FidelityFX™ SDK to enable foveated rendering in your project, and it also highlights the modifications done to the FidelityFX™ SDK to enable this feature.

FidelityFX Foveated VRS Sample

Note

Adding foveated rendering support to your project is compatible with super resolution techniques as well as other VRS techniques, as described in Foveated rendering.

How to: add foveated rendering to your project

To integrate foveated rendering support in your project, you can follow these steps:

  1. Integrate our AMD FidelityFX™ SDK fork from github (or cherry-pick this commit).

  2. Add FidelityFX™ Variable Shading to your rendering pipeline (see documentation ).

  3. Integrate the Beam Eye Tracker, making an instance of the API.

Then, at each frame, you will need to:

  • Retrieve the latest FoveatedRenderingState from the Beam Eye Tracker API.

  • Fill in the FfxVrsDispatchDescription struct, mapping from the FoveatedRenderingState data.

Note

Please take into account the User experience considerations when integrating foveated rendering.

Warning

The prebuilt libraries are not updated with the latest changes, so you will need to build the SDK yourself.

Modifications done to the FidelityFX™ SDK

The modified version of the FidelityFX™ SDK is at https://github.com/eyeware/FidelityFX-SDK. Please refer to that page and, in particular, the Variable Shading sample and the instructions to build the samples which are already well documented therein.

What is of interest here is to explain the changes that were made to extend FidelityFX™ SDK with foveated rendering support. This was done in a two step process:

Description

Commit

Added foveated rendering support to ffx_vrs.h

651fbd34f17205f3040122a69d8fa5e01f58081b

Beam Eye Tracker SDK integration

56a203e540065eda66ed7a8cbf13417b41c40f3d

This was done to separate the changes to core library from those specific to the Beam Eye Tracker SDK integration. We will explain these work packages in more detail in the following sections.

Foveated rendering support

To add foveated rendering support we needed to extend the core library ffx_vrs.h, which is the API for the FidelityFX™ Variable Shading implementation. After our modifications, the high level process of creating the VRS context with ffxVrsContextCreate, setting the dispatch parameters through the FfxVrsDispatchDescription struct and dispatching the call to create the shading rate image (SRI) with ffxVrsContextDispatch is essentially the same.

However, the main change to the ffx_vrs.h API is that the FfxVrsDispatchDescription struct was extended with the following parameters:

  • vrsAlgorithm: allows to select between either the prior method based on motion vectors and luminance, foveation, or a combination of both using the max operator.

  • foveationCenter: the center of the foveated region, same as normalized_foveation_center but in pixel coordinates.

  • foveationRadiiSquared: the radii of the foveated regions, same as FoveationRadii, but in pixel coordinates and squared for shader performance reasons.

To achieve this, the shader (found in ffx_variable_shading.h) was extended with the following function:

FfxUInt32 VrsComputeFoveatedShadingRate(FfxUInt32x3 Gid, FfxUInt32x3 Gtid, FfxUInt32 Gidx)

which computes the shading rate for the given thread. Then, the main shader function void VrsGenerateVrsImage(FfxUInt32x3 Gid, FfxUInt32x3 Gtid, FfxUInt32 Gidx) calls VrsComputeFoveatedShadingRate and combines its result with the prior method based on motion vectors and luminance using the max operator, depending on the vrsAlgorithm parameter.

Given this modified API, the VRS sample available in the SDK was extended with a UI interface selection of the vrsAlgorithm parameter, as explained in the UI Elements section. Predefined (constant)values for the foveationCenter and foveationRadiiSquared parameters were also added.

Beam Eye Tracker SDK integration

The integration of the Beam Eye Tracker is mostly relevant to the sample’s code, as it does not relate to the core library itself. However, as the Beam Eye Tracker API requires knowledge of the Viewport geometry, the Cauldron Framework was further extended to deliver a callback OnMove() whenever the window is moved, so that we can call the update_viewport_geometry() method accordingly.

Finally, the process described in How to: add foveated rendering to your project is applied at each frame, and the FfxVrsDispatchDescription struct is filled in with the latest FoveatedRenderingState from the Beam Eye Tracker API, after mapping from the normalized coordinates to pixel coordinates and squaring the radii.