Good morning everyone!

I would like to make a minor adjustment to my talk title: “Modelling an accurate multispectral atmosphere in a real-time physics simulation which is fit for purpose.”

This adjustment reflects the core focus of my presentation: achieving an optimal balance between simulation accuracy and computational performance. The criterion for determining whether this balance has been achieved is whether the simulation is “fit for purpose.” However, what precisely constitutes “fit for purpose” requires clarification.

I come from a physics background, and from this perspective, accuracy is paramount. Accurate simulations yield credible and reliable results. In contrast, my colleagues within the software engineering team prioritise performance, recognising its critical importance for efficiency, reliability, and user experience, including smooth visuals and immediate feedback.

These dual considerations are central to the simulation environment I will discuss today: Infinite Studio. It is a suite of software plugins for Unreal Engine 5 designed to simulate electro-optical (EO) and infrared (IR) systems across diverse environmental conditions.

Infinite Studio delivers realistic scenes with real-time performance. The appropriate equilibrium between accuracy and performance is confirmed when both criteria are satisfied, ensuring that Infinite Studio is fit for purpose.

As a scientist in the team, my discussion begins with the idealised, physically accurate representation of an atmosphere. Subsequently, I explore how this representation must be adapted to enable real-time performance while maintaining realism.

First, consider the modeling of light from a source in the absence of an atmosphere. This scenario simplifies to the product of the source radiance and the sensor response.

Introducing an atmosphere significantly complicates the problem.

Let us examine the processes involved step-by-step:

The sun irradiates the scene, shown in yellow. Some of this solar radiation is reflected by the target or causes it to emit thermal radiation toward the sensor. Consequently, the source radiance now incorporates contributions from both reflection and emission by the target, shown in red. This source radiance must propagate through the atmosphere to reach the sensor, undergoing various interactions along the way.

Firstly, the source radiance interacts with particles or molecules and is redirected out of the original path, a phenomenon known as out-scattering. Secondly, the source radiance interacts with air molecules and is absorbed by them. Both scattering and absorption reduce the incident radiance along the path to the sensor. The extent of this reduction is termed transmission, shown in blue.

The atmosphere also contributes radiance to the path. Solar irradiance can be scattered into the path either directly or after multiple scattering events. Additionally, air molecules along the path emit their own radiation. These mechanisms contribute to what is known as path radiance, shown in purple.

To model the light received at the sensor with an atmosphere present, one must account not only for the camera response and source radiance but also for atmospheric transmission and path radiance.

Radiance and transmittance are spectral quantities, meaning their values depend on the specific region of the electromagnetic spectrum under consideration. This simplified view of the spectrum includes visible light in the rainbow region and infrared light in the dark red region beyond.

When rendering a scene for human vision, only the visible portion of the spectrum is relevant. Out-scattering primarily influences transmittance, while in-scattering affects path radiance. Approximate models for these phenomena can be computed in real-time, such as the SkyAtmosphere implementation in Unreal Engine. These approximations are sufficiently accurate to reproduce visually convincing atmospheric effects.

However, modeling in the infrared region demands knowledge of transmittance and radiance throughout the entire spectrum. Infrared radiation is predominantly affected by absorption and emission rather than scattering. Modeling in this region presents challenges due to high variability in absorption coefficients and the lack of intuitive understanding of infrared appearance compared to visible light.

Modeling across both spectral domains is non-trivial yet essential. Applications span remote sensing, defense systems, surveillance, and night vision, underscoring the necessity of simulating both visible and invisible light. Products like Infinite Studio address these requirements directly.

Therefore, how do we compute the required spectral transmittance and path radiance accurately while ensuring real-time performance?

Several atmospheric radiative transfer software programs exist for this purpose. One widely recognised and validated tool is MODTRAN, developed by Spectral Sciences. It calculates all relevant scattering, absorption, and emission components discussed previously, as well as numerous others.

MODTRAN requires input parameters such as the viewing direction, atmospheric characteristics (e.g., temperature profile), solar position, and model settings (including desired resolution). It solves the radiative transfer equation by computing all the aforementioned components and outputs the necessary spectral transmittance and path radiance.

To fully characterise the atmosphere for our simulation, MODTRAN must be executed for numerous viewing directions to account for users moving the camera around.

This involves calculating radiance for varying path lengths, angles, and observer heights.

This process generates a substantial amount of spectral data, which takes considerable time to compute even with the reduced resolution Infinite Studio runs at to maintain performance.

To improve performance while preserving accuracy, this data is typically precomputed and stored in multidimensional lookup tables, which is an established practice.

If performance were not a constraint, one could utilise this precomputed data directly to render scenes with atmospheres. Such an approach would involve:

• Reading spectral data,
• Calculating spectral radiance at the observer,
• Integrating over the sensor’s spectral bandwidth.

However, executing this for every pixel in every frame would severely compromise real-time performance, making it unsuitable for practical applications. Thus, although such a method would produce realistic scenes, it fails to meet real-time performance requirements and is therefore not fit for purpose. We must therefore find ways to compress this data into forms suitable for GPU processing. One approach involves approximating spectral components with in-band values.

In-band values for path radiance are obtained by integrating the spectral path radiance over the camera’s response function.

This process is repeated across all spectral data points, reducing the dimensionality and enabling storage in 2D lookup textures usable by the GPU for real-time rendering. While this technique preserves accuracy for path radiance, certain performance considerations arise that warrant further discussion.

For transmission, the calculation is more complex since transmission depends on the source radiance.

However, when modeling the atmosphere alone, no source is present. Thus, an approximation without a source is used to derive in-band transmittance values.

These in-band values are then mapped to texture entries for real-time GPU usage.

This method achieves real-time performance but raises questions regarding accuracy.

To evaluate the impact of this approximation, we compare the final radiance obtained using the exact spectral transmittance against that derived using the approximate transmittance. We conducted this comparison for sources of varying temperatures and plotted the resulting percentage error. Results indicate that for low-temperature sources, the error is relatively high. However, for temperatures exceeding 350K (~77°C), the error drops below 10%, and for temperatures above 500K, it remains under 5%. So, is a 10% error acceptable? To assess this, let us consider a 10% error in transmittance.

Consider a waterfall captured with a standard visible camera.

And the same scene captured in infrared, showing rock surfaces reflecting significant radiance and cooler water with minimal emission. Now we apply a 10% error to objects below approximately 13°C to visualise the impact, shown in the middle image. Does the resulting image appear significantly different or unrealistic? To provide context, a 50% error shown in the right image produces a clearly distorted result, confirming that a 10% error is imperceptible in most cases. Therefore, while a 50% error compromises realism, a 10% or even 5% error remains acceptable.

In summary, modeling an accurate multispectral atmosphere in a real-time physics simulation is a complex endeavor, requiring careful handling of spectral radiance and transmittance. This presentation has only touched upon the fundamental aspects.

Despite the challenges, we successfully modeled the thermal component of path radiance accurately without sacrificing performance. However, I noted earlier that this assertion holds primarily for the thermal radiance component. The scattering component introduces additional complexity due to its dependence on azimuthal angle, resulting in increased data volume and a separate set of trade-offs between accuracy and performance. Furthermore, we computed in-band transmittance for the atmosphere, which performs efficiently in real-time and maintains sufficient accuracy for practical use.

Thank you for attending today’s session.