Thanks for the introduction.

Importance of Infrared

Heat imaging technology has been of interest to Australian Defence researchers since the 1950s when scientists in the Weapons Research Establishment at DSTG first started developing uncooled resistance bolometer arrays.

Despite DSTG ceasing this research in 1996, heat imaging technology is currently prolific in both Defence and commercial industries worldwide, and countless technologies including thermal seekers, signature analysers, thermal night vision goggles, and weapon sights rely on these infrared or heat imagers. It is therefore crucial that radiometric simulations are capable of accurately simulating infrared signals if they wish to test and evaluate these technologies.

Infrared Bands

The infrared part of the electromagnetic spectrum is beyond the red part of the visible spectrum. It is often broken down into smaller wavebands called short-wave, mid-wave, and long-wave, based on certain transmissive regions of the atmosphere. The infrared part of the spectrum has lower energy than the visible wavelengths. The Very Long Wave IR band is generally outside of the range that sensor materials can measure the radiation using photonic techniques.

Benefits of simulation

Simulations have grown to mainstream use because they allow users to synthetically generate data and test products virtually rather than physically which comes with many advantages. Some of the key benefits include that it is generally cheaper than real laboratory testing, one can generate more focused results without physical limitations, whole test cycles are quicker, and it allows for a full testing ecosystem to be generated from conception to sustainment.

Digital modelling and simulation has become crucial for the rapid advancement of new technology and is the backbone of digital engineering. One such radiometric simulation capable of simulating infrared is Infinite Studio.

What is Infinite Studio?

Infinite Studio is a simulation program built on Unreal Engine 5 ​as a joint development between Aurizn and the Defence Science and Technology Group. Infinite Studio is capable of simulating across two major regions of the electromagnetic spectrum – visible wavelengths and infrared wavelengths. Infinite Studio claims to provide credible physics-based synthetic imagery​ incorporating material spectral curves and camera sensor spectral dependencies, but how can one actually know that? We set out to prove that Infinite Studio is capable of accurately simulating infrared signals.

To be able to validate Infinite Studio, we need to understand how Infinite Studio works

To start with we need to think about how a real sensor works. Picture a scene in reality: when a sensor measures it, there is some physics that goes on in the “black box” that is the sensor before an image is returned.

Similarly in Infinite Studio, there is a scene with sources. Each source produces its own radiance, which is propagated to the sensor following the principles of physically-based rendering to generate an at-sensor irradiance, which is passed through the photon transfer technique to generate an image.

But what is the photon transfer technique?

The Photon Transfer Technique is a common method used to approximate the physics that converts incident radiation to camera outputs.

The photon transfer technique:

1. Takes an incident number of photons
2. Which interact with the sensor material within the exposure time of the camera shutter
3. Which creates a number of electrons with efficiency η which is called the quantum efficiency.
4. These electrons form a charge which is converted to a voltage by a capacitor
5. Which is amplified
6. And digitised
7. to form a digital grey value, with gain K which is called the system gain

So how did we validate Infinite Studio?

The goal was to recreate a real laboratory scene with an infrared radiation source in Infinite Studio. The radiation source we used is called a black body. We had a sensor under test in both reality and simulation, and we matched the scenes as closely as possible with a simulated black body and simulated sensor actor. The “Black Box” physics in reality is approximated within Infinite Studio by the photon transfer technique, but with identical black bodies and identical sensors, will Infinite Studio recreate the exact data measured in the laboratory?

There are two key parts to this test that need to be simulated in Infinite Studio as close as possible to their real counterparts – the source of infrared radiation and the sensor itself. If there are differences here, the test is invalid.

Firstly, for the source of radiation, what is a black body?

A black body is a type of radiation source that emits electromagnetic radiation based on its temperature and its surface material. In general, most objects act as black bodies with varying degrees of emissivity.

1. The graph shows wavelength on the x axis and spectral radiance on the y. The rainbow section is the visible spectrum which is the part our eyes can see, on the right of that is infrared. For a 25°C object, we see it peaks out in the LWIR region with a low peak value
2. As an object gets hotter the amount of radiance it emits increases shown by the increasing area under the curve, and it also peaks towards the shorter wavelengths, so 200°C peaks in the MWIR at about 5 microns
3. For even hotter target at 1000°C we see the peak is in the SWIR infrared region
4. And finally for the 5000°C target we see it peaks in the visible wavelength region, and has a significantly larger area under the curve. For example, the surface of the sun is about 5000°C and hence it is very bright, and our eyes can see the visible wavelength light from it. However, humans which could be approximately 25°C, emit very little radiance at a wavelength we can’t see, hence they don’t glow to our eyes. But to a LWIR sensor, such as night-vision goggles, a human does in fact “glow”.

Planck’s law can be simply understood as the principle behind why things glow red hot when they heat up. As for modelling a black body in Infinite Studio, their emitted radiance it emits is governed by Planck’s Law, which makes it very easy to simulate them if you know their temperature and their material. Therefore, Infinite Studio should be more than capable of accurately modelling these devices. So now we think about how well Infinite Studio is capable of mimicking the sensor.

We then need to make sure that the sensor actor in Infinite Studio has the same performance parameters as the real sensor

Fortunately the EMVA Standard 1288 was designed such that customers can reliably interrogate their real sensors to characterise them according to an agreed-upon list of parameters. The parameters generated by the EMVA Standard 1288 are the same parameters that are used in the photon transfer technique, and as such following the EMVA Standard 1288 allows users to directly import their sensors into Infinite Studio which is convenient.

So how did we characterise the cameras in the lab according to the Standard? We followed method 2 which requires changing the incident photons to the sensor and measuring the resultant change in digital grey values.

We changed the incident photons to the sensor under test by changing the temperature of the black body source while maintaining a constant sensor exposure time such that any time-dependent noise sources in the sensor were held constant. This meant that no noise corrections needed to be applied to the pixels for our tests to work, which was relevant given that there was no validation of the corrections supplied.

Assuming things are going well, we would expect to see data that looks something like this – linear. This linear dataset has a slope which is equal to the quantum efficiency of the sensor multiplied by the system gain of the sensor. The offset is primarily comprised of three terms – from the lens itself which acts as a weak black body, from the atmosphere which also is a weak black body as well as has spectral emissions, and a term called the cold point which is this bottomed out value that the sensor can go to.

Overall, this measurement process is simple and was used in the lab to yield real performance values of the cameras under test. However, while simple, changing the temperature of the black body added potential error to the measurement which we needed to address.

Augmented Dickey Fuller test

If the temperature of the black body in the laboratory was not stable, it would introduce errors to the measurements. We tested for the instability of the black body temperature by using the Augmented Dickey-Fuller test to confirm that our measured data from the laboratory black body was stationary i.e. that it wasn’t experiencing background trends.

For example, if a data series in red was following a trend but an instability arose, if it stays on that new trend then it would have a unit root and fail to pass the Augmented Dickey fuller Test. But if it returns to its original trend then it has no unit root and it passes. We see a failed test on the left with a passing test on the right

The fluctuations in the data should be oscillating about a constant value, not like the plot on the left where the background trend is increasing (as if the black body is still heating up) rather than on the right where the temperature is stable. So in the laboratory measurements we made sure to give the black body several hours between tests to make sure that the data would pass the ADF.

Overview of the experiment design

We have seen the method we used to measure the sensors in the laboratory, such that we can characterise sensors according to the EMVA 1288 standard, to produce sensor responsivity values and sensor offset values for each sensor tested.

These processed results, along with the relevant black body temperatures, were used as inputs for Infinite Studio. We essentially treated Infinite Studio as a black box for these tests. Therefore, Infinite Studio should create digital grey value outputs of simulated sensors with identical conditions, so that we can ask: are these the same?

SWIR and MWIR Results

Both show excellent agreement between lab data and IS data down to the sub-percentage level

We can make digital twins in Infinite Studio of photonic sensors in IR!

LWIR Results

Shows excellent agreement between lab data and IS data down to the few percent level

Offset of lab data being higher likely implies that there were sources of radiation in the lab that we didn’t account for that the Infinite Studio model does not include, such that the offset arose.

This is because of the increased difficulty of validation in the LWIR waveband

For a normal visible scene shown top left, seeing it in the NIR shown top right is perfectly easy and retains most of its contrast. However, for a nighttime scene in the visible, shown bottom left, we can’t see much other than what the lights on the boat show. However, that same scene in the LWIR shown bottom right now reveals the clouds above, which is great, but a lot of the detail in the boat is completely washed out. This is because everything emits in the LWIR due to the temperatures it cares about, and in general, the boat and its environment are at thermal equilibrium so there is no contrast.

Our measurements in the lab were fine, but we didn’t account for every source of radiation when we moved to Infinite Studio and as such an offset error has come about. This doesn’t mean Infinite Studio is not valid, it just means we need to investigate what other sources could potentially be causing this offset.

Future works for this project

We have been able to prove that the sensor model within Infinite Studio is valid and that it can faithfully recreate real results measured in the laboratory under the same conditions.

Future work includes completing the same characterisation and validation for visible sensors. Even though the visible spectrum part of Infinite Studio also uses the photon transfer technique, it is still worth measuring and validating to confirm its performance.

We are also developing a new type of sensor model that is non-photonic for infrared radiation based on bolometry.