Time-Resolved MNIST Dataset for Single-Photon Recognition

Abstract

Time-resolved single photon imaging is a promising imaging modality characterized by the unique capability of timestamping the arrivals of single photons. Single-Photon Avalanche Diodes (SPADs) are the leading technology for implementing modern time-resolved pixels, suitable for passive imaging with asynchronous readout. However, they are currently limited to small sized arrays, thus there is a lack of datasets for passive time-resolved SPAD imaging, which in turn hinders research on this peculiar imaging data.

In this paper we describe a realistic simulation process for SPAD imag- ing, which takes into account both the stochastic nature of photon ar- rivals and all the noise sources involved in the acquisition process of time-resolved SPAD arrays. We have implemented this simulator in a software prototype able to generate arbitrary-sized time-resolved SPAD arrays operating in passive mode. Starting from a reference image, our simulator generates a realistic stream of timestamped photon detections. We use our simulator to generate a time-resolved version of MNIST, which we make publicly available. Our dataset has the purpose of en- couraging novel research directions in time-resolved SPAD imaging, as well as investigating the performance of CNN classifiers in extremely low-light conditions.

Fig.1: Samples from TR-MNIST-rec of reconstructions at different lux levels with 1ms integration time.

SPAD Cameras

Recent advancements in electronic technologies, allowed the manufacturing of sensor units sensible to single photons, with extremely high readout speed. Among these novel sensors, the Single-Photon Avalanche Diodes (SPADs) are able to detect the impinge of a single photon by generating an exponential gain of the voltage produced by a single photoelectron. Time-Resolved SPADs (TR- SPADs) are SPADs supported by a fast time-to-digital converter in order to precisely timestamp the arrival of each photon, with precision in the order of picoseconds. Asynchronous SPAD pixels are readout as soon as a photon is detected, without the need to wait for an exposure time to complete. Thanks to these features, SPAD pixels are revolutionary with respect to traditional pixels, and they are attracting increasing interest from the imaging community.

Challenges

One of the major factors inhibiting the research in time-resolved data is the lack of open-access datasets. The scarcity of training sets is also due to the fact there are no high resolution asynchronous SPAD sensors in commerce. On the other hand, time-resolved SPAD arrays pose new challenges to computer vision and learning-based algorithms, requiring new research approaches. These challenges include handling asynchronous photon arrivals, which prevents the straightforward use of algorithms meant for arrays, and the data size, which grows significantly in mid-luminance scenes acquired with non-trivial sized sensors, as raw data can easily contain thousands of timestamps per pixel. Existing reconstruction algorithms can mitigate these issues by estimating the underlying photon flux from the statistics of the photon arrivals, returning a conventional digital image. However, this approach hides raw data's asynchronous and online nature, and adds a computational step that might be unnecessary for some vision tasks.

Simulator

In this work, we address the data scarcity problem by presenting a simulation procedure for generating synthetic photon detection timestamps of a time-resolved SPAD array, modeling all the relevant noise sources. Starting from a reference image, the simulator generates a stream of photon detections whose distribution is compatible with the scene depicted by the reference image. Addi- tionally, the simulator can adjust the rate of photon arrivals based on a specified scene illuminance. We foresee two impactful use of datasets generated by our SPAD array simulator: first, enabling researching machine learning models na- tively processing streams of photon arrivals, second investigating established CNN performance in arbitrarily low light conditions, on the reconstructed flux.

Fig.2: Illustration of pixel-wise SPAD model used by our simulator. An input image is transformed to a photon flux, via a reference lux level, which is then used to simulate photon detections. The pixel-wise SPAD model is applied to all pixels in the image.

Dataset

We publicly release a dataset synthetically generated with our simulator from the traditional MNIST, at different illuminance levels. The dataset comes in two versions: the raw time-resolved version (TR-MNIST), con- sisting of raw photon detections, and a collection of reconstructed images using state-of-the-art flux estimation techniques (TR-MNIST-rec). Furthermore, TR-MNIST-rec provides a benchmark for testing the robustness of image classifiers when applied to flux estimated in extremely low-light conditions. In this regard, we have trained and tested a baseline CNN classifier on the TR-MNIST-rec dataset, assessing the classification performance drop when reducing the luminance for different flux reconstruction methods.

Fig.3: Classification accuracy of LeNet on TR-SPAD-rec, with different reconstruction methods and at different lux levels. The reference accuracy is obtained from the original MNIST dataset.

The dataset is available for download HERE

References

[Suonsivu et al. 2024] Time-Resolved MNIST Dataset for Single-Photon Recognition
Aleksi Suonsivu, Lauri Salmela, Edoardo Peretti, Leevi Uosukainen, Radu Ciprian Bilcu, and Giacomo Boracchi, European Conference on Computer Vision, Synthethic Data for Computer Vision Workshop, 2024 doi:??
(Preprint), (Original)