An extremely low-light real-world image from SID Dataset [1] with a ×300 amplification ratio. The image processed by our proposed method exhibits clear improvement in structural content, showcasing finer details compared to previous works.
Abstract
Enhancing a low-light noisy RAW image into a well-exposed and clean sRGB image is a significant challenge for modern digital cameras. Prior approaches have difficulties in recovering fine-grained details and true colors of the scene under extremely low-light environments due to near-to-zero SNR. Meanwhile, diffusion models have shown significant progress towards general domain image generation. In this paper, we propose to leverage the pre-trained latent diffusion model to perform the neural ISP for enhancing extremely low-light images. Specifically, to tailor the pre-trained latent diffusion model to operate on the RAW domain, we train a set of lightweight taming modules to inject the RAW information into the diffusion denoising process via modulating the intermediate features of UNet. We further observe different roles of UNet denoising and decoder reconstruction in the latent diffusion model, which inspires us to decompose the low-light image enhancement task into latent-space low-frequency content generation and decoding-phase high-frequency detail maintenance. Through extensive experiments on representative datasets, we demonstrate our simple design not only achieves state-of-the-art performance in quantitative evaluations but also shows significant superiority in visual comparisons over strong baselines, which highlight the effectiveness of powerful generative priors for neural ISP under extremely low-light environments.
Method
The overview of our proposed LLIE method, LDM-ISP. The Bayer RAW image is processed by similar operations mentioned in [1]. A series of 2D discrete wavelet transforms (DWT) is applied to the processed image for capturing the low-frequency (LL) and high-frequency subbands (LH, HL, HH). The low-frequency subband (LL) serves to modulate the feature at each layer in the LDM. Specifically, each feature has its corresponding taming module, whose key part is an SFT layer, to map the sub-band into a pair of scale γ and shift β parameters. Similar to the low-frequency taming, the features of the decoder 𝒟 are modulated by another set of taming modules, where the LL sub-band is mapped to the scale γ and the concatenation of LH, HL, HH is mapped to the shift β. All parameters from the pre-trained Stable Diffusion are frozen and only taming modules are trainable.
Visualized Comparison
Reference
[1] Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. Learning to see in the dark. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018.
[2] Xin Jin, Jia-Wen Xiao, Ling-Hao Han, Chunle Guo, Ruixun Zhang, Xialei Liu, Chongyi Li. Lighting Every Darkness in Two Pairs: A Calibration-Free Pipeline for RAW Denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023.
[3] Xingbo Dong, Wanyan Xu, Zhihui Miao, Lan Ma, Chao Zhang, Jiewen Yang, Zhe Jin, Andrew Beng Jin Teoh, Jiajun Shen. Abandoning the Bayer-Filter to See in the Dark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[4] Xin Jin, Linghao Han, Zhen Li, Zhi Chai, Chunle Guo, Chongyi Li. DNF: Decouple and Feedback Network for Seeing in the Dark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
BibTex
@misc{wen2024ldmisp,
title={LDM-ISP: Enhancing Neural ISP for Low Light with Latent Diffusion Models},
author={Qiang Wen and Yazhou Xing and Zhefan Rao and Qifeng Chen},
year={2024},
eprint={2312.01027},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
}
Contact
Feel free to contact us at csqiangwen[at]gmail[dot]com.
