Uncertainties and Differentiable Imaging
December 25, 2024 | Ni Chen

---

• The Laser Allergy
• Years of Wandering
• Seeds of Understanding
• The Nature of Uncertainty
• Code Is Not Computational Imaging
• From Holography to Imaging
• Closing the Loop
• Looking Back
• References

This post is also published in Nature Research Communities. You can access it via the link.

The Laser Allergy

After my BS in computer science, I switched to optical engineering for graduate school. My body did not seem to like the new place. Every time I walked into the lab, I felt sick, with headaches and dizziness. After months of this, I came to a conclusion. I must be allergic to lasers.

The truth was simpler but just as hard. I did not have the basic optics knowledge that my classmates took for granted. My advisor, Prof. Park, saw that I was struggling and tried to help by giving me more computational work and less lab work. But in optical engineering you cannot avoid experiments. Scared and uncomfortable as I was, I had to face them.

The problems kept piling up. SNU’s safety rules did not allow anyone to do experiments alone. The lab was far from our office. As the only woman on the team, I found it hard to gather the three people usually needed to watch over my work. One weekend stands out among many hard moments. I sneaked into the lab alone, hoping to finish some measurements. Instead, I put the wrong screw into the optical table, right on the optical axis. Metric and inch screws had been mixed up, and no matter what I tried, I could not get it out. I spent hours on it and got nowhere. Behind the lab’s black curtains, I broke down and cried alone in the dark room, without a single piece of data.

That was only one of many hard days in those years. The irony was clear to me. I had chosen computational imaging as my research topic, a field that barely had a name then. It needed strong computing skills and strong experimental skills, and I had neither. Crying in that dark lab, I felt that my worst fear had come true. I had chosen the wrong path.

Years of Wandering

My struggles went far beyond the optical bench. For years I felt stuck between two worlds, and neither fully accepted me. To the optics people, I was the computer scientist who could not align a system. To the computing people, I was the one who cared too much about physics they would rather leave out.

Finding the right algorithms for inverse imaging was its own long search. Traditional optimization methods felt cut off from the physics of light, and machine learning went further, treating imaging as just another dataset. I spent many hours implementing algorithms that worked well in simulation and then failed badly on real data.

Each failure taught me something. The gap between our computational models and the physical world was not only a technical problem. It was a philosophical problem. We were trying to force a messy, uncertain physical world into neat computational boxes, and the world would not fit.

Seeds of Understanding

The change started quietly. When I first worked on differentiable holography¹ in late 2020, I understood uncertainty much less than I do now. At that time I saw differentiable holography as a clever technical trick, a way to optimize holographic reconstruction with differentiable programming. I was inspired by Yann LeCun’s 2018 post, “Deep Learning est mort. Vive Differentiable Programming!” He saw differentiable programming as one framework for many problems. That idea stayed with me, because it suggested a way to connect the computational and physical worlds I had been struggling to join.

Yann LeCun’s post in 2018: Deep Learning est mort. Vive Differentiable Programming 2

Differentiable programming is where the idea of differentiable imaging³ started. If error feedback can flow through a neural network, it can also flow through a whole computational imaging system, from the optics and the propagation of light all the way to the reconstruction.

This view goes further. A whole computational imaging system can be seen as a neural network. Each part of the imaging chain, from the lens to the code, acts like a “layer” that transforms information, and error feedback tunes the whole system. This view drove my early work on differentiable holography. It also showed me how closely computational imaging follows the logic of neural networks, although back then I did not see how far this would go.

Only between 2020 and 2022, as I worked more on making holographic systems differentiable, did I start to see the role of uncertainty. Each time I tried to build an end-to-end differentiable system, I ran into the same uncertainties that had troubled my experiments years before. But now I no longer saw them as obstacles. I started to see them as a basic part of reality.

The Nature of Uncertainty

The idea that changed my view was simple. Uncertainty is the essence of the universe. We find it everywhere, from quantum mechanics to cosmology. Uncertainty is not a flaw in our measurements or models. It is a basic property of reality.

This changed how I think about computational imaging. Traditional methods treat uncertainty as noise to be removed. But then we are fighting the nature of the universe. Instead, we could design systems that expect uncertainty and even make use of it.

When an editor from Advanced Physics Research invited me to write for their new journal, I saw a chance to put this view into words. The result was a perspective paper that turned my years of failure and frustration into the basis of a new approach to computational imaging.

Code Is Not Computational Imaging

As I learned more, I noticed a worrying trend in our field. Many researchers fall into what I call the “reductionist fallacy”, the belief that computational imaging is the same as its code

\[\text{CODE} \equiv \text{COMPUTATIONAL IMAGING}\]

This belief causes real problems, not just academic arguments.

It explains a problem that comes up again and again. A researcher carefully develops an imaging algorithm and shows that it works in their own lab. Then the code is shared, and others who run it on different setups often get much worse results, or nothing at all. People may then wrongly doubt the original work or the quality of the code.

The cause is that computational imaging code depends on the exact physical parameters of the system it was written for. A slightly misaligned component or a different camera sensor can move the system far from its original conditions. An algorithm tuned for one set of physical assumptions may not be able to handle these changes. So an algorithm that works well in one setup may work poorly, or not at all, in another. The algorithm is not wrong. The physical system is simply different from the one it assumed.

Microscopists learned this lesson centuries ago. When you change the sample, you refocus the microscope. The physical world always needs adjustment, yet we often expect fixed code to work everywhere without any. It cannot. Real computational imaging needs both theory and practice. Writing good algorithms is not enough. One must also understand the real optics and sensors, and how their errors spread through the whole system.

There is an unexpected upside. Because the system and the code cannot be separated, computational imaging protects intellectual property in a natural way. Ordinary software can be copied and run anywhere. A computational imaging method stays tied to where it came from. Without the optical design and the calibration it was made for, even the best algorithm is of little use. In this sense, the link protects years of hard-won knowledge. One cannot just take the code and expect it to work.

This limitation could also be turned around. Imagine that the researchers who once said a method could not be reproduced had imaging systems that adapt to their own lab setups. The code would no longer be a fixed set of instructions. It would measure its own uncertainty and tune itself to each physical setting. The weakness of computational imaging, its dependence on physical details, would become its strength. Users would no longer need to master both optics and computation, and an algorithm that fails in a rigid setup could learn to work in almost any reasonable one. This would open advanced imaging to many more people, and let researchers focus on their science instead of the details of one system.

My own years of struggle taught me this lesson the hard way. Misaligned parts ruined whole days of work, and algorithms that worked in simulation failed on real data. Again and again, the physical world did not fit into our clean computational models. The uncertainties I kept meeting were not flaws in the system. They were part of reality. I now believe that differentiable imaging can turn this hard-won lesson into reliable systems. Others would then not have to go through the same painful learning, and computational imaging could serve everyone who needs it.

From Holography to Imaging

As I kept working on differentiable holography, the way forward became clearer and I began to see its wider meaning. Differentiable programming was not only about optimizing neural networks. It gave us one language to describe both physical and computational processes. This idea grew into the wider vision of differentiable imaging.

Since 2020, this work has taken me across three countries / four regions, where I have been lucky to meet wonderful scientists from many communities. Each meeting gave me new ideas. These exchanges across cultures and fields taught me a great deal, and showed me how the same basic ideas look different from different angles.

2024 Nobel Prize in Physics

When the 2024 Nobel Prize in Physics went to Geoffrey Hinton for his work, I had a happy moment of recognition. Watching the news, I thought his face looked familiar. Then I realized that I had been using his photo in my slides since 2023, on the slide that explains deep learning as a special case of differentiable programming. It felt like a kind of confirmation. The work I had been citing to explain our basic ideas had just received the highest award in science.

One of the slides from my iCANX Youth Talk in April 2023 4

This moment brought things full circle. Differentiable imaging is a natural next step for these ideas. It is more than using gradients to tune parameters. It builds systems where information flows freely between the physical and the computational sides. The Nobel Prize confirmed what many of us already believed, that combining computation and physics is not just a technical trick but a new way to understand and work with the world.

Closing the Loop

Differentiable imaging is more than another optimization technique. It closes the loop between the physical world and the computational model, so that the system can learn from its measurements and their uncertainty. Noise and variation become information the system can use ⁵⁶⁷¹.

Imaging then changes from one-way capture into a two-way exchange between the system and its environment. A differentiable imaging system can use computational feedback to improve its optics and its algorithms. It can also estimate its own uncertainty, so that it knows which details it cannot resolve.

I want to stress that research on differentiable imaging is still at an early stage. I believe we are at the start of a big change for the whole field, and much is still unexplored. Even so, I am convinced that differentiable imaging can help with many basic problems that computational imaging has had for decades, such as the strict separation of optics and computation and the difficulty of adapting to changing conditions.

What I like most about this framework is that it fits how reality works. Once we accept that uncertainty is basic, we can stop fighting the universe and start working with it. The same principles hold for a microscope, a camera or a medical scanner. Embrace uncertainty, make things differentiable, and let the system learn.

Looking forward, our recent papers look at how this approach⁸, together with ideas such as digital twins, could lead to imaging systems that run on their own. These systems would not just take images. They would know where their limits are and adjust themselves to each new situation. We have only scratched the surface, but the early results point to imaging systems that are as adaptive and intelligent as the researchers who use them.

Looking Back

As I write this, I think back to that weekend with the stuck screw, and to the years of feeling caught between two communities. Each failure taught me something about uncertainty.

My “laser allergy” was real, just not in the way I thought. It was an allergy to a rigid, deterministic view of imaging systems that pretends uncertainty does not exist. The road from those early struggles to differentiable imaging has been a process of learning to accept what I once tried to avoid.

Making systems differentiable and adaptive is more than a technical fix. Uncertainty is part of the real world, and our systems have to learn to live with it. Looking back, much of what I understand now came from the hardest of those years.

The tears in that dark lab have turned into a vision of systems that expect the unexpected. They do not try to remove all uncertainty. They accept it and learn from it.

References

---