Photonic convolutional neural network with pre-trained in situ training
Convolutional neural networks (CNNs) have transformed image processing, but the energy consumption and inference latency of electronic based implementations remain fundamental bottlenecks. These limitations have motivated the search for alternative hardware architectures beyond Complementary metal-oxide-semiconductor (CMOS) chips. Optical systems can perform linear matrix operations at the speed of light with extremely low energy dissipation, making them attractive for CNN acceleration. However, building a fully coherent photonic CNN that performs both linear and nonlinear operations and training it efficiently remains an open challenge. Here we present a fully photonic convolutional neural network (PCNN) that executes image classification in the optical domain, including convolution, max-pooling, nonlinear activation, and fully connected layers. The network achieves 94.49 percent accuracy on the MNIST dataset distributed across Mach Zehnder Interferometer (MZI) meshes, weighted Multimode Interferometer (MMI) trees, and a microring resonator based nonlinearity. A mathematically exact differentiable digital twin, enables backpropagation for ex situ pre training, reaches 97.45 percent digital accuracy. Trained phases are transferred one-to-one to the photonic hardware and refined via a gradient free algorithm that estimates the full gradient with only two forward passes. The architecture exhibits inherent robustness to non idealities, under the compound effect of propagation loss, MZI insertion loss, fabrication disorder, and thermal crosstalk. A bottom-up power analysis yields 10.83 W static chip consumption and 843 ns inference latency, translating to 220 to 330 times greater energy efficiency than state of the art electronic GPUs for single-image inference.