Overview
The integration of artificial intelligence into microscopy systems significantly enhances performance, optimizing both the image acquisition and analysis phases. Development of AI-assisted super-resolution microscopy is often limited by the access to large biological datasets, as well as by the difficulties to benchmark and compare approaches on heterogeneous samples. We demonstrate the benefits of a realistic STED simulation platform, pySTED, for the development and deployment of AI-strategies for super-resolution microscopy. The simulation environment provided by pySTED allows to perform data augmentation for the training of deep neural networks, to develop online optimization strategies, and to train reinforcement learning models, that can be deployed successfully on a real microscope.
All code used in this manuscript are open source. Code for the STED simulation platform is available from pySTED. The bandit algorithms and training procedures are available from optim-sted. The gymsted environment and training routines are available from gym-sted and gym-sted-pfrl respectively.
The pySTED simulation tool was implemented in a Google Colab notebook. Which can be accessed from the following button.
Paper
The paper is available here.
If you use any of the material please cite the article:
Bilodeau, A., Michaud-Gagnon, A., Chabbert, J. et al. Development of AI-assisted microscopy frameworks through realistic simulation with pySTED. Nat Mach Intell (2024). https://doi.org/10.1038/s42256-024-00903-w
Data and models availability
This section contains the link to the models and the data that were generated for each of the presented experiments of the paper.
Simulation and reality experiments (124 MB)
U-Net datamap model (115 MB)
Bandit optimization (4.7 GB)
Contextual-Bandit optimization (4.7 GB)
Reinforcement Learning in simulation (6.9 GB)
Reinforcement Learning on a real microscope (4.6 GB)
Datasets
This section contains the link to the datasets that were used for the training of the models presented in the paper.
Dataset of datamaps used to trained the bandit and RL model (284 MB).
Acknowledgments
Sarah Pensivy and Tommy Roy from the Neuronal Cell Culture Platform of the CERVO Brain Research Center for the preparation of the dissociated hippocampal cultures. Valérie Clavet-Fournier for immunostainings of neuronal proteins. Mélissa Bilodeau for the help with the pySTED logo. Funding was provided by grants from the Natural Sciences and Engineering Research Council of Canada (NSERC) (RGPIN-06704-2019 to F.L.C. and RGPIN-2019-06706 to C.G.), Fonds de Recherche Nature et Technologie (FRQNT) Team Grant (2021-PR-284335 to F.L.C and A.D.), the Canadian Institute for Health Research (CIHR) (F.L.C.), and the Neuronex Initiative (National Science Foundation 2014862, Fond de recherche du Québec - Santé 295824) (F.L.C.). A.D. is a CIFAR Canada AI Chair and F.L.C. is a Canada Research Chair Tier II. A.B. was supported by a scholarship from NSERC. A.B. and A.M.G. were awarded an excellence scholarship from the FRQNT strategic cluster UNIQUE.
