Crowdsourcing to solve problems in CryoET

To study and understand disease scientists need to watch how proteins and cellular structures interact. But seeing structures as tiny as proteins has been a long-standing challenge for cell biologists.

Breakthroughs in an imaging technique known as cryo-electron tomography, or cryoET, over the last decade have given scientists a glimpse of the cell’s complex inner workings. In cryoET, cells are frozen, halting proteins in their tracks. Images of these frozen samples are then captured from a variety of different perspectives, resulting in detailed three-dimensional images called tomograms.

Tomograms let us see the tiniest details inside of cells, but the method comes with plenty of challenges. While tomograms are information-dense, they can also be low-contrast, crowded and unclear. Scientists must pick through images and label, or annotate, every single particle — from tiny proteins to bulky organelles — by hand, which can take months. This bottleneck makes cryoET almost prohibitively difficult for many researchers.

The CZ Imaging Institute wanted to get fresh perspectives on how to overcome this bottleneck. In November of 2024, they reached beyond the cryoET community to Kaggle, an online platform where AI and machine learning experts compete to solve challenges in a variety of disciplines. The Imaging Institute issued their own challenge for Kaggle users: Come up with an algorithm that can annotate a sample tomogram quickly and accurately.

The Kaggle challenge was a competition, but one rooted in collaboration. Over 1100 Kaggle members, from over 70 countries, joined and formed 932 teams to compete for the top prize. The teams were highly engaged with the challenge and with one another, discussing their approaches and sharing code on conversation forums.

Over the course of the challenge, the competition saw nearly 28,000 submissions, with the most prolific team submitting more than 400 proposed solutions. That dedication returned astounding results: almost three-quarters of the submitted solutions performed better than the Imaging Institute’s own benchmark solution.

Building a Challenge – From the Ground Up

One reason annotating tomograms is so challenging is the sheer complexity of the images. Cells are crowded. They’re full of flexible membranes, “off-target” proteins that distract from the ones scientists are actually looking for, and structures that are hard to tell apart. This makes it nearly impossible to correctly identify every single particle in a tomogram.

Typically, computer-generated simulated samples are used to train algorithms to annotate tomograms, but these samples lack the complexity of real cells. Algorithms trained on this kind of data are overly simplistic and lack the precision required to annotate a real tomogram.

The Imaging Institute took a different approach. They created their own sample by mixing known purified proteins with cell lysate, a messy mixture that contains organelle membranes, off-target proteins and other complicating components. While not as complex as a cell, this sample allowed for rigorous annotation on a realistic timeline. “This is a real sample that is a middle ground between crowded cells and purified proteins in a test tube,” said Reza Paraan, a scientist at the Imaging Institute who helped develop the challenge.

Once the Imaging Institute team acquired tomograms for the competition, they had to annotate each image, generating “ground truth” labels that contestants’ algorithms could be judged against. Creating and validating these necessary ground truth labels was a daunting task that occupied the entire CZII team for months.

Along the way, the team developed their own in-house tools just to help them manage the labeling process. Some provided preliminary labels for known particles, and others helped ease the logistical burden of the process. All the programs they developed were made available to challenge contestants, and are available to scientists worldwide.

“The fact that finding ground truth took so much effort proves just how important it is to make labeling more efficient,” says Kyle Harrington, a research scientist at CZI who helped design the challenge.

The data, which is publicly available on CZI’s cryoET data portal, will provide an ongoing benchmark for assessing the quality of simulated samples or cell reconstructions in the future. And with an array of unlabeled particles in the data set, there’s still plenty left to explore. In a few weeks, the team’s Nature Methods manuscript — a detailed account of how they created the challenge sample — will be published. The preprint is available now.

Remarkable Results

Once the competition launched, the competing teams poured massive amounts of time into the challenge. They were able to submit in-progress solutions throughout the challenge to see how they scored, so they were constantly refining and improving their algorithms. “It can be like a part-time job,” said David List, a software engineer who participated in the challenge, at a workshop after the competition. His team’s solution won 11th place.

The winning solutions give valuable insight into what works in an annotation algorithm. Many of the top-scoring teams employed similar strategies, using backbones built to divide three-dimensional images or compensate for limited training data. Those lessons will continue to inform algorithm development in the future.

With a set of successful solutions in hand, the team at the Imaging Institute has worked to make the algorithms more versatile. While the winning solutions were optimized for success in the Kaggle challenge, they still need to be adapted and integrated into supporting tools and pipelines — such as the cryoET data portal — to enable biologists to easily use them with their own datasets. The updated solutions are publicly available on GitHub, and will soon be made available on CZI’s Virtual Cell Platform, a centralized resource for AI models and datasets. CZ scientists are also working to refine their benchmark model, now called Octopi, by trying out strategies from the winning algorithms and incorporating those that improve the model. Octopi is also available on GitHub and will soon be added to the Virtual Cell Platform.

After the competition ended, CZI hosted a workshop for Kaggle and machine learning experts as well as cryoET experts. The takeaway: The challenge was a stellar success, and more Kaggle challenges in the future could continue to push the boundaries of machine learning in cryoET.

All the submitted solutions are viewable on CZI’s cryoET data portal, and will be published on the Virtual Cell Platform. By making these tools and data available, the Imaging Institute hopes that experts in both cryoET and machine learning will be inspired to continue working on the problem of tomogram annotation.

“The experiment of crowdsourcing science has been more than worthwhile,” says Bridget Carragher, founding technical director of the Imaging Institute. “We look forward to seeing where this strategy can take us next.”