Google AI introduces “SegCLR”, a self-supervised machine learning technique that produces highly informative representations of cells directly from 3D electron microscope images and segmentations

If we can analyze the organization of neural circuits, this will play a crucial role in better understanding the thought process. This is where maps come in. Maps of the nervous system contain information about the identity of individual cells, such as their type, subcellular component, and neuron connectivity.

But how do you get these cards?

Nanometer-Resolution Volumetric Imaging brain tissue is a technique that provides the raw data needed to construct these maps. But deducing all the relevant information is a laborious and difficult task due to the multiple scales of brain structures (e.g., nm for a synapse versus mm for an axon). This requires hours of manual ground truth labeling by expert annotators.

However, some computer vision and machine learning methods help with these annotations, but these are not very reliable and require proofreading of the final ground truth annotations. Moreover, tasks such as identifying the cell type from a very small neuron fragment are difficult even for human experts.

To further automate this task and solve the problems mentioned above, the authors of this paper proposed a self-supervised machine learning technique, “SegCLRwhich stands for Contrastive learning guided by segmentation of representations. SegCLR takes a 3d volume of VEM as input and produces an embedding as output.

The proposed approach is scalable in three important respects:

1. The coating produced can be used for different tasks like identifying cell sub-compartments, cell types, etc.
2. The representation learned by SegCLR is very compact. It is directly usable in downstream analyzes with linear classifiers or shallow networks, eliminating the need to train a large model repeatedly to learn representations for each new task.
3. Additionally, SegCLR reduces the need for labeled ground truth data by an order of magnitude of 4.

Moreover, SegCLR allows reliable cell annotation even from very short fragments (~10-50 μm) of cortical cells. In the end, the authors also showed that SegCLR could be combined with Gaussian processes to estimate the uncertainty of predictions.

Let’s talk a bit about the history of neuropil annotation. In the past, machine learning methods used features designed by hand or derived from supervised learning. A random forest classifier trained on hand-derived features and a 2d convolutional network trained on neuropil projections or 3d convolutional trained directly on voxels.

What does SegCLR do?

SegCLR produces embeddings: which are rich biological features in a low-dimensional space. The embeddings produced also have the quality of contrastive learning, that is, vector maps of distance to biological distinction. Now, various downstream tasks can use these seamless vector representations.

SegCLR integration represents a local 3D view of EM data and focuses on an individual cell or cell fragment within the neuropil with accompanying segmentation.

A ResNet-18 based encoder is trained to produce 64-dimensional embedments while using a contrastive loss function as the cost function and a projection head, which further reduces the embedding dimensions from 64 to 16 (see Fig. 1). The encoder was trained on two publicly available EM connectomics datasets; one is from the human temporal cortex and the other from the temporal cortex of a mouse. Now the trained encoder and the produced seamless vector representations are used for the following downstream tasks:

1. Classification of cellular sub-compartments: it consists in identifying cellular sub-compartments such as axons, dendrites and somas. A linear classifier trained on the integrations of the human cortical dataset obtained an F1 score of 0.988. While on the mouse dataset, the classification reached 0.958 F1-Score mean. The classifier matches the performance of the direct supervised model requiring about 4000 times fewer labeled training examples and outperforming it when trained on full data (see Fig. 2).

2. Classification of neuron and glial cell subtypes for fragments of large and small cells: it is very similar to the classification of cell subcompartments, but the individual incorporations of SegCLR represent a local 3D view of the 4-5 micrometer side only, which is not sufficient for cell typing. To counter this author proposed a technique to aggregate integration information over larger spatial extents by collecting nearby integrations in radius R and then taking the average integration value over each feature. After various experiments, for the human dataset, the classifier obtained an F1-Score of 0.938 for R = 10 μm for six classes; for the mouse dataset, the F1-Score is 0.748 for R = 25 μm for 13 classes.
3. Unsupervised data mining: UMAP projections are used to visualize embedding samples, and separate clusters in UMAP space are easily observed for glia versus neurons and axons, dendrites and somas (see fig 3)

4. Out-of-distribution input detection via Gaussian processes: The remaining problem with all of these applications was whether the image content was outside the distribution of the training data. How can we quantify the distance between a given image and the train data? To solve this problem, the author used the spectral normalized neural Gaussian, which added prediction uncertainty to the model output and calibrated this uncertainty to reflect the distance between the test data and the training distribution. This uncertainty calibration allows OOD entries to be rejected rather than ambiguous classifications. We can set an appropriate threshold on the uncertainty for a given task.

In conclusion, SegCLR captures rich cellular functionality and can greatly simplify downstream analyzes compared to working directly with raw image and segmentation data. However, SegCLR has two major limitations besides requiring instance segmentation for the voxel. First, the 32–40 nm voxel resolution of input views hinders the capture of finer EM ultrastructures, such as vesicles, subtypes, or ciliary microtubule structures. Second, masking the input excludes context outside of the current segment, which can be useful in case of segmentation and classification.

The most powerful application of SegCLR demonstrated in the paper is to classify neuronal and glial subtypes even from small fragments, which is a difficult task even for human experts.

---

Check Paper and Reference article. All credit for this research goes to the researchers on this project. Also don’t forget to register. our Reddit page and discord channelwhere we share the latest AI research news, cool AI projects, and more.