First Connectomics Challenge: From Imaging to Connectivity

Abstract

We organized a Challenge to unravel the connectivity of simulated neuronal networks. The provided data was solely based on fluorescence time series of spontaneous activity in a network constituted by 1000 neurons. The task of the participants was to compute the effective connectivity between neurons, with the goal to reconstruct as accurately as possible the ground truth topology of the network. The procured dataset is similar to the one measured in in vivo and in vitro recordings of calcium fluorescence imaging, and therefore the algorithms developed by the participants may largely contribute in the future to unravel major topological features of living neuronal networks from just the analysis of recorded data, and without the need of slow, painstaking experimental connectivity labeling methods. Among 143 entrants, 16 teams participated in the final round of the challenge to compete for prizes. The winners significantly outperformed the baseline method provided by the organizers. To measure influences between neurons the participants used an array of diverse methods, including transfer entropy, regression algorithms, correlation, deep learning, and network deconvolution. The development of “connectivity reconstruction” techniques is a major step in brain science, with many ramifications in the comprehension of neuronal computation, as well as the understanding of network dysfunctions in neuropathologies.

Editors: Demian Battaglia, Isabelle Guyon, Vincent Lemaire, Javier Orlandi, Bisakha Ray, Jordi Soriano

The original form of this article appears in JMLR W&CP Volume 46.

Appendices

Appendix A. Challenge Verification Results

• 1. Winners prize #1 (first place, verified) 500 USD and 1000 USD travel award $+$ Award certificate

  AAAGV

  The code from the winning team AAAGV, publicly available at https://github.com/asutera/kaggle-connectomics, was run successfully on a desktop PC, it used 7 GB of RAM and it took 30 h to run in single core mode on a 3 GHZ i7 CPU for each dataset. The code is built in Python and only uses standard dependencies. There was a issue with a specific library version but this has been resolved. Also we only need to run 1 script for the whole computation (main.py). From the valid dataset we obtained an AUC of 0.9426 and for the valid dataset and 0.9416 for the test dataset, which are the same as the ones reported in Kaggle.

• 2. Winners prize #2 (third place, verified) 250 USD and 750 USD travel award $+$ Award certificate

  Ildefons

  The code from Ildefons team is publicly available at https://github.com/ildefons/connectomics and consists of 6 separate scripts. The following are the time and memory requirements for each of the scripts. The main challenges were installing the required R package gbm and his script makeFeatures.R which needed 128 G. This R script started a MATLAB server in the SGE (Sun Grid Engine) background. We had to execute makeFeatures.R separately for normal-1, normal-2, valid, and test. His code was executed on the standard compute nodes on the cluster. The compute nodes have 2 INTEL CPUs, 16 processing cores, and 128 GB RAM. The statistics for the execution of his code can be found in Table 4.

  The code passed verification successfully. His AUC for the Kaggle submission generated by us is 0.94066. This is better than his leader board score of 0.93900. The difference between the two scores is 0.00166.

• 3. Winners prize #3 (fourth place, verified) 100 USD and 400 USD travel award $+$ Award certificate

  Lukasz Romaszko

  The code for Luaksz Romaszko team can be obtained at https://github.com/lr292358/connectomics. The details for Lukasz’s code can be found in Table 5. His solution involved predicting the outcomes eight different times and averaging. All of his code passed verification successfully. The bottlenecks were installing theano (Python module) on the GPU units and gaining access to the GPU units. We have 5 cluster nodes with GPU accelerators. Each node has 1 accelerator. Each GPU has 2496 cores. The accelerator is NVIDIA Tesla Kepler (K20).

  After merging, his score is 0.93931, which is slightly better than his score of 0.93920 on the leader board. The difference between the two is 0.00011 or, in other words, negligible.

Table 4 Memory requirements and time for Ildefons’ code

Table 5 Memory requirements and time for Lukasz’s code

Appendix B. Description of Sample Methods and Sample Code

Matlab: We provide Matlab sample code to:

• read the data

• prepare a sample submission

• visualize data

• compute the GTE Stetter et al. (2012) coefficient and a few other causal direction coefficients

• train and test a predictor based on such coefficients.

The Matlab sample code is suitable to get started. We provide a script (challengeFastBaseline) that computes a solution to the challenge (big “valid” and “test” datasets) in a few minutes, on a regular laptop computer. This uses Pearson’s correlation coefficient (Correlation benchmark, AUC = 0.87322 on the public leaderboard). The data are first discretized with a simple method. Using more elaborate discretization methods such as OOPSI may work better. The other network reconstruction methods, including GTE, are not optimized: they are slow and requires a lot of memory.

C $++$: Network-reconstruction.org provides C$++$ code which would help participants to:

• read the data

• prepare a sample submission

• compute the GTE coefficient and a few other causal direction coefficients.

Note: The fluorescence matrices for small networks have dimension 179498$\,\times \,$100 and of large networks 179500$\,\times \,$1000. Even though the GTE code is “optimized” it is still slow and requires 10–12 h of computation for the big 1000 neuron networks on a compute cluster.

Python: We are providing scripts that:

• read the data

• discretizes

• prepare a sample submission using correlation.

One participant also made Python code available.

The baseline network reconstruction method, which we implemented, is described in details in (Stetter et al. 2012). It is based on Generalized Transfer Entropy (GTE), which is an extension of Transfer Entropy first introduced by Schreiber (Schreiber 2000), a measure that quantifies predictive information flow between stationary systems evolving in time. It is given by the Kulback–Leibler divergence between two models of a given time series, conditioned on a given dynamical state of the system, which in the case of fluorescence signals corresponds to the population average. Transfer Entropy captures linear and non-linear interactions between any pair of neurons in the network and is model-free, i.e. it does not require any a priori knowledge on the type of interaction between neurons. Apart from GTE, we have also provided the implementation of cross correlation and two information gain (IG) measures based on entropy and gini for network reconstruction. Cross correlation gives best results when there are zero time delays, which reduces it to a simple correlation coefficient measure. Hence, all these methods treat the data as independent instances/points in space instead of time series data. Another module that we have added to our software kit is a supervised learner, which extracts features from a network whose ground truth values are known and builds a simple linear classifier for learning whether a connection is present between two neurons or not. Currently, the features extracted are GTE, correlation, information gain using gini and information gain using entropy.

Appendix C. Description of the Algorithms of the Winners

We provide a high level description of the method of the top ranking participants provided in their fact sheets.

Team: AAAGV

The key point is building an undirected network through partial correlations, estimated through inverse covariance matrix. As preprocessing they use a combination of low and high pass filters to filter the signals and they try to filter out bursts or peak neural activities. They stress that their main contribution is the preprocessing of the data. The calcium fluorescence signal is generally very noisy due to light scattering artifacts. In the first step, a low pass filter is used to smooth the signal and filter out high frequency noise. To only retain high frequency around spikes, the time series is transformed into its backward difference. A hard-threshold filter is next applied to eliminate small variances and negative values. In a final step, another function is applied to magnify spikes that occur in cases of low global activity.

For inference, this team assumed that the fluorescence of the neurons at each point can be modeled as random variables independently drawn from the same time-invariant joint probability distribution. They then used partial correlation to detect direct associations between neurons and filter out spurious ones. Partial correlation measures contain dependence between variables and has been used for inference in gene regulatory networks De La Fuente et al. (2004); Schäfer and Strimmer (2005).

As the partial correlation matrix is symmetric, this method was not useful in detecting directionality. Some improvement was obtained by choosing an appropriate number of principal components. The method was sensitive to the choice of filter parameters.

Team: Matthias Ossadnik

He uses multivariate logistic regression of inferred spike trains (thresholded derivative signals). Then the scores of the regressive model are fed into a modified AdaBoost Freund and Schapire (1995) classifier together with other information, such as neuronal firing rates.

Team: Ildefons Magrans

Ildefons designed a feature engineering pipeline based on information about connectivity between neurons and optimized for a particular noise level and firing rate between neurons. Instead of using a single connectivity indicator, he optimizes several indicators. As a first step, he used OOPSI, which is based on the sequential Monte-Carlo methods, in his spike inference module. Spikes below a noise-level are treated as background noise and removed. After that, time steps containing spiking activity above the synchronization rate are removed as inter-bursts recordings are more informative for topology reconstruction. As connectivity indicator, he used plain correlation which however did not provide any directionality information. In order to eliminate arbitrary path lengths caused by direct and indirect effects, he used network deconvolution Feizi et al. (2013) which takes into account the entire connectivity matrix. The classifiers he uses with the features generated from correlation are Random Forests Liaw and Wiener (2002) and Gradient Boosting Machines Ridgeway (2006).

This method also could not identify directions of connections and correlation and the singular value decomposition step of network deconvolution had an extremely high computational complexity.

Team: Lukasz8000

Convolutional Neural Networks (CNN) go beyond feed forward neural networks in their ability to identify spatial dependencies and pattern recognition. CNNs recognize smaller patterns or feature maps in each layer eventually generalizing to more complex patterns in subsequent layers. Each convolutional layer is defined by the number and shapes of filters it has alongwith its ability to learn patterns. In addition, max pooling Boureau et al. (2010) is used to reduce the size of the generated feature maps.

He uses a deep convolutional neuronal network LeCun et al. (1998) to learn features of pairs of time-series hinting at the existence of a connection. In addition he also introduces an additional input layer, the average activity of network. Lukasz used preprocessing to retain regions of higher activity conditioned on a particular threshold. These active regions help to detect interdependencies. The other important choice which influenced results was that of an activation function. He used tanh in the first convolutional layer followed by Rectified Linear Unit Nair and Hinton (2010) in the next two layers. To improve the network structure, he used max pooling. Gradient descent was combined with momentum Polyak (1964) and this helped to navigate past local extrema.