Parallel computing

Speeding up Recon-all computation

Software installation

Homebrew & GNU Parallel

Install Homebrew with the following command and add it to your PATH:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

test -d ~/.linuxbrew && eval "$(~/.linuxbrew/bin/brew shellenv)"
test -d /home/linuxbrew/.linuxbrew && eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"
test -r ~/.bash_profile && echo "eval \"\$($(brew --prefix)/bin/brew shellenv)\"" >> ~/.bash_profile
echo "eval \"\$($(brew --prefix)/bin/brew shellenv)\"" >> ~/.profile

Then install gcc and parallel:

brew install gcc
brew install parallel

Using the parallel command

Parallel is run by piping the ls command into the parallel command:

NPROC=$(nproc)

ls *.nii | parallel --jobs $NPROC recon-all -s {.}_recon -i {} -all

--jobs NPROC indicates that NPROC cores will be used to analyze the data and that each instance of recon-all will be assigned to a different core.

The full bash script consists in copying the T1.nii.gz images in the new directory (/FS), unzip them, run and store the output of recon-all in this directory. Then, we move the newly created recon directory into freesurfer’s $SUBJECTS_DIR.

#!/bin/bash

# Assuming you are in the raw_data directory which lists all the subjects folders
mkdir FS

# grab the list of subject name
ls . | grep ^sub- > subjList.txt

# Path to the anatomical T1 image
for sub in `cat subjList.txt`; do
    cp ./${sub}/anat/${sub}_T1w.nii.gz ./FS
done

cd FS

# Unzip
gunzip *.gz

# Story output in the current directory
# This avoids permission error
SUBJECTS_DIR=`pwd`

ls *.nii | parallel --jobs $NPROC recon-all -s {.}_recon -i {} -all
# Remove the copied T1.nii.gz files
rm *.nii

# Move into freesurfer's subjects directory. Substitute <password> for your UNIX password
echo <password> | sudo -S mv *recon $SUBJECTS_DIR
# Grant permission to write the file
echo <password> | sudo -S chmod -R a+w $FREESURFER_HOME

You can then carry on with mapping the glasser annotation file:

################################################################
# For the command "mri_aparc2aseg", there can be an error which is due to the way multithreading is handled. Just rerun the command manually for the subjects which did not have an output hcpmmp1.mgz
# You can find these subjects by typing "find . -name *.mgz" in a terminal in the directory that contains all your subjects folders
################################################################

# Map the annotation files of the HCP MMP 1.0 atlas from fsaverage to you rsubject for both hemispheres:

ls . | grep ^sub- > subjList.txt

for sub in `cat subjList.txt`; do
  # Left hemisphere
  mri_surf2surf --srcsubject fsaverage --trgsubject ${sub}_recon --hemi lh --sval-annot $SUBJECTS_DIR/fsaverage/label/lh.hcpmmp1.annot --tval $SUBJECTS_DIR/${sub}_T1w_recon/label/lh.hcpmmp1.annot

  # Right hemisphere
  mri_surf2surf --srcsubject fsaverage --trgsubject ${sub}_recon --hemi rh --sval-annot $SUBJECTS_DIR/fsaverage/label/rh.hcpmmp1.annot --tval $SUBJECTS_DIR/${sub}_T1w_recon/label/rh.hcpmmp1.annot
  
  cd ./{sub}/dwi
  mri_aparc2aseg --old-ribbon --s ${sub}_recon --annot hcpmmp1 --o ${sub}/dwi/hcpmmp1.mgz --nthreads $NPROC/2
  cd ../..
  
done