Access to Data

Filesystem Layout

Each computer in the system has access to two system-wide network filesystems. User home directories $HOME are mounted from the user server (currently cicus03) on /home/cic/<username>. Home directories are suitable for storing the regular configuration files, as well as papers and similar files, it should not be used to store data.

The high performance filesystem (currently hosted on cicss05) is available under the /data path. All users have access to scratch storage at /data/scratch2, which is suitable for storing data during processing. Scratch does not keep any historical versions and is not backed up, so it should not be relied upon for long-term storage. In the future it is expected a age-based deletion policy will be implemented to maintain sufficient free space and encourage users to use storage properly.

Paid storage for individual lab groups is also available under the /data/ path, the exact name depends upon what the group decided during creation.

Mounting on demand

Network filesystems are not automatically mounted on boot-up on machines, but rather mounted lazily on-demand. As such, /data will appear empty before attempts are made to access a specific filesystem. GUI file managers do not work particularly well in this scenario, as there will be no folders to interact with. It is recommended to perform file management using the command line tools.

Transferring Data

Access to filesystems is enabled via the scp/sftp functionality of ssh, as well as the rsync program over ssh. Linux and OSX users can find the sftp scp and rsync commands in their terminal. Windows users can use WinSCP or FileZilla for a graphical tool to access data or MobaXterm for a proper Linux-like terminal.

Here are few example commands for data transfer, note that these commands assume that you are within the Douglas:

# Copying a directory of files to cicss05 in /data/scratch/<username>
$ rsync -avz directory <username>@cicss05:/data/scratch/<username>
# Copying another directory from cicss05 to your local system
$ rsync -avz <username>@cicss05:/data/scratch/<username>/another_dir .

Resumable file transfers

rsync is strongly recommended for all data transfers, as it supports resuming interrupted transfers

Bulk data transfer

For bulk data transfers, please connect directly to cicss05 to bypass any round-trip data would need to travel if performing transfers to workstations.

Real time data access

The sshfs project allows for filesystems to be mounted remotely via ssh. This allows you to access files without having to explicitly transfer them back-and-forth. See the site for details and for windows see here.

Accessing Human MRI Scanner DICOMS

The Siemens MAGNETOM Prisma Human MRI scanner sends collected data to cicus03 which acts a pseudo-PACS system, collecting the files and storing them at /home/cic/dicom/transfers accessible for 14 calendar days before being moved to long-term cold storage. Users are expected to access their data during that time and copy it to appropriate long term storage. Recovery from cold storage is available with delayed access and a recovery fee associated with staff time.

Accessing Animal MRI Scanner Raw Data

The Bruker BioSpec 70/30 scanner produces data in the raw Bruker data format. The data sets are accessible on the Bruker controlling computer for 14 calendar days from the date of the acquisition at /opt/<PV version>/data/<username>, with <PV version> the ParaVision software version (PV5.1 or PV6.0.1) and <username> the login of the user performing the scans. Users are expected to access their data during that time and copy it to appropriate long term storage. Recovery is available with delayed access and a recovery fee associated with staff time. Data can be accessed by 1) accessing the Neuroinformatics Platform and then 2) accessing the bruker7t computer. The tools for data transfer will work, provided you take into account you need to connect to an additional computer (the bruker7t).

Conversion to other formats (DICOM and NIFTI) is available upon request.