ibllib.oneibl.data_handlers

Functions

get_local_data_repository

Classes

DataHandler
LocalDataHandler
RemoteAwsDataHandler
RemoteGlobusDataHandler	Data handler for running tasks on remote compute node.
RemoteHttpDataHandler
SDSCDataHandler	Data handler for running tasks on SDSC compute node
ServerDataHandler
ServerGlobusDataHandler

get_local_data_repository(one)

class DataHandler(session_path, signature, one=None)

Bases: ABC

setUp()

Function to optionally overload to download required data to run task :return:

getData(one=None)

Finds the datasets required for task based on input signatures :return:

uploadData(outputs, version)

Function to optionally overload to upload and register data

Parameters

• outputs – output files from task to register

• version – ibllib version

Returns

cleanUp()

Function to optionally overload to cleanup files after running task :return:

class LocalDataHandler(session_path, signatures, one=None)

Bases: DataHandler

class ServerDataHandler(session_path, signatures, one=None)

Bases: DataHandler

uploadData(outputs, version, **kwargs)

Function to upload and register data of completed task

Parameters

• outputs – output files from task to register

• version – ibllib version

Returns

output info of registered datasets

class ServerGlobusDataHandler(session_path, signatures, one=None)

Bases: DataHandler

setUp()

Function to download necessary data to run tasks using globus-sdk :return:

uploadData(outputs, version, **kwargs)

Function to upload and register data of completed task

Parameters

• outputs – output files from task to register

• version – ibllib version

Returns

output info of registered datasets

cleanUp()

Clean up, remove the files that were downloaded from globus once task has completed :return:

class RemoteHttpDataHandler(session_path, signature, one=None)

Bases: DataHandler

setUp()

Function to download necessary data to run tasks using ONE :return:

uploadData(outputs, version, **kwargs)

Function to upload and register data of completed task via FTP patcher

Parameters

• outputs – output files from task to register

• version – ibllib version

Returns

output info of registered datasets

class RemoteAwsDataHandler(task, session_path, signature, one=None)

Bases: DataHandler

setUp()

Function to download necessary data to run tasks using AWS boto3 :return:

uploadData(outputs, version, **kwargs)

Function to upload and register data of completed task via FTP patcher

Parameters

• outputs – output files from task to register

• version – ibllib version

Returns

output info of registered datasets

cleanUp()

Clean up, remove the files that were downloaded from globus once task has completed :return:

class RemoteGlobusDataHandler(session_path, signature, one=None)

Bases: DataHandler

Data handler for running tasks on remote compute node. Will download missing data using globus

Parameters

• session_path – path to session

• signature – input and output file signatures

• one – ONE instance

setUp()

Function to download necessary data to run tasks using globus :return:

uploadData(outputs, version, **kwargs)

Function to upload and register data of completed task via FTP patcher

Parameters

• outputs – output files from task to register

• version – ibllib version

Returns

output info of registered datasets

class SDSCDataHandler(task, session_path, signatures, one=None)

Bases: DataHandler

Data handler for running tasks on SDSC compute node

Parameters

• session_path – path to session

• signature – input and output file signatures

• one – ONE instance

setUp()

Function to create symlinks to necessary data to run tasks :return:

uploadData(outputs, version, **kwargs)

Function to upload and register data of completed task via SDSC patcher

Parameters

• outputs – output files from task to register

• version – ibllib version

Returns

output info of registered datasets

cleanUp()

Function to clean up symlinks created to run task :return: