ibllib.pipes.misc

Functions

`backup_session`	Used to move the contents of a session to a backup folder, likely before the folder is removed.
`behavior_exists`
`check_create_raw_session_flag`
`check_transfer`	Check all the files in the source directory match those in the destination directory.
`cli_ask_default`	Prompt the user for input, display the default option and return user input or default
`cli_ask_options`
`confirm_ephys_remote_folder`
`copy_wiring_files`	Run after moving files to probe folders
`copy_with_check`
`create_alyx_probe_insertions`
`create_basic_transfer_params`	Create some basic parameters common to all acquisition rigs.
`create_custom_ephys_wirings`
`create_ephys_flags`	Create flags for processing an ephys session.
`create_ephys_transfer_done_flag`
`create_ephyspc_params`
`create_transfer_done_flag`
`create_video_transfer_done_flag`
`create_videopc_params`
`get_directory_size`	Used to determine total size of all files in a given session_path, including all child directories
`get_iblscripts_folder`
`get_new_filename`	get_new_filename is system agnostic (3A, 3B1, 3B2).
`get_session_numbers_from_date_path`	Retrieves session numbers when given a date path
`load_ephyspc_params`
`load_params_dict`
`load_videopc_params`
`move_ephys_files`	move_ephys_files is system agnostic (3A, 3B1, 3B2).
`multi_parts_flags_creation`	Creates the sequence files to run spike sorting in batches A sequence file is a json file with the following fields: sha1: a unique hash of the metafiles involved probe: a string with the probe name index: the index within the sequence nrecs: the length of the sequence files: a list of files
`probe_labels_from_session_path`	Finds ephys probes according to the metadata spikeglx files.
`rdiff_install`	For windows: * if the rdiff-backup executable does not already exist on the system * downloads rdiff-backup zip file * copies the executable to the C: ools folder
`rename_ephys_files`	rename_ephys_files is system agnostic (3A, 3B1, 3B2).
`rename_session`	Rename a session.
`rsync_paths`	Used to run the rsync algorithm via a rdiff-backup command on the paths contained on the provided source and destination.
`subjects_data_folder`	Given a root_data_folder will try to find a 'Subjects' data folder.
`transfer_folder`	functionality has been replaced by transfer_session_folders function
`transfer_session_folders`	Used to determine which local session folders should be transferred to which remote session folders, will prompt the user when necessary.

Classes

WindowsInhibitor

Prevent OS sleep/hibernate in windows; code from: https://github.com/h3llrais3r/Deluge-PreventSuspendPlus/blob/master/preventsuspendplus/core.py API documentation: https://msdn.microsoft.com/en-us/library/windows/desktop/aa373208(v=vs.85).aspx

subjects_data_folder(folder: Path, rglob: bool = False) → Path[source]: Given a root_data_folder will try to find a ‘Subjects’ data folder. If Subjects folder is passed will return it directly.

cli_ask_default(prompt: str, default: str)[source]

Prompt the user for input, display the default option and return user input or default

Parameters

prompt – String to display to user
default – The default value to return if user doesn’t enter anything

Returns

User input or default

cli_ask_options(prompt: str, options: list, default_idx: int = 0) → str[source]

behavior_exists(session_path: str) → bool[source]

check_transfer(src_session_path, dst_session_path)[source]

Check all the files in the source directory match those in the destination directory. Function will throw assertion errors/exceptions if number of files do not match, file names do not match, or if file sizes do not match.

Parameters

src_session_path – The source directory that was copied
dst_session_path – The copy target directory

rename_session(session_path: str, new_subject=None, new_date=None, new_number=None, ask: bool = False) → Path[source]

Rename a session. Prompts the user for the new subject name, data and number and then moves session path to new session path.

Parameters

session_path (str) – A session path to rename
new_subject – A new subject name, if none provided, the user is prompted for one
new_date – A new session date, if none provided, the user is prompted for one
new_number – A new session number, if none provided, the user is prompted for one
ask (bool) – used to ensure prompt input from user, defaults to False

Returns

The renamed session path

Return type

Path

backup_session(session_path)[source]

Used to move the contents of a session to a backup folder, likely before the folder is removed.

Parameters: session_path – A session path to be backed up
Returns: True if directory was backed up or exits if something went wrong
Return type: Bool

copy_with_check(src, dst, **kwargs)[source]

transfer_session_folders(local_sessions: list, remote_subject_folder, subfolder_to_transfer)[source]

Used to determine which local session folders should be transferred to which remote session folders, will prompt the user when necessary.

Parameters

local_sessions (list) – Required list of local session folder paths to sync to local server.
remote_subject_folder (str, pathlib.Path) – The remote location of the subject folder (typically pulled from the params).
subfolder_to_transfer (str) – Which subfolder to sync

Returns

list of tuples – For each session, a tuple of (source, destination) of attempted file transfers.
list of bool – A boolean True/False for success/failure of the transfer.

transfer_folder(src: Path, dst: Path, force: bool = False) → None[source]: functionality has been replaced by transfer_session_folders function

load_params_dict(params_fname: str) → dict[source]

load_videopc_params()[source]

load_ephyspc_params()[source]

create_basic_transfer_params(param_str='transfer_params', local_data_path=None, remote_data_path=None, clobber=False, **kwargs)[source]

Create some basic parameters common to all acquisition rigs.

Namely prompt user for the local root data path and the remote (lab server) data path. NB: All params stored in uppercase by convention.

Parameters

param_str (str) – The name of the parameters to load/save.
local_data_path (str, pathlib.Path) – The local root data path, stored with the DATA_FOLDER_PATH key. If None, user is prompted.
remote_data_path (str, pathlib.Path) – The local root data path, stored with the REMOTE_DATA_FOLDER_PATH key. If None, user is prompted.
clobber (bool) – If True, any parameters in existing parameter file not found as keyword args will be removed, otherwise the user is prompted for these also.
**kwargs – Extra parameters to set. If value is None, the user is prompted.

Returns

The parameters written to disc.

Return type

dict

Examples

Set up basic transfer parameters for modality acquisition PC

>>> par = create_basic_transfer_params()

Set up basic transfer paramers without prompting the user

>>> par = create_basic_transfer_params(
...     local_data_path='/iblrig_data/Subjects',
...     remote_data_path='/mnt/iblserver.champalimaud.pt/ibldata/Subjects')

Prompt user for extra parameter using custom prompt (will call function with current default)

>>> from functools import partial
>>> par = create_basic_transfer_params(
...     custom_arg=partial(cli_ask_default, 'Please enter custom arg value'))

create_videopc_params(force=False, silent=False)[source]

create_ephyspc_params(force=False, silent=False)[source]

rdiff_install() → bool[source]

For windows: * if the rdiff-backup executable does not already exist on the system

downloads rdiff-backup zip file

copies the executable to the C: ools folder

For linux/mac: * runs a pip install rdiff-backup

Returns: True when install is successful, False when an error is encountered

get_directory_size(dir_path: Path, in_gb=False) → float[source]

Used to determine total size of all files in a given session_path, including all child directories

Parameters

dir_path (Path) – path we want to get the total size of
in_gb (bool) – set to True for returned value to be in gigabytes

Returns

sum of all files in the given directory path (in bytes by default, in GB if specified)

Return type

float

get_session_numbers_from_date_path(date_path: Path) → list[source]

Retrieves session numbers when given a date path

Parameters: date_path (Path) – path to date, i.e. \serversome_labSubjectsDate”
Returns: Found sessions as a sorted list
Return type: (list)

rsync_paths(src: Path, dst: Path) → bool[source]

Used to run the rsync algorithm via a rdiff-backup command on the paths contained on the provided source and destination. This function relies on the rdiff-backup package and is run from the command line, i.e. subprocess.run(). Full documentation can be found here - https://rdiff-backup.net/docs/rdiff-backup.1.html

Parameters

src (Path) – source path that contains data to be transferred
dst (Path) – destination path that will receive the transferred data

Returns

True for success, False for failure

Return type

bool

Raises

FileNotFoundError, subprocess.CalledProcessError –

confirm_ephys_remote_folder(local_folder=False, remote_folder=False, force=False, iblscripts_folder=False)[source]

probe_labels_from_session_path(session_path: Union[str, Path]) → List[str][source]

Finds ephys probes according to the metadata spikeglx files. Only returns first subfolder name under raw_ephys_data folder, ie. raw_ephys_data/probe00/copy_of_probe00 won’t be returned If there is a NP2.4 probe with several shanks, create several probes

Parameters: session_path –
Returns: list of strings

create_alyx_probe_insertions(session_path: str, force: bool = False, one: Optional[object] = None, model: Optional[str] = None, labels: Optional[list] = None)[source]

create_ephys_flags(session_folder: str)[source]

Create flags for processing an ephys session. Should be called after move_ephys_files

Parameters: session_folder – A path to an ephys session
Returns

create_ephys_transfer_done_flag(session_folder: str) → None[source]

create_video_transfer_done_flag(session_folder: str) → None[source]

create_transfer_done_flag(session_folder: str, flag_name: str) → None[source]

check_create_raw_session_flag(session_folder: str) → None[source]

rename_ephys_files(session_folder: str) → None[source]

rename_ephys_files is system agnostic (3A, 3B1, 3B2). Renames all ephys files to Alyx compatible filenames. Uses get_new_filename.

Parameters: session_folder (str) – Session folder path
Returns: None - Changes names of files on filesystem
Return type: None

get_new_filename(filename: str) → str[source]

get_new_filename is system agnostic (3A, 3B1, 3B2). Gets an alyx compatible filename from any spikeglx ephys file.

Parameters: filename – Name of an ephys file
Returns: New name for ephys file

move_ephys_files(session_folder: str) → None[source]

move_ephys_files is system agnostic (3A, 3B1, 3B2). Moves all properly named ephys files to appropriate locations for transfer. Use rename_ephys_files function before this one.

Parameters: session_folder (str) – Session folder path
Returns: None - Moves files on filesystem
Return type: None

create_custom_ephys_wirings(iblscripts_folder: str)[source]

get_iblscripts_folder()[source]

copy_wiring_files(session_folder, iblscripts_folder)[source]: Run after moving files to probe folders

multi_parts_flags_creation(root_paths: Union[list, str, Path]) → List[Path][source]

Creates the sequence files to run spike sorting in batches A sequence file is a json file with the following fields:

sha1: a unique hash of the metafiles involved probe: a string with the probe name index: the index within the sequence nrecs: the length of the sequence files: a list of files

Parameters: root_paths –
Returns

class WindowsInhibitor[source]

Bases: object

Prevent OS sleep/hibernate in windows; code from: https://github.com/h3llrais3r/Deluge-PreventSuspendPlus/blob/master/preventsuspendplus/core.py API documentation: https://msdn.microsoft.com/en-us/library/windows/desktop/aa373208(v=vs.85).aspx

ES_CONTINUOUS = 2147483648

ES_SYSTEM_REQUIRED = 1

inhibit()[source]

uninhibit()[source]