Core Functions¶
Functions that are required to operate the package at a basic level
caiman.source_extraction.cnmf.CNMF (n_processes) |
Source extraction using constrained non-negative matrix factorization. |
caiman.source_extraction.cnmf.CNMF.fit (images) |
This method uses the cnmf algorithm to find sources in data. |
caiman.source_extraction.cnmf.online_cnmf.OnACID ([…]) |
Source extraction of streaming data using online matrix factorization. |
caiman.source_extraction.cnmf.online_cnmf.OnACID.fit_online (…) |
Implements the caiman online algorithm on the list of files fls. |
caiman.source_extraction.cnmf.params.CNMFParams ([…]) |
Class for setting and changing the various parameters. |
caiman.source_extraction.cnmf.estimates.Estimates ([…]) |
Class for storing and reusing the analysis results and performing basic processing and plotting operations. |
caiman.motion_correction.MotionCorrect (fname) |
class implementing motion correction operations |
caiman.motion_correction.MotionCorrect.motion_correct ([…]) |
general function for performing all types of motion correction. |
caiman.base.movies.load (file_name, …[, …]) |
load movie from file. |
caiman.base.movies.movie.play (gain[, fr, …]) |
Play the movie using opencv |
caiman.base.rois.register_ROIs (A1, A2, dims) |
Register ROIs across different sessions using an intersection over union metric and the Hungarian algorithm for optimal matching |
caiman.base.rois.register_multisession (A, dims) |
Register ROIs across multiple sessions using an intersection over union metric and the Hungarian algorithm for optimal matching. |
caiman.source_extraction.cnmf.utilities.detrend_df_f (A, …) |
Compute DF/F signal without using the original data. |
Movie Handling¶
-
class
caiman.base.movies.
movie
¶ Class representing a movie. This class subclasses timeseries, that in turn subclasses ndarray
movie(input_arr, fr=None,start_time=0,file_name=None, meta_data=None)
- Example of usage:
- input_arr = 3d ndarray fr=33; # 33 Hz start_time=0 m=movie(input_arr, start_time=0,fr=33);
See https://docs.scipy.org/doc/numpy/user/basics.subclassing.html for notes on objects that are descended from ndarray
Attributes: T
The transposed array.
base
Base object if memory is from some other object.
ctypes
An object to simplify the interaction of the array with the ctypes module.
data
Python buffer object pointing to the start of the array’s data.
dtype
Data-type of the array’s elements.
flags
Information about the memory layout of the array.
flat
A 1-D iterator over the array.
imag
The imaginary part of the array.
itemsize
Length of one array element in bytes.
nbytes
Total bytes consumed by the elements of the array.
ndim
Number of array dimensions.
real
The real part of the array.
shape
Tuple of array dimensions.
size
Number of elements in the array.
strides
Tuple of bytes to step in each dimension when traversing an array.
- time
Methods
IPCA
(components, batch)Iterative Principal Component analysis, see sklearn.decomposition.incremental_pca IPCA_denoise
(components, batch)Create a denoised version of the movie using only the first ‘components’ components IPCA_io
(n_components, fun, max_iter[, tol])DO NOT USE STILL UNDER DEVELOPMENT IPCA_stICA
(componentsPCA, componentsICA, …)Compute PCA + ICA a la Mukamel 2009. NonnegativeMatrixFactorization
(n_components, …)See documentation for scikit-learn NMF all
([axis, out, keepdims])Returns True if all elements evaluate to True. any
([axis, out, keepdims])Returns True if any of the elements of a evaluate to True. apply_shifts
(shifts, interpolation, method, …)Apply precomputed shifts to a movie, using subpixels adjustment (cv2.INTER_CUBIC function) argmax
([axis, out])Return indices of the maximum values along the given axis. argmin
([axis, out])Return indices of the minimum values along the given axis of a. argpartition
(kth[, axis, kind, order])Returns the indices that would partition this array. argsort
([axis, kind, order])Returns the indices that would sort this array. astype
(dtype[, order, casting, subok, copy])Copy of the array, cast to a specified type. bilateral_blur_2D
(diameter, sigmaColor[, …])performs bilateral filtering on each frame. bin_median
(window)compute median of 3D array in along axis o by binning values bin_median_3d
([window])compute median of 4D array in along axis o by binning values byteswap
([inplace])Swap the bytes of the array elements choose
(choices[, out, mode])Use an index array to construct a new array from a set of choices. clip
([min, max, out])Return an array whose values are limited to [min, max]
.compress
(condition[, axis, out])Return selected slices of this array along given axis. computeDFF
(secsWindow, quantilMin, method, …)compute the DFF of the movie or remove baseline conj
()Complex-conjugate all elements. conjugate
()Return the complex conjugate, element-wise. copy
([order])Return a copy of the array. crop
([crop_top, crop_bottom, crop_left, …])Crop movie (inline) cumprod
([axis, dtype, out])Return the cumulative product of the elements along the given axis. cumsum
([axis, dtype, out])Return the cumulative sum of the elements along the given axis. debleach
()Debleach by fiting a model to the median intensity. diagonal
([offset, axis1, axis2])Return specified diagonals. dot
(b[, out])Dot product of two arrays. dump
(file)Dump a pickle of the array to the specified file. dumps
()Returns the pickle of the array as a string. extract_shifts
(max_shift_w, max_shift_h[, …])Performs motion correction using the opencv matchtemplate function. extract_traces_from_masks
(masks)Args: fill
(value)Fill the array with a scalar value. flatten
([order])Return a copy of the array collapsed into one dimension. gaussian_blur_2D
([kernel_size_x, …])Compute gaussian blut in 2D. getfield
(dtype[, offset])Returns a field of the given array as a certain type. guided_filter_blur_2D
(guide_filter, radius)performs guided filtering on each frame. item
(*args)Copy an element of an array to a standard Python scalar and return it. itemset
(*args)Insert scalar into an array (scalar is cast to array’s dtype, if possible) local_correlations
(eight_neighbours, …)Computes the correlation image (CI) for the input movie. max
([axis, out, keepdims, initial, where])Return the maximum along a given axis. mean
([axis, dtype, out, keepdims])Returns the average of the array elements along given axis. median_blur_2D
(kernel_size)Compute gaussian blut in 2D. min
([axis, out, keepdims, initial, where])Return the minimum along a given axis. motion_correct
([max_shift_w, max_shift_h, …])Extract shifts and motion corrected movie automatically, motion_correct_3d
([max_shift_z, …])Extract shifts and motion corrected movie automatically, newbyteorder
([new_order])Return the array with the same data viewed with a different byte order. nonzero
()Return the indices of the elements that are non-zero. online_NMF
(n_components, method, lambda1, …)Method performing online matrix factorization and using the spams partition
(kth[, axis, kind, order])Rearranges the elements in the array in such a way that the value of the element in kth position is in the position it would be in a sorted array. partition_FOV_KMeans
(tradeoff_weight, fx, …)Partition the FOV in clusters that are grouping pixels close in space and in mutual correlation play
(gain[, fr, magnification, offset, …])Play the movie using opencv prod
([axis, dtype, out, keepdims, initial, …])Return the product of the array elements over the given axis ptp
([axis, out, keepdims])Peak to peak (maximum - minimum) value along a given axis. put
(indices, values[, mode])Set a.flat[n] = values[n]
for all n in indices.ravel
([order])Return a flattened array. removeBL
(windowSize, quantilMin, in_place, …)Remove baseline from movie using percentiles over a window Args: windowSize: int window size over which to compute the baseline (the larger the faster the algorithm and the less granular repeat
(repeats[, axis])Repeat elements of an array. reshape
(shape[, order])Returns an array containing the same data with a new shape. resize
([fx, fy, fz, interpolation])Resizing caiman movie into a new one. round
([decimals, out])Return a with each element rounded to the given number of decimals. save
(file_name[, to32, order, imagej, …])Save the timeseries in single precision. searchsorted
(v[, side, sorter])Find indices where elements of v should be inserted in a to maintain order. setfield
(val, dtype[, offset])Put a value into a specified place in a field defined by a data-type. setflags
([write, align, uic])Set array flags WRITEABLE, ALIGNED, (WRITEBACKIFCOPY and UPDATEIFCOPY), respectively. sort
([axis, kind, order])Sort an array in-place. squeeze
([axis])Remove single-dimensional entries from the shape of a. std
([axis, dtype, out, ddof, keepdims])Returns the standard deviation of the array elements along given axis. sum
([axis, dtype, out, keepdims, initial, where])Return the sum of the array elements over the given axis. swapaxes
(axis1, axis2)Return a view of the array with axis1 and axis2 interchanged. take
(indices[, axis, out, mode])Return an array formed from the elements of a at the given indices. to2DPixelxTime
([order])Transform 3D movie into 2D to3DFromPixelxTime
(shape[, order])Transform 2D movie into 3D tobytes
([order])Construct Python bytes containing the raw data bytes in the array. tofile
(fid[, sep, format])Write array to a file as text or binary (default). tolist
()Return the array as an a.ndim
-levels deep nested list of Python scalars.tostring
([order])A compatibility alias for tobytes, with exactly the same behavior. trace
([offset, axis1, axis2, dtype, out])Return the sum along diagonals of the array. transpose
(*axes)Returns a view of the array with axes transposed. var
([axis, dtype, out, ddof, keepdims])Returns the variance of the array elements, along given axis. view
([dtype][, type])New view of array with the same data. zproject
(method[, cmap, aspect])Compute and plot projection across time: apply_shifts_online calc_min to_2D
-
movie.
play
(gain: float = 1, fr=None, magnification=1, offset=0, interpolation=1, backend: str = 'opencv', do_loop: bool = False, bord_px=None, q_max=99.75, q_min=1, plot_text: bool = False, save_movie: bool = False, opencv_codec: str = 'H264', movie_name: str = 'movie.avi') → None¶ Play the movie using opencv
- Args:
gain: adjust movie brightness
fr: framerate, playing speed if different from original (inter frame interval in seconds)
- magnification: int
- magnification factor
offset: (undocumented)
interpolation: (undocumented)
backend: ‘pylab’ or ‘opencv’, the latter much faster
do_loop: Whether to loop the video
- bord_px: int
- truncate pixels from the borders
- q_max, q_min: float in [0, 100]
- percentile for maximum/minimum plotting value
- plot_text: bool
- show some text
- save_movie: bool
- flag to save an avi file of the movie
- opencv_codec: str
- FourCC video codec for saving movie. Check http://www.fourcc.org/codecs.php
- movie_name: str
- name of saved file
- Raises:
- Exception ‘Unknown backend!’
-
movie.
resize
(fx=1, fy=1, fz=1, interpolation=3)¶ Resizing caiman movie into a new one. Note that the temporal dimension is controlled by fz and fx, fy, fz correspond to magnification factors. For example to downsample in time by a factor of 2, you need to set fz = 0.5.
- Args:
- fx (float):
- Magnification factor along x-dimension
- fy (float):
- Magnification factor along y-dimension
- fz (float):
- Magnification factor along temporal dimension
- Returns:
- self (caiman movie)
-
movie.
computeDFF
(secsWindow: int = 5, quantilMin: int = 8, method: str = 'only_baseline', in_place: bool = False, order: str = 'F') → Tuple[Any, Any]¶ compute the DFF of the movie or remove baseline
In order to compute the baseline frames are binned according to the window length parameter and then the intermediate values are interpolated.
- Args:
secsWindow: length of the windows used to compute the quantile
quantilMin : value of the quantile
method=’only_baseline’,’delta_f_over_f’,’delta_f_over_sqrt_f’
in_place: compute baseline in a memory efficient way by updating movie in place
- Returns:
self: DF or DF/F or DF/sqrt(F) movies
movBL=baseline movie
- Raises:
- Exception ‘Unknown method’
-
caiman.base.movies.
load
(file_name: Union[str, List[str]], fr: float = 30, start_time: float = 0, meta_data: Dict[KT, VT] = None, subindices=None, shape: Tuple[int, int] = None, var_name_hdf5: str = 'mov', in_memory: bool = False, is_behavior: bool = False, bottom=0, top=0, left=0, right=0, channel=None, outtype=<class 'numpy.float32'>, is3D: bool = False) → Any¶ load movie from file. Supports a variety of formats. tif, hdf5, npy and memory mapped. Matlab is experimental.
- Args:
- file_name: string or List[str]
- name of file. Possible extensions are tif, avi, npy, (npz and hdf5 are usable only if saved by calblitz)
- fr: float
- frame rate
- start_time: float
- initial time for frame 1
- meta_data: dict
- dictionary containing meta information about the movie
- subindices: iterable indexes
- for loading only portion of the movie
- shape: tuple of two values
- dimension of the movie along x and y if loading from a two dimensional numpy array
- var_name_hdf5: str
- if loading from hdf5 name of the variable to load
in_memory: (undocumented)
is_behavior: (undocumented)
bottom,top,left,right: (undocumented)
channel: (undocumented)
outtype: The data type for the movie
- Returns:
- mov: caiman.movie
- Raises:
Exception ‘Subindices not implemented’
Exception ‘Subindices not implemented’
Exception ‘sima module unavailable’
Exception ‘Unknown file type’
Exception ‘File not found!’
-
caiman.base.movies.
load_movie_chain
(file_list: List[str], fr: float = 30, start_time=0, meta_data=None, subindices=None, var_name_hdf5: str = 'mov', bottom=0, top=0, left=0, right=0, z_top=0, z_bottom=0, is3D: bool = False, channel=None, outtype=<class 'numpy.float32'>) → Any¶ load movies from list of file names
- Args:
- file_list: list
- file names in string format
the other parameters as in load_movie except
- bottom, top, left, right, z_top, z_bottom : int
- to load only portion of the field of view
- is3D : bool
- flag for 3d data (adds a fourth dimension)
- Returns:
- movie: movie
- movie corresponding to the concatenation og the input files
Timeseries Handling¶
-
class
caiman.base.timeseries.
timeseries
¶ Class representing a time series.
Attributes: T
The transposed array.
base
Base object if memory is from some other object.
ctypes
An object to simplify the interaction of the array with the ctypes module.
data
Python buffer object pointing to the start of the array’s data.
dtype
Data-type of the array’s elements.
flags
Information about the memory layout of the array.
flat
A 1-D iterator over the array.
imag
The imaginary part of the array.
itemsize
Length of one array element in bytes.
nbytes
Total bytes consumed by the elements of the array.
ndim
Number of array dimensions.
real
The real part of the array.
shape
Tuple of array dimensions.
size
Number of elements in the array.
strides
Tuple of bytes to step in each dimension when traversing an array.
- time
Methods
all
([axis, out, keepdims])Returns True if all elements evaluate to True. any
([axis, out, keepdims])Returns True if any of the elements of a evaluate to True. argmax
([axis, out])Return indices of the maximum values along the given axis. argmin
([axis, out])Return indices of the minimum values along the given axis of a. argpartition
(kth[, axis, kind, order])Returns the indices that would partition this array. argsort
([axis, kind, order])Returns the indices that would sort this array. astype
(dtype[, order, casting, subok, copy])Copy of the array, cast to a specified type. byteswap
([inplace])Swap the bytes of the array elements choose
(choices[, out, mode])Use an index array to construct a new array from a set of choices. clip
([min, max, out])Return an array whose values are limited to [min, max]
.compress
(condition[, axis, out])Return selected slices of this array along given axis. conj
()Complex-conjugate all elements. conjugate
()Return the complex conjugate, element-wise. copy
([order])Return a copy of the array. cumprod
([axis, dtype, out])Return the cumulative product of the elements along the given axis. cumsum
([axis, dtype, out])Return the cumulative sum of the elements along the given axis. diagonal
([offset, axis1, axis2])Return specified diagonals. dot
(b[, out])Dot product of two arrays. dump
(file)Dump a pickle of the array to the specified file. dumps
()Returns the pickle of the array as a string. fill
(value)Fill the array with a scalar value. flatten
([order])Return a copy of the array collapsed into one dimension. getfield
(dtype[, offset])Returns a field of the given array as a certain type. item
(*args)Copy an element of an array to a standard Python scalar and return it. itemset
(*args)Insert scalar into an array (scalar is cast to array’s dtype, if possible) max
([axis, out, keepdims, initial, where])Return the maximum along a given axis. mean
([axis, dtype, out, keepdims])Returns the average of the array elements along given axis. min
([axis, out, keepdims, initial, where])Return the minimum along a given axis. newbyteorder
([new_order])Return the array with the same data viewed with a different byte order. nonzero
()Return the indices of the elements that are non-zero. partition
(kth[, axis, kind, order])Rearranges the elements in the array in such a way that the value of the element in kth position is in the position it would be in a sorted array. prod
([axis, dtype, out, keepdims, initial, …])Return the product of the array elements over the given axis ptp
([axis, out, keepdims])Peak to peak (maximum - minimum) value along a given axis. put
(indices, values[, mode])Set a.flat[n] = values[n]
for all n in indices.ravel
([order])Return a flattened array. repeat
(repeats[, axis])Repeat elements of an array. reshape
(shape[, order])Returns an array containing the same data with a new shape. resize
(new_shape[, refcheck])Change shape and size of array in-place. round
([decimals, out])Return a with each element rounded to the given number of decimals. save
(file_name[, to32, order, imagej, …])Save the timeseries in single precision. searchsorted
(v[, side, sorter])Find indices where elements of v should be inserted in a to maintain order. setfield
(val, dtype[, offset])Put a value into a specified place in a field defined by a data-type. setflags
([write, align, uic])Set array flags WRITEABLE, ALIGNED, (WRITEBACKIFCOPY and UPDATEIFCOPY), respectively. sort
([axis, kind, order])Sort an array in-place. squeeze
([axis])Remove single-dimensional entries from the shape of a. std
([axis, dtype, out, ddof, keepdims])Returns the standard deviation of the array elements along given axis. sum
([axis, dtype, out, keepdims, initial, where])Return the sum of the array elements over the given axis. swapaxes
(axis1, axis2)Return a view of the array with axis1 and axis2 interchanged. take
(indices[, axis, out, mode])Return an array formed from the elements of a at the given indices. tobytes
([order])Construct Python bytes containing the raw data bytes in the array. tofile
(fid[, sep, format])Write array to a file as text or binary (default). tolist
()Return the array as an a.ndim
-levels deep nested list of Python scalars.tostring
([order])A compatibility alias for tobytes, with exactly the same behavior. trace
([offset, axis1, axis2, dtype, out])Return the sum along diagonals of the array. transpose
(*axes)Returns a view of the array with axes transposed. var
([axis, dtype, out, ddof, keepdims])Returns the variance of the array elements, along given axis. view
([dtype][, type])New view of array with the same data.
-
timeseries.
save
(file_name, to32=True, order='F', imagej=False, bigtiff=True, excitation_lambda=488.0, compress=0, q_max=99.75, q_min=1, var_name_hdf5='mov', sess_desc='some_description', identifier='some identifier', imaging_plane_description='some imaging plane description', emission_lambda=520.0, indicator='OGB-1', location='brain', starting_time=0.0, experimenter='Dr Who', lab_name=None, institution=None, experiment_description='Experiment Description', session_id='Session ID')¶ Save the timeseries in single precision. Supported formats include TIFF, NPZ, AVI, MAT, HDF5/H5, MMAP, and NWB
- Args:
- file_name: str
- name of file. Possible formats are tif, avi, npz, mmap and hdf5
- to32: Bool
- whether to transform to 32 bits
- order: ‘F’ or ‘C’
- C or Fortran order
- var_name_hdf5: str
- Name of hdf5 file subdirectory
- q_max, q_min: float in [0, 100]
- percentile for maximum/minimum clipping value if saving as avi (If set to None, no automatic scaling to the dynamic range [0, 255] is performed)
- Raises:
- Exception ‘Extension Unknown’
Motion Correction¶
-
class
caiman.motion_correction.
MotionCorrect
(fname, min_mov=None, dview=None, max_shifts=(6, 6), niter_rig=1, splits_rig=14, num_splits_to_process_rig=None, strides=(96, 96), overlaps=(32, 32), splits_els=14, num_splits_to_process_els=None, upsample_factor_grid=4, max_deviation_rigid=3, shifts_opencv=True, nonneg_movie=True, gSig_filt=None, use_cuda=False, border_nan=True, pw_rigid=False, num_frames_split=80, var_name_hdf5='mov', is3D=False, indices=(slice(None, None, None), slice(None, None, None)))¶ class implementing motion correction operations
Methods
apply_shifts_movie
(fname, rigid_shifts, …)Applies shifts found by registering one file to a different file. motion_correct
([template, save_movie])general function for performing all types of motion correction. motion_correct_pwrigid
(save_movie, template, …)Perform pw-rigid motion correction motion_correct_rigid
([template, save_movie])Perform rigid motion correction
-
MotionCorrect.
motion_correct
(template=None, save_movie=False)¶ general function for performing all types of motion correction. The function will perform either rigid or piecewise rigid motion correction depending on the attribute self.pw_rigid and will perform high pass spatial filtering for determining the motion (used in 1p data) if the attribute self.gSig_filt is not None. A template can be passed, and the output can be saved as a memory mapped file.
- Args:
- template: ndarray, default: None
- template provided by user for motion correction
- save_movie: bool, default: False
- flag for saving motion corrected file(s) as memory mapped file(s)
- Returns:
- self
-
MotionCorrect.
motion_correct_rigid
(template=None, save_movie=False) → None¶ Perform rigid motion correction
- Args:
- template: ndarray 2D (or 3D)
- if known, one can pass a template to register the frames to
- save_movie_rigid:Bool
- save the movies vs just get the template
- Important Fields:
self.fname_tot_rig: name of the mmap file saved
self.total_template_rig: template updated by iterating over the chunks
self.templates_rig: list of templates. one for each chunk
self.shifts_rig: shifts in x and y (and z if 3D) per frame
-
MotionCorrect.
motion_correct_pwrigid
(save_movie: bool = True, template: numpy.ndarray = None, show_template: bool = False) → None¶ Perform pw-rigid motion correction
- Args:
- save_movie:Bool
- save the movies vs just get the template
- template: ndarray 2D (or 3D)
- if known, one can pass a template to register the frames to
- show_template: boolean
- whether to show the updated template at each iteration
- Important Fields:
- self.fname_tot_els: name of the mmap file saved self.templates_els: template updated by iterating over the chunks self.x_shifts_els: shifts in x per frame per patch self.y_shifts_els: shifts in y per frame per patch self.z_shifts_els: shifts in z per frame per patch (if 3D) self.coord_shifts_els: coordinates associated to the patch for values in x_shifts_els and y_shifts_els (and z_shifts_els if 3D) self.total_template_els: list of templates. one for each chunk
- Raises:
- Exception: ‘Error: Template contains NaNs, Please review the parameters’
-
MotionCorrect.
apply_shifts_movie
(fname, rigid_shifts: bool = None, save_memmap: bool = False, save_base_name: str = 'MC', order: str = 'F', remove_min: bool = True)¶ Applies shifts found by registering one file to a different file. Useful for cases when shifts computed from a structural channel are applied to a functional channel. Currently only application of shifts through openCV is supported. Returns either cm.movie or the path to a memory mapped file.
- Args:
- fname: str of List[str]
- name(s) of the movie to motion correct. It should not contain nans. All the loadable formats from CaImAn are acceptable
- rigid_shifts: bool (True)
- apply rigid or pw-rigid shifts (must exist in the mc object) deprectated (read directly from mc.pw_rigid)
- save_memmap: bool (False)
- flag for saving the resulting file in memory mapped form
- save_base_name: str [‘MC’]
- base name for memory mapped file name
- order: ‘F’ or ‘C’ [‘F’]
- order of resulting memory mapped file
- remove_min: bool (True)
- If minimum value is negative, subtract it from the data
- Returns:
- m_reg: caiman movie object
- caiman movie object with applied shifts (not memory mapped)
Estimates¶
-
class
caiman.source_extraction.cnmf.estimates.
Estimates
(A=None, b=None, C=None, f=None, R=None, dims=None)¶ Class for storing and reusing the analysis results and performing basic processing and plotting operations.
Methods
compute_background
(Yr)compute background (has big memory requirements) compute_residuals
(Yr)compute residual for each component (variable R) deconvolve
(params[, dview, dff_flag])performs deconvolution on the estimated traces using the parameters specified in params. detrend_df_f
([quantileMin, frames_window, …])Computes DF/F normalized fluorescence for the extracted traces. evaluate_components
(imgs, params[, dview])Computes the quality metrics for each component and stores the indices of the components that pass user specified thresholds. evaluate_components_CNN
(params[, neuron_class])Estimates the quality of inferred spatial components using a pretrained CNN classifier. filter_components
(imgs, params[, new_dict, …])Filters components based on given thresholds without re-computing the quality metrics. hv_view_components
([Yr, img, idx, …])view spatial and temporal components interactively in a notebook make_color_movie
(imgs[, q_max, q_min, …])Displays a color movie where each component is given an arbitrary color. manual_merge
(components, params)merge a given list of components. masks_2_neurofinder
(dataset_name)Return masks to neurofinder format merge_components
(Y, params[, mx, …])merges components nb_view_components
([Yr, img, idx, …])view spatial and temporal components interactively in a notebook nb_view_components_3d
([Yr, image_type, …])view spatial and temporal components interactively in a notebook (version for 3d data) normalize_components
()Normalizes components such that spatial components have l_2 norm 1 play_movie
(imgs[, q_max, q_min, gain_res, …])Displays a movie with three panels (original data (left panel), reconstructed data (middle panel), residual (right panel)) plot_contours
([img, idx, thr_method, thr, …])view contours of all spatial footprints. plot_contours_nb
([img, idx, thr_method, …])view contours of all spatial footprints (notebook environment). remove_duplicates
([predictions, r_values, …])remove neurons that heavily overlap and might be duplicates. remove_small_large_neurons
(min_size_neuro, …)remove neurons that are too large or too small restore_discarded_components
()Recover components that are filtered out with the select_components method save_NWB
(filename[, imaging_plane_name, …])writes NWB file select_components
([idx_components, …])Keeps only a selected subset of components and removes the rest. threshold_spatial_components
([maxthr, dview])threshold spatial components. view_components
([Yr, img, idx])view spatial and temporal components interactively
-
Estimates.
compute_residuals
(Yr)¶ compute residual for each component (variable R)
- Args:
- Yr : np.ndarray
- movie in format pixels (d) x frames (T)
-
Estimates.
deconvolve
(params, dview=None, dff_flag=False)¶ performs deconvolution on the estimated traces using the parameters specified in params. Deconvolution on detrended and normalized (DF/F) traces can be performed by setting dff_flag=True. In this case the results of the deconvolution are stored in F_dff_dec and S_dff
- Args:
- params: params object
- Parameters of the algorithm
- dff_flag: bool (True)
- Flag for deconvolving the DF/F traces
- Returns:
- self: estimates object
-
Estimates.
detrend_df_f
(quantileMin=8, frames_window=500, flag_auto=True, use_fast=False, use_residuals=True, detrend_only=False)¶ Computes DF/F normalized fluorescence for the extracted traces. See caiman.source.extraction.utilities.detrend_df_f for details
- Args:
- quantile_min: float
- quantile used to estimate the baseline (values in [0,100])
- frames_window: int
- number of frames for computing running quantile
- flag_auto: bool
- flag for determining quantile automatically (different for each trace)
- use_fast: bool
- flag for using approximate fast percentile filtering
- use_residuals: bool
- flag for using non-deconvolved traces in DF/F calculation
- detrend_only: bool (False)
- flag for only subtracting baseline and not normalizing by it. Used in 1p data processing where baseline fluorescence cannot be determined.
- Returns:
- self: CNMF object
- self.F_dff contains the DF/F normalized traces
-
Estimates.
evaluate_components
(imgs, params, dview=None)¶ Computes the quality metrics for each component and stores the indices of the components that pass user specified thresholds. The various thresholds and parameters can be passed as inputs. If left empty then they are read from self.params.quality’]
- Args:
- imgs: np.array (possibly memory mapped, t,x,y[,z])
- Imaging data
- params: params object
Parameters of the algorithm. The parameters in play here are contained in the subdictionary params.quality:
- min_SNR: float
- trace SNR threshold
- rval_thr: float
- space correlation threshold
- use_cnn: bool
- flag for using the CNN classifier
- min_cnn_thr: float
- CNN classifier threshold
- Returns:
- self: estimates object
- self.idx_components: np.array
- indices of accepted components
- self.idx_components_bad: np.array
- indices of rejected components
- self.SNR_comp: np.array
- SNR values for each temporal trace
- self.r_values: np.array
- space correlation values for each component
- self.cnn_preds: np.array
- CNN classifier values for each component
-
Estimates.
evaluate_components_CNN
(params, neuron_class=1)¶ Estimates the quality of inferred spatial components using a pretrained CNN classifier.
- Args:
- params: params object
- see .params for details
- neuron_class: int
- class label for neuron shapes
- Returns:
- self: Estimates object
- self.idx_components contains the indeced of components above the required treshold.
-
Estimates.
filter_components
(imgs, params, new_dict={}, dview=None, select_mode='All')¶ Filters components based on given thresholds without re-computing the quality metrics. If the quality metrics are not present then it calls self.evaluate components.
- Args:
- imgs: np.array (possibly memory mapped, t,x,y[,z])
- Imaging data
- params: params object
- Parameters of the algorithm
- select_mode: str
- Can be ‘All’ (no subselection is made, but quality filtering is performed), ‘Accepted’ (subselection of accepted components, a field named self.accepted_list must exist), ‘Rejected’ (subselection of rejected components, a field named self.rejected_list must exist), ‘Unassigned’ (both fields above need to exist)
- new_dict: dict
New dictionary with parameters to be called. The dictionary modifies the params.quality subdictionary in the following entries:
- min_SNR: float
- trace SNR threshold
- SNR_lowest: float
- minimum required trace SNR
- rval_thr: float
- space correlation threshold
- rval_lowest: float
- minimum required space correlation
- use_cnn: bool
- flag for using the CNN classifier
- min_cnn_thr: float
- CNN classifier threshold
- cnn_lowest: float
- minimum required CNN threshold
- gSig_range: list
- gSig scale values for CNN classifier
- Returns:
- self: estimates object
- self.idx_components: np.array
- indices of accepted components
- self.idx_components_bad: np.array
- indices of rejected components
- self.SNR_comp: np.array
- SNR values for each temporal trace
- self.r_values: np.array
- space correlation values for each component
- self.cnn_preds: np.array
- CNN classifier values for each component
-
Estimates.
hv_view_components
(Yr=None, img=None, idx=None, denoised_color=None, cmap='viridis')¶ view spatial and temporal components interactively in a notebook
- Args:
- Yr : np.ndarray
- movie in format pixels (d) x frames (T)
- img : np.ndarray
- background image for contour plotting. Default is the mean image of all spatial components (d1 x d2)
- idx : list
- list of components to be plotted
- denoised_color: string or None
- color name (e.g. ‘red’) or hex color code (e.g. ‘#F0027F’)
- cmap: string
- name of colormap (e.g. ‘viridis’) used to plot image_neurons
-
Estimates.
nb_view_components
(Yr=None, img=None, idx=None, denoised_color=None, cmap='jet', thr=0.99)¶ view spatial and temporal components interactively in a notebook
- Args:
- Yr : np.ndarray
- movie in format pixels (d) x frames (T)
- img : np.ndarray
- background image for contour plotting. Default is the mean image of all spatial components (d1 x d2)
- idx : list
- list of components to be plotted
- thr: double
- threshold regulating the extent of the displayed patches
- denoised_color: string or None
- color name (e.g. ‘red’) or hex color code (e.g. ‘#F0027F’)
- cmap: string
- name of colormap (e.g. ‘viridis’) used to plot image_neurons
-
Estimates.
nb_view_components_3d
(Yr=None, image_type='mean', dims=None, max_projection=False, axis=0, denoised_color=None, cmap='jet', thr=0.9)¶ view spatial and temporal components interactively in a notebook (version for 3d data)
- Args:
- Yr : np.ndarray
- movie in format pixels (d) x frames (T) (only required to compute the correlation image)
- dims: tuple of ints
- dimensions of movie (x, y and z)
- image_type: ‘mean’|’max’|’corr’
- image to be overlaid to neurons (average of shapes, maximum of shapes or nearest neigbor correlation of raw data)
- max_projection: bool
- plot max projection along specified axis if True, o/w plot layers
- axis: int (0, 1 or 2)
- axis along which max projection is performed or layers are shown
- thr: scalar between 0 and 1
- Energy threshold for computing contours
- denoised_color: string or None
- color name (e.g. ‘red’) or hex color code (e.g. ‘#F0027F’)
- cmap: string
- name of colormap (e.g. ‘viridis’) used to plot image_neurons
-
Estimates.
normalize_components
()¶ Normalizes components such that spatial components have l_2 norm 1
-
Estimates.
play_movie
(imgs, q_max=99.75, q_min=2, gain_res=1, magnification=1, include_bck=True, frame_range=slice(None, None, None), bpx=0, thr=0.0, save_movie=False, movie_name='results_movie.avi', display=True, opencv_codec='H264', use_color=False, gain_color=4, gain_bck=0.2)¶ Displays a movie with three panels (original data (left panel), reconstructed data (middle panel), residual (right panel))
- Args:
- imgs: np.array (possibly memory mapped, t,x,y[,z])
- Imaging data
- q_max: float (values in [0, 100], default: 99.75)
- percentile for maximum plotting value
- q_min: float (values in [0, 100], default: 1)
- percentile for minimum plotting value
- gain_res: float (1)
- amplification factor for residual movie
- magnification: float (1)
- magnification factor for whole movie
- include_bck: bool (True)
- flag for including background in original and reconstructed movie
- frame_range: range or slice or list (default: slice(None))
- display only a subset of frames
- bpx: int (default: 0)
- number of pixels to exclude on each border
- thr: float (values in [0, 1[) (default: 0)
- threshold value for contours, no contours if thr=0
- save_movie: bool (default: False)
- flag to save an avi file of the movie
- movie_name: str (default: ‘results_movie.avi’)
- name of saved file
- display: bool (default: True)
- flag for playing the movie (to stop the movie press ‘q’)
- opencv_codec: str (default: ‘H264’)
- FourCC video codec for saving movie. Check http://www.fourcc.org/codecs.php
- use_color: bool (default: False)
- flag for making a color movie. If True a random color will be assigned for each of the components
- gain_color: float (default: 4)
- amplify colors in the movie to make them brighter
- gain_bck: float (default: 0.2)
- dampen background in the movie to expose components (applicable only when color is used.)
- Returns:
- mov: The concatenated output movie
-
Estimates.
plot_contours
(img=None, idx=None, thr_method='max', thr=0.2, display_numbers=True, params=None, cmap='viridis')¶ view contours of all spatial footprints.
- Args:
- img : np.ndarray
- background image for contour plotting. Default is the mean image of all spatial components (d1 x d2)
- idx : list
- list of accepted components
- thr_method : str
- thresholding method for computing contours (‘max’, ‘nrg’) if list of coordinates self.coordinates is None, i.e. not already computed
- thr : float
- threshold value only effective if self.coordinates is None, i.e. not already computed
- display_numbers : bool
- flag for displaying the id number of each contour
- params : params object
- set of dictionary containing the various parameters
-
Estimates.
plot_contours_nb
(img=None, idx=None, thr_method='max', thr=0.2, params=None, line_color='white', cmap='viridis')¶ view contours of all spatial footprints (notebook environment).
- Args:
- img : np.ndarray
- background image for contour plotting. Default is the mean image of all spatial components (d1 x d2)
- idx : list
- list of accepted components
- thr_method : str
- thresholding method for computing contours (‘max’, ‘nrg’) if list of coordinates self.coordinates is None, i.e. not already computed
- thr : float
- threshold value only effective if self.coordinates is None, i.e. not already computed
- params : params object
- set of dictionary containing the various parameters
-
Estimates.
remove_duplicates
(predictions=None, r_values=None, dist_thr=0.1, min_dist=10, thresh_subset=0.6, plot_duplicates=False, select_comp=False)¶ remove neurons that heavily overlap and might be duplicates.
- Args:
- predictions r_values dist_thr min_dist thresh_subset plot_duplicates
-
Estimates.
remove_small_large_neurons
(min_size_neuro, max_size_neuro, select_comp=False)¶ remove neurons that are too large or too small
- Args:
- min_size_neuro: int
- min size in pixels
- max_size_neuro: int
- max size in pixels
- select_comp: bool
- remove components that are too small/large from main estimates fields. See estimates.selecte_components() for more details.
- Returns:
- neurons_to_keep: np.array
- indeces of components with size within the acceptable range
-
Estimates.
select_components
(idx_components=None, use_object=False, save_discarded_components=True)¶ Keeps only a selected subset of components and removes the rest. The subset can be either user defined with the variable idx_components or read from the estimates object. The flag use_object determines this choice. If no subset is present then all components are kept.
- Args:
- idx_components: list
- indices of components to be kept
- use_object: bool
- Flag to use self.idx_components for reading the indices.
- save_discarded_components: bool
- whether to save the components from initialization so that they can be restored using the restore_discarded_components method
- Returns:
- self: Estimates object
-
Estimates.
restore_discarded_components
()¶ Recover components that are filtered out with the select_components method
-
Estimates.
save_NWB
(filename, imaging_plane_name=None, imaging_series_name=None, sess_desc='CaImAn Results', exp_desc=None, identifier=None, imaging_rate=30.0, starting_time=0.0, session_start_time=None, excitation_lambda=488.0, imaging_plane_description='some imaging plane description', emission_lambda=520.0, indicator='OGB-1', location='brain', raw_data_file=None)¶ writes NWB file
- Args:
filename: str
imaging_plane_name: str, optional
imaging_series_name: str, optional
sess_desc: str, optional
exp_desc: str, optional
identifier: str, optional
- imaging_rate: float, optional
- default: 30 (Hz)
- starting_time: float, optional
- default: 0.0 (seconds)
location: str, optional
- session_start_time: datetime.datetime, optional
- Only required for new files
excitation_lambda: float
imaging_plane_description: str
emission_lambda: float
indicator: str
location: str
-
Estimates.
view_components
(Yr=None, img=None, idx=None)¶ view spatial and temporal components interactively
- Args:
- Yr : np.ndarray
- movie in format pixels (d) x frames (T)
- img : np.ndarray
- background image for contour plotting. Default is the mean image of all spatial components (d1 x d2)
- idx : list
- list of components to be plotted
Deconvolution¶
-
caiman.source_extraction.cnmf.deconvolution.
constrained_foopsi
(fluor, bl=None, c1=None, g=None, sn=None, p=None, method_deconvolution='oasis', bas_nonneg=True, noise_range=[0.25, 0.5], noise_method='logmexp', lags=5, fudge_factor=1.0, verbosity=False, solvers=None, optimize_g=0, s_min=None, **kwargs)¶ Infer the most likely discretized spike train underlying a fluorescence trace
It relies on a noise constrained deconvolution approach
- Args:
- fluor: np.ndarray
- One dimensional array containing the fluorescence intensities with one entry per time-bin.
- bl: [optional] float
- Fluorescence baseline value. If no value is given, then bl is estimated from the data.
- c1: [optional] float
- value of calcium at time 0
- g: [optional] list,float
- Parameters of the AR process that models the fluorescence impulse response. Estimated from the data if no value is given
- sn: float, optional
- Standard deviation of the noise distribution. If no value is given, then sn is estimated from the data.
- p: int
- order of the autoregression model
- method_deconvolution: [optional] string
- solution method for basis projection pursuit ‘cvx’ or ‘cvxpy’ or ‘oasis’
- bas_nonneg: bool
- baseline strictly non-negative
- noise_range: list of two elms
- frequency range for averaging noise PSD
- noise_method: string
- method of averaging noise PSD
- lags: int
- number of lags for estimating time constants
- fudge_factor: float
- fudge factor for reducing time constant bias
- verbosity: bool
- display optimization details
- solvers: list string
- primary and secondary (if problem unfeasible for approx solution) solvers to be used with cvxpy, default is [‘ECOS’,’SCS’]
- optimize_g : [optional] int, only applies to method ‘oasis’
- Number of large, isolated events to consider for optimizing g. If optimize_g=0 (default) the provided or estimated g is not further optimized.
- s_min : float, optional, only applies to method ‘oasis’
- Minimal non-zero activity within each bin (minimal ‘spike size’). For negative values the threshold is abs(s_min) * sn * sqrt(1-g) If None (default) the standard L1 penalty is used If 0 the threshold is determined automatically such that RSS <= sn^2 T
- Returns:
- c: np.ndarray float
- The inferred denoised fluorescence signal at each time-bin.
bl, c1, g, sn : As explained above
- sp: ndarray of float
- Discretized deconvolved neural activity (spikes)
- lam: float
- Regularization parameter
- Raises:
Exception(“You must specify the value of p”)
Exception(‘OASIS is currently only implemented for p=1 and p=2’)
Exception(‘Undefined Deconvolution Method’)
- References:
- Pnevmatikakis et al. 2016. Neuron, in press, http://dx.doi.org/10.1016/j.neuron.2015.11.037
- Machado et al. 2015. Cell 162(2):338-350
image: docs/img/deconvolution.png image: docs/img/evaluationcomponent.png
-
caiman.source_extraction.cnmf.deconvolution.
constrained_oasisAR2
(y, g, sn, optimize_b=True, b_nonneg=True, optimize_g=0, decimate=5, shift=100, window=None, tol=1e-09, max_iter=1, penalty=1, s_min=0)¶ Infer the most likely discretized spike train underlying an AR(2) fluorescence trace
Solves the noise constrained sparse non-negative deconvolution problem min (s)_1 subject to (c-y)^2 = sn^2 T and s_t = c_t-g1 c_{t-1}-g2 c_{t-2} >= 0
- Args:
- y : array of float
- One dimensional array containing the fluorescence intensities (with baseline already subtracted) with one entry per time-bin.
- g : (float, float)
- Parameters of the AR(2) process that models the fluorescence impulse response.
- sn : float
- Standard deviation of the noise distribution.
- optimize_b : bool, optional, default True
- Optimize baseline if True else it is set to 0, see y.
- b_nonneg: bool, optional, default True
- Enforce strictly non-negative baseline if True.
- optimize_g : int, optional, default 0
- Number of large, isolated events to consider for optimizing g. No optimization if optimize_g=0.
- decimate : int, optional, default 5
- Decimation factor for estimating hyper-parameters faster on decimated data.
- shift : int, optional, default 100
- Number of frames by which to shift window from on run of NNLS to the next.
- window : int, optional, default None (200 or larger dependend on g)
- Window size.
- tol : float, optional, default 1e-9
- Tolerance parameter.
- max_iter : int, optional, default 1
- Maximal number of iterations.
- penalty : int, optional, default 1
- Sparsity penalty. 1: min (s)_1 0: min (s)_0
- s_min : float, optional, default 0
- Minimal non-zero activity within each bin (minimal ‘spike size’). For negative values the threshold is |s_min| * sn * sqrt(1-decay_constant) If 0 the threshold is determined automatically such that RSS <= sn^2 T
- Returns:
- c : array of float
- The inferred denoised fluorescence signal at each time-bin.
- s : array of float
- Discretized deconvolved neural activity (spikes).
- b : float
- Fluorescence baseline value.
- (g1, g2) : tuple of float
- Parameters of the AR(2) process that models the fluorescence impulse response.
- lam : float
- Sparsity penalty parameter lambda of dual problem.
- References:
- Friedrich J and Paninski L, NIPS 2016 Friedrich J, Zhou P, and Paninski L, arXiv 2016
Parameter Setting¶
-
class
caiman.source_extraction.cnmf.params.
CNMFParams
(fnames=None, dims=None, dxy=(1, 1), border_pix=0, del_duplicates=False, low_rank_background=True, memory_fact=1, n_processes=1, nb_patch=1, p_ssub=2, p_tsub=2, remove_very_bad_comps=False, rf=None, stride=None, check_nan=True, n_pixels_per_process=None, k=30, alpha_snmf=100, center_psf=False, gSig=[5, 5], gSiz=None, init_iter=2, method_init='greedy_roi', min_corr=0.85, min_pnr=20, gnb=1, normalize_init=True, options_local_NMF=None, ring_size_factor=1.5, rolling_length=100, rolling_sum=True, ssub=2, ssub_B=2, tsub=2, block_size_spat=5000, num_blocks_per_run_spat=20, block_size_temp=5000, num_blocks_per_run_temp=20, update_background_components=True, method_deconvolution='oasis', p=2, s_min=None, do_merge=True, merge_thresh=0.8, decay_time=0.4, fr=30, min_SNR=2.5, rval_thr=0.8, N_samples_exceptionality=None, batch_update_suff_stat=False, expected_comps=500, iters_shape=5, max_comp_update_shape=inf, max_num_added=5, min_num_trial=5, minibatch_shape=100, minibatch_suff_stat=5, n_refit=0, num_times_comp_updated=inf, simultaneously=False, sniper_mode=False, test_both=False, thresh_CNN_noisy=0.5, thresh_fitness_delta=-50, thresh_fitness_raw=None, thresh_overlap=0.5, update_freq=200, update_num_comps=True, use_dense=True, use_peak_max=True, only_init_patch=True, var_name_hdf5='mov', max_merge_area=None, use_corr_img=False, params_dict={})¶ Class for setting and changing the various parameters.
Methods
change_params
(params_dict[, verbose])Method for updating the params object by providing a single dictionary. check_consistency
()Populates the params object with some dataset dependent values and ensures that certain constraints are satisfied. get
(group, key)Get a value for a given group and key. get_group
(group)Get the dictionary of key-value pairs for a group. set
(group, val_dict[, set_if_not_exists, …])Add key-value pairs to a group. Existing key-value pairs will be overwritten to_dict
()Returns the params class as a dictionary with subdictionaries for each catergory.
-
CNMFParams.
__init__
(fnames=None, dims=None, dxy=(1, 1), border_pix=0, del_duplicates=False, low_rank_background=True, memory_fact=1, n_processes=1, nb_patch=1, p_ssub=2, p_tsub=2, remove_very_bad_comps=False, rf=None, stride=None, check_nan=True, n_pixels_per_process=None, k=30, alpha_snmf=100, center_psf=False, gSig=[5, 5], gSiz=None, init_iter=2, method_init='greedy_roi', min_corr=0.85, min_pnr=20, gnb=1, normalize_init=True, options_local_NMF=None, ring_size_factor=1.5, rolling_length=100, rolling_sum=True, ssub=2, ssub_B=2, tsub=2, block_size_spat=5000, num_blocks_per_run_spat=20, block_size_temp=5000, num_blocks_per_run_temp=20, update_background_components=True, method_deconvolution='oasis', p=2, s_min=None, do_merge=True, merge_thresh=0.8, decay_time=0.4, fr=30, min_SNR=2.5, rval_thr=0.8, N_samples_exceptionality=None, batch_update_suff_stat=False, expected_comps=500, iters_shape=5, max_comp_update_shape=inf, max_num_added=5, min_num_trial=5, minibatch_shape=100, minibatch_suff_stat=5, n_refit=0, num_times_comp_updated=inf, simultaneously=False, sniper_mode=False, test_both=False, thresh_CNN_noisy=0.5, thresh_fitness_delta=-50, thresh_fitness_raw=None, thresh_overlap=0.5, update_freq=200, update_num_comps=True, use_dense=True, use_peak_max=True, only_init_patch=True, var_name_hdf5='mov', max_merge_area=None, use_corr_img=False, params_dict={})¶ Class for setting the processing parameters. All parameters for CNMF, online-CNMF, quality testing, and motion correction can be set here and then used in the various processing pipeline steps. The prefered way to set parameters is by using the set function, where a subclass is determined and a dictionary is passed. The whole dictionary can also be initialized at once by passing a dictionary params_dict when initializing the CNMFParams object. Direct setting of the positional arguments in CNMFParams is only present for backwards compatibility reasons and should not be used if possible.
- Args:
- Any parameter that is not set get a default value specified by the dictionary default options
DATA PARAMETERS (CNMFParams.data) #####
- fnames: list[str]
- list of complete paths to files that need to be processed
- dims: (int, int), default: computed from fnames
- dimensions of the FOV in pixels
- fr: float, default: 30
- imaging rate in frames per second
- decay_time: float, default: 0.4
- length of typical transient in seconds
- dxy: (float, float)
- spatial resolution of FOV in pixels per um
- var_name_hdf5: str, default: ‘mov’
- if loading from hdf5 name of the variable to load
- caiman_version: str
- version of CaImAn being used
- last_commit: str
- hash of last commit in the caiman repo
- mmap_F: list[str]
- paths to F-order memory mapped files after motion correction
- mmap_C: str
- path to C-order memory mapped file after motion correction
PATCH PARAMS (CNMFParams.patch)######
- rf: int or list or None, default: None
- Half-size of patch in pixels. If None, no patches are constructed and the whole FOV is processed jointly. If list, it should be a list of two elements corresponding to the height and width of patches
- stride: int or None, default: None
- Overlap between neighboring patches in pixels.
- nb_patch: int, default: 1
- Number of (local) background components per patch
- border_pix: int, default: 0
- Number of pixels to exclude around each border.
- low_rank_background: bool, default: True
- Whether to update the background using a low rank approximation. If False all the nonzero elements of the background components are updated using hals (to be used with one background per patch)
- del_duplicates: bool, default: False
- Delete duplicate components in the overlaping regions between neighboring patches. If False, then merging is used.
- only_init: bool, default: True
- whether to run only the initialization
- p_patch: int, default: 0
- order of AR dynamics when processing within a patch
- skip_refinement: bool, default: False
- Whether to skip refinement of components (deprecated?)
- remove_very_bad_comps: bool, default: True
- Whether to remove (very) bad quality components during patch processing
- p_ssub: float, default: 2
- Spatial downsampling factor
- p_tsub: float, default: 2
- Temporal downsampling factor
- memory_fact: float, default: 1
- unitless number for increasing the amount of available memory
- n_processes: int
- Number of processes used for processing patches in parallel
- in_memory: bool, default: True
- Whether to load patches in memory
PRE-PROCESS PARAMS (CNMFParams.preprocess) #############
- sn: np.array or None, default: None
- noise level for each pixel
- noise_range: [float, float], default: [.25, .5]
- range of normalized frequencies over which to compute the PSD for noise determination
- noise_method: ‘mean’|’median’|’logmexp’, default: ‘mean’
- PSD averaging method for computing the noise std
- max_num_samples_fft: int, default: 3*1024
- Chunk size for computing the PSD of the data (for memory considerations)
- n_pixels_per_process: int, default: 1000
- Number of pixels to be allocated to each process
- compute_g’: bool, default: False
- whether to estimate global time constant
- p: int, default: 2
- order of AR indicator dynamics
- lags: int, default: 5
- number of lags to be considered for time constant estimation
- include_noise: bool, default: False
- flag for using noise values when estimating g
- pixels: list, default: None
- pixels to be excluded due to saturation
- check_nan: bool, default: True
- whether to check for NaNs
INIT PARAMS (CNMFParams.init)###############
- K: int, default: 30
- number of components to be found (per patch or whole FOV depending on whether rf=None)
- SC_kernel: {‘heat’, ‘cos’, binary’}, default: ‘heat’
- kernel for graph affinity matrix
- SC_sigma: float, default: 1
- variance for SC kernel
- SC_thr: float, default: 0,
- threshold for affinity matrix
- SC_normalize: bool, default: True
- standardize entries prior to computing the affinity matrix
- SC_use_NN: bool, default: False
- sparsify affinity matrix by using only nearest neighbors
- SC_nnn: int, default: 20
- number of nearest neighbors to use
- gSig: [int, int], default: [5, 5]
- radius of average neurons (in pixels)
- gSiz: [int, int], default: [int(round((x * 2) + 1)) for x in gSig],
- half-size of bounding box for each neuron
- center_psf: bool, default: False
- whether to use 1p data processing mode. Set to true for 1p
- ssub: float, default: 2
- spatial downsampling factor
- tsub: float, default: 2
- temporal downsampling factor
- nb: int, default: 1
- number of background components
- lambda_gnmf: float, default: 1.
- regularization weight for graph NMF
- maxIter: int, default: 5
- number of HALS iterations during initialization
- method_init: ‘greedy_roi’|’corr_pnr’|’sparse_NMF’|’local_NMF’ default: ‘greedy_roi’
- initialization method. use ‘corr_pnr’ for 1p processing and ‘sparse_NMF’ for dendritic processing.
- min_corr: float, default: 0.85
- minimum value of correlation image for determining a candidate component during corr_pnr
- min_pnr: float, default: 20
- minimum value of psnr image for determining a candidate component during corr_pnr
- seed_method: str {‘auto’, ‘manual’, ‘semi’}
- methods for choosing seed pixels during greedy_roi or corr_pnr initialization ‘semi’ detects nr components automatically and allows to add more manually if running as notebook ‘semi’ and ‘manual’ require a backend that does not inline figures, e.g. %matplotlib tk
- ring_size_factor: float, default: 1.5
- radius of ring (*gSig) for computing background during corr_pnr
- ssub_B: float, default: 2
- downsampling factor for background during corr_pnr
- init_iter: int, default: 2
- number of iterations during corr_pnr (1p) initialization
- nIter: int, default: 5
- number of rank-1 refinement iterations during greedy_roi initialization
- rolling_sum: bool, default: True
- use rolling sum (as opposed to full sum) for determining candidate centroids during greedy_roi
- rolling_length: int, default: 100
- width of rolling window for rolling sum option
- kernel: np.array or None, default: None
- user specified template for greedyROI
- max_iter_snmf : int, default: 500
- maximum number of iterations for sparse NMF initialization
- alpha_snmf: float, default: 100
- sparse NMF sparsity regularization weight
- sigma_smooth_snmf : (float, float, float), default: (.5,.5,.5)
- std of Gaussian kernel for smoothing data in sparse_NMF
- perc_baseline_snmf: float, default: 20
- percentile to be removed from the data in sparse_NMF prior to decomposition
- normalize_init: bool, default: True
- whether to equalize the movies during initialization
- options_local_NMF: dict
- dictionary with parameters to pass to local_NMF initializer
SPATIAL PARAMS (CNMFParams.spatial) ##########
- method_exp: ‘dilate’|’ellipse’, default: ‘dilate’
- method for expanding footprint of spatial components
- dist: float, default: 3
- expansion factor of ellipse
- expandCore: morphological element, default: None(?)
- morphological element for expanding footprints under dilate
- nb: int, default: 1
- number of global background components
- n_pixels_per_process: int, default: 1000
- number of pixels to be processed by each worker
- thr_method: ‘nrg’|’max’, default: ‘nrg’
- thresholding method
- maxthr: float, default: 0.1
- Max threshold
- nrgthr: float, default: 0.9999
- Energy threshold
- extract_cc: bool, default: True
- whether to extract connected components during thresholding (might want to turn to False for dendritic imaging)
- medw: (int, int) default: None
- window of median filter (set to (3,)*len(dims) in cnmf.fit)
- se: np.array or None, default: None
- Morphological closing structuring element (set to np.ones((3,)*len(dims), dtype=np.uint8) in cnmf.fit)
- ss: np.array or None, default: None
- Binary element for determining connectivity (set to np.ones((3,)*len(dims), dtype=np.uint8) in cnmf.fit)
- update_background_components: bool, default: True
- whether to update the spatial background components
- method_ls: ‘lasso_lars’|’nnls_L0’, default: ‘lasso_lars’
- ‘nnls_L0’. Nonnegative least square with L0 penalty ‘lasso_lars’ lasso lars function from scikit learn
- block_size : int, default: 5000
- Number of pixels to process at the same time for dot product. Reduce if you face memory problems
- num_blocks_per_run: int, default: 20
- Parallelization of A’*Y operation
- normalize_yyt_one: bool, default: True
- Whether to normalize the C and A matrices so that diag(C*C.T) = 1 during update spatial
TEMPORAL PARAMS (CNMFParams.temporal)###########
- ITER: int, default: 2
- block coordinate descent iterations
- method_deconvolution: ‘oasis’|’cvxpy’|’oasis’, default: ‘oasis’
- method for solving the constrained deconvolution problem (‘oasis’,’cvx’ or ‘cvxpy’) if method cvxpy, primary and secondary (if problem unfeasible for approx solution)
- solvers: ‘ECOS’|’SCS’, default: [‘ECOS’, ‘SCS’]
- solvers to be used with cvxpy, can be ‘ECOS’,’SCS’ or ‘CVXOPT’
- p: 0|1|2, default: 2
- order of AR indicator dynamics
memory_efficient: False
- bas_nonneg: bool, default: True
- whether to set a non-negative baseline (otherwise b >= min(y))
- noise_range: [float, float], default: [.25, .5]
- range of normalized frequencies over which to compute the PSD for noise determination
- noise_method: ‘mean’|’median’|’logmexp’, default: ‘mean’
- PSD averaging method for computing the noise std
- lags: int, default: 5
- number of autocovariance lags to be considered for time constant estimation
- optimize_g: bool, default: False
- flag for optimizing time constants
- fudge_factor: float (close but smaller than 1) default: .96
- bias correction factor for discrete time constants
- nb: int, default: 1
- number of global background components
- verbosity: bool, default: False
- whether to be verbose
- block_size : int, default: 5000
- Number of pixels to process at the same time for dot product. Reduce if you face memory problems
- num_blocks_per_run: int, default: 20
- Parallelization of A’*Y operation
- s_min: float or None, default: None
- Minimum spike threshold amplitude (computed in the code if used).
- MERGE PARAMS (CNMFParams.merge)#####
- do_merge: bool, default: True
- Whether or not to merge
- thr: float, default: 0.8
- Trace correlation threshold for merging two components.
- merge_parallel: bool, default: False
- Perform merging in parallel
- max_merge_area: int or None, default: None
- maximum area (in pixels) of merged components, used to determine whether to merge components during fitting process
QUALITY EVALUATION PARAMETERS (CNMFParams.quality)###########
- min_SNR: float, default: 2.5
- trace SNR threshold. Traces with SNR above this will get accepted
- SNR_lowest: float, default: 0.5
- minimum required trace SNR. Traces with SNR below this will get rejected
- rval_thr: float, default: 0.8
- space correlation threshold. Components with correlation higher than this will get accepted
- rval_lowest: float, default: -1
- minimum required space correlation. Components with correlation below this will get rejected
- use_cnn: bool, default: True
- flag for using the CNN classifier.
- min_cnn_thr: float, default: 0.9
- CNN classifier threshold. Components with score higher than this will get accepted
- cnn_lowest: float, default: 0.1
- minimum required CNN threshold. Components with score lower than this will get rejected.
- gSig_range: list or integers, default: None
- gSig scale values for CNN classifier. In not None, multiple values are tested in the CNN classifier.
ONLINE CNMF (ONACID) PARAMETERS (CNMFParams.online)#####
- N_samples_exceptionality: int, default: np.ceil(decay_time*fr),
- Number of frames over which trace SNR is computed (usually length of a typical transient)
- batch_update_suff_stat: bool, default: False
- Whether to update sufficient statistics in batch mode
- ds_factor: int, default: 1,
- spatial downsampling factor for faster processing (if > 1)
- dist_shape_update: bool, default: False,
- update shapes in a distributed fashion
- epochs: int, default: 1,
- number of times to go over data
- expected_comps: int, default: 500
- number of expected components (for memory allocation purposes)
- full_XXt: bool, default: False
- save the full residual sufficient statistic matrix for updating W in 1p. If set to False, a list of submatrices is saved (typically faster).
- init_batch: int, default: 200,
- length of mini batch used for initialization
- init_method: ‘bare’|’cnmf’|’seeded’, default: ‘bare’,
- initialization method
- iters_shape: int, default: 5
- Number of block-coordinate decent iterations for each shape update
- max_comp_update_shape: int, default: np.inf
- Maximum number of spatial components to be updated at each time
- max_num_added: int, default: 5
- Maximum number of new components to be added in each frame
- max_shifts_online: int, default: 10,
- Maximum shifts for motion correction during online processing
- min_SNR: float, default: 2.5
- Trace SNR threshold for accepting a new component
- min_num_trial: int, default: 5
- Number of mew possible components for each frame
- minibatch_shape: int, default: 100
- Number of frames stored in rolling buffer
- minibatch_suff_stat: int, default: 5
- mini batch size for updating sufficient statistics
- motion_correct: bool, default: True
- Whether to perform motion correction during online processing
- movie_name_online: str, default: ‘online_movie.avi’
- Name of saved movie (appended in the data directory)
- normalize: bool, default: False
- Whether to normalize each frame prior to online processing
- n_refit: int, default: 0
- Number of additional iterations for computing traces
num_times_comp_updated: int, default: np.inf
- opencv_codec: str, default: ‘H264’
- FourCC video codec for saving movie. Check http://www.fourcc.org/codecs.php
- path_to_model: str, default: os.path.join(caiman_datadir(), ‘model’, ‘cnn_model_online.h5’)
- Path to online CNN classifier
- rval_thr: float, default: 0.8
- space correlation threshold for accepting a new component
- save_online_movie: bool, default: False
- Whether to save the results movie
- show_movie: bool, default: False
- Whether to display movie of online processing
- simultaneously: bool, default: False
- Whether to demix and deconvolve simultaneously
- sniper_mode: bool, default: False
- Whether to use the online CNN classifier for screening candidate components (otherwise space correlation is used)
- test_both: bool, default: False
- Whether to use both the CNN and space correlation for screening new components
- thresh_CNN_noisy: float, default: 0,5,
- Threshold for the online CNN classifier
- thresh_fitness_delta: float (negative)
- Derivative test for detecting traces
- thresh_fitness_raw: float (negative), default: computed from min_SNR
- Threshold value for testing trace SNR
- thresh_overlap: float, default: 0.5
- Intersection-over-Union space overlap threshold for screening new components
- update_freq: int, default: 200
- Update each shape at least once every X frames when in distributed mode
- update_num_comps: bool, default: True
- Whether to search for new components
- use_dense: bool, default: True
- Whether to store and represent A and b as a dense matrix
- use_peak_max: bool, default: True
- Whether to find candidate centroids using skimage’s find local peaks function
MOTION CORRECTION PARAMETERS (CNMFParams.motion)####
- border_nan: bool or str, default: ‘copy’
- flag for allowing NaN in the boundaries. True allows NaN, whereas ‘copy’ copies the value of the nearest data point.
- gSig_filt: int or None, default: None
- size of kernel for high pass spatial filtering in 1p data. If None no spatial filtering is performed
- is3D: bool, default: False
- flag for 3D recordings for motion correction
- max_deviation_rigid: int, default: 3
- maximum deviation in pixels between rigid shifts and shifts of individual patches
- max_shifts: (int, int), default: (6,6)
- maximum shifts per dimension in pixels.
- min_mov: float or None, default: None
- minimum value of movie. If None it get computed.
- niter_rig: int, default: 1
- number of iterations rigid motion correction.
- nonneg_movie: bool, default: True
- flag for producing a non-negative movie.
- num_frames_split: int, default: 80
- split movie every x frames for parallel processing
num_splits_to_process_els, default: [7, None] num_splits_to_process_rig, default: None
- overlaps: (int, int), default: (24, 24)
- overlap between patches in pixels in pw-rigid motion correction.
- pw_rigid: bool, default: False
- flag for performing pw-rigid motion correction.
- shifts_opencv: bool, default: True
- flag for applying shifts using cubic interpolation (otherwise FFT)
- splits_els: int, default: 14
- number of splits across time for pw-rigid registration
- splits_rig: int, default: 14
- number of splits across time for rigid registration
- strides: (int, int), default: (96, 96)
- how often to start a new patch in pw-rigid registration. Size of each patch will be strides + overlaps
- upsample_factor_grid” int, default: 4
- motion field upsampling factor during FFT shifts.
- use_cuda: bool, default: False
- flag for using a GPU.
- indices: tuple(slice), default: (slice(None), slice(None))
- Use that to apply motion correction only on a part of the FOV
RING CNN PARAMETERS (CNMFParams.ring_CNN)
- n_channels: int, default: 2
- Number of “ring” kernels
- use_bias: bool, default: False
- Flag for using bias in the convolutions
- use_add: bool, default: False
- Flag for using an additive layer
- pct: float between 0 and 1, default: 0.01
- Quantile used during training with quantile loss function
- patience: int, default: 3
- Number of epochs to wait before early stopping
- max_epochs: int, default: 100
- Maximum number of epochs to be used during training
- width: int, default: 5
- Width of “ring” kernel
- loss_fn: str, default: ‘pct’
- Loss function specification (‘pct’ for quantile loss function, ‘mse’ for mean squared error)
- lr: float, default: 1e-3
- (initial) learning rate
- lr_scheduler: function, default: None
- Learning rate scheduler function
- path_to_model: str, default: None
- Path to saved weights (if training then path to saved model weights)
- remove_activity: bool, default: False
- Flag for removing activity of last frame prior to background extraction
- reuse_model: bool, default: False
- Flag for reusing an already trained model (saved in path to model)
-
CNMFParams.
set
(group, val_dict, set_if_not_exists=False, verbose=False)¶ - Add key-value pairs to a group. Existing key-value pairs will be overwritten
- if specified in val_dict, but not deleted.
- Args:
- group: The name of the group. val_dict: A dictionary with key-value pairs to be set for the group. set_if_not_exists: Whether to set a key-value pair in a group if the key does not currently exist in the group.
-
CNMFParams.
get
(group, key)¶ Get a value for a given group and key. Raises an exception if no such group/key combination exists.
- Args:
- group: The name of the group. key: The key for the property in the group of interest.
Returns: The value for the group/key combination.
-
CNMFParams.
get_group
(group)¶ Get the dictionary of key-value pairs for a group.
- Args:
- group: The name of the group.
-
CNMFParams.
change_params
(params_dict, verbose=False)¶ Method for updating the params object by providing a single dictionary. For each key in the provided dictionary the method will search in all subdictionaries and will update the value if it finds a match.
- Args:
- params_dict: dictionary with parameters to be changed and new values verbose: bool (False). Print message for all keys
-
CNMFParams.
to_dict
()¶ Returns the params class as a dictionary with subdictionaries for each catergory.
CNMF¶
-
class
caiman.source_extraction.cnmf.cnmf.
CNMF
(n_processes, k=5, gSig=[4, 4], gSiz=None, merge_thresh=0.8, p=2, dview=None, Ain=None, Cin=None, b_in=None, f_in=None, do_merge=True, ssub=2, tsub=2, p_ssub=1, p_tsub=1, method_init='greedy_roi', alpha_snmf=100, rf=None, stride=None, memory_fact=1, gnb=1, nb_patch=1, only_init_patch=False, method_deconvolution='oasis', n_pixels_per_process=4000, block_size_temp=5000, num_blocks_per_run_temp=20, block_size_spat=5000, num_blocks_per_run_spat=20, check_nan=True, skip_refinement=False, normalize_init=True, options_local_NMF=None, minibatch_shape=100, minibatch_suff_stat=3, update_num_comps=True, rval_thr=0.9, thresh_fitness_delta=-20, thresh_fitness_raw=None, thresh_overlap=0.5, max_comp_update_shape=inf, num_times_comp_updated=inf, batch_update_suff_stat=False, s_min=None, remove_very_bad_comps=False, border_pix=0, low_rank_background=True, update_background_components=True, rolling_sum=True, rolling_length=100, min_corr=0.85, min_pnr=20, ring_size_factor=1.5, center_psf=False, use_dense=True, deconv_flag=True, simultaneously=False, n_refit=0, del_duplicates=False, N_samples_exceptionality=None, max_num_added=3, min_num_trial=2, thresh_CNN_noisy=0.5, fr=30, decay_time=0.4, min_SNR=2.5, ssub_B=2, init_iter=2, sniper_mode=False, use_peak_max=False, test_both=False, expected_comps=500, max_merge_area=None, params=None)¶ Source extraction using constrained non-negative matrix factorization.
The general class which is used to produce a factorization of the Y matrix being the video it computes it using all the files inside of cnmf folder. Its architecture is similar to the one of scikit-learn calling the function fit to run everything which is part of the structure of the class
it is calling everyfunction from the cnmf folder you can find out more at how the functions are called and how they are laid out at the ipython notebook
See Also: @url http://www.cell.com/neuron/fulltext/S0896-6273(15)01084-3 .. image:: docs/img/quickintro.png @author andrea giovannucci
Methods
HALS4footprints
(Yr[, update_bck, num_iter])Uses hierarchical alternating least squares to update shapes and background HALS4traces
(Yr[, groups, use_groups, order, …])Solves C, f = argmin_C ||Yr-AC-bf|| using block-coordinate decent. compute_residuals
(Yr)Compute residual trace for each component (variable YrA). deconvolve
([p, method_deconvolution, …])Performs deconvolution on already extracted traces using constrained foopsi. fit
(images[, indices])This method uses the cnmf algorithm to find sources in data. fit_file
([motion_correct, indices, include_eval])This method packages the analysis pipeline (motion correction, memory mapping, patch based CNMF processing and component evaluation) in a single method that can be called on a specific (sequence of) file(s). initialize
(Y, **kwargs)Component initialization merge_comps
(Y[, mx, fast_merge, max_merge_area])merges components preprocess
(Yr)Examines data to remove corrupted pixels and computes the noise level estimate for each pixel. refit
(images[, dview])Refits the data using CNMF initialized from a previous interation remove_components
(ind_rm)Remove a specified list of components from the CNMF object. save
(filename)save object in hdf5 file format update_spatial
(Y[, use_init])Updates spatial components update_temporal
(Y[, use_init])Updates temporal components
-
CNMF.
fit
(images, indices=(slice(None, None, None), slice(None, None, None)))¶ This method uses the cnmf algorithm to find sources in data. it is calling every function from the cnmf folder you can find out more at how the functions are called and how they are laid out at the ipython notebook
- Args:
images : mapped np.ndarray of shape (t,x,y[,z]) containing the images that vary over time.
indices: list of slice objects along dimensions (x,y[,z]) for processing only part of the FOV
- Returns:
- self: updated using the cnmf algorithm with C,A,S,b,f computed according to the given initial values
Raises: Exception ‘You need to provide a memory mapped file as input if you use patches!!’
See Also: ..image::docs/img/quickintro.png
-
CNMF.
refit
(images, dview=None)¶ Refits the data using CNMF initialized from a previous interation
- Args:
- images dview
- Returns:
- cnm
- A new CNMF object
-
CNMF.
fit_file
(motion_correct=False, indices=None, include_eval=False)¶ This method packages the analysis pipeline (motion correction, memory mapping, patch based CNMF processing and component evaluation) in a single method that can be called on a specific (sequence of) file(s). It is assumed that the CNMF object already contains a params object where the location of the files and all the relevant parameters have been specified. The method will perform the last step, i.e. component evaluation, if the flag “include_eval” is set to True.
- Args:
- motion_correct (bool)
- flag for performing motion correction
- indices (list of slice objects)
- perform analysis only on a part of the FOV
- include_eval (bool)
- flag for performing component evaluation
- Returns:
- cnmf object with the current estimates
-
CNMF.
save
(filename)¶ save object in hdf5 file format
- Args:
- filename: str
- path to the hdf5 file containing the saved object
-
CNMF.
deconvolve
(p=None, method_deconvolution=None, bas_nonneg=None, noise_method=None, optimize_g=0, s_min=None, **kwargs)¶ Performs deconvolution on already extracted traces using constrained foopsi.
-
CNMF.
update_spatial
(Y, use_init=True, **kwargs)¶ Updates spatial components
- Args:
- Y: np.array (d1*d2) x T
- input data
- use_init: bool
- use Cin, f_in for computing A, b otherwise use C, f
- Returns:
- self
- modified values self.estimates.A, self.estimates.b possibly self.estimates.C, self.estimates.f
-
CNMF.
update_temporal
(Y, use_init=True, **kwargs)¶ Updates temporal components
- Args:
- Y: np.array (d1*d2) x T
- input data
-
CNMF.
compute_residuals
(Yr)¶ Compute residual trace for each component (variable YrA). WARNING: At the moment this method is valid only for the 2p processing pipeline
- Args:
- Yr : np.ndarray
- movie in format pixels (d) x frames (T)
-
CNMF.
remove_components
(ind_rm)¶ Remove a specified list of components from the CNMF object.
- Args:
- ind_rm : list
- indices of components to be removed
-
CNMF.
HALS4traces
(Yr, groups=None, use_groups=False, order=None, update_bck=True, bck_non_neg=True, **kwargs)¶ Solves C, f = argmin_C ||Yr-AC-bf|| using block-coordinate decent. Can use groups to update non-overlapping components in parallel or a specified order.
- Args:
- Yr : np.array (possibly memory mapped, (x,y,[,z]) x t)
- Imaging data reshaped in matrix format
- groups : list of sets
- grouped components to be updated simultaneously
- use_groups : bool
- flag for using groups
- order : list
- Update components in that order (used if nonempty and groups=None)
- update_bck : bool
- Flag for updating temporal background components
- bck_non_neg : bool
- Require temporal background to be non-negative
- Returns:
- self (updated values for self.estimates.C, self.estimates.f, self.estimates.YrA)
-
CNMF.
HALS4footprints
(Yr, update_bck=True, num_iter=2)¶ Uses hierarchical alternating least squares to update shapes and background
- Args:
- Yr: np.array (possibly memory mapped, (x,y,[,z]) x t)
- Imaging data reshaped in matrix format
- update_bck: bool
- flag for updating spatial background components
- num_iter: int
- number of iterations
- Returns:
- self (updated values for self.estimates.A and self.estimates.b)
-
CNMF.
merge_comps
(Y, mx=50, fast_merge=True, max_merge_area=None)¶ merges components
-
CNMF.
initialize
(Y, **kwargs)¶ Component initialization
-
CNMF.
preprocess
(Yr)¶ Examines data to remove corrupted pixels and computes the noise level estimate for each pixel.
- Args:
- Yr: np.array (or memmap.array)
- 2d array of data (pixels x timesteps) typically in memory mapped form
-
caiman.source_extraction.cnmf.cnmf.
load_CNMF
(filename, n_processes=1, dview=None)¶ load object saved with the CNMF save method
- Args:
- filename: str
- hdf5 (or nwb) file name containing the saved object
- dview: multiprocessing or ipyparallel object
- useful to set up parllelization in the objects
Online CNMF (OnACID)¶
-
class
caiman.source_extraction.cnmf.online_cnmf.
OnACID
(params=None, estimates=None, path=None, dview=None, Ain=None)¶ Source extraction of streaming data using online matrix factorization. The class can be initialized by passing a “params” object for setting up the relevant parameters and an “Estimates” object for setting an initial state of the algorithm (optional)
- Methods:
- initialize_online:
- Initialize the online algorithm using a provided method, and prepare the online object
- _prepare_object:
- Prepare the online object given a set of estimates
- fit_next:
- Fit the algorithm on the next data frame
- fit_online:
- Run the entire online pipeline on a given list of files
Methods
fit_next
(t, frame_in[, num_iters_hals])This method fits the next frame using the CaImAn online algorithm and updates the object. fit_online
(**kwargs)Implements the caiman online algorithm on the list of files fls. save
(filename)save object in hdf5 file format create_frame initialize_online mc_next
-
OnACID.
fit_online
(**kwargs)¶ Implements the caiman online algorithm on the list of files fls. The files are taken in alpha numerical order and are assumed to each have the same number of frames (except the last one that can be shorter). Caiman online is initialized using the seeded or bare initialization methods.
- Args:
- fls: list
- list of files to be processed
- init_batch: int
- number of frames to be processed during initialization
- epochs: int
- number of passes over the data
- motion_correct: bool
- flag for performing motion correction
- kwargs: dict
- additional parameters used to modify self.params.online’] see options.[‘online’] for details
- Returns:
- self (results of caiman online)
-
OnACID.
fit_next
(t, frame_in, num_iters_hals=3)¶ This method fits the next frame using the CaImAn online algorithm and updates the object.
- Args
- t : int
- time measured in number of frames
- frame_in : array
- flattened array of shape (x * y [ * z],) containing the t-th image.
- num_iters_hals: int, optional
- maximal number of iterations for HALS (NNLS via blockCD)
-
OnACID.
save
(filename)¶ save object in hdf5 file format
- Args:
- filename: str
- path to the hdf5 file containing the saved object
-
OnACID.
initialize_online
(model_LN=None, T=None)¶
-
caiman.source_extraction.cnmf.online_cnmf.
load_OnlineCNMF
(filename, dview=None)¶ load object saved with the CNMF save method
- Args:
- filename: str
- hdf5 file name containing the saved object
- dview: multiprocessing or ipyparallel object
- useful to set up parllelization in the objects
Preprocessing¶
-
caiman.source_extraction.cnmf.pre_processing.
preprocess_data
(Y, sn=None, dview=None, n_pixels_per_process=100, noise_range=[0.25, 0.5], noise_method='logmexp', compute_g=False, p=2, lags=5, include_noise=False, pixels=None, max_num_samples_fft=3000, check_nan=True)¶ Performs the pre-processing operations described above.
- Args:
- Y: ndarray
- input movie (n_pixels x Time). Can be also memory mapped file.
- n_processes: [optional] int
- number of processes/threads to use concurrently
- n_pixels_per_process: [optional] int
- number of pixels to be simultaneously processed by each process
- p: positive integer
- order of AR process, default: 2
- lags: positive integer
- number of lags in the past to consider for determining time constants. Default 5
- include_noise: Boolean
- Flag to include pre-estimated noise value when determining time constants. Default: False
- noise_range: np.ndarray [2 x 1] between 0 and 0.5
- Range of frequencies compared to Nyquist rate over which the power spectrum is averaged default: [0.25,0.5]
- noise method: string
- method of averaging the noise. Choices: ‘mean’: Mean ‘median’: Median ‘logmexp’: Exponential of the mean of the logarithm of PSD (default)
- Returns:
- Y: ndarray
- movie preprocessed (n_pixels x Time). Can be also memory mapped file.
- g: np.ndarray (p x 1)
- Discrete time constants
- psx: ndarray
- position of thoses pixels
- sn_s: ndarray (memory mapped)
- file where to store the results of computation.
Initialization¶
-
caiman.source_extraction.cnmf.initialization.
initialize_components
(Y, K=30, gSig=[5, 5], gSiz=None, ssub=1, tsub=1, nIter=5, maxIter=5, nb=1, kernel=None, use_hals=True, normalize_init=True, img=None, method_init='greedy_roi', max_iter_snmf=500, alpha_snmf=1000.0, sigma_smooth_snmf=(0.5, 0.5, 0.5), perc_baseline_snmf=20, options_local_NMF=None, rolling_sum=False, rolling_length=100, sn=None, options_total=None, min_corr=0.8, min_pnr=10, seed_method='auto', ring_size_factor=1.5, center_psf=False, ssub_B=2, init_iter=2, remove_baseline=True, SC_kernel='heat', SC_sigma=1, SC_thr=0, SC_normalize=True, SC_use_NN=False, SC_nnn=20, lambda_gnmf=1)¶ Initalize components. This function initializes the spatial footprints, temporal components, and background which are then further refined by the CNMF iterations. There are four different initialization methods depending on the data you’re processing:
‘greedy_roi’: GreedyROI method used in standard 2p processing (default) ‘corr_pnr’: GreedyCorr method used for processing 1p data ‘sparse_nmf’: Sparse NMF method suitable for dendritic/axonal imaging ‘graph_nmf’: Graph NMF method also suitable for dendritic/axonal imagingThe GreedyROI method by default is not using the RollingGreedyROI method. This can be changed through the binary flag ‘rolling_sum’.
All the methods can be used for volumetric data except ‘corr_pnr’ which is only available for 2D data.
It is also by default followed by hierarchical alternative least squares (HALS) NMF. Optional use of spatio-temporal downsampling to boost speed.
- Args:
- Y: np.ndarray
- d1 x d2 [x d3] x T movie, raw data.
- K: [optional] int
- number of neurons to extract (default value: 30). Maximal number for method ‘corr_pnr’.
- tau: [optional] list,tuple
- standard deviation of neuron size along x and y [and z] (default value: (5,5).
- gSiz: [optional] list,tuple
- size of kernel (default 2*tau + 1).
- nIter: [optional] int
- number of iterations for shape tuning (default 5).
- maxIter: [optional] int
- number of iterations for HALS algorithm (default 5).
- ssub: [optional] int
- spatial downsampling factor recommended for large datasets (default 1, no downsampling).
- tsub: [optional] int
- temporal downsampling factor recommended for long datasets (default 1, no downsampling).
- kernel: [optional] np.ndarray
- User specified kernel for greedyROI (default None, greedy ROI searches for Gaussian shaped neurons)
- use_hals: [optional] bool
- Whether to refine components with the hals method
- normalize_init: [optional] bool
- Whether to normalize_init data before running the initialization
- img: optional [np 2d array]
- Image with which to normalize. If not present use the mean + offset
- method_init: {‘greedy_roi’, ‘corr_pnr’, ‘sparse_nmf’, ‘graph_nmf’, ‘pca_ica’}
- Initialization method (default: ‘greedy_roi’)
- max_iter_snmf: int
- Maximum number of sparse NMF iterations
- alpha_snmf: scalar
- Sparsity penalty
- rolling_sum: boolean
- Detect new components based on a rolling sum of pixel activity (default: False)
- rolling_length: int
- Length of rolling window (default: 100)
- center_psf: Boolean
- True indicates centering the filtering kernel for background removal. This is useful for data with large background fluctuations.
- min_corr: float
- minimum local correlation coefficients for selecting a seed pixel.
- min_pnr: float
- minimum peak-to-noise ratio for selecting a seed pixel.
- seed_method: str {‘auto’, ‘manual’, ‘semi’}
- methods for choosing seed pixels ‘semi’ detects K components automatically and allows to add more manually if running as notebook ‘semi’ and ‘manual’ require a backend that does not inline figures, e.g. %matplotlib tk
- ring_size_factor: float
- it’s the ratio between the ring radius and neuron diameters.
- nb: integer
- number of background components for approximating the background using NMF model
- sn: ndarray
- per pixel noise
- options_total: dict
- the option dictionary
- ssub_B: int, optional
- downsampling factor for 1-photon imaging background computation
- init_iter: int, optional
- number of iterations for 1-photon imaging initialization
- Returns:
- Ain: np.ndarray
- (d1 * d2 [ * d3]) x K , spatial filter of each neuron.
- Cin: np.ndarray
- T x K , calcium activity of each neuron.
- center: np.ndarray
- K x 2 [or 3] , inferred center of each neuron.
- bin: np.ndarray
- (d1 * d2 [ * d3]) x nb, initialization of spatial background.
- fin: np.ndarray
- nb x T matrix, initalization of temporal background
- Raises:
Exception “Unsupported method”
Exception ‘You need to define arguments for local NMF’
-
caiman.source_extraction.cnmf.initialization.
greedyROI
(Y, nr=30, gSig=[5, 5], gSiz=[11, 11], nIter=5, kernel=None, nb=1, rolling_sum=False, rolling_length=100, seed_method='auto')¶ Greedy initialization of spatial and temporal components using spatial Gaussian filtering
- Args:
- Y: np.array
- 3d or 4d array of fluorescence data with time appearing in the last axis.
- nr: int
- number of components to be found
- gSig: scalar or list of integers
- standard deviation of Gaussian kernel along each axis
- gSiz: scalar or list of integers
- size of spatial component
- nIter: int
- number of iterations when refining estimates
- kernel: np.ndarray
- User specified kernel to be used, if present, instead of Gaussian (default None)
- nb: int
- Number of background components
- rolling_max: boolean
- Detect new components based on a rolling sum of pixel activity (default: True)
- rolling_length: int
- Length of rolling window (default: 100)
- seed_method: str {‘auto’, ‘manual’, ‘semi’}
- methods for choosing seed pixels ‘semi’ detects nr components automatically and allows to add more manually if running as notebook ‘semi’ and ‘manual’ require a backend that does not inline figures, e.g. %matplotlib tk
- Returns:
- A: np.array
- 2d array of size (# of pixels) x nr with the spatial components. Each column is ordered columnwise (matlab format, order=’F’)
- C: np.array
- 2d array of size nr X T with the temporal components
- center: np.array
- 2d array of size nr x 2 [ or 3] with the components centroids
- Author:
- Eftychios A. Pnevmatikakis and Andrea Giovannucci based on a matlab implementation by Yuanjun Gao
- Simons Foundation, 2015
- See Also:
- http://www.cell.com/neuron/pdf/S0896-6273(15)01084-3.pdf
-
caiman.source_extraction.cnmf.initialization.
greedyROI_corr
(Y, Y_ds, max_number=None, gSiz=None, gSig=None, center_psf=True, min_corr=None, min_pnr=None, seed_method='auto', min_pixel=3, bd=0, thresh_init=2, ring_size_factor=None, nb=1, options=None, sn=None, save_video=False, video_name='initialization.mp4', ssub=1, ssub_B=2, init_iter=2)¶ initialize neurons based on pixels’ local correlations and peak-to-noise ratios.
- Args:
* see init_neurons_corr_pnr for descriptions of following input arguments * data: max_number: gSiz: gSig: center_psf: min_corr: min_pnr: seed_method: min_pixel: bd: thresh_init: swap_dim: save_video: video_name: * see init_neurons_corr_pnr for descriptions of above input arguments *
- ring_size_factor: float
- it’s the ratio between the ring radius and neuron diameters.
- ring_model: Boolean
- True indicates using ring model to estimate the background components.
- nb: integer
- number of background components for approximating the background using NMF model for nb=0 the exact background of the ringmodel (b0 and W) is returned for nb=-1 the full rank background B is returned for nb<-1 no background is returned
- ssub_B: int, optional
- downsampling factor for 1-photon imaging background computation
- init_iter: int, optional
- number of iterations for 1-photon imaging initialization
-
caiman.source_extraction.cnmf.initialization.
graphNMF
(Y_ds, nr, max_iter_snmf=500, lambda_gnmf=1, sigma_smooth=(0.5, 0.5, 0.5), remove_baseline=True, perc_baseline=20, nb=1, truncate=2, tol=0.001, SC_kernel='heat', SC_normalize=True, SC_thr=0, SC_sigma=1, SC_use_NN=False, SC_nnn=20)¶
-
caiman.source_extraction.cnmf.initialization.
sparseNMF
(Y_ds, nr, max_iter_snmf=500, alpha=1000.0, sigma_smooth=(0.5, 0.5, 0.5), remove_baseline=True, perc_baseline=20, nb=1, truncate=2)¶ Initialization using sparse NMF
- Args:
- Y_ds: nd.array or movie (T, x, y [,z])
- data
- nr: int
- number of components
- max_iter_snm: int
- number of iterations
- alpha_snmf:
- sparsity regularizer
- sigma_smooth_snmf:
- smoothing along z,x, and y (.5,.5,.5)
- perc_baseline_snmf:
- percentile to remove frmo movie before NMF
- nb: int
- Number of background components
- Returns:
- A: np.array
- 2d array of size (# of pixels) x nr with the spatial components. Each column is ordered columnwise (matlab format, order=’F’)
- C: np.array
- 2d array of size nr X T with the temporal components
- center: np.array
- 2d array of size nr x 2 [ or 3] with the components centroids
Spatial Components¶
-
caiman.source_extraction.cnmf.spatial.
update_spatial_components
(Y, C=None, f=None, A_in=None, sn=None, dims=None, min_size=3, max_size=8, dist=3, normalize_yyt_one=True, method_exp='dilate', expandCore=None, dview=None, n_pixels_per_process=128, medw=(3, 3), thr_method='max', maxthr=0.1, nrgthr=0.9999, extract_cc=True, b_in=None, se=array([[1, 1, 1], [1, 1, 1], [1, 1, 1]]), ss=array([[1, 1, 1], [1, 1, 1], [1, 1, 1]]), nb=1, method_ls='lasso_lars', update_background_components=True, low_rank_background=True, block_size_spat=1000, num_blocks_per_run_spat=20)¶ update spatial footprints and background through Basis Pursuit Denoising
- for each pixel i solve the problem
- [A(i,:),b(i)] = argmin sum(A(i,:))
- subject to
- || Y(i,:) - A(i,:)*C + b(i)*f || <= sn(i)*sqrt(T);
for each pixel the search is limited to a few spatial components
- Args:
- Y: np.ndarray (2D or 3D)
- movie, raw data in 2D or 3D (pixels x time).
- C: np.ndarray
- calcium activity of each neuron.
- f: np.ndarray
- temporal profile of background activity.
- A_in: np.ndarray
- spatial profile of background activity. If A_in is boolean then it defines the spatial support of A. Otherwise it is used to determine it through determine_search_location
- b_in: np.ndarray
- you can pass background as input, especially in the case of one background per patch, since it will update using hals
- dims: [optional] tuple
- x, y[, z] movie dimensions
min_size: [optional] int
max_size: [optional] int
dist: [optional] int
- sn: [optional] float
- noise associated with each pixel if known
- backend [optional] str
- ‘ipyparallel’, ‘single_thread’ single_thread:no parallelization. It can be used with small datasets. ipyparallel: uses ipython clusters and then send jobs to each of them SLURM: use the slurm scheduler
- n_pixels_per_process: [optional] int
- number of pixels to be processed by each thread
- method: [optional] string
- method used to expand the search for pixels ‘ellipse’ or ‘dilate’
- expandCore: [optional] scipy.ndimage.morphology
- if method is dilate this represents the kernel used for expansion
- dview: view on ipyparallel client
- you need to create an ipyparallel client and pass a view on the processors (client = Client(), dview=client[:])
- medw, thr_method, maxthr, nrgthr, extract_cc, se, ss: [optional]
- Parameters for components post-processing. Refer to spatial.threshold_components for more details
- nb: [optional] int
- Number of background components
- method_ls:
- method to perform the regression for the basis pursuit denoising.
- ‘nnls_L0’. Nonnegative least square with L0 penalty ‘lasso_lars’ lasso lars function from scikit learn
- normalize_yyt_one: bool
- whether to normalize the C and A matrices so that diag(C*C.T) are ones
- update_background_components:bool
- whether to update the background components in the spatial phase
- low_rank_background:bool
- whether to update the using a low rank approximation. In the False case all the nonzero elements of the background components are updated using hals (to be used with one background per patch)
- Returns:
- A: np.ndarray
- new estimate of spatial footprints
- b: np.ndarray
- new estimate of spatial background
- C: np.ndarray
- temporal components (updated only when spatial components are completely removed)
- f: np.ndarray
- same as f_in except if empty component deleted.
- Raises:
Exception ‘You need to define the input dimensions’
Exception ‘Dimension of Matrix Y must be pixels x time’
Exception ‘Dimension of Matrix C must be neurons x time’
Exception ‘Dimension of Matrix f must be background comps x time ‘
Exception ‘Either A or C need to be determined’
Exception ‘Dimension of Matrix A must be pixels x neurons’
Exception ‘You need to provide estimate of C and f’
Exception ‘Not implemented consistently’
Exception “Failed to delete: ” + folder
Temporal Components¶
-
caiman.source_extraction.cnmf.temporal.
update_temporal_components
(Y, A, b, Cin, fin, bl=None, c1=None, g=None, sn=None, nb=1, ITER=2, block_size_temp=5000, num_blocks_per_run_temp=20, debug=False, dview=None, **kwargs)¶ Update temporal components and background given spatial components using a block coordinate descent approach.
- Args:
- Y: np.ndarray (2D)
- input data with time in the last axis (d x T)
- A: sparse matrix (crc format)
- matrix of temporal components (d x K)
- b: ndarray (dx1)
- current estimate of background component
- Cin: np.ndarray
- current estimate of temporal components (K x T)
- fin: np.ndarray
- current estimate of temporal background (vector of length T)
- g: np.ndarray
- Global time constant (not used)
- bl: np.ndarray
- baseline for fluorescence trace for each column in A
- c1: np.ndarray
- initial concentration for each column in A
- g: np.ndarray
- discrete time constant for each column in A
- sn: np.ndarray
- noise level for each column in A
- nb: [optional] int
- Number of background components
- ITER: positive integer
- Maximum number of block coordinate descent loops.
- method_foopsi: string
- Method of deconvolution of neural activity. constrained_foopsi is the only method supported at the moment.
- n_processes: int
- number of processes to use for parallel computation.
- Should be less than the number of processes started with ipcluster.
- backend: ‘str’
- single_thread no parallelization ipyparallel, parallelization using the ipyparallel cluster. You should start the cluster (install ipyparallel and then type ipcluster -n 6, where 6 is the number of processes). SLURM: using SLURM scheduler
- memory_efficient: Bool
- whether or not to optimize for memory usage (longer running times). necessary with very large datasets
- kwargs: dict
- all parameters passed to constrained_foopsi except bl,c1,g,sn (see documentation).
- Some useful parameters are
- p: int
- order of the autoregression model
- method: [optional] string
- solution method for constrained foopsi. Choices are
- ‘cvx’: using cvxopt and picos (slow especially without the MOSEK solver) ‘cvxpy’: using cvxopt and cvxpy with the ECOS solver (faster, default)
- solvers: list string
- primary and secondary (if problem unfeasible for approx solution)
- solvers to be used with cvxpy, default is [‘ECOS’,’SCS’]
- Note:
- The temporal components are updated in parallel by default by forming of sequence of vertex covers.
- Returns:
- C: np.ndarray
- matrix of temporal components (K x T)
- f: np.array
- vector of temporal background (length T)
- S: np.ndarray
- matrix of merged deconvolved activity (spikes) (K x T)
- bl: float
- same as input
- c1: float
- same as input
- g: float
- same as input
- sn: float
- same as input
- YrA: np.ndarray
- matrix of spatial component filtered raw data, after all contributions have been removed. YrA corresponds to the residual trace for each component and is used for faster plotting (K x T)
- lam: np.ndarray
- Automatically tuned sparsity parameter
Merge components¶
-
caiman.source_extraction.cnmf.merging.
merge_components
(Y, A, b, C, R, f, S, sn_pix, temporal_params, spatial_params, dview=None, thr=0.85, fast_merge=True, mx=1000, bl=None, c1=None, sn=None, g=None, merge_parallel=False, max_merge_area=None)¶ Merging of spatially overlapping components that have highly correlated temporal activity
The correlation threshold for merging overlapping components is user specified in thr
- Args:
- Y: np.ndarray
- residual movie after subtracting all found components (Y_res = Y - A*C - b*f) (d x T)
- A: sparse matrix
- matrix of spatial components (d x K)
- b: np.ndarray
- spatial background (vector of length d)
- C: np.ndarray
- matrix of temporal components (K x T)
- R: np.ndarray
- array of residuals (K x T)
- f: np.ndarray
- temporal background (vector of length T)
- S: np.ndarray
- matrix of deconvolved activity (spikes) (K x T)
- sn_pix: ndarray
- noise standard deviation for each pixel
- temporal_params: dictionary
- all the parameters that can be passed to the update_temporal_components function
- spatial_params: dictionary
- all the parameters that can be passed to the update_spatial_components function
- thr: scalar between 0 and 1
- correlation threshold for merging (default 0.85)
- mx: int
- maximum number of merging operations (default 50)
- sn_pix: nd.array
- noise level for each pixel (vector of length d)
- fast_merge: bool
- if true perform rank 1 merging, otherwise takes best neuron
- bl:
- baseline for fluorescence trace for each row in C
- c1:
- initial concentration for each row in C
- g:
- discrete time constant for each row in C
- sn:
- noise level for each row in C
- merge_parallel: bool
- perform merging in parallel
- max_merge_area: int
- maximum area (in pixels) of merged components, used to determine whether to merge
- Returns:
- A: sparse matrix
- matrix of merged spatial components (d x K)
- C: np.ndarray
- matrix of merged temporal components (K x T)
- nr: int
- number of components after merging
- merged_ROIs: list
- index of components that have been merged
- S: np.ndarray
- matrix of merged deconvolved activity (spikes) (K x T)
- bl: float
- baseline for fluorescence trace
- c1: float
- initial concentration
- g: float
- discrete time constant
- sn: float
- noise level
- R: np.ndarray
- residuals
- Raises:
- Exception “The number of elements of blc1gsn must match the number of components”
Utilities¶
-
caiman.source_extraction.cnmf.utilities.
detrend_df_f
(A, b, C, f, YrA=None, quantileMin=8, frames_window=500, flag_auto=True, use_fast=False, detrend_only=False)¶ Compute DF/F signal without using the original data. In general much faster than extract_DF_F
- Args:
- A: scipy.sparse.csc_matrix
- spatial components (from cnmf cnm.A)
- b: ndarray
- spatial background components
- C: ndarray
- temporal components (from cnmf cnm.C)
- f: ndarray
- temporal background components
- YrA: ndarray
- residual signals
- quantile_min: float
- quantile used to estimate the baseline (values in [0,100]) used only if ‘flag_auto’ is False, i.e. ignored by default
- frames_window: int
- number of frames for computing running quantile
- flag_auto: bool
- flag for determining quantile automatically
- use_fast: bool
- flag for using approximate fast percentile filtering
- detrend_only: bool (False)
- flag for only subtracting baseline and not normalizing by it. Used in 1p data processing where baseline fluorescence cannot be determined.
- Returns:
- F_df:
- the computed Calcium activity to the derivative of f
-
caiman.source_extraction.cnmf.utilities.
update_order
(A, new_a=None, prev_list=None, method='greedy')¶ Determines the update order of the temporal components given the spatial components by creating a nest of random approximate vertex covers
- Args:
- A: np.ndarray
- matrix of spatial components (d x K)
- new_a: sparse array
- spatial component that is added, in order to efficiently update the orders in online scenarios
- prev_list: list of list
- orders from previous iteration, you need to pass if new_a is not None
- Returns:
- O: list of sets
- list of subsets of components. The components of each subset can be updated in parallel
- lo: list
- length of each subset
Written by Eftychios A. Pnevmatikakis, Simons Foundation, 2015
-
caiman.source_extraction.cnmf.utilities.
get_file_size
(file_name, var_name_hdf5='mov')¶ Computes the dimensions of a file or a list of files without loading it/them in memory. An exception is thrown if the files have FOVs with different sizes
- Args:
- file_name: str/filePath or various list types
- locations of file(s)
- var_name_hdf5: ‘str’
- if loading from hdf5 name of the dataset to load
- Returns:
- dims: list
- dimensions of FOV
- T: list
- number of timesteps in each file
ROIs¶
-
caiman.base.rois.
register_ROIs
(A1, A2, dims, template1=None, template2=None, align_flag=True, D=None, max_thr=0, use_opt_flow=True, thresh_cost=0.7, max_dist=10, enclosed_thr=None, print_assignment=False, plot_results=False, Cn=None, cmap='viridis')¶ Register ROIs across different sessions using an intersection over union metric and the Hungarian algorithm for optimal matching
- Args:
- A1: ndarray or csc_matrix # pixels x # of components
- ROIs from session 1
- A2: ndarray or csc_matrix # pixels x # of components
- ROIs from session 2
- dims: list or tuple
- dimensionality of the FOV
- template1: ndarray dims
- template from session 1
- template2: ndarray dims
- template from session 2
- align_flag: bool
- align the templates before matching
- D: ndarray
- matrix of distances in the event they are pre-computed
- max_thr: scalar
- max threshold parameter before binarization
- use_opt_flow: bool
- use dense optical flow to align templates
- thresh_cost: scalar
- maximum distance considered
- max_dist: scalar
- max distance between centroids
- enclosed_thr: float
- if not None set distance to at most the specified value when ground truth is a subset of inferred
- print_assignment: bool
- print pairs of matched ROIs
- plot_results: bool
- create a plot of matches and mismatches
- Cn: ndarray
- background image for plotting purposes
- cmap: string
- colormap for background image
- Returns:
- matched_ROIs1: list
- indices of matched ROIs from session 1
- matched_ROIs2: list
- indices of matched ROIs from session 2
- non_matched1: list
- indices of non-matched ROIs from session 1
- non_matched2: list
- indices of non-matched ROIs from session 2
- performance: list
- (precision, recall, accuracy, f_1 score) with A1 taken as ground truth
- A2: csc_matrix # pixels x # of components
- ROIs from session 2 aligned to session 1
-
caiman.base.rois.
register_multisession
(A, dims, templates=[None], align_flag=True, max_thr=0, use_opt_flow=True, thresh_cost=0.7, max_dist=10, enclosed_thr=None)¶ Register ROIs across multiple sessions using an intersection over union metric and the Hungarian algorithm for optimal matching. Registration occurs by aligning session 1 to session 2, keeping the union of the matched and non-matched components to register with session 3 and so on.
- Args:
- A: list of ndarray or csc_matrix matrices # pixels x # of components
- ROIs from each session
- dims: list or tuple
- dimensionality of the FOV
- template: list of ndarray matrices of size dims
- templates from each session
- align_flag: bool
- align the templates before matching
- max_thr: scalar
- max threshold parameter before binarization
- use_opt_flow: bool
- use dense optical flow to align templates
- thresh_cost: scalar
- maximum distance considered
- max_dist: scalar
- max distance between centroids
- enclosed_thr: float
- if not None set distance to at most the specified value when ground truth is a subset of inferred
- Returns:
- A_union: csc_matrix # pixels x # of total distinct components
- union of all kept ROIs
- assignments: ndarray int of size # of total distinct components x # sessions
- element [i,j] = k if component k from session j is mapped to component i in the A_union matrix. If there is no much the value is NaN
- matchings: list of lists
- matchings[i][j] = k means that component j from session i is represented by component k in A_union
-
caiman.base.rois.
com
(A: numpy.ndarray, d1: int, d2: int, d3: Optional[int] = None) → numpy.array¶ Calculation of the center of mass for spatial components
- Args:
- A: np.ndarray
- matrix of spatial components (d x K)
- d1: int
- number of pixels in x-direction
- d2: int
- number of pixels in y-direction
- d3: int
- number of pixels in z-direction
- Returns:
- cm: np.ndarray
- center of mass for spatial components (K x 2 or 3)
-
caiman.base.rois.
extract_binary_masks_from_structural_channel
(Y, min_area_size: int = 30, min_hole_size: int = 15, gSig: int = 5, expand_method: str = 'closing', selem: numpy.array = array([[1., 1., 1.], [1., 1., 1.], [1., 1., 1.]])) → Tuple[numpy.ndarray, numpy.array]¶ Extract binary masks by using adaptive thresholding on a structural channel
- Args:
- Y: caiman movie object
- movie of the structural channel (assumed motion corrected)
- min_area_size: int
- ignore components with smaller size
- min_hole_size: int
- fill in holes up to that size (donuts)
- gSig: int
- average radius of cell
- expand_method: string
- method to expand binary masks (morphological closing or dilation)
- selem: np.array
- morphological element with which to expand binary masks
- Returns:
- A: sparse column format matrix
- matrix of binary masks to be used for CNMF seeding
- mR: np.array
- mean image used to detect cell boundaries
Memory mapping¶
-
caiman.mmapping.
load_memmap
(filename: str, mode: str = 'r') → Tuple[Any, Tuple, int]¶ Load a memory mapped file created by the function save_memmap
- Args:
- filename: str
- path of the file to be loaded
- mode: str
- One of ‘r’, ‘r+’, ‘w+’. How to interact with files
- Returns:
- Yr:
- memory mapped variable
- dims: tuple
- frame dimensions
- T: int
- number of frames
- Raises:
- ValueError “Unknown file extension”
-
caiman.mmapping.
save_memmap_join
(mmap_fnames: List[str], base_name: str = None, n_chunks: int = 20, dview=None, add_to_mov=0) → str¶ Makes a large file memmap from a number of smaller files
- Args:
mmap_fnames: list of memory mapped files
base_name: string, will be the first portion of name to be solved
n_chunks: number of chunks in which to subdivide when saving, smaller requires more memory
dview: cluster handle
add_to_mov: (undocumented)
-
caiman.mmapping.
save_memmap
(filenames: List[str], base_name: str = 'Yr', resize_fact: Tuple = (1, 1, 1), remove_init: int = 0, idx_xy: Tuple = None, order: str = 'F', var_name_hdf5: str = 'mov', xy_shifts: Optional[List[T]] = None, is_3D: bool = False, add_to_movie: float = 0, border_to_0=0, dview=None, n_chunks: int = 100, slices=None) → str¶ Efficiently write data from a list of tif files into a memory mappable file
- Args:
- filenames: list
- list of tif files or list of numpy arrays
- base_name: str
- the base used to build the file name. IT MUST NOT CONTAIN “_”
- resize_fact: tuple
- x,y, and z downsampling factors (0.5 means downsampled by a factor 2)
- remove_init: int
- number of frames to remove at the begining of each tif file (used for resonant scanning images if laser in rutned on trial by trial)
- idx_xy: tuple size 2 [or 3 for 3D data]
- for selecting slices of the original FOV, for instance idx_xy = (slice(150,350,None), slice(150,350,None))
- order: string
- whether to save the file in ‘C’ or ‘F’ order
- xy_shifts: list
- x and y shifts computed by a motion correction algorithm to be applied before memory mapping
- is_3D: boolean
- whether it is 3D data
- add_to_movie: floating-point
- value to add to each image point, typically to keep negative values out.
border_to_0: (undocumented)
dview: (undocumented)
n_chunks: (undocumented)
- slices: slice object or list of slice objects
- slice can be used to select portion of the movies in time and x,y directions. For instance slices = [slice(0,200),slice(0,100),slice(0,100)] will take the first 200 frames and the 100 pixels along x and y dimensions.
- Returns:
- fname_new: the name of the mapped file, the format is such that
- the name will contain the frame dimensions and the number of frames
Image statistics¶
-
caiman.summary_images.
local_correlations
(Y, eight_neighbours: bool = True, swap_dim: bool = True, order_mean=1) → numpy.ndarray¶ Computes the correlation image for the input dataset Y
- Args:
- Y: np.ndarray (3D or 4D)
- Input movie data in 3D or 4D format
- eight_neighbours: Boolean
- Use 8 neighbors if true, and 4 if false for 3D data (default = True) Use 6 neighbors for 4D data, irrespectively
- swap_dim: Boolean
- True indicates that time is listed in the last axis of Y (matlab format) and moves it in the front
order_mean: (undocumented)
- Returns:
- rho: d1 x d2 [x d3] matrix, cross-correlation with adjacent pixels
-
caiman.summary_images.
max_correlation_image
(Y, bin_size: int = 1000, eight_neighbours: bool = True, swap_dim: bool = True) → numpy.ndarray¶ Computes the max-correlation image for the input dataset Y with bin_size
- Args:
- Y: np.ndarray (3D or 4D)
- Input movie data in 3D or 4D format
- bin_size: scalar (integer)
- Length of bin_size (if last bin is smaller than bin_size < 2 bin_size is increased to impose uniform bins)
- eight_neighbours: Boolean
- Use 8 neighbors if true, and 4 if false for 3D data (default = True) Use 6 neighbors for 4D data, irrespectively
- swap_dim: Boolean
- True indicates that time is listed in the last axis of Y (matlab format) and moves it in the front
- Returns:
- Cn: d1 x d2 [x d3] matrix,
- max correlation image
-
caiman.summary_images.
correlation_pnr
(Y, gSig=None, center_psf: bool = True, swap_dim: bool = True, background_filter: str = 'disk') → Tuple[numpy.ndarray, numpy.ndarray]¶ compute the correlation image and the peak-to-noise ratio (PNR) image. If gSig is provided, then spatially filtered the video.
- Args:
- Y: np.ndarray (3D or 4D).
- Input movie data in 3D or 4D format
- gSig: scalar or vector.
- gaussian width. If gSig == None, no spatial filtering
- center_psf: Boolean
- True indicates subtracting the mean of the filtering kernel
- swap_dim: Boolean
- True indicates that time is listed in the last axis of Y (matlab format) and moves it in the front
- background_filter: str
- (undocumented)
- Returns:
- cn: np.ndarray (2D or 3D).
- local correlation image of the spatially filtered (or not) data
- pnr: np.ndarray (2D or 3D).
- peak-to-noise ratios of all pixels/voxels
Parallel Processing functions¶
-
caiman.cluster.
apply_to_patch
(mmap_file, shape: Tuple[Any, Any, Any], dview, rf, stride, function, *args, **kwargs) → Tuple[List[T], Any, Tuple]¶ apply function to patches in parallel or not
- Args:
- mmap_file: Memory-mapped variable
- Variable handle, first thing returned by load_memmap()
- shape: tuple of three elements
- dimensions of the original movie across y, x, and time
- dview: ipyparallel view on client
- if None
- rf: int
- half-size of the square patch in pixel
- stride: int
- amount of overlap between patches
- Returns:
- results
- Raises:
- Exception ‘Something went wrong’
-
caiman.cluster.
start_server
(slurm_script: str = None, ipcluster: str = 'ipcluster', ncpus: int = None) → None¶ programmatically start the ipyparallel server
- Args:
- ncpus: int
- number of processors
- ipcluster : str
- ipcluster binary file name; requires 4 path separators on Windows. ipcluster=”C:\Anaconda3\Scripts\ipcluster.exe” Default: “ipcluster”
-
caiman.cluster.
stop_server
(ipcluster: str = 'ipcluster', pdir: str = None, profile: str = None, dview=None) → None¶ programmatically stops the ipyparallel server
- Args:
- ipcluster : str
- ipcluster binary file name; requires 4 path separators on Windows Default: “ipcluster”a
pdir : Undocumented profile: Undocumented dview: Undocumented
Ring-CNN functions¶
-
class
caiman.utils.nn_models.
Masked_Conv2D
(output_dim=1, kernel_size=(5, 5), strides=(1, 1), radius_min=2, radius_max=3, initializer='uniform', use_bias=True)¶ Creates a trainable ring convolutional kernel with non zero entries between user specified radius_min and radius_max. Uses a random uniform non-negative initializer unless specified otherwise.
- Args:
- output_dim: int, default: 1
- number of output channels (number of kernels)
- kernel_size: (int, int), default: (5, 5)
- dimension of 2d boundaing box
- strides: (int, int), default: (1, 1)
- stride for convolution (modifying that will downsample)
- radius_min: int, default: 2
- inner radius of kernel
- radius_max: int, default: 3
- outer radius of kernel (typically: 2*radius_max - 1 = kernel_size[0])
- initializer: ‘uniform’ or Keras initializer, default: ‘uniform’
- initializer for ring weights. ‘uniform’ will choose from a non-negative random uniform distribution such that the expected value of the sum is 2.
- use_bias: bool, default: True
- add a bias term to each convolution kernel
- Returns:
- Masked_Conv2D: tensorflow.keras.layer
- A trainable layer implementing the convolution with a ring
Attributes: activity_regularizer
Optional regularizer function for the output of this layer.
compute_dtype
The dtype of the layer’s computations.
dtype
The dtype of the layer weights.
dtype_policy
The dtype policy associated with this layer.
dynamic
Whether the layer is dynamic (eager-only); set in the constructor.
inbound_nodes
Deprecated, do NOT use! Only for compatibility with external Keras.
input
Retrieves the input tensor(s) of a layer.
input_mask
Retrieves the input mask tensor(s) of a layer.
input_shape
Retrieves the input shape(s) of a layer.
input_spec
InputSpec instance(s) describing the input format for this layer.
losses
List of losses added using the add_loss() API.
metrics
List of metrics added using the add_metric() API.
name
Name of the layer (string), set in the constructor.
name_scope
Returns a tf.name_scope instance for this class.
- non_trainable_variables
non_trainable_weights
List of all non-trainable weights tracked by this layer.
outbound_nodes
Deprecated, do NOT use! Only for compatibility with external Keras.
output
Retrieves the output tensor(s) of a layer.
output_mask
Retrieves the output mask tensor(s) of a layer.
output_shape
Retrieves the output shape(s) of a layer.
- stateful
submodules
Sequence of all sub-modules.
supports_masking
Whether this layer supports computing a mask using compute_mask.
- trainable
trainable_variables
Sequence of trainable variables owned by this module and its submodules.
trainable_weights
List of all trainable weights tracked by this layer.
- updates
variable_dtype
Alias of Layer.dtype, the dtype of the weights.
variables
Returns the list of all layer variables/weights.
weights
Returns the list of all layer variables/weights.
Methods
__call__
(*args, **kwargs)Wraps call, applying pre- and post-processing steps. add_loss
(losses, **kwargs)Add loss tensor(s), potentially dependent on layer inputs. add_metric
(value[, name])Adds metric tensor to the layer. add_update
(updates[, inputs])Add update op(s), potentially dependent on layer inputs. add_variable
(*args, **kwargs)Deprecated, do NOT use! Alias for add_weight. add_weight
([name, shape, dtype, …])Adds a new variable to the layer. apply
(inputs, *args, **kwargs)Deprecated, do NOT use! build
(input_shape)Creates the variables of the layer (optional, for subclass implementers). call
(x)This is where the layer’s logic lives. compute_mask
(inputs[, mask])Computes an output mask tensor. compute_output_shape
(input_shape)Computes the output shape of the layer. compute_output_signature
(input_signature)Compute the output tensor signature of the layer based on the inputs. count_params
()Count the total number of scalars composing the weights. from_config
(config)Creates a layer from its config. get_config
()Returns the config of the layer. get_input_at
(node_index)Retrieves the input tensor(s) of a layer at a given node. get_input_mask_at
(node_index)Retrieves the input mask tensor(s) of a layer at a given node. get_input_shape_at
(node_index)Retrieves the input shape(s) of a layer at a given node. get_losses_for
(inputs)Deprecated, do NOT use! get_output_at
(node_index)Retrieves the output tensor(s) of a layer at a given node. get_output_mask_at
(node_index)Retrieves the output mask tensor(s) of a layer at a given node. get_output_shape_at
(node_index)Retrieves the output shape(s) of a layer at a given node. get_updates_for
(inputs)Deprecated, do NOT use! get_weights
()Returns the current weights of the layer. set_weights
(weights)Sets the weights of the layer, from Numpy arrays. with_name_scope
(method)Decorator to automatically enter the module name scope.
-
class
caiman.utils.nn_models.
Hadamard
(initializer=<tensorflow.python.keras.initializers.initializers_v2.Constant object>, **kwargs)¶ Creates a tensorflow.keras multiplicative layer that performs pointwise multiplication with a set of learnable weights.
- Args:
- initializer: keras initializer, deafult: Constant(0.1)
Attributes: activity_regularizer
Optional regularizer function for the output of this layer.
compute_dtype
The dtype of the layer’s computations.
dtype
The dtype of the layer weights.
dtype_policy
The dtype policy associated with this layer.
dynamic
Whether the layer is dynamic (eager-only); set in the constructor.
inbound_nodes
Deprecated, do NOT use! Only for compatibility with external Keras.
input
Retrieves the input tensor(s) of a layer.
input_mask
Retrieves the input mask tensor(s) of a layer.
input_shape
Retrieves the input shape(s) of a layer.
input_spec
InputSpec instance(s) describing the input format for this layer.
losses
List of losses added using the add_loss() API.
metrics
List of metrics added using the add_metric() API.
name
Name of the layer (string), set in the constructor.
name_scope
Returns a tf.name_scope instance for this class.
- non_trainable_variables
non_trainable_weights
List of all non-trainable weights tracked by this layer.
outbound_nodes
Deprecated, do NOT use! Only for compatibility with external Keras.
output
Retrieves the output tensor(s) of a layer.
output_mask
Retrieves the output mask tensor(s) of a layer.
output_shape
Retrieves the output shape(s) of a layer.
- stateful
submodules
Sequence of all sub-modules.
supports_masking
Whether this layer supports computing a mask using compute_mask.
- trainable
trainable_variables
Sequence of trainable variables owned by this module and its submodules.
trainable_weights
List of all trainable weights tracked by this layer.
- updates
variable_dtype
Alias of Layer.dtype, the dtype of the weights.
variables
Returns the list of all layer variables/weights.
weights
Returns the list of all layer variables/weights.
Methods
__call__
(*args, **kwargs)Wraps call, applying pre- and post-processing steps. add_loss
(losses, **kwargs)Add loss tensor(s), potentially dependent on layer inputs. add_metric
(value[, name])Adds metric tensor to the layer. add_update
(updates[, inputs])Add update op(s), potentially dependent on layer inputs. add_variable
(*args, **kwargs)Deprecated, do NOT use! Alias for add_weight. add_weight
([name, shape, dtype, …])Adds a new variable to the layer. apply
(inputs, *args, **kwargs)Deprecated, do NOT use! build
(input_shape)Creates the variables of the layer (optional, for subclass implementers). call
(x)This is where the layer’s logic lives. compute_mask
(inputs[, mask])Computes an output mask tensor. compute_output_shape
(input_shape)Computes the output shape of the layer. compute_output_signature
(input_signature)Compute the output tensor signature of the layer based on the inputs. count_params
()Count the total number of scalars composing the weights. from_config
(config)Creates a layer from its config. get_config
()Returns the config of the layer. get_input_at
(node_index)Retrieves the input tensor(s) of a layer at a given node. get_input_mask_at
(node_index)Retrieves the input mask tensor(s) of a layer at a given node. get_input_shape_at
(node_index)Retrieves the input shape(s) of a layer at a given node. get_losses_for
(inputs)Deprecated, do NOT use! get_output_at
(node_index)Retrieves the output tensor(s) of a layer at a given node. get_output_mask_at
(node_index)Retrieves the output mask tensor(s) of a layer at a given node. get_output_shape_at
(node_index)Retrieves the output shape(s) of a layer at a given node. get_updates_for
(inputs)Deprecated, do NOT use! get_weights
()Returns the current weights of the layer. set_weights
(weights)Sets the weights of the layer, from Numpy arrays. with_name_scope
(method)Decorator to automatically enter the module name scope.
-
class
caiman.utils.nn_models.
Additive
(data=None, initializer=<tensorflow.python.keras.initializers.initializers_v2.Constant object>, pct=1, **kwargs)¶ Creates a tensorflow.keras additive layer that performs pointwise addition with a set of learnable weights.
- Args:
- initializer: keras initializer, deafult: Constant(0)
Attributes: activity_regularizer
Optional regularizer function for the output of this layer.
compute_dtype
The dtype of the layer’s computations.
dtype
The dtype of the layer weights.
dtype_policy
The dtype policy associated with this layer.
dynamic
Whether the layer is dynamic (eager-only); set in the constructor.
inbound_nodes
Deprecated, do NOT use! Only for compatibility with external Keras.
input
Retrieves the input tensor(s) of a layer.
input_mask
Retrieves the input mask tensor(s) of a layer.
input_shape
Retrieves the input shape(s) of a layer.
input_spec
InputSpec instance(s) describing the input format for this layer.
losses
List of losses added using the add_loss() API.
metrics
List of metrics added using the add_metric() API.
name
Name of the layer (string), set in the constructor.
name_scope
Returns a tf.name_scope instance for this class.
- non_trainable_variables
non_trainable_weights
List of all non-trainable weights tracked by this layer.
outbound_nodes
Deprecated, do NOT use! Only for compatibility with external Keras.
output
Retrieves the output tensor(s) of a layer.
output_mask
Retrieves the output mask tensor(s) of a layer.
output_shape
Retrieves the output shape(s) of a layer.
- stateful
submodules
Sequence of all sub-modules.
supports_masking
Whether this layer supports computing a mask using compute_mask.
- trainable
trainable_variables
Sequence of trainable variables owned by this module and its submodules.
trainable_weights
List of all trainable weights tracked by this layer.
- updates
variable_dtype
Alias of Layer.dtype, the dtype of the weights.
variables
Returns the list of all layer variables/weights.
weights
Returns the list of all layer variables/weights.
Methods
__call__
(*args, **kwargs)Wraps call, applying pre- and post-processing steps. add_loss
(losses, **kwargs)Add loss tensor(s), potentially dependent on layer inputs. add_metric
(value[, name])Adds metric tensor to the layer. add_update
(updates[, inputs])Add update op(s), potentially dependent on layer inputs. add_variable
(*args, **kwargs)Deprecated, do NOT use! Alias for add_weight. add_weight
([name, shape, dtype, …])Adds a new variable to the layer. apply
(inputs, *args, **kwargs)Deprecated, do NOT use! build
(input_shape)Creates the variables of the layer (optional, for subclass implementers). call
(x)This is where the layer’s logic lives. compute_mask
(inputs[, mask])Computes an output mask tensor. compute_output_shape
(input_shape)Computes the output shape of the layer. compute_output_signature
(input_signature)Compute the output tensor signature of the layer based on the inputs. count_params
()Count the total number of scalars composing the weights. from_config
(config)Creates a layer from its config. get_config
()Returns the config of the layer. get_input_at
(node_index)Retrieves the input tensor(s) of a layer at a given node. get_input_mask_at
(node_index)Retrieves the input mask tensor(s) of a layer at a given node. get_input_shape_at
(node_index)Retrieves the input shape(s) of a layer at a given node. get_losses_for
(inputs)Deprecated, do NOT use! get_output_at
(node_index)Retrieves the output tensor(s) of a layer at a given node. get_output_mask_at
(node_index)Retrieves the output mask tensor(s) of a layer at a given node. get_output_shape_at
(node_index)Retrieves the output shape(s) of a layer at a given node. get_updates_for
(inputs)Deprecated, do NOT use! get_weights
()Returns the current weights of the layer. set_weights
(weights)Sets the weights of the layer, from Numpy arrays. with_name_scope
(method)Decorator to automatically enter the module name scope.
-
caiman.utils.nn_models.
create_LN_model
(Y=None, shape=(None, None, 1), n_channels=2, gSig=5, r_factor=1.5, use_add=True, initializer='uniform', lr=0.0001, pct=10, loss='mse', width=5, use_bias=False)¶ Creates a convolutional neural network with ring shape convolutions and multiplicative layers. User needs to specify the radius of the average neuron through gSig and the number of channels. The other parameters can be modified or left to the default values. The inner and outer radius of the ring kernel will be int(gSig*r_factor) and int(gSig*r_factor) + width, respectively.
- Args:
- Y: np.array, default: None
- dataset to be fit, used only if a percentile based initializer is used for the additive layer and can be left to None
- shape: tuple, default: (None, None, 1)
- dimensions of the FOV. Can be left to its default value
- n_channels: int, default: 2
- number of convolutional kernels
- gSig: int, default: 5
- radius of average neuron
- r_factor: float, default: 1.5
- expansion factor to deteremine inner radius
- width: int, default: 5
- width of ring kernel
- use_add: bool, default: True
- flag for using an additive layer
- initializer: ‘uniform’ or Keras initializer, default: ‘uniform’
- initializer for ring weights. ‘uniform’ will choose from a non-negative random uniform distribution such that the expected value of the sum is 2.
- lr: float, default: 1e-4
- (initial) learning rate
- pct: float, default: 10
- percentile used for initializing additive layer
- loss: str or keras loss function
- loss function used for training
- use_bias: bool, default: False
- add a bias term to each convolution kernel
- Returns:
- model_LIN: tf.keras model compiled and ready to be trained.
-
caiman.utils.nn_models.
fit_NL_model
(model_NL, Y, patience=5, val_split=0.2, batch_size=32, epochs=500, schedule=None)¶ Fit either the linear or the non-linear model. The model is fit for a use specified maximum number of epochs and early stopping is used based on the validation loss. A Tensorboard compatible log is also created. Args:
- model_LN: Keras Ring-CNN model
- see create_LN_model and create_NL_model above
- patience: int, default: 5
- patience value for early stopping criterion
- val_split: float, default: 0.2
- fraction of data to keep for validation (value between 0 and 1)
- batch_size: int, default: 32
- batch size during training
- epochs: int, default: 500
- maximum number of epochs
schedule: keras learning rate scheduler
- Returns:
- model_NL: Keras Ring-CNN model
- trained model loaded with best weights according to validation loss
history_NL: contains data related to the training history
- path_to_model: str
- path to where the weights are stored.
-
caiman.utils.nn_models.
quantile_loss
(qnt=0.5)¶ Returns a quantile loss function that can be used for training.
- Args:
- qnt: float, default: 0.5
- desired quantile (0 < qnt < 1)
- Returns:
- my_qnt_loss: quantile loss function
VolPy¶
-
class
caiman.source_extraction.volpy.volpy.
VOLPY
(n_processes, dview=None, context_size=35, censor_size=12, flip_signal=True, hp_freq_pb=0.3333333333333333, nPC_bg=8, ridge_bg=0.01, hp_freq=1, clip=2000, threshold_method='simple', min_spikes=10, threshold=4, sigmas=array([1., 1.5, 2. ]), n_iter=2, weight_update='ridge', do_plot=True, do_cross_val=False, sub_freq=75, method='spikepursuit', superfactor=10, params=None)¶ Spike Detection in Voltage Imaging The general file class which is used to find spikes of voltage imaging. Its architecture is similar to the one of scikit-learn calling the function fit to run everything which is part of the structure of the class. The output will be recorded in self.estimates. In order to use VolPy within CaImAn, you must install Keras into your conda environment. You can do this by activating your environment, and then issuing the command “conda install -c conda-forge keras”.
Methods
fit
([n_processes, dview])Run the volspike function to detect spikes and save the result into self.estimates
-
VOLPY.
__init__
(n_processes, dview=None, context_size=35, censor_size=12, flip_signal=True, hp_freq_pb=0.3333333333333333, nPC_bg=8, ridge_bg=0.01, hp_freq=1, clip=2000, threshold_method='simple', min_spikes=10, threshold=4, sigmas=array([1., 1.5, 2. ]), n_iter=2, weight_update='ridge', do_plot=True, do_cross_val=False, sub_freq=75, method='spikepursuit', superfactor=10, params=None)¶ - Args:
- n_processes: int
- number of processes used
- dview: Direct View object
- for parallelization pruposes when using ipyparallel
- context_size: int
- number of pixels surrounding the ROI to use as context
- censor_size: int
- number of pixels surrounding the ROI to censor from the background PCA; roughly the spatial scale of scattered/dendritic neural signals, in pixels
- flip_signal: boolean
- whether to flip signal upside down for spike detection True for voltron, False for others
- hp_freq_pb: float
- high-pass frequency for removing photobleaching
- nPC_bg: int
- number of principal components used for background subtraction
- ridge_bg: float
- regularization strength for ridge regression in background removal
- hp_freq: float
- high-pass cutoff frequency to filter the signal after computing the trace
- clip: int
- maximum number of spikes for producing templates
- threshold_method: str
- ‘simple’ or ‘adaptive_threshold’ method for thresholding signals ‘simple’ method threshold based on estimated noise level ‘adaptive_threshold’ method threshold based on estimated peak distribution
- min_spikes: int
- minimal number of spikes to be detected
- threshold: float
- threshold for spike detection in ‘simple’ threshold method The real threshold is the value multiplied by the estimated noise level
- sigmas: 1-d array
- spatial smoothing radius imposed on high-pass filtered movie only for finding weights
- n_iter: int
- number of iterations alternating between estimating spike times and spatial filters
- weight_update: str
- ‘ridge’ or ‘NMF’ for weight update
- do_plot: boolean
- if Ture, plot trace of signals and spiketimes, peak triggered average, histogram of heights in the last iteration
- do_cross_val: boolean
- whether to use cross validation to optimize regression regularization parameters
- sub_freq: float
- frequency for subthreshold extraction
- method: str
- ‘spikepursuit’ or ‘atm’ method
- superfactor: int
- used in ‘atm’ method for regression
-
VOLPY.
fit
(n_processes=None, dview=None)¶ Run the volspike function to detect spikes and save the result into self.estimates
-
class
caiman.source_extraction.volpy.volparams.
volparams
(fnames=None, fr=None, index=None, ROIs=None, weights=None, context_size=35, censor_size=12, flip_signal=True, hp_freq_pb=0.3333333333333333, nPC_bg=8, ridge_bg=0.01, hp_freq=1, clip=2000, threshold_method='simple', min_spikes=10, threshold=4, sigmas=array([1., 1.5, 2. ]), n_iter=2, weight_update='ridge', do_plot=True, do_cross_val=False, sub_freq=75, method='spikepursuit', superfactor=10, params_dict={})¶ Methods
get
(group, key)Get a value for a given group and key. get_group
(group)Get the dictionary of key-value pairs for a group. set
(group, val_dict[, set_if_not_exists, …])Add key-value pairs to a group. Existing key-value pairs will be overwritten change_params
-
volparams.
__init__
(fnames=None, fr=None, index=None, ROIs=None, weights=None, context_size=35, censor_size=12, flip_signal=True, hp_freq_pb=0.3333333333333333, nPC_bg=8, ridge_bg=0.01, hp_freq=1, clip=2000, threshold_method='simple', min_spikes=10, threshold=4, sigmas=array([1., 1.5, 2. ]), n_iter=2, weight_update='ridge', do_plot=True, do_cross_val=False, sub_freq=75, method='spikepursuit', superfactor=10, params_dict={})¶ Class for setting parameters for voltage imaging. Including parameters for the data, motion correction and spike detection. The prefered way to set parameters is by using the set function, where a subclass is determined and a dictionary is passed. The whole dictionary can also be initialized at once by passing a dictionary params_dict when initializing the CNMFParams object.
-
volparams.
set
(group, val_dict, set_if_not_exists=False, verbose=False)¶ - Add key-value pairs to a group. Existing key-value pairs will be overwritten
- if specified in val_dict, but not deleted.
- Args:
- group: The name of the group. val_dict: A dictionary with key-value pairs to be set for the group. set_if_not_exists: Whether to set a key-value pair in a group if the key does not currently exist in the group.
-
volparams.
get
(group, key)¶ Get a value for a given group and key. Raises an exception if no such group/key combination exists.
- Args:
- group: The name of the group. key: The key for the property in the group of interest.
Returns: The value for the group/key combination.
-
volparams.
get_group
(group)¶ Get the dictionary of key-value pairs for a group.
- Args:
- group: The name of the group.
-
volparams.
change_params
(params_dict, verbose=False)¶
-
caiman.source_extraction.volpy.spikepursuit.
volspike
(pars)¶ Function for finding spikes of single neuron with given ROI in voltage imaging. Use function denoise_spikes to find spikes from one dimensional signal, and use ridge regression to find the best weight. Do these two steps iteratively to find best spike times.
- Args:
- pars: list
- fnames: str
- name of the memory mapping file in C order
- fr: int
- frame rate of the movie
- cell_n: int
- index of the cell processing
- ROIs: 3-d array
- all regions of interests
- weights: 3-d array
- spatial weights of different cells generated by previous data blocks as initialization
- args: dictionary
- context_size: int
- number of pixels surrounding the ROI to use as context
- censor_size: int
- number of pixels surrounding the ROI to censor from the background PCA; roughly the spatial scale of scattered/dendritic neural signals, in pixels
- flip_signal: boolean
- whether to flip signal upside down for spike detection True for voltron, False for others
- hp_freq_pb: float
- high-pass frequency for removing photobleaching
- nPC_bg: int
- number of principle components used for background subtraction
- ridge_bg: float
- regularization strength for ridge regression in background removal
- hp_freq: float
- high-pass cutoff frequency to filter the signal after computing the trace
- clip: int
- maximum number of spikes for producing templates
- threshold_method: str
- ‘simple’ or ‘adaptive_threshold’ method for thresholding signals ‘simple’ method threshold based on estimated noise level ‘adaptive_threshold’ method threshold based on estimated peak distribution
- min_spikes: int
- minimal number of spikes to be detected
- threshold: float
- threshold for spike detection in ‘simple’ threshold method The real threshold is the value multiplied by the estimated noise level
- sigmas: 1-d array
- spatial smoothing radius imposed on high-pass filtered movie only for finding weights
- n_iter: int
- number of iterations alternating between estimating spike times and spatial filters
- weight_update: str
- ‘ridge’ or ‘NMF’ for weight update
- do_plot: boolean
- if Ture, plot trace of signals and spiketimes, peak triggered average, histogram of heights in the last iteration
- do_cross_val: boolean
- whether to use cross validation to optimize regression regularization parameters
- sub_freq: float
- frequency for subthreshold extraction
- Returns:
- output: dictionary
- cell_n: int
- index of cell
- t: 1-d array
- trace without applying whitened matched filter
- ts: 1-d array
- trace after applying whitened matched filter
- t_rec: 1-d array
- reconstructed signal of the neuron
- t_sub: 1-d array
- subthreshold signal of the neuron
- spikes: 1-d array
- spike time of the neuron
- num_spikes: list
- number of spikes detected in each iteration
- low_spikes: boolean
- True if detected number spikes is less than min_spikes
- template: 1-d array
- temporal template of the neuron
- snr: float
- signal to noise ratio of the processed signal
- thresh: float
- threshold of the signal
- spatial_filter: 2-d array
- spatial filter of the neuron in the whole FOV
- weights: 2-d array
- ridge regression coefficients for fitting reconstructed signal
- locality: boolean
- False if the maximum of spatial filter is not in the initial ROI
- context_coord: 2-d array
- boundary of context region in x,y coordinates
- mean_im: 1-d array
- mean of the signal in ROI after removing photobleaching, used for producing F0
- F0: 1-d array
- baseline signal
- dFF: 1-d array
- scaled signal
- rawROI: dictionary
- including the result after the first spike extraction