.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "generated\examples\tutorials\plot_large_data.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_generated_examples_tutorials_plot_large_data.py: =========================================================== Datasets larger than RAM =========================================================== When data are entered as numpy arrays, mreg will load the entire array in memory, perform the computations and generate new datasets in memory, such as model fit arrays or deformation fields. When working with large datasets, such as 3D time series of images, it may happen that this requires more memory than a machine has available. Even if the original data fit in memory, you will need more than three times that to fit all the results in. A solution in such conditions is to store the data on disk as a zarray rather than a numpy array. Most mdreg functions can operate on zarrays, and will consume a lot less memory because only the data that are actually being processed are loaded. This example illustrates how to work with zarrays for the example of non-linear model fitting to a variable flip angle series. .. GENERATED FROM PYTHON SOURCE LINES 26-28 Setup ----- .. GENERATED FROM PYTHON SOURCE LINES 28-35 .. code-block:: Python import os import time import shutil import numpy as np import mdreg .. GENERATED FROM PYTHON SOURCE LINES 36-38 Fetch the low resolution VFA data saved as zarray. Since the zarray exist on disk this step is not actually loading any data in memory. .. GENERATED FROM PYTHON SOURCE LINES 38-40 .. code-block:: Python data = mdreg.fetch_zarr('VFA') .. GENERATED FROM PYTHON SOURCE LINES 41-42 In a zarray any header information or metadata are in attributes: .. GENERATED FROM PYTHON SOURCE LINES 42-44 .. code-block:: Python FA = data.attrs['FA'] # The FA values in degrees .. GENERATED FROM PYTHON SOURCE LINES 45-47 For the purpose of this example we will always use the same pixel-wise model fit. We define the keyword arguments up front to save some repetition later: .. GENERATED FROM PYTHON SOURCE LINES 47-55 .. code-block:: Python modelfit = { 'model': mdreg.spgr_vfa, # VFA signal model 'xdata': FA, # Flip angle 'func_init': mdreg.spgr_vfa_init, # Initializer 'p0': [1, 0.5], # Initial values 'bounds': ([0, 0], [np.inf, 1]), # Parameter bounds } .. GENERATED FROM PYTHON SOURCE LINES 56-61 Fit numpy arrays ---------------- Loading an entire array into memory can be done by indexing with data[:], which returns a numpy array. Therefore the following operation simply fits the signal model to a numpy array in memory: .. GENERATED FROM PYTHON SOURCE LINES 61-69 .. code-block:: Python t = time.time() coreg, fit, defo, pars = mdreg.fit( data[:], fit_pixels=modelfit, maxit=1, ) print(f"Computation time: {round(time.time()-t)} seconds.") .. rst-class:: sphx-glr-script-out .. code-block:: none Computation time: 3651 seconds. .. GENERATED FROM PYTHON SOURCE LINES 70-71 Since the argument is a numpy array, the return values are numpy arrays too: .. GENERATED FROM PYTHON SOURCE LINES 71-73 .. code-block:: Python print('Data type of return values: ', type(fit)) .. rst-class:: sphx-glr-script-out .. code-block:: none Data type of return values: .. GENERATED FROM PYTHON SOURCE LINES 74-75 Let's visualise the S0 map for reference: .. GENERATED FROM PYTHON SOURCE LINES 77-80 .. code-block:: Python fig = mdreg.plot.par(pars[...,0], title='S0', vmin=0, vmax=np.percentile(pars[...,0], 95)) .. image-sg:: /generated/examples/tutorials/images/sphx_glr_plot_large_data_001.png :alt: S0 , Slice 1, Slice 2, Slice 3, Slice 4, Slice 5, Slice 6, Slice 7, Slice 8, Slice 9, Slice 10, Slice 11, Slice 12, Slice 13, Slice 14, Slice 15, Slice 16, Slice 17, Slice 18, Slice 19, Slice 20, Slice 21, Slice 22, Slice 23, Slice 24, Slice 25, Slice 26, Slice 27, Slice 28, Slice 29, Slice 30, Slice 31, Slice 32, Slice 33, Slice 34, Slice 35, Slice 36 :srcset: /generated/examples/tutorials/images/sphx_glr_plot_large_data_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 81-84 Fitting zarrays: results in memory ---------------------------------- We can also feed in the zarray directly as argument: .. GENERATED FROM PYTHON SOURCE LINES 84-92 .. code-block:: Python t = time.time() coreg, fit, defo, pars = mdreg.fit( data, fit_pixels=modelfit, maxit=1, ) print(f"Computation time: {round(time.time()-t)} seconds.") .. rst-class:: sphx-glr-script-out .. code-block:: none Fitting zarray: 0%| | 0/36 [00:00 .. GENERATED FROM PYTHON SOURCE LINES 98-100 The result is the same as with numpy arrays, and can be accessed in the same way: .. GENERATED FROM PYTHON SOURCE LINES 102-105 .. code-block:: Python fig = mdreg.plot.par(pars[...,0], title='S0', vmin=0, vmax=np.percentile(pars[...,0], 95)) .. image-sg:: /generated/examples/tutorials/images/sphx_glr_plot_large_data_002.png :alt: S0 , Slice 1, Slice 2, Slice 3, Slice 4, Slice 5, Slice 6, Slice 7, Slice 8, Slice 9, Slice 10, Slice 11, Slice 12, Slice 13, Slice 14, Slice 15, Slice 16, Slice 17, Slice 18, Slice 19, Slice 20, Slice 21, Slice 22, Slice 23, Slice 24, Slice 25, Slice 26, Slice 27, Slice 28, Slice 29, Slice 30, Slice 31, Slice 32, Slice 33, Slice 34, Slice 35, Slice 36 :srcset: /generated/examples/tutorials/images/sphx_glr_plot_large_data_002.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 106-110 Fitting zarrays: results on disk -------------------------------- Since we did not specify a path before, the zarrays returned by mfit() are stored in memory: .. GENERATED FROM PYTHON SOURCE LINES 110-112 .. code-block:: Python print('Storage type of return values: ', type(fit.store)) .. rst-class:: sphx-glr-script-out .. code-block:: none Storage type of return values: .. GENERATED FROM PYTHON SOURCE LINES 113-115 If we want the return values to be stored on disk instead, we need to provide a path in the function call: .. GENERATED FROM PYTHON SOURCE LINES 115-126 .. code-block:: Python path = os.path.join(os.getcwd(), 'tmp') t = time.time() coreg, fit, defo, pars = mdreg.fit( data, path=path, fit_pixels=modelfit, maxit=1, ) print(f"Computation time: {round(time.time()-t)} seconds.") .. rst-class:: sphx-glr-script-out .. code-block:: none Fitting zarray: 0%| | 0/36 [00:00 .. GENERATED FROM PYTHON SOURCE LINES 131-132 Clean up the directory for the next computation .. GENERATED FROM PYTHON SOURCE LINES 132-134 .. code-block:: Python shutil.rmtree(path) .. GENERATED FROM PYTHON SOURCE LINES 135-140 Row-by-row computation ---------------------- By default zarrays are processed slice-by-slice (memdim=2). If a single slice is still too large for memory, the *memdim* argument can be set to 1 to perform the computations row-by-row: .. GENERATED FROM PYTHON SOURCE LINES 140-151 .. code-block:: Python modelfit['memdim']=1 t = time.time() coreg, fit, defo, pars = mdreg.fit( data, path=path, fit_pixels=modelfit, maxit=1, ) print(f"Computation time: {round(time.time()-t)} seconds.") .. rst-class:: sphx-glr-script-out .. code-block:: none Fitting zarray: 0%| | 0/3456 [00:00` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_large_data.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_large_data.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_