Sambuca Core Documentation

Build Status Documentation Status

Contains the core model components for the Semi-Analytical Model for Bathymetry, Un-mixing, and Concentration Assessment (SAMBUCA).

Installation

pip install sambuca_core

For everything else, please refer to the project documentation.

Contents

Usage

To use sambuca_core in a project:

import sambuca_core

Or:

import sambuca_core as sbc

Contributing

Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.

Bug reports

When reporting a bug please include:

  • Your operating system name and version.
  • Any details about your local setup that might be helpful in troubleshooting.
  • The Sambuca Core package version.
  • Detailed steps to reproduce the bug.

Documentation improvements

Sambuca Core could always use more documentation, whether as part of the official Sambuca Core docs, in docstrings, or even on the web in blog posts, articles, and such.

Note

This project uses Google-style docstrings. Contributed code should follow the same conventions. For examples, please see the Napoleon examples, or the Google Python Style Guide.

Feature requests and feedback

The best way to send feedback is to file an issue

If you are proposing a feature:

  • Explain in detail how it would work.
  • Keep the scope as narrow as possible, to make it easier to implement.
  • Remember that this is a volunteer-driven project, and that contributions are welcome :)

Or, implement the feature yourself and submit a pull request.

Development

To set up Sambuca Core for local development:

  1. Fork the sambuca_core repo on GitHub.

  2. Clone your fork locally.

  3. Create a feature branch.

  4. When you’re done making changes, run all the tests, doc builder and pylint checks:

    py.test
tox
pylint ./src/sambuca_core/
sphinx-build -b html docs build/docs

    Or, using the project makefile:

    make clean lint tests html
tox

  5. Commit your changes and push your branch to GitHub.

  6. Submit a pull request through the GitHub website.

There is a makefile in the project root with targets for the most common development operations such as lint checks, running unit tests, building the documentation, and building installing packages.

Bumpversion is used to manage the package version numbers. This ensures that the version number is correctly incremented in all required files. Please see the bumpversion documentation for usage instructions, and do not edit the version strings directly.

Note

Sphinx requires a working LaTeX installation to build the documentation.

Pull Request Guidelines

If you need some code review or feedback while you’re developing the code just make the pull request.

For merging, you should:

  1. Include passing tests (run py.test).
  2. Update documentation when there’s new API, functionality etc.
  3. Add a note to CHANGELOG.rst about the changes.
  4. Add yourself to AUTHORS.rst.

Authors

Changelog

0.1.0 (2015-06-23)

  • Initial commit of core Sambuca code after split from single package.

1.0.0 (2015-12-01)

  • Data loading functions for sensor filters, and general spectra implemented. Supported formats are ENVI spectral libraries, and tabular data in Excel and CSV formats.
  • Full physical model added to the forward model.

1.1.0 (2015-12-03)

  • Additional SIOP and IOP outputs added to forward model. These values were being calculated anyway, and they were required by Bioopti.
  • Docstrings are now almost complete.
  • Some naming inconsistencies have been addressed.

1.2.0 (2015-12-04)

  • forward model changes:
    • renamed slope_cdom to a_cdom_slope to indicate that it is an absorption value, and for consistency with the main naming convention.
    • renamed slope_nap to a_nap_slope for the same reasons.
    • split backscatter slope into two values, one for phytoplankton, and one for NAP. Both are optional, and if the NAP backscatter slope value is not supplied, the phytoplankton backscatter slope value will be reused. This preserves the behaviour of the IDL model which only has a single backscatter slope value for both phytoplankton and NAP. The new slope parameter names conform to the primary naming convention.

1.2.1 (2015-12-04)

  • Implemented q_factor and R(0-) calculations.

1.2.2 (2015-12-04)

  • Fixed filename case-sensitivity issue between Windows and Linux.

1.3.0 (2018-06-18)

  • Added licence, cleaned up docs and pushed to GitHub.

1.3.1 (2018-06-18)

  • Added Readthedocs link.
  • updated changelog.

1.3.2 (Wednesday June 13, 2018)

  • Minor documentation fixes.
  • Setup Travis CI integration.
  • Completed Read the Docs integration.
  • Removed Python 3.4 from CI, as it is not supported by pandas.

1.3.3 (Wednesday June 13, 2018)

  • Updating setup.py metadata
  • First version pushed to PyPI

Licence

CSIRO Open Source Software Licence Agreement (variation of the BSD / MIT License)

Copyright (c) 2014, Commonwealth Scientific and Industrial Research Organisation (CSIRO) ABN 41 687 119 230. All rights reserved. CSIRO is willing to grant you a licence to use Sambuca Core on the following terms, except where otherwise indicated for third party material. Redistribution and use of this software in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  • Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
  • Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
  • Neither the name of CSIRO nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission of CSIRO.

EXCEPT AS EXPRESSLY STATED IN THIS AGREEMENT AND TO THE FULL EXTENT PERMITTED BY APPLICABLE LAW, THE SOFTWARE IS PROVIDED “AS-IS”. CSIRO MAKES NO REPRESENTATIONS, WARRANTIES OR CONDITIONS OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY REPRESENTATIONS, WARRANTIES OR CONDITIONS REGARDING THE CONTENTS OR ACCURACY OF THE SOFTWARE, OR OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, THE ABSENCE OF LATENT OR OTHER DEFECTS, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT DISCOVERABLE.

TO THE FULL EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT SHALL CSIRO BE LIABLE ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION, IN AN ACTION FOR BREACH OF CONTRACT, NEGLIGENCE OR OTHERWISE) FOR ANY CLAIM, LOSS, DAMAGES OR OTHER LIABILITY HOWSOEVER INCURRED. WITHOUT LIMITING THE SCOPE OF THE PREVIOUS SENTENCE THE EXCLUSION OF LIABILITY SHALL INCLUDE: LOSS OF PRODUCTION OR OPERATION TIME, LOSS, DAMAGE OR CORRUPTION OF DATA OR RECORDS; OR LOSS OF ANTICIPATED SAVINGS, OPPORTUNITY, REVENUE, PROFIT OR GOODWILL, OR OTHER ECONOMIC LOSS; OR ANY SPECIAL, INCIDENTAL, INDIRECT, CONSEQUENTIAL, PUNITIVE OR EXEMPLARY DAMAGES, ARISING OUT OF OR IN CONNECTION WITH THIS AGREEMENT, ACCESS OF THE SOFTWARE OR ANY OTHER DEALINGS WITH THE SOFTWARE, EVEN IF CSIRO HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH CLAIM, LOSS, DAMAGES OR OTHER LIABILITY.

APPLICABLE LEGISLATION SUCH AS THE AUSTRALIAN CONSUMER LAW MAY APPLY REPRESENTATIONS, WARRANTIES, OR CONDITIONS, OR IMPOSES OBLIGATIONS OR LIABILITY ON CSIRO THAT CANNOT BE EXCLUDED, RESTRICTED OR MODIFIED TO THE FULL EXTENT SET OUT IN THE EXPRESS TERMS OF THIS CLAUSE ABOVE “CONSUMER GUARANTEES”. TO THE EXTENT THAT SUCH CONSUMER GUARANTEES CONTINUE TO APPLY, THEN TO THE FULL EXTENT PERMITTED BY THE APPLICABLE LEGISLATION, THE LIABILITY OF CSIRO UNDER THE RELEVANT CONSUMER GUARANTEE IS LIMITED (WHERE PERMITTED AT CSIRO’S OPTION) TO ONE OF FOLLOWING REMEDIES OR SUBSTANTIALLY EQUIVALENT REMEDIES:

  1. THE REPLACEMENT OF THE SOFTWARE, THE SUPPLY OF EQUIVALENT SOFTWARE, OR SUPPLYING RELEVANT SERVICES AGAIN;
  2. THE REPAIR OF THE SOFTWARE;
  3. THE PAYMENT OF THE COST OF REPLACING THE SOFTWARE, OF ACQUIRING EQUIVALENT SOFTWARE, HAVING THE RELEVANT SERVICES SUPPLIED AGAIN, OR HAVING THE SOFTWARE REPAIRED.

IN THIS CLAUSE, CSIRO INCLUDES ANY THIRD PARTY AUTHOR OR OWNER OF ANY PART OF THE SOFTWARE OR MATERIAL DISTRIBUTED WITH IT. CSIRO MAY ENFORCE ANY RIGHTS ON BEHALF OF THE RELEVANT THIRD PARTY.

API Reference

sambuca_core.exceptions

Sambuca exception definitions.

exception sambuca_core.exceptions.DataValidationError[source]

The data file failed validation.

exception sambuca_core.exceptions.SambucaException[source]

Root exception class for Sambuca exceptions.

Only used as a base class for any Sambuca errors. This exception is never raised directly.

exception sambuca_core.exceptions.UnsupportedDataFormatError[source]

The file format is not supported by Sambuca.

sambuca_core.forward_model

class sambuca_core.forward_model.ForwardModelResults(r_substratum, rrs, rrsdp, r_0_minus, rdp_0_minus, kd, kub, kuc, a, a_ph_star, a_cdom_star, a_nap_star, a_ph, a_cdom, a_nap, a_water, bb, bb_ph_star, bb_nap_star, bb_ph, bb_nap, bb_water)

A namedtuple containing the forward model results.

r_substratum

The combined substrate, or substrate1 if the optional second substrate was not provided.

Type:numpy.ndarray
rrs

Modelled remotely-sensed reflectance.

Type:numpy.ndarray
rrsdp

Modelled optically-deep remotely-sensed reflectance.

Type:numpy.ndarray
r_0_minus

Modelled remotely-sensed closed reflectance (R(0-)).

Type:numpy.ndarray
rdp_0_minus

Modelled optically-deep remotely-sensed closed reflectance (Rdp(0-)).

Type:numpy.ndarray
kd

TODO

Type:numpy.ndarray
kub

TODO

Type:numpy.ndarray
kuc

TODO

Type:numpy.ndarray
a

Modelled total absorption (absorption due to water + a_ph + a_cdom + a_nap)

Type:numpy.ndarray
a_ph_star

Specific absorption of phytoplankton. Although this is an input to the Sambuca model, it is included here for ease of access by client code.

Type:numpy.ndarray
a_cdom_star

Modelled specific absorption of coloured dissolved organic particulates (CDOM).

Type:numpy.ndarray
a_nap_star

Modelled specific absorption of non-algal particulates (NAP).

Type:numpy.ndarray
a_ph

Modelled absorption of phytoplankton.

Type:numpy.ndarray
a_cdom

Modelled absorption of CDOM.

Type:numpy.ndarray
a_nap

Modelled absorption of NAP.

Type:numpy.ndarray
a_water

Absorption coefficient of water. Another model input included in the results structure for convenience.

Type:numpy.ndarray
bb

Modelled total backscatter (bb_water + bb_ph + bb_nap).

Type:numpy.ndarray
bb_ph_star

Modelled specific backscatter of phytoplankton.

Type:numpy.ndarray
bb_nap_star

Modelled specific backscatter of NAP.

Type:numpy.ndarray
bb_ph

Modelled backscatter of phytoplankton.

Type:numpy.ndarray
bb_nap

Modelled backscatter of NAP.

Type:numpy.ndarray
bb_water

Modelled backscatter of water.

Type:numpy.ndarray
sambuca_core.forward_model(chl, cdom, nap, depth, substrate1, wavelengths, a_water, a_ph_star, num_bands, substrate_fraction=1, substrate2=None, a_cdom_slope=0.0168052, a_nap_slope=0.00977262, bb_ph_slope=0.878138, bb_nap_slope=None, lambda0cdom=550.0, lambda0nap=550.0, lambda0x=546.0, x_ph_lambda0x=0.00157747, x_nap_lambda0x=0.0225353, a_cdom_lambda0cdom=1.0, a_nap_lambda0nap=0.00433, bb_lambda_ref=550, water_refractive_index=1.33784, theta_air=30.0, off_nadir=0.0, q_factor=3.141592653589793)[source]

Semi-analytical Lee/Sambuca forward model.

TODO: Extended description goes here.

TODO: For those arguments which have units, the units should be stated.

Parameters:
  • chl (float) – Concentration of chlorophyll (algal organic particulates).
  • cdom (float) – Concentration of coloured dissolved organic particulates (CDOM).
  • nap (float) – Concentration of non-algal particulates (NAP).
  • depth (float) – Water column depth.
  • substrate1 (array-like) – A benthic substrate.
  • wavelengths (array-like) – Central wavelengths of the modelled spectral bands.
  • a_water (array-like) – Absorption coefficient of pure water
  • a_ph_star (array-like) – Specific absorption of phytoplankton.
  • num_bands (int) – The number of spectral bands.
  • substrate_fraction (float) – Substrate proportion, used to generate a convex combination of substrate1 and substrate2.
  • substrate2 (array-like, optional) – A benthic substrate.
  • a_cdom_slope (float, optional) – slope of CDOM absorption
  • a_nap_slope (float, optional) – slope of NAP absorption
  • bb_ph_slope (float, optional) – Power law exponent for the phytoplankton backscattering coefficient.
  • bb_nap_slope (float, optional) – Power law exponent for the NAP backscattering coefficient. If no value is supplied, the default behaviour is to use the bb_ph_slope value.
  • lambda0cdom (float, optional) – Reference wavelength for CDOM absorption.
  • lambda0nap (float, optional) – Reference wavelength for NAP absorption.
  • lambda0x (float, optional) – Backscattering reference wavelength.
  • x_ph_lambda0x (float, optional) – Specific backscatter of chlorophyl at lambda0x.
  • x_nap_lambda0x (float, optional) – Specific backscatter of NAP at lambda0x.
  • a_cdom_lambda0cdom (float, optional) – Absorption of CDOM at lambda0cdom.
  • a_nap_lambda0nap (float, optional) – Absorption of NAP at lambda0nap.
  • bb_lambda_ref (float, optional) – Reference wavelength for backscattering coefficient.
  • water_refractive_index (float, optional) – refractive index of water.
  • theta_air (float, optional) – solar zenith angle in degrees.
  • off_nadir (float, optional) – off-nadir angle.
  • q_factor (float, optional) – q value for producing the R(0-) values from modelled remotely-sensed reflectance (rrs) values.
Returns:

A namedtuple containing the model outputs.
Return type:

ForwardModelResults

sambuca_core.sensor_filter

Contains functions for working with Sensor Filters.

sambuca_core.sensor_filter.apply_sensor_filter(spectra, normalised_response_function)[source]

Applies a sensor filter to a spectra using the given spectral response function.

Parameters:
  • spectra (array-like) – The input spectra.
  • normalised_response_function (matrix-like) – The spectral sensitivity matrix. The first dimension determines the number of output bands. The second dimension represents the proportional contribution of each of the input bands to an output band. The size must match the number of bands in the input spectra.
Returns:

The filtered spectra.
Return type:

ndarray
sambuca_core.sensor_filter.load_sensor_filter_spectral_library(directory, base_filename, normalise=False)[source]

Loads a single sensor filter from an ENVI spectral library.

Parameters:
  • directory (str) – Directory containing the sensor filter file.
  • base_filename (str) – The filename without the extension or ‘.’ preceeding the extension.
  • normalise (bool) – If true, the filter will be normalised.
Returns:

The band-centre wavelengths. numpy.array: The sensor filter.
Return type:

numpy.array
sambuca_core.sensor_filter.load_sensor_filters(path, normalise=False, spectral_library_name_parser=None)[source]

” Loads all valid sensor filters from the given location.

Parameters:
  • path (str) – The directory path to scan for sensor filters.
  • normalise (boolean) – Determines whether the filter bands will be normalised after loading.
  • spectral_library_name_parser (function) – If supplied, this function accepts a single string argument (the full path to a spectral library file) and returns the sensor filter name that will be used in the dictionary of results.
Returns:

A dictionary of 2-tuples of numpy.ndarrays.

The first element contains the band centre wavelengths of the input bands, while the second element contains the filter. Dictionary is keyed by filter name inferred from the sheet name.

Note that names are not disambiguated, so that if more than one filter has the same name, only the first will be returned and no error will be raised (although it will be logged).
Return type:

dict
sambuca_core.sensor_filter.load_sensor_filters_excel(filename, normalise=False, sheet_names=None)[source]

Loads sensor filters from an Excel file. Both new style XLSX and old-style XLS formats are supported.

Parameters:
  • filename (str) – full path to the Excel file.
  • normalise (boolean) – Determines whether the filter bands will be normalised after loading.
  • sheet_names (list) – Optional list of worksheet names to load. The default is to attempt to load all worksheets.
Returns:

A dictionary of 2-tuples of numpy.ndarrays.

The first element contains the band centre wavelengths of the input bands, while the second element contains the filter. Dictionary is keyed by filter name inferred from the sheet name.
Return type:

dict

sambuca_core.spectra_operations

Contains functions for manipulating the (wavelength, value) tuples returned by the spectra readers.

sambuca_core.spectra_operations.spectra_apply_wavelength_mask(spectra, mask)[source]

Applies a wavelength mask to a spectra ((wavelengths, values) tuple). All values in the spectra that are not in the mask will be removed in the returned values. The input spectra is not modified.

Parameters:
  • spectra (tuple) – the (wavelengths, values) spectra tuple.
  • mask (array-like) – The wavelength values that should be retained.
Returns:

The masked tuple of (wavelengths, values).
sambuca_core.spectra_operations.spectra_find_common_wavelengths(*args)[source]

Finds the common subset of wavelengths for the given inputs. I could have called this intersect, but chose the name based on purpose.

Parameters:*args (array-like) – a vector of wavelength values, or a (wavelength, values) tuple.
Returns:
The common subset of wavelengths, which can be used as an
input to spectra_apply_wavelength_mask.
Return type:numpy.ndarray

sambuca_core.spectra_readers

Contains functions for loading collections of spectra from Sambuca spectral database directories.

sambuca_core.spectra_readers.load_all_spectral_libraries(path, validate=True)[source]

Loads all valid spectra from the given location.

Parameters:
  • path (str) – The directory path to scan for supported spectra files.
  • validate (bool) – If true, data validation will be performed.
Returns:

A dictionary of 2-tuples of numpy.ndarrays.

The first element contains the band centre wavelengths of the input bands, while the second element contains the spectra values. Dictionary is keyed by spectra name built from the file and band/sheet names, separated by a colon.

Note that names are not disambiguated, so that if more than one filter has the same name, only the first will be returned and no error will be raised (although it will be logged).

Note that the filename component is always converted to lower case. This is required for consistent results on Linux and Windows.
Return type:

dict
sambuca_core.spectra_readers.load_csv_spectral_library(filename, validate=True)[source]

Loads a spectral library from a CSV file. The CSV file must have a header row, and the wavelengths must be the first column.

Parameters:
  • filename (str) – full path to the Excel file.
  • validate (bool) – If true, data validation will be performed.
Returns:

A dictionary of 2-tuples of numpy.ndarrays.

The first element contains the band centre wavelengths, while the second element contains the spectra. The dictionary is keyed by spectra name, formed by concatenation of the file and band names. This allows multiple spectra from multiple files to be unambigiously collected into a dictionary. Note that the filename component is always converted to lower case. This is required for consistent results on Linux and Windows.
Return type:

dict
sambuca_core.spectra_readers.load_envi_spectral_library(directory, base_filename, validate=True)[source]

Loads spectra from an ENVI spectral library.

Parameters:
  • directory (str) – Directory containing the spectral library file.
  • base_filename (str) – The filename without the extension or ‘.’ preceeding the extension.
  • validate (bool) – If true, data validation will be performed.
Returns:

A dictionary of 2-tuples of numpy.ndarrays.

The first element contains the band centre wavelengths, while the second element contains the spectra. The dictionary is keyed by spectra name, formed by concatenation of the file and band names. This allows multiple spectra from multiple files to be unambigiously collected into a dictionary. Note that the filename component is always converted to lower case. This is required for consistent results on Linux and Windows.
Return type:

dict
sambuca_core.spectra_readers.load_excel_spectral_library(filename, sheet_names=None, validate=True)[source]

Loads a spectral library from an Excel file. Both new style XLSX and old-style XLS formats are supported.

Parameters:
  • filename (str) – full path to the Excel file.
  • sheet_names (list) – Optional list of worksheet names to load. The default is to attempt to load all worksheets.
  • validate (bool) – If true, data validation will be performed.
Returns:

A dictionary of 2-tuples of numpy.ndarrays.

The first element contains the band centre wavelengths, while the second element contains the spectra. The dictionary is keyed by spectra name, formed by concatenation of the file and band names. This allows multiple spectra from multiple files to be unambigiously collected into a dictionary. Note that the filename component is always converted to lower case. This is required for consistent results on Linux and Windows.
Return type:

dict
sambuca_core.spectra_readers.load_spectral_library(filename, validate=True)[source]

Loads a single spectral library from the given file name from any supported format (selected by file extension).

Parameters:
  • filename (str) – full path to the file.
  • validate (bool) – If true, data validation will be performed.
Returns:

A dictionary of 2-tuples of numpy.ndarrays.

The first element contains the band centre wavelengths of the input bands, while the second element contains the spectra values. Dictionary is keyed by spectra name built from the file and band/sheet names, separated by a colon. For example: Moreton_Bay_speclib:white_sand

Note that names are not disambiguated, so that if more than one filter has the same name, only the first will be returned and no error will be raised (although it will be logged).

Note that the filename component is always converted to lower case. This is required for consistent results on Linux and Windows.
Return type:

dict

sambuca_core.utility

Sambuca_Core utility code module

Collections

Operating system-specific utility functions.

sambuca_core.utility.collections.merge_dictionary(target, new_items)[source]

Merges a dictionary into the master set, warning when a duplicate name is detected. Keys from new_items that are already present in target will generate warnings without modifying target.

And yes, I know there are builtin methods to merge dictionaries (update), but I wanted finer control over handling for existing keys.

Parameters:
  • target (dictionary) – The destination dictionary.
  • new_items (dictionary) – The dictionary of new items to merge.
Returns:

target, with all unique items merged from new items.
Return type:

dict
sambuca_core.utility.collections.pairwise(iterable)[source]

s -> (s0,s1), (s1,s2), (s2, s3), …

Numpy

Numpy-specific utility functions.

sambuca_core.utility.numpy.strictly_decreasing(x)[source]

Tests if a 1D vector is strictly decreasing, where x[i+1] < x[i] for i in [0 .. len(x)].

Parameters:x (array-like) – The vector to test.
Returns:True if x is strictly decreasing; false otherwise.
Return type:bool
sambuca_core.utility.numpy.strictly_increasing(x)[source]

Tests if a 1D vector is strictly increasing, where x[i+1] > x[i] for i in [0 .. len(x)].

Parameters:x (array-like) – The vector to test.
Returns:True if x is strictly increasing; false otherwise.
Return type:bool
OS

Operating system-specific utility functions.

sambuca_core.utility.os.list_files(directory, extensions=None)[source]

Get a list of files in a directory, filtered by an optional list of extensions.

Parameters:
  • directory (str) – The directory to list.
  • extensions (list) – Optional list of file extensions.
Returns:

The list of matching file info objects.
Return type:

list

Indices and tables