Changelog

Changelog#

v1.0.0rc1 - 26.09.2025#

This version is the official release candidate for mesmer v1.0.0. It restructures much of the code base to be more modular, flexible and stable. Mesmer now fully relies on xarray data structures, including xr.DataTree. As a consequence, mesmer now needs xarray >= 2025.03. We integrated MESMER-M, allowing the users to now emulate monthly temperatures, and MESMER-X, which enables emulation of any climate variable using conditional distributions. This version introduces support of python 3.13 and dropping support of python 3.9 and 3.10.

New Features#

Breaking changes#

  • Switch random number generation for drawing emulations from np.random.seed() to np.random.default_rng() (#495). By Victoria Bauer.

  • Using Cholesky decomposition for finding covariance localization radius and drawing from the multivariate normal distribution (#408) By Victoria Bauer.

  • Removed support for python 3.9 and python 3.10 (#513, and #733) By Mathias Hauser.

  • Removed the deprecated function mask_percentage() (#654) By Mathias Hauser.

  • Removed Expression.evaluate - use Expression.evaluate_params instead (#786) By Mathias Hauser.

  • The supported versions of some dependencies were changed (#399, #405, #503, #621, #627, #683, #686, and #740):

    Package

    Old

    New

    cartopy

    not specified

    0.23

    dask

    not specified

    2024.7

    filefisher

    not required

    1.1

    joblib

    not specified

    1.4

    netcdf4

    not specified

    1.7

    numpy

    not specified

    1.26

    packaging

    not specified

    24.1

    pandas

    2.0

    2.2

    pooch

    not specified

    1.8

    properscoring

    not specified

    0.1

    pyproj

    not specified

    3.6

    regionmask

    0.8

    0.12

    scikit-learn

    not specified

    1.5

    scipy

    not specified

    1.14

    shapely

    not specified

    2.0

    statsmodels

    not specified

    0.14

    xarray

    2023.04

    2025.03

Deprecations#

  • Deprecated mask_3D_frac_approx as the functionality is now offered in regionmask v0.12.0 (#451).

Bug fixes#

  • Averaging standard deviations for the AR parameters of global variability over several ensemble members and scenarios now averages the variances (#499). By Victoria Bauer.

Documentation#

  • Updated and extended the development Guide (#511, #523)

  • Added example notebooks for calibrating on multiple scenarios and ensemble members and emulating multiple scenarios (#521).

Internal Changes#

  • Start testing the minimum versions of required dependencies (#398). By Mathias Hauser.

  • Restore compatibility with pandas version v2.2 and xarray version v2024.02 (#404). By Mathias Hauser.

  • Explicitly include all required dependencies (#448).

  • Unshallow the mesmer git repository on rtd (#456). By Victoria Bauer.

  • Use ruff instead of isort and flake8 to lint the code base (#490). By Mathias Hauser.

  • Consolidate package metadata and configuration in pyproject.toml (#650). By Mathias Hauser.

  • Made the create_equal_dim_names() private (#653). By Mathias Hauser.

  • Removed the regionmaskcompat.py module. It is no longer needed after requiring regionmask v0.12 (#683). By Mathias Hauser.

  • Moved data modules to mesmer namespace (#773). By Mathias Hauser.

  • Made the core data module explicitly private by renaming it to _core (#774). By Mathias Hauser.

Data structure using DataTree#

This release uses xr.DataTree as data structure to handle multiple scenarios. This was originally done with the prototype xarray-datatree package. After the port of DataTree to xarray stabilized, mesmer briefly supported both DataTree versions (the one in xarray-datatree and in xarray) before dropping support for xarray-datatree.

  • Switch to storing several predictors in one xr.Dataset` per scenario node in a DataTree (#677).

  • Enable passing a DataTree to the auto regression functions (#570, #677).

  • Enable passing DataTree and xr.Dataset to LinearRegression (#566, and #720).

  • Add weighting function for several scenarios (#567).

  • Add function to compute anomalies over several scenarios stored in a DataTree (#625).

  • Add utility functions for DataTree (#556).

  • Add a wrapper to allow DataTree in many data handling functions (#632, #643, #641, #644, and #682).

  • Add calibration integration tests for multiple scenarios and change parameter files to netcdfs with new naming structure (#537)

  • Add new integration tests for drawing realisations (#599)

  • Add helper function to merge DataTree objects (#701)

  • PRs related to xarray and xarray-datatree:

    • Add xarray-datatree as dependency (#554)

    • Add upper pin to xarray version to support xarray-datatree (#559).

    • Port the functionality to xarray’s DataTree implementation (#607).

    • Drop support for xarray-datatree (#627).

  • Add filefisher as dependency to handle file paths of several scenarios (#586, #592, and #629).

By Victoria Bauer and Mathias Hauser.

Integration of MESMER-X#

In the release the MESMER-X functionality is integrated into the MESMER Codebase.

  • Add MESMER-X functionality to the code base (#432)

  • Overall restructuring and refactoring of the MESMER-X code base (#645)

  • Some general refactoring and clean-up (#437, #465, #466, #467, #468, #469, #470, #502)

  • Add unit tests (#526, #533, #534, #540, #577)

  • Add integration tests (#524, #550, #553)

  • Enable to pass set values for loc and scale (only integers) and make scale parameter optional (#597).

  • Enable threshold_min_proba to be None in distrib_cov (#598).

  • Also use Nelder-Mead fit in distrib_cov._minimize() for option_NelderMead == "best_run" when Powell fit was not successful (#600).

  • Return logpmf for discrete distributions in distrib_cov._fg_fun_nll_cubed() (#602).

  • Speed-up MESMER-X

    • add method to calculate params of a distribution (#539)

    • avoiding frozen distributions (#532)

    • not broadcasting scalars (#613)

    • compiling the expression (#614).

By Yann Quilcaille with Victoria Bauer and Mathias Hauser.

Integration of MESMER-M#

This release integrates MESMER-M into the existing MESMER infrastructure. This includes some refactoring, bugfixes and enhancements of the MESMER-M functionality. Note that this led to some numerical changes compared to the MESMER-M publication (Nath et al., 2022).

  • move MESMER-M scripts into mesmer (#419, and #421).

  • move the harmonic model and power transformer functionalities to the stats module (#484).

  • add example script for MESMER-M workflow (#491)

  • add integration tests for MESMER-M (#501)

  • enable calibrating MESMER-M on several scenarios and ensemble members (#678) and add an example (#572).

Auto-Regression#

  • Implement functions performing the monthly (cyclo-stationary) auto-regression and adapt these functions to work with xarray. This includes extracting the drawing of spatially correlated innovations to a stand-alone function. (#473)

  • Remove the bounds of -1 and 1 on the slope of the cyclo-stationary AR(1) process. This bound is not necessary since cyclo-stationarity is also given if the slopes of a few months are (slightly) larger than one. We now return the residuals of the cyclo-stationary AR(1) process to fit the covariance matrix on these residuals. As a consequence, adjustment of the covariance matrix with the AR slopes is no longer necessary. (#480) Compare discussion in #472.

  • Implement function to localize the empirical covarince matrix for each month individually to use in drawing of spatially correlated noise in the AR process. (#479)

  • Enable passing data with a dimension without coordinates (i.e. sample dimension) to fit_auto_regression_monthly (#706).

  • Ensure residuals are ordered correctly in fit_auto_regression_monthly (#708).

Yeo-Johnson power transformer#

  • Ensure the power transformer yields the correct normalization for more cases (#440):

    • expand the upper bound of the first coefficient from \(1\) to \(\infty\), i.e. to 1e10 (#446, #501)

    • remove jacobian rosen_der from fit (#447)

    • change optimization method from SLSQP to Nelder-Mead (#455)

  • adjust the first guess to assume the data is normally distributed (#429)

  • make (back-) transformations more stable by using np.expm1() and np.log1p() (#494)

  • rewrite power transformer to work with xarray, and refactor from a class structure to functions (#442, and #474)

  • fix small code issues and clean the docstrings (#436, #444, #439, #475, and #425)

  • add tests (#430)

  • Converted Yeo-Johnson power transformer functions back into a class, which allows to add additional lambda functions (#716).

  • Added a constant lambda function (#718).

  • Enable passing data with a dimension without coordinates (i.e. sample dimension) to power transformer functions (#703).

Harmonic model#

  • Performance and other optimizations:

    • only fit orders up to local minimum and use coeffs from previous order as first guess (#443)

    • infer the harmonic model order from the coefficients (#434)

    • optimization of _generate_fourier_series_order_np() (#516 and #578)

  • return residuals instead of the loss for the optimization (#460)

  • remove fitting of linear regression with yearly temperature (#415 and #488) in line with (Nath et al. 2022).

  • add helper function to upsample yearly data to monthly resolution (#418, #435, and #688).

  • de-duplicate the expression of months in their harmonic form (#415) move creation of the month array to the deepest level (#487).

  • fix indexing of harmonic model coefficients (#415)

  • Refactor variable names, small code improvements, optimization, fixes and clean docstring (#415, #424, #433, #512, #574, and #589).

  • add tests (#431, and #458)

  • add function to generate fourier series using xarray (#478)

  • Enable passing data with a dimension without coordinates (i.e. sample dimension) to harmonic model functions (#705).

By Victoria Bauer and Mathias Hauser.

Data#

  • Directly source the stratospheric aerosol optical depth data from NASA instead of using the version from Climate Explorer (#665). By Mathias Hauser.

v0.10.0 - 2024.01.04#

version 0.10.0 fixes the bug in the legacy calibration and is thus not numerically backward compatible. It also updated the supported python, pandas and xarray versions. Updating the pandas version will create an error when trying to load pickled mesmer bundles, requiring to use mesmer version v0.9.0 for this.

Bug fixes#

Ensure de-duplicating the historical ensemble members conserves their order. Previously, the legacy calibration routines used np.unique, which shuffles them. See #338 for details. (#339). By Mathias Hauser.

Breaking changes#

  • Removed support for python 3.7 and python 3.8 (#163, #365, #367, and #371). By Mathias Hauser.

  • The supported versions of some dependencies were changed (#369):

    Package

    Old

    New

    pandas

    <2.0

    >=2.0

    xarray

    not specified

    >=2023.04

New Features#

v0.9.0 - 2023.12.19#

version 0.9.0 is a big step towards rewriting mesmer. All statistical functionality was extracted and works for xarray data objects. It also contains data handling functions to prepare climate model data using xarray.

  • The restructured code is fully functional and can be used to calibrate and emulate temperature. However, it is still missing wrappers which encapsulate the full chain and helpers to simplify calibrating several scenarios and ensemble members.

  • This version still contains the legacy routines to train and emulate temperature. It should have no numerical changes, only minimal changes in usage, and offers speed gains over v0.8.3.

Known bugs#

For the legacy training, the influence of the global variability is underestimated, because the historical ensemble members are shuffled “randomly”. This is kept in v0.9.0 for backward compatibility and will be fixed in a follow-up bug fix release. For details see #338.

New Features#

Breaking changes#

Deprecations#

  • The function mesmer.create_emulations.create_emus_gt has been renamed to create_emulations.gather_gt_data() (#246). By Mathias Hauser.

  • The function mesmer.utils.select.extract_time_period is now deprecated and will be removed in a future version. Please raise an issue if you use this function (#243). By Mathias Hauser.

Bug fixes#

  • Fix three issues with core.regionmaskcompat.mask_3D_frac_approx(). Note that these issues are only relevant if passing xarray objects and/ or masks close to the poles (#202 and #218). By Mathias Hauser.

Documentation#

Internal Changes#

v0.8.3 - 2021-12-23#

New Features#

  • Add mesmer.stats._linear_regression (renamed to mesmer.stats._fit_linear_regression_np in #142). Starts the process of refactoring the codebase (#116). By Zeb Nicholls.

Bug fixes#

  • Initialize llh_max to -inf to ensure the cross validation loop is entered (#110). By Jonas Schwaab.

Documentation#

  • Fix copyright notice and release version in documentation (#127). By Zeb Nicholls.

Internal Changes#

  • Automatically upload the code coverage to codecov.io after the test suite has run (#99). By Mathias Hauser.

  • Internal refactor: moved a number of inline comments to their own line (especially if this allows to have the code on one line instead of several) and other minor cleanups (#98). By Mathias Hauser.

  • Refactor io.load_cmipng_tas and io.load_cmipng_hfds to de-duplicate their code and add tests for them (#55). By Mathias Hauser.

v0.8.2 - 2021-10-07#

Bug fixes#

  • Reintroduce ability to read in cmip5 data from the cmip5-ng archive at ETH (#90). By Lea Beusch.

Internal Changes#

  • Reproduce the test files because of a change in regionmask which affected the mesmer tests (#95). By Mathias Hauser.

  • Refactor and speed up of the Gaspari-Cohn function and the calculation of the great circle distance (#85, #88). By Mathias Hauser.

  • The geopy package is no longer a dependency of mesmer (#88). By Mathias Hauser.

  • Convert README from Markdown to reStructuredText to fix package build errors. Also allows to include the README in the docs to avoid duplication (#102). By Mathias Hauser.

v0.8.1 - 2021-07-15#

  • Update example script (#80).

v0.8.0 - 2021-07-13#

  • First release on PyPI and conda (#79).

Contents