15-16 November 2016, Ludwig-Maximilians-Universität München (LMU), Germany
Conveners: Veronika Eyring, Björn Brötz, Axel Lauer, Alexander Löw, Ben Müller, and Mattia Righi
The 2nd workshop on the technical development of the Earth System Model Evaluation Tool (ESMValTool) was held at the Ludwig-Maximilians-Universität München (LMU) in the Department of Geography from 15-16 November 2016. The ESMValTool is developed as a community system, open to both users and developers, hence encouraging open exchange of evaluation methods and results (Eyring et al., 2016c). This will facilitate and improve Earth System Model (ESM) evaluation beyond the state-of-the-art and aims at supporting model evaluation and development activities at individual modelling centres and within the Coupled Model Intercomparison Project (CMIP). It is envisaged to run the tool routinely on model output submitted to CMIP Phase 6 (CMIP6, Eyring et al. (2016a)), utilizing observations and reanalyses available through the Earth System Grid Federation (ESGF) in standard formats (obs4MIPs/ana4MIPs) or provided by the user.
To download the workshop summary as pdf click here
The workshop invitations were restricted to a subgroup of the ESMValTool Development Team that works on general technical issues and the structure of the tool, as well as the coupling of the tool to the ESGF. A total of 24 participants from 10 institutions (9 from Europe, 1 from the US) participating in several projects that fund the ESMValTool development came together to define the required steps and a timeline ahead towards the next ESMValTool release as open-source software. These projects include (a) the EU Horizon 2020 Coordinated Research in Earth Systems and Climate: Experiments, kNowledge, Dissemination and Outreach (CRESCENDO) project, (b) the EU Horizon 2020 PRocess-based climate sIMulation: AdVances in high-resolution modelling and European climate Risk Assessment (PRIMAVERA) project, (c) the Copernicus Climate Change Service (C3S) Lot 1 and Lot 2 Metrics and Access to Global Indices for Climate Projections (C3S-MAGIC) projects, (d) the BMMF CMIP6-DICAD project that funds CMIP6 activities in Germany, and (e) the ESA Climate Change Initiative Climate Model User Group (ESA CCI CMUG) project.
In BLOCK 1 the present status of the ESMValTool and future perspectives were presented. Veronika Eyring summarized the decisions of the ‘1st Technical ESMValTool Workshop’ that was held in March 2015 and reviewed what has been achieved and what is still ongoing. Several new institutions have joined the ESMValTool development team since then, supported by various projects to advance the technical and scientific development of the tool. Veronika outlined the workshop goal to define a common strategy across projects and institutions towards a more efficient performance and improved provenance of the ESMValTool in time for the analysis of CMIP6 simulations. A perspective how to achieve this has been developed in a white paper as part of the European Network for Earth System Modelling Phase 2 (ISENES-2) project (Eyring et al., 2016b). Axel Lauer showed the current status of the tool concerning technical improvements, diagnostics and implemented observations, and one participant from each institution summarized their main ESMValTool related work and projects:
- DLR (presented by Veronika Eyring) is the ESMValTool PI and core development team of the ESMValTool. As such, it is responsible for the coordination and common strategy across multiple partners and projects but is also heavily involved in the technical development of the tool and the expansion of the ESMValTool with new diagnostics and metrics from different research fields (e.g. aerosols, chemistry, clouds, greenhouse gases, sea-ice, emergent constraints, weighting of model projections). DLR funds the ESMValTool development through internal projects to secure long-term support and maintenance, and the ESMVal group is involved in several third-party funded projects that support the development (C3S-MAGIC, CMIP6-DICAD, CRESCENDO, and ESA CCI CMUG).
- BSC (presented by Javier Vegas-Regidor) contributes to PRIMAVERA WP1&2, with the objective to develop diagnostics and metrics and to share the developments with the community. They are also part of C3S-MAGIC where specific metrics will be added to the ESMValTool using s2dverification, an R package developed at BSC. The ESMValTool has the potential to be a good long-term solution for the BSC diagnostics needs. Concerns were expressed regarding the efficiency of the tool in particular for the use of high-resolution data in PRIMAVERA (CPU-time and disk space, e.g. time-series plot, pre-processing is slow, need to reprocess data just for extending the time-period.). BSC has implemented the capability to run multiple instances from one installation and created an xml scheme for the ESMValTool. Both are available in the ESMValTool subversion (svn) development environment.
- DKRZ (presented by Stephan Kindermann) participates in the ESMValTool development as a contribution to the CMIP6-DICAD and Copernicus Lot 1 projects. DKRZ hosts a large data pool (~5 Pb) for generic CMIP5/6 related data access and data processing. Within the CMIP6-DICAD project, the focus is on the organization, data management, and ESGF integration (publication, replication), and the coupling of the ESMValTool to the ESGF infrastructure. As part of Copernicus Lot 1 global and regional data integration into the C3S will be realized and the Birdhouse Web Processing System (WPS) will be implemented as a compute solution at DKRZ, CEDA, and IPSL.
- Esciencecenter (presented by Willem van Hage) has recently joined the ESMValTool development team as part of C3S-MAGIC with a focus on improving the technical development of the tool towards enhanced efficiency and modularity. The Esciencecenter will help bridging the scientists’ work with the computing centers. They will bring in their long-term experience in scientific programming to provide consulting and advice for the technical development of the tool.
- FUB (presented by Christopher Kadow) established cooperation between the ESMValTool development team and Freva as part of the CMIP6-DICAD project. In this project, Freva will be used to visualize the ESMValTool results and to facilitate discussions of the ESMValTool results among the ESMValTool development team and the climate modeling groups who submit the models’ output to the ESGF.
- GFDL (presented by John Krasting) is one of the first groups that integrated the ESMValTool into its modeling workflow. It also educates the PIs about the tool. In addition, GFDL contributes new diagnostics to the ESMValTool, for example the diagnostics on snowfall climatology (Krasting et al., 2013) and at a later stage also diagnostics from MOM6, a next-generation open-source ocean model that uses the World Ocean Atlas 2009 data for the evaluation. There is a clear preference from GFDL that the entire ESMValTool development system is moved to GitHub (open and private) due to security restrictions regarding access to DLR servers. Concerns are expressed on the memory footprint and speed of the tool (e.g. diurnal precipitation diagnostics are slow and memory intensive).
- IPSL (presented by Nikolay Kadygrov) is another modeling group that uses the ESMValTool for operational support of IPSL-CM6 model evaluation. IPSL holds 450 Tb of replicated model data, which will reach 3-4 Pb in CMIP6. The aim is to set up well-tuned diagnostics on the IPSL ESGF node that can run on any set of the CMIP models and to let scientists at IPSL easily run the ESMValTool (in parallel), to compare and evaluate different configurations of the IPSL-CM6 model. Concerns were raised regarding the not-unique file-naming and a proposal how to address this issue with a slightly different structure of the ESMValTool namelists has been presented.
- ISAC (presented by Jost von Hardenberg) has newly joined the ESMValTool development team and plans to contribute to the ESMValTool development as part of PRIMAVERA, CRESCENDO, C3S-MAGIC, and EC-EARTH3 tuning. ISAC will, for example, add extreme events, stratosphere-troposphere coupling, and blocking events diagnostics. The implementation requires language support of the ESMValTool for R and Python and the question was raised whether additional support for other languages (Julia and Octave) could be provided.
- LMU (presented by Ben Müller) is part of the ESMValTool core development team in CRESCENDO. LMU is mainly responsible for implementing a reporting and testing system into the ESMValTool. As part of the ESA CCI CMUG project, the focus is on the implementation of diagnostics for soil moisture, sea surface temperature, land cover, ocean color, fire, surface water, and energy fluxes into the ESMValTool.
- MetOffice/ University of Reading (presented by Jeremy Walton) contributes to the coupling of the ESMValTool to the ESGF as part of CRESCENDO, works on an improved ESMValTool backend as part of C3S-MAGIC, and implements the AutoAssess radiation and other diagnostics into ESMValTool as part of the ESA CCI CMUG project. A key goal over the coming year is the merging of the ESMValTool and AutoAssess based on a rewrite of the ESMValTool backend using Iris (see below for details).
- SMHI (presented by Martin Evaldsson) is part of the ESMValTool core development team in CRESCENDO and contributes to the technical development of the tool. Within CRESCENDO, SMHI provides technical support and guidance for integration of diagnostics from WP2. In addition, SMHI works on the implementation of new diagnostics on storm tracks, quality work on satellite based cloud data sets, and blocking events into the tool. Possible overlap with ISAC plans in PRIMAVERA for the diagnostics on blocking events needs to be sorted out soon.
BLOCK 2 discussed the Workflow and Interfaces in ESMValTool and AutoAssess. As a basis for discussing a strategy for the merge of the ESMValTool and AutoAssess, the current ESMValTool workflow was summarized by Mattia Righi. The workflow is controlled by the main namelist and additional variable and diagnostic specific configuration files. The backend currently reformats the input data to meet CF/CMOR standard, fixes known errors in specific model output files and calculates derived variables if needed. The goal is to get the ESMValTool ready for routine evaluation in CMIP6 including the coupling to the ESGF. The AutoAssess workflow was summarized by Thomas Voigt. It is handled by the graphical user interface “Rose suites” and includes the following steps: configuration, data retrieval, creation of Iris data cubes, running the assessments, creating plots, and creation of a webpage. AutoAssess has been specifically developed as a tool for assessing and comparing a standard MetOffice climate model run to a previous run. Options of merging AutoAssess and the ESMValTool are 1) complete merger into a single tool and 2) scientific merger resulting in separate tools but with greater interoperability of metric/diagnostic code. This was further discussed during the workshop (see below).
BLOCK 3 reviewed the ESMValTool backend for data I/O and preprocessing. The current implementation of the ESMValTool backend handles the processing of input data so that they can be used by the diagnostics (Mattia Righi). This includes reformatting and on-the-fly CMORization-light of model data and observations, and time extraction. Features currently not included in the backend are regridding, consistent masking of missing values in input data, multi-model mean and ensemble statistics. Besides these missing features, a general problem of the ESMValTool backend is that the implementation is currently rather slow. It was concluded that the requirements for a revised ESMValTool backend include a full control through namelists (regridding, definition of the target grid, type/level of missing value masking, calculation of multi-model and ensemble statistics, smart file caching). Speed and efficiency will be a main priority, i.e. parallelization is a must. The use of Iris as the backbone of the revised ESMValTool backend was discussed. It was concluded that the revised ESMValTool backend will require recoding of some of the already implemented diagnostics.
An Iris overview was presented by Paul Earnshaw. Iris is an open source Python library for meteorology and climatology (http://scitools.org.uk) that is community developed and hosted on GitHub. Iris provides Iris data cubes (i.e. CF meta data layer over NumPy data arrays), multi-format reading and writing, cube interfaces to several Python libraries (NumPy, SciPy, Pandas, Matplotlib), dataset organization (e.g. spatial / temporal subset), unit conversion, and regridding / interpolation / aggregation operations (basic statistics).
Regridding in the ESMValTool is currently handled in each diagnostic and therefore specific for each language (Martin Evaldsson). It was concluded that a priority for the next months is to move regridding from diagnostics to the backend (i.e., independent of the language) on top of the reformatting. Questions discussed include the reusability of regridded data and the usage of standard grids. Identified regridding issues include the handling of land-/sea masks, auxiliary grids, and missing values.
A decision was made that we will actively work towards a merge of ESMValTool and Auto-Assess using Iris. It was agreed that for the rewriting of the ESMValTool backend as preparatory work for the merge of the ESMValTool with AutoAssess a coding workshop is needed. A major focus of the revised ESMValTool backend will be improvements in the performance. The revised backend should offer the same functionality as the existing ESMValTool and AutoAssess backends plus additional abilities, preferably in a way that diagnostics could remain as much as possible unchanged. A report will be written that outlines the requirements of the revised ESMValTool backend and also develops an implementation plan in time for the coding workshop.
BLOCK 4 reviewed the status of the coupling of ESMValTool to the ESGF. The implementation of the ESMValTool on the ESGF nodes will be based on a WPS or similar GUI (Graphical User Interface), depending on the progress achieved by collaboration between DLR, DKRZ and FUB (Freva). For any of these solutions, automated namelist generation is needed to run the ESMValTool from an XML based input. Furthermore, standard namelists, including their degree of flexibility, have to be defined. Priority data will be mirrored locally for faster processing. A very first WPS based prototype is already available and runs a single instance of the tool in a protocoled, specific runtime environment without caching of states or memory. Multi-user support and full reproducibility is provided. Within a cached environment, the tool’s backend has to support multi-user functionality, which will be included in the revised ESMValTool backend. The ESMValTool will include options for full download, reports, transferring to cloud or node storage of the results.
BLOCK 5 discussed reporting (visualization), testing and documentation. For code development, a unit testing scheme instead of an integration testing scheme will be implemented for selected diagnostics in the beginning. The challenge of this scheme is the proper implementation in mixed code environments (ncl, python, R) and the testing of the backend. The backend should only be touched within rewriting and testing needs to be properly implemented. For user (installation) tests, these could be made available with small examples of integration tests. Therefore, small data sets need to be provided with the code, optionally adding more data sets via a random data generator. For the integration testing scheme, a list of reasonable tests for resulting data (images, data sets, file numbers) was set up. A critical point was the definition of the reference for the testing and how to decide what is correct. Unit tests for selected diagnostics will be explored in addition to an integration testing scheme. The first unit test implementation should also provide prototypes of code snippets for the development of the revised backend.
For reporting, a python based HTML5-generator will be provided that is capable of gathering images and tables after running the ESMValTool. Tag based content will facilitate a more structured presentation of the results. Therefore, the diagnostics need to provide additional information using tags, like “land surface”, “ocean”, etc., as well as content layout, figure / table captions, and layout. This information needs to be written by the diagnostics. The use of metadata, both within image files and as additional linked XML files was encouraged. A recoding of already available diagnostics will be requested after a prototype has been set up. Similarly, the ESMValTool needs to be capable of reading and propagating tags from namelists.
Regarding visualization, it was agreed that the ESMValTool results for CMIP6 will not be published immediately and that initially access will only be given to the participating modeling groups and the ESMValTool development team to ensure quality control before the results become public. The structuring of the output including tagging of diagnostics, figure captions, and additionally needed metadata was discussed. In the near-term, Freva will be used mostly as a communication platform and for spreading of ESMValTool results within modeling community. There will be no pre-selection of the output by Freva. It was concluded that additionally needed metadata for the visualization will be provided both as ASCII files and EXIF or XMP headers for image files. In a first version, the system will be similar to WPS but extends functionality to ESGF node data browsing. Reporting might be included as a post-processing plug-in or the structural information from the ESMValTool could be used directly.
It was further discussed how to merge the currently-used documentation systems (wiki, user’s guide, and in-code documentation) into one system. It was agreed that the new documentation system will be based on Sphinx. The script that generates documentation via Sphinx for ESMValTool functions and procedures by extracting their comments is not capable of “error and consistency checks”, i.e. it only reads the code’s comments and structures/functions, and doesn’t check the correctness of the comments themselves, since checks of this kind require the intervention of a human proofreader. Future code needs to comply with standards (yet to be written) for in-line documentation of the code such that documentation can be generated automatically via Sphinx. The existing user guide and technical documentation will be converted to restructured text format to be then further used as input to Sphinx. In order to improve the provenance, more information should be added to the log-files. This includes a list of output files created, the repository id from svn / git as well as preserving more metadata available in the original input files.
During BLOCK 6 breakout sessions were held on the following three topics: (a) I/O backends, (b) documentation, visualization, testing and provenance; (c) coupling of the ESMValTool to ESGF and ESGF specific provenance. The results of the breakout groups were presented and discussed in a plenary session afterwards. The discussions fed into the final decisions (see BLOCK 10).
BLOCK 7 focused on diagnostics, coupling of external tools or libraries, and multi-language issues. Keith Williams summarized the current status of the coupling between AutoAssess and the ESMValTool. Two possible approaches were identified, a complete merge resulting in a single tool or a scientific merge with two separate tools but with greater interoperability. The work is ongoing and focuses on: (i) rewriting part of the AutoAssess code to work with CMOR-netCDF data rather than the PP format used by MetOffice; (ii) development of control scripts for running on a cluster; (iii) adaptation of exiting diagnostics and addition of new ones. The coupling will be further supported by the PRIMAVERA project, where users are expected to implement new diagnostics into one or the other tool to run on PRIMAVERA data. It was agreed to work towards a common backend to facilitate the interoperability of the two tools.
John Krasting showed the use of the ESMValTool in the GFDL model workflow and related issues, including problems in managing the package installation within conda due to security restrictions at GFDL. It was suggested to move towards a more python-friendly ESMValTool, including a better handling of python modules and a high-level anaconda package that could help end users to meet python module requirements more easily and also ensure the usage of consistent module versions.
New functionalities for a semi-automated workflow were discussed by Martin Evaldsson for the EC-EARTH model, which includes a new script to skip the reformat phase and provide model output directly to the tool, while also automatically generating the main namelist. Another important topic discussed is the possibility to run multiple instances of the ESMValTool at the same time. The problem is that only one interface_data/ncl.interface script is written at runtime and could be overwritten if multiple instances are run at the same time. As a solution, the interface_data path could be replaced with a folder with the namelist name or with a unique and random suffix. Another approach to improve the performance (introduced by SMHI) is to split the main namelist and run each diag-block separately: a script has been developed for this purpose (split-on-diag.py) but at the workshop it was decided that this solution is not further followed. SMHI and DLR will develop a concept how to achieve efficient execution of multiple namelists in the revised ESMValTool backend.
In BLOCK 8 Stephan Kindermann summarized conclusions on the ESMValTool interfaces and workflows that resulted from the workshop discussions and presentations so far. He first defined “interface” as a shared boundary across which two separate components of a computer system exchange information. The exchange can be between software, computer hardware, peripheral devices, humans and combinations of these. Relevant ESMValTool interface topics include the overall architecture, the „human-human interface“ (i.e. code maintenance and contribution workflow), the ESGF-coupling interface, the backend interfaces, and „plugging“ interfaces such as logging, multi-user / parallel environment, and deployment.
- It was concluded that clear interfaces already exist for the ESGF coupling. The operational application of the ESMValTool within the DKRZ ESGF-infrastructure will make use of the synda based system. Together with automated execution (e.g. by cron) the ESGF will be monitored for changes in target data and replication will be triggered as well as the execution of the ESMValTool upon successful replication. An alpha version of this integration is expected to be in place by the end of 2016.
- The human/human interface (communication between developers) needs to be improved. It was recommended to switch the ESMValTool development from svn to GitHub and make use of the tools available there (issue tracker, continuous integration, etc.). It was concluded that a private GitHub development is well suited for a multi-institutional software development due to the secure and non-bureaucratic access right management.
- It was concluded that the interface to/from the diagnostics in terms of data will remain in the form of netCDF files as output of the ESMValTool backend. As the next step towards the backend rewrite a document of the minimum requirements of the revised ESMValTool backend will be created and a coding workshop will be held (see BLOCK 3and 10).
In BLOCK 9 Björn Brötz summarized the current status of the technical infrastructure for joint development. The current infrastructure is based on a Wiki page and the Mantis Bug Tracker, both accessible to developers only. The code development is handled within subversion (svn), consisting of a stable trunk (managed by the core development team) and several development branches. The whole system in hosted at DLR. There is a strong desire from the community to move towards git, which is considered a better tool for joint development of software. It was discussed how to organize the transition from svn to git. A critical issue is the requirement by several developers to have some of their contributions not visible to the public: this is the case, for example, for the development of unpublished diagnostics or for the work of PhD students. Other developers, however, prefer to develop in a public workflow and to make their work quickly available to the community. It was decided to work towards a solution supporting both approaches (private and public GitHub) while keeping the latest stable version of the ESMValTool always synchronized. This will require a redefinition of the current development and integration workflow. DLR will develop a strategy for this transition and will circulate it among the developers.
Finally BLOCK 10 concluded the workshop with a final discussion and decisions.
- A decision was made that we will actively work towards a merge of ESMValTool and Auto-Assess using Iris. It was agreed that for the rewriting of the ESMValTool backend as preparatory work for the merge of the ESMValTool with AutoAssess a coding workshop is needed.
- It was decided that a 5-day coding workshop for rewriting the ESMValTool backend will be held at the MetOffice on 6-10 February 2017 in Exeter, UK. Participants will include scientists from the Iris and AutoAssess development teams of the MetOffice and ESMValTool developers from DLR, University of Reading, BSC, Esciencecenter, SMHI, and possibly from DKRZ and ISAC.
- To prepare for this coding workshop, a report will be written by the end of January. The writing will be led by DLR and other partners who are involved in the rewrite of the backend will contribute. The report will include (a) requirements for the revised ESMValTool backend; (b) current issues; (c) an implementation plan.
- Regarding the harmonization of the currently used documentation systems (website, wiki, user’s guide), it was concluded that the future ESMValTool documentation should be based on the already implemented Sphinx documentation system that will be extended to include information from the wiki and the user’s guide. If needed, another coding workshop specifically on this topic could be held around April 2017. This activity will be led by MetOffice, LMU and DLR.
- A reporting facility will be developed by LMU and DLR as part of CRESCENDO and implemented by end of March 2017. Standards for future diagnostics will be developed and included into the coding standards for the ESMValTool.
- The ESMValTool will be run at DKRZ for routine evaluation of the CMIP6 models as soon as the output is submitted to the ESGF. The results will be first reviewed and iterated on a password restricted website with access for the CMIP6 modeling groups and the ESMValTool Development Team to ensure quality control before the ESMValTool results will be made publically available. This strategy has been supported, encouraged, and approved also by the Working Group of Coupled Modelling (WGCM).
- For automatic testing, two options were discussed. (a) Continue current approach with reference data, extending testing with new routines (comparison of graphic files), possibly integration of synthetic data (LMU, DLR and NLesc to help developing the concept). (b) Implement unit testing for selected diagnostics (e.g. perfmetrics) and provide this as a unit testing template for Python and NCL. For this a wrapper that allows the testing of NCL routines would need to be developed.
- DLR and DKRZ will further work on improving the provenance of the ESMValTool by adding more information to the log-file such as a listing of the output files (could be done by the reporting framework), repository id from svn / git, and selected information on the software environment.
- DLR will provide a high-level anaconda package for the next major release, as requested by GFDL.
- In addition, the transition from svn to GitHub was discussed. The current version on GitHub will be updated with official releases of the ESMValTool. A decision was made that we will keep a private (i.e. access restricted to the ESMValTool development team) environment for the ESMValTool development and that the current licenses will be kept. Official releases of the open source version on GitHub will preferably be accompanied with a documentation paper. The workflow document for moving the entire ESMValTool environment to GitHub (open and private) will be defined by DLR alongside a timeline for the svn to GitHub transition.
- For the way ahead, it was decided to provide regular releases of new versions of the ESMValTool once major improvements took place. The next release with minor technical updates but additional diagnostics (e.g. from ESA CCI CMUG) will be made around end of January 2017. A major release (version 2.0) alongside with a documentation paper is planned for the version that will be run on CMIP6 models around summer or autumn in 2017. Regular workshops on the technical development of the ESMValTool will be held supported by coding workshops for specific topics.
- More generally, the ESMValTool Development Team will further provide user support and will substantially enhance the tool with new diagnostics as science progresses.
The workshop was held under the auspices of the ISENES-2 and CRESCENDO projects, the Institute of Atmospheric Physics of the German Aerospace Center (DLR), and the Department of Geography at LMU Munich.
- Eyring, V., Bony, S., Meehl, G.A., Senior, C.A., Stevens, B., Stouffer, R.J. and Taylor, K.E., 2016a. Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization. Geosci. Model Dev., 9(5): 1937-1958.
- Eyring, V., Gleckler, P.J., Heinze, C., Stouffer, R.J., Taylor, K.E., Balaji, V., Guilyardi, E., Joussaume, S., Kindermann, S., Lawrence, B.N., Meehl, G.A., Righi, M. and Williams, D.N., 2016b. Towards improved and more routine Earth system model evaluation in CMIP. Earth Syst. Dynam., 7(4): 813-830.
- Eyring, V., Righi, M., Lauer, A., Evaldsson, M., Wenzel, S., Jones, C., Anav, A., Andrews, O., Cionni, I., Davin, E.L., Deser, C., Ehbrecht, C., Friedlingstein, P., Gleckler, P., Gottschaldt, K.D., Hagemann, S., Juckes, M., Kindermann, S., Krasting, J., Kunert, D., Levine, R., Loew, A., Mäkelä, J., Martin, G., Mason, E., Phillips, A.S., Read, S., Rio, C., Roehrig, R., Senftleben, D., Sterl, A., van Ulft, L.H., Walton, J., Wang, S. and Williams, K.D., 2016c. ESMValTool (v1.0) – a community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP. Geosci. Model Dev., 9(5): 1747-1802.
- Krasting, J.P., Broccoli, A.J., Dixon, K.W. and Lanzante, J.R., 2013. Future Changes in Northern Hemisphere Snowfall. Journal of Climate, 26(20): 7813-7828.