Skip to content

Commit 1d76ec9

Browse files
authored
Feature 530 plotly kaleido for 3.0.1 (#532)
* Update kaleido and Plotly versions to 1.0.0 and 6.1.1 respectively * Update versions of kaleido and Plotly to 1.0.0 and 6.1.1 respectively. * Added strtobool functionality that used to be available in distutils. This is helpful in checking for truth values in environment variables. * Address loss of kaleido 0.x functionality and add support for determining whether to download Chrome at runtime. * Update the json file since we are using a different version of Plotly, resulting in slight changes * Add to this directory to ensure the test fixture is found. Sometimes the fixture isn't found even though it exists in a higher directory. * Update json file * Update versions of kaleido and Plotly to 1.0.0 and 6.1.1 respectively * Release notes for the 6.0.1 release * Updated dates and version for the 3.0.1 bugfix release
1 parent c0dfdec commit 1d76ec9

11 files changed

Lines changed: 260 additions & 36 deletions

File tree

.github/workflows/unit_tests.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,9 @@ jobs:
4949
run: |
5050
python -m pip install --upgrade pip
5151
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
52+
# Ensure that kaleido 1.0.0 and Plotly 6.1.1 are in use
53+
python -m pip install kaleido==1.0.0 plotly==6.1.1
54+
5255
5356
# Checking the branch name, not necessary but useful when setting things up.
5457
# - name: Extract branch name

docs/Users_Guide/release-notes.rst

Lines changed: 19 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,8 @@ describes the bugfix, enhancement, or new feature:
1010
METplotpy Release Notes
1111
=======================
1212

13-
METplotpy Version 3.0.0 release notes (20241218)
13+
14+
METplotpy Version 3.0.1 release notes (20250722)
1415
------------------------------------------------------
1516

1617
.. dropdown:: New Plots
@@ -20,40 +21,37 @@ METplotpy Version 3.0.0 release notes (20241218)
2021

2122
.. dropdown:: Enhancements
2223

23-
* Improve logging for STIGS on remaining plots (`METplus-Internal#57 <https://github.com/dtcenter/METplus-internal/issues/57>`_)
24-
* Plot new TC-Diagnostics output from TC-Pairs (`#233 <https://github.com/dtcenter/METplotpy/issues/233>`_)
25-
* **Enhance TCMPR plotter to read TCDiag lines and filter by one column and plot another column** (`#342 <https://github.com/dtcenter/METplotpy/issues/342>`_).
26-
* Create documentation and testing for TCMPR plotting code (`#383 <https://github.com/dtcenter/METplotpy/issues/383>`_).
27-
* Update fv3_physics_tend for new FV3 output format (`#380 <https://github.com/dtcenter/METplotpy/issues/380>`_).
28-
* Update GitHub actions workflows to switch from node 16 to node 20 (`#414 <https://github.com/dtcenter/METplotpy/issues/414>`_).
29-
* **Add GitHub action to run SonarQube for METplotpy pull requests and feature branches** (`#429 <https://github.com/dtcenter/METplotpy/issues/429>`_).
30-
* **Add a summary curve to the ROC diagram** (`#399 <https://github.com/dtcenter/METplotpy/issues/399>`_).
31-
* **Hide/show the legend entries line-by-line** (`#355 <https://github.com/dtcenter/METplotpy/issues/355>`_).
32-
* **Specify the color of the no resolution and no skill lines for the reliability diagram** (`#329 <https://github.com/dtcenter/METplotpy/issues/329>`_).
3324

25+
None
3426

3527

3628
.. dropdown:: Internal
3729

38-
* Improve unit test infrastructure and coverage (`#461 <https://github.com/dtcenter/METplotpy/issues/461>`_)
39-
* Update GitHub issue and pull request templates to reflect the current development workflow details (`#388 <https://github.com/dtcenter/METplotpy/issues/388>`_).
40-
* Consider using only .yml or only .yaml extensions (`#417 <https://github.com/dtcenter/METplotpy/issues/417>`_).
41-
* METplotpy: Code coverage statistics (`#55 <https://github.com/dtcenter/METplus-Internal/issues/55>`_).
30+
None
4231

4332

4433

4534
.. dropdown:: Bugfixes
4635

47-
* TC-RMW plot needs updating to support changes to input (`#425 <https://github.com/dtcenter/METplotpy/issues/425>`_).
48-
* **Documentation - Fix METviewer link in line plot** (`#385 <https://github.com/dtcenter/METplotpy/issues/385>`_).
49-
* **Inconsistency with generating plot in METviewer vs command line** (`#391 <https://github.com/dtcenter/METplotpy/issues/391>`_).
50-
* **Bugfix for ROC plot image saving to file** (`#394 <https://github.com/dtcenter/METplotpy/issues/394>`_).
51-
* Bugfix for errors in line and revision_series tests (`#401 <https://github.com/dtcenter/METplotpy/issues/401>`_).
36+
* **Incorporate Plotly/kaleido fixes to the main_3.0 code** (`#530 <https://github.com/dtcenter/METplotpy/issues/530>`_).
5237

5338

5439
METplotpy Upgrade Instructions
5540
==============================
5641

57-
None
42+
.. note::
43+
44+
In June 2025, Plotly made significant updates to the kaleido package with the 1.0.0
45+
release by removing Google Chrome code. Now, users will need to have Google Chrome
46+
installed in directories specified in this Plotly documentation (based on operating
47+
system):
48+
https://plotly.com/python/static-image-export/
49+
50+
The METplotpy code downloads Chrome at runtime by invoking the kaleido.get_chrome_sync()
51+
method call.
52+
53+
If users do not wish to have Chrome downloaded at run time and already have Chrome installed
54+
in one of the expected locations (specified in the Plotly link above), then the PRE_LOAD_CHROME environment variable
55+
will need to be set to 'True' (case insensitive string).
5856

5957

docs/conf.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -26,12 +26,12 @@
2626
author = 'UCAR/NSF NCAR, NOAA, CSU/CIRA, and CU/CIRES'
2727
author_list = 'Adriaansen, D., C. Kalb, D. Fillmore, T. Jensen, L. Goodrich, M. Win-Gildenmeister, T. Burek, and H. Fisher'
2828

29-
version = '3.0.0'
29+
version = '3.0.1'
3030
verinfo = version
3131
release = f'{version}'
32-
release_year = '2024'
32+
release_year = '2025'
3333

34-
release_date = f'{release_year}-12-18'
34+
release_date = f'{release_year}-07-22'
3535

3636
copyright = f'{release_year}, {author}'
3737

metplotpy/plots/base_plot.py

Lines changed: 43 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -20,15 +20,53 @@
2020
import numpy as np
2121
import yaml
2222
from typing import Union
23-
24-
import plotly.io as pio
25-
23+
import kaleido
2624
import metplotpy.plots.util
25+
from metplotpy.plots.util import strtobool
2726
from .config import Config
2827
from metplotpy.plots.context_filter import ContextFilter
2928

30-
# set kaleido to use single process to prevent GPU errors in containers
31-
pio.kaleido.scope.chromium_args += ("--single-process",)
29+
# kaleido 0.x will be deprecated after September 2025 and Chrome will no longer
30+
# be included with kaleido from version 1.0.0. Explicitly get Chrome via call to kaleido.
31+
32+
# In some instances, we do NOT want Chrome to be installed at run-time. If the
33+
# PRE_LOAD_CHROME environment variable exists, or set to TRUE,
34+
# then Chrome will be assumed to have been pre-loaded. Otherwise,
35+
# invoke get_chrome_sync() to install Chrome in the
36+
# /path-to-python-libs/pythonx.yz/site-packages/... directory
37+
38+
# Check if the PRE_LOAD_CHROME env variable exists
39+
aquire_chrome = False
40+
41+
turn_on_logging = strtobool('LOG_BASE_PLOT')
42+
# Log when Chrome is downloaded at runtime
43+
if turn_on_logging is True:
44+
log = logging.getLogger("base_plot")
45+
log.setLevel(logging.INFO)
46+
47+
formatter = logging.Formatter("%(asctime)s [%(levelname)s] | %(name)s | %(message)s")
48+
49+
# set the WRITE_LOG env var to True to save the log message to a
50+
# separate log file
51+
write_log = strtobool('WRITE_LOG')
52+
if write_log is True:
53+
file_handler = logging.FileHandler("./base_plot.log")
54+
file_handler.setFormatter(formatter)
55+
log.addHandler(file_handler)
56+
57+
# Only load Chrome at run-time if PRE_LOAD_CHROME is False or not defined.
58+
# Some applications may not want to load Chrome at runtime and
59+
# will set the PRE_LOAD_CHROME to True to indicate that it is already
60+
# loaded/downloaded prior to runtime.
61+
chrome_env =strtobool ('PRE_LOAD_CHROME')
62+
if ('PRE_LOAD_CHROME' not in os.environ)or (chrome_env is False):
63+
aquire_chrome=True
64+
kaleido.get_chrome_sync()
65+
66+
67+
# Log when kaleido is downloading Chrome
68+
if aquire_chrome is True and turn_on_logging is True:
69+
log.info("Plotly kaleido is loading Chrome at run time")
3270

3371
class BasePlot:
3472
"""A class that provides methods for building Plotly plot's common features

metplotpy/plots/util.py

Lines changed: 48 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -483,7 +483,7 @@ def filter_by_fixed_vars(input_df: pd.DataFrame, settings_dict: dict) -> pd.Data
483483
else:
484484
updated_vals.append(val)
485485

486-
# Create the query string based on whether or not there is/are NA values.
486+
# Create the query string based on whether NA values exist.
487487
if na_found:
488488
if len(updated_vals) == 0:
489489
# NA was the only value for this column, create the query
@@ -623,3 +623,50 @@ def prepare_ctc_roc(subset_df, is_ascending):
623623
thresh = pd.concat([thresh, pd.Series([''])], ignore_index=True)
624624

625625
return pody, pofd, thresh
626+
627+
628+
def strtobool(env_var:str)->bool:
629+
"""
630+
Since distutils.util.strtobool was deprecated in Python 3.12, implement
631+
our own version.
632+
633+
In the distutils.util.strtobool, a simple one line command was used to determine
634+
whether an environment variable was set to True or False. In this
635+
example, the default value is set to False in the event that the environment
636+
variable is not defined:
637+
638+
turn_on_logging = strtobool(os.getenv('LOG_BASE_PLOT', 'False') )
639+
640+
Environment variables can be set as string or bool. Evaluate whether a string
641+
value for true or false (support case-insensitive text) is True/False and
642+
set the default value.
643+
644+
Args:
645+
@parm env_vars: string name of the environment variable to evaluate
646+
647+
turn_on_logging = strtobool(os.getenv('LOG_BASE_PLOT') )
648+
"""
649+
650+
true_list = ['true', 't', '1',]
651+
false_list = ['false', 'f', '0' ]
652+
# if the environment variable does not exist, then return False
653+
try:
654+
val = os.environ[env_var]
655+
except KeyError:
656+
return False
657+
658+
# If the environment variable is None, return false
659+
if val is None:
660+
return False
661+
else:
662+
# Check for variations of truth values
663+
lower = val.lower()
664+
if lower in true_list:
665+
return True
666+
elif lower in false_list:
667+
return False
668+
else:
669+
msg = "Value does not represent a truth value (i.e. true or false)"
670+
raise ValueError(msg)
671+
672+

nco_requirements.txt

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
1-
kaleido>=0.2.1
1+
kaleido>=1.0.0
22
matplotlib>=3.6.3
33
metpy>=1.4.0
44
netcdf4>=1.6.2
55
numpy==1.24.1
66
scipy>=1.11.1
77
pandas>=1.5.2
88
pint>=0.20.1
9-
plotly>=5.13.0
9+
plotly>=6.1.1
1010
pytest>=7.2.1
1111
pyyaml>=6.0
1212
xarray>=2023.1.0

requirements.txt

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,14 @@
11
imageio==2.25.0
22
imutils==0.5.4
3-
kaleido>=0.2.1
3+
kaleido>=1.0.0
44
matplotlib==3.6.3
55
metpy==1.4.0
66
netcdf4==1.6.2
77
numpy==1.24.2
88
opencv-python>=4.7.0.72
99
pandas==1.5.2
1010
pint==0.20.1
11-
plotly==5.13.0
11+
plotly==6.1.1
1212
pytest>=7.2.1
1313
pyyaml==6.0
1414
scikit-image==0.19.3

test/hovmoeller/conftest.py

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
import pytest
2+
import os
3+
from unittest.mock import patch
4+
import shutil
5+
import json
6+
import xarray as xr
7+
from pandas import DatetimeIndex
8+
9+
# This fixture temporarily sets the working directory
10+
# to the dir containing the test file. This means
11+
# realative file locations can be used for each test
12+
# file.
13+
# NOTE: autouse=True means this applies to ALL tests.
14+
# Code that updates the cwd inside test is now redundant
15+
# and can be deleted.
16+
@pytest.fixture(autouse=True)
17+
def change_test_dir(request, monkeypatch):
18+
monkeypatch.chdir(request.fspath.dirname)
19+
20+
21+
@pytest.fixture(autouse=True)
22+
def patch_CompareImages(request):
23+
"""This fixture controls the use of CompareImages in the
24+
test suite. By default, all calls to CompareImages will
25+
result in the test skipping. To change this behaviour set
26+
an env var METPLOTPY_COMPAREIMAGES
27+
"""
28+
if bool(os.getenv("METPLOTPY_COMPAREIMAGES")):
29+
yield
30+
else:
31+
class mock_CompareImages:
32+
def __init__(self, img1, img2):
33+
# TODO: rather than skip we could inject an alternative
34+
# comparison that is more relaxed. To do this, extend
35+
# this this class to generate a self.mssim value.
36+
pytest.skip("CompareImages not enabled in pytest. "
37+
"To enable `export METPLOTPY_COMPAREIMAGES=$true`")
38+
try:
39+
with patch.object(request.module, 'CompareImages', mock_CompareImages) as mock_ci:
40+
yield mock_ci
41+
except AttributeError:
42+
# test module doesn't import CompareImages. Do nothing.
43+
yield
44+
45+
46+
def ordered(obj):
47+
"""Recursive function to sort JSON, even lists of dicts with the same keys"""
48+
if isinstance(obj, dict):
49+
return sorted((k, ordered(v)) for k, v in obj.items())
50+
if isinstance(obj, list):
51+
return sorted(ordered(x) for x in obj)
52+
else:
53+
return obj
54+
55+
@pytest.fixture
56+
def assert_json_equal():
57+
def compare_json(fig, expected_json_file):
58+
"""Takes a plotly figure and a json file
59+
"""
60+
# Treat everything as str for comparison purposes.
61+
actual = json.loads(fig.to_json(), parse_float=str, parse_int=str)
62+
with open(expected_json_file) as f:
63+
expected = json.load(f,parse_float=str, parse_int=str)
64+
# Fail with a nice message
65+
if ordered(actual) == ordered(expected):
66+
return True
67+
else:
68+
message = "This test will fail when there have been changes to plot code but the corresponding" \
69+
"json test file hasn't been updates. To update the test file run `fig.write_json`"\
70+
" e.g. `scatter.figure.write_json('custom_scatter_expected.json')`"
71+
raise AssertionError(message)
72+
73+
return compare_json
74+
75+
76+
@pytest.fixture
77+
def setup_env():
78+
def set_environ(test_dir):
79+
print("Setting up environment")
80+
os.environ['METPLOTPY_BASE'] = f"{test_dir}/../../"
81+
os.environ['TEST_DIR'] = test_dir
82+
return set_environ
83+
84+
85+
@pytest.fixture()
86+
def remove_files():
87+
def remove_the_files(test_dir, file_list):
88+
print("Removing the files")
89+
# loop over list of files under test_dir and remove them
90+
for file in file_list:
91+
try:
92+
os.remove(os.path.join(test_dir, file))
93+
except OSError:
94+
pass
95+
96+
# also remove intermed_files directory if it exists
97+
print("Removing intermed_files directory if it exists")
98+
try:
99+
shutil.rmtree(f"{test_dir}/intermed_files")
100+
except FileNotFoundError:
101+
pass
102+
103+
return remove_the_files
104+
105+
106+
# data for netCDF file
107+
TEST_NC_DATA = xr.Dataset(
108+
{
109+
"precip": xr.DataArray(
110+
[
111+
[[0.1, 0.2, 0.3], [0, 1.3, 4], [0, 20, 0]],
112+
[[0, 0, 0], [0, 0, 0], [0, 0, 0]],
113+
],
114+
coords={
115+
"lat": [-1, 0, 1],
116+
"lon": [112, 113, 114],
117+
"time": DatetimeIndex(["2024-09-25 00:00:00", "2024-09-25 03:00:33"]),
118+
},
119+
dims=["time", "lat", "lon"],
120+
attrs={"long_name": "variable long name"},
121+
),
122+
},
123+
attrs={"Conventions": "CF-99.9", "history": "History string"},
124+
)
125+
126+
@pytest.fixture()
127+
def nc_test_file(tmp_path_factory):
128+
"""Create a netCDF file with a very small amount of data.
129+
File is written to a temp directory and the path to the
130+
file returned as the fixture value.
131+
"""
132+
file_name = tmp_path_factory.mktemp("data") / "test_data.nc"
133+
TEST_NC_DATA.to_netcdf(file_name)
134+
return file_name
135+

0 commit comments

Comments
 (0)