Verification and Validation¶

After generating synthetic flows, two questions follow (Stedinger and Taylor, 1982):

Verification: does the ensemble reproduce the statistics the generator was designed to reproduce (moments, correlations, distributional shape)?
Validation: does the ensemble reproduce characteristics the generator was not explicitly fit to, such as drought duration and severity?

synhydro.verify and synhydro.validate answer these with the same reporting idiom: each statistic is computed once on the observed record and once on every realization, and the observed value is compared against the distribution of the statistic across realizations.

Note: Run this notebook from the examples/ directory. Figures are saved to figures/05_verification/ and data outputs to outputs/05_verification/ (both are gitignored).

In [1]:

Copied!





from pathlib import Path

FIG_DIR = Path("figures") / "05_verification"
OUT_DIR = Path("outputs") / "05_verification"
FIG_DIR.mkdir(parents=True, exist_ok=True)
OUT_DIR.mkdir(parents=True, exist_ok=True)
from pathlib import Path

FIG_DIR = Path("figures") / "05_verification"
OUT_DIR = Path("outputs") / "05_verification"
FIG_DIR.mkdir(parents=True, exist_ok=True)
OUT_DIR.mkdir(parents=True, exist_ok=True)

Setup¶

In [2]:

Copied!





import synhydro

Q_daily = synhydro.load_example_data()
Q_monthly = Q_daily.resample("MS").sum()

gen = synhydro.KirschGenerator()
gen.fit(Q_monthly)
ensemble = gen.generate(n_realizations=100, n_years=50, seed=42)
import synhydro

Q_daily = synhydro.load_example_data()
Q_monthly = Q_daily.resample("MS").sum()

gen = synhydro.KirschGenerator()
gen.fit(Q_monthly)
ensemble = gen.generate(n_realizations=100, n_years=50, seed=42)

Run verification¶

Metric selection is explicit: pass metrics="all", or a list of category names, metric names, and callables. Categories: marginal, temporal, seasonal, annual, spatial, fdc, lmoments, extremes, spectral. See the verification API reference for the full metric table with citations.

In [3]:

Copied!

result = synhydro.verify(ensemble, Q_monthly, metrics="all")
result = synhydro.verify(ensemble, Q_monthly, metrics="all")

Or select specific categories and metrics:

In [4]:

Copied!





result_subset = synhydro.verify(
    ensemble, Q_monthly,
    metrics=["marginal", "seasonal", "acf"],
)
result_subset = synhydro.verify(
    ensemble, Q_monthly,
    metrics=["marginal", "seasonal", "acf"],
)

Read the summary¶

result.summary() returns one row per metric (and per component for curve metrics such as monthly means), with the observed value, the spread across realizations, and two consistency measures:

obs_percentile: the observed value's position within the synthetic sample, (n_below + 0.5 n_equal + 0.5) / (n + 1). Values near 0.5 mean the observed statistic is central in the ensemble; values near 0 or 1 mean it lies in the tail.
in_90_band: whether the observed value falls between the ensemble's 5th and 95th percentiles.

In [5]:

Copied!

summary = result.summary()
summary[summary["in_90_band"] == False]        # observed outside the ensemble band
summary = result.summary()
summary[summary["in_90_band"] == False]        # observed outside the ensemble band

Out[5]:

	category	metric	kind	site	component	observed	syn_median	syn_mean	syn_q05	syn_q25	syn_q75	syn_q95	obs_percentile	in_90_band	relative_diff	reject_rate	n_realizations	units
100	temporal	hurst	scalar	USGS-01434000	NaN	0.924442	0.721804	0.722484	0.577860	0.661962	0.770410	0.866366	0.965347	False	-0.219201	NaN	100	dimensionless
101	temporal	hurst	scalar	USGS-01438500	NaN	0.937157	0.729652	0.723501	0.587401	0.656173	0.768782	0.871130	0.985149	False	-0.221420	NaN	100	dimensionless
351	seasonal	monthly_lag1_correlation	curve	USGS-01434000	8	0.329677	0.584245	0.576486	0.330793	0.514499	0.670833	0.767854	0.054455	False	0.772178	NaN	100	dimensionless
363	seasonal	monthly_lag1_correlation	curve	USGS-01438500	8	0.343944	0.610894	0.600584	0.375711	0.522726	0.698202	0.769410	0.044554	False	0.776141	NaN	100	dimensionless
375	seasonal	monthly_lag1_correlation	curve	USGS-01440000	8	0.251185	0.512202	0.518075	0.304973	0.429196	0.615761	0.717570	0.034653	False	1.039142	NaN	100	dimensionless
387	seasonal	monthly_lag1_correlation	curve	USGS-01463500	8	0.322661	0.568731	0.567934	0.348476	0.494085	0.657244	0.744181	0.044554	False	0.762625	NaN	100	dimensionless
508	annual	annual_minimum	scalar	USGS-01434000	NaN	22118.940160	28795.066891	28570.705246	23473.432194	26478.393337	30874.248916	34036.350851	0.044554	False	0.301829	NaN	100	flow_cumulative
509	annual	annual_minimum	scalar	USGS-01438500	NaN	25035.037339	32062.066658	32230.800438	26304.444749	29683.988784	34896.355904	39555.682397	0.034653	False	0.280688	NaN	100	flow_cumulative
510	annual	annual_minimum	scalar	USGS-01440000	NaN	466.483236	581.020168	583.436063	469.594264	533.676734	633.511944	708.720353	0.054455	False	0.245533	NaN	100	flow_cumulative
511	annual	annual_minimum	scalar	USGS-01463500	NaN	51437.835003	66323.423604	66854.639899	54574.151065	60448.868687	72361.835900	81957.916476	0.024752	False	0.289390	NaN	100	flow_cumulative
516	spatial	cross_correlation	matrix	USGS-01434000\|USGS-01438500	NaN	0.997877	0.991325	0.991020	0.988531	0.990191	0.991978	0.992936	0.995050	False	-0.006565	NaN	100	dimensionless
517	spatial	cross_correlation	matrix	USGS-01434000\|USGS-01440000	NaN	0.880228	0.835611	0.825374	0.772282	0.816059	0.847314	0.865310	0.995050	False	-0.050688	NaN	100	dimensionless
518	spatial	cross_correlation	matrix	USGS-01434000\|USGS-01463500	NaN	0.967247	0.938339	0.937086	0.917702	0.932370	0.944588	0.949035	0.995050	False	-0.029886	NaN	100	dimensionless
519	spatial	cross_correlation	matrix	USGS-01438500\|USGS-01440000	NaN	0.891883	0.845942	0.836710	0.788279	0.828061	0.857833	0.875181	0.995050	False	-0.051510	NaN	100	dimensionless
520	spatial	cross_correlation	matrix	USGS-01438500\|USGS-01463500	NaN	0.972801	0.947492	0.946809	0.933292	0.942635	0.952237	0.957426	0.995050	False	-0.026016	NaN	100	dimensionless
521	spatial	cross_correlation	matrix	USGS-01440000\|USGS-01463500	NaN	0.944060	0.912164	0.906032	0.861854	0.902047	0.920824	0.932514	0.995050	False	-0.033786	NaN	100	dimensionless
633	spectral	spectral_density	curve	USGS-01434000	1-2y	0.245055	0.094915	0.095096	0.063096	0.081092	0.107344	0.127558	0.995050	False	-0.612681	NaN	100	dimensionless
634	spectral	spectral_density	curve	USGS-01434000	0.5-1y	0.203623	0.358308	0.358046	0.305586	0.331112	0.379710	0.412556	0.004950	False	0.759663	NaN	100	dimensionless
639	spectral	spectral_density	curve	USGS-01438500	1-2y	0.244810	0.094434	0.096022	0.065721	0.082471	0.107753	0.130186	0.995050	False	-0.614255	NaN	100	dimensionless
640	spectral	spectral_density	curve	USGS-01438500	0.5-1y	0.206544	0.359417	0.361220	0.308891	0.336600	0.384525	0.416615	0.004950	False	0.740144	NaN	100	dimensionless
645	spectral	spectral_density	curve	USGS-01440000	1-2y	0.263857	0.105161	0.105622	0.070894	0.090376	0.120861	0.144980	0.995050	False	-0.601446	NaN	100	dimensionless
646	spectral	spectral_density	curve	USGS-01440000	0.5-1y	0.244650	0.402046	0.392758	0.299572	0.356787	0.428519	0.468345	0.014851	False	0.643351	NaN	100	dimensionless
651	spectral	spectral_density	curve	USGS-01463500	1-2y	0.245970	0.103260	0.105278	0.076123	0.090628	0.118018	0.138595	0.995050	False	-0.580192	NaN	100	dimensionless
652	spectral	spectral_density	curve	USGS-01463500	0.5-1y	0.221060	0.369097	0.365439	0.307547	0.334335	0.390580	0.421311	0.004950	False	0.669666	NaN	100	dimensionless

In [6]:

Copied!

summary[summary["obs_percentile"] < 0.05]      # ensemble overstates the statistic
summary[summary["obs_percentile"] < 0.05]      # ensemble overstates the statistic

Out[6]:

	category	metric	kind	site	component	observed	syn_median	syn_mean	syn_q05	syn_q25	syn_q75	syn_q95	obs_percentile	in_90_band	relative_diff	reject_rate	n_realizations	units
363	seasonal	monthly_lag1_correlation	curve	USGS-01438500	8	0.343944	0.610894	0.600584	0.375711	0.522726	0.698202	0.769410	0.044554	False	0.776141	NaN	100	dimensionless
375	seasonal	monthly_lag1_correlation	curve	USGS-01440000	8	0.251185	0.512202	0.518075	0.304973	0.429196	0.615761	0.717570	0.034653	False	1.039142	NaN	100	dimensionless
387	seasonal	monthly_lag1_correlation	curve	USGS-01463500	8	0.322661	0.568731	0.567934	0.348476	0.494085	0.657244	0.744181	0.044554	False	0.762625	NaN	100	dimensionless
508	annual	annual_minimum	scalar	USGS-01434000	NaN	22118.940160	28795.066891	28570.705246	23473.432194	26478.393337	30874.248916	34036.350851	0.044554	False	0.301829	NaN	100	flow_cumulative
509	annual	annual_minimum	scalar	USGS-01438500	NaN	25035.037339	32062.066658	32230.800438	26304.444749	29683.988784	34896.355904	39555.682397	0.034653	False	0.280688	NaN	100	flow_cumulative
511	annual	annual_minimum	scalar	USGS-01463500	NaN	51437.835003	66323.423604	66854.639899	54574.151065	60448.868687	72361.835900	81957.916476	0.024752	False	0.289390	NaN	100	flow_cumulative
634	spectral	spectral_density	curve	USGS-01434000	0.5-1y	0.203623	0.358308	0.358046	0.305586	0.331112	0.379710	0.412556	0.004950	False	0.759663	NaN	100	dimensionless
640	spectral	spectral_density	curve	USGS-01438500	0.5-1y	0.206544	0.359417	0.361220	0.308891	0.336600	0.384525	0.416615	0.004950	False	0.740144	NaN	100	dimensionless
646	spectral	spectral_density	curve	USGS-01440000	0.5-1y	0.244650	0.402046	0.392758	0.299572	0.356787	0.428519	0.468345	0.014851	False	0.643351	NaN	100	dimensionless
652	spectral	spectral_density	curve	USGS-01463500	0.5-1y	0.221060	0.369097	0.365439	0.307547	0.334335	0.390580	0.421311	0.004950	False	0.669666	NaN	100	dimensionless

result.category_summary() rolls the table up to one row per (category, site), using only unit-free quantities so metrics with different physical units are never averaged together. There is deliberately no single overall score.

In [7]:

Copied!

result.category_summary()
result.category_summary()

Out[7]:

	category	site	n_metrics	n_compared	n_in_90_band	median_abs_relative_diff	median_obs_percentile_distance
0	marginal	USGS-01434000	11	10	10	0.007455	0.039604
1	marginal	USGS-01438500	11	10	10	0.005551	0.039604
2	marginal	USGS-01440000	11	10	10	0.009228	0.044554
3	marginal	USGS-01463500	11	10	10	0.006317	0.039604
4	temporal	USGS-01434000	4	15	14	0.116953	0.168317
5	temporal	USGS-01438500	4	15	14	0.128946	0.217822
6	temporal	USGS-01440000	4	15	15	0.093398	0.168317
7	temporal	USGS-01463500	4	15	15	0.129717	0.168317
8	seasonal	USGS-01434000	8	72	71	0.040380	0.099010
9	seasonal	USGS-01438500	8	72	71	0.043666	0.103960
10	seasonal	USGS-01440000	8	72	71	0.067917	0.168317
11	seasonal	USGS-01463500	8	72	71	0.048011	0.108911
12	annual	USGS-01434000	7	7	6	0.033675	0.138614
13	annual	USGS-01438500	7	7	6	0.092388	0.108911
14	annual	USGS-01440000	7	7	6	0.117017	0.089109
15	annual	USGS-01463500	7	7	6	0.042445	0.069307
16	spatial	USGS-01434000\|USGS-01438500	1	1	0	0.006565	0.495050
17	spatial	USGS-01434000\|USGS-01440000	1	1	0	0.050688	0.495050
18	spatial	USGS-01434000\|USGS-01463500	1	1	0	0.029886	0.495050
19	spatial	USGS-01438500\|USGS-01440000	1	1	0	0.051510	0.495050
20	spatial	USGS-01438500\|USGS-01463500	1	1	0	0.026016	0.495050
21	spatial	USGS-01440000\|USGS-01463500	1	1	0	0.033786	0.495050
22	spatial	USGS-01434000->USGS-01438500	1	1	1	0.029952	0.148515
23	spatial	USGS-01434000->USGS-01440000	1	1	1	0.102150	0.405941
24	spatial	USGS-01434000->USGS-01463500	1	1	1	0.007905	0.089109
25	spatial	USGS-01438500->USGS-01434000	1	1	1	0.050792	0.217822
26	spatial	USGS-01438500->USGS-01440000	1	1	1	0.087512	0.376238
27	spatial	USGS-01438500->USGS-01463500	1	1	1	0.007770	0.069307
28	spatial	USGS-01440000->USGS-01434000	1	1	1	0.034829	0.089109
29	spatial	USGS-01440000->USGS-01438500	1	1	1	0.042020	0.089109
30	spatial	USGS-01440000->USGS-01463500	1	1	1	0.044300	0.138614
31	spatial	USGS-01463500->USGS-01434000	1	1	1	0.038387	0.237624
32	spatial	USGS-01463500->USGS-01438500	1	1	1	0.043721	0.227723
33	spatial	USGS-01463500->USGS-01440000	1	1	1	0.036383	0.207921
34	fdc	USGS-01434000	2	13	13	0.005949	0.059406
35	fdc	USGS-01438500	2	13	13	0.005487	0.049505
36	fdc	USGS-01440000	2	13	13	0.001720	0.019802
37	fdc	USGS-01463500	2	13	13	0.004311	0.029703
38	lmoments	USGS-01434000	3	3	3	0.011172	0.079208
39	lmoments	USGS-01438500	3	3	3	0.008201	0.039604
40	lmoments	USGS-01440000	3	3	3	0.024160	0.049505
41	lmoments	USGS-01463500	3	3	3	0.017172	0.029703
42	extremes	USGS-01434000	7	7	7	0.020083	0.079208
43	extremes	USGS-01438500	7	7	7	0.011645	0.089109
44	extremes	USGS-01440000	7	7	7	0.013716	0.138614
45	extremes	USGS-01463500	7	7	7	0.017258	0.099010
46	spectral	USGS-01434000	2	7	5	0.332761	0.207921
47	spectral	USGS-01438500	2	7	5	0.388682	0.227723
48	spectral	USGS-01440000	2	7	5	0.091140	0.118812
49	spectral	USGS-01463500	2	7	5	0.164557	0.168317

In a notebook, displaying result renders the summary as a styled table with the percentile column shaded.

Metrics that could not be computed (for example daily-only low-flow metrics on a monthly ensemble) are listed with reasons:

In [8]:

Copied!

result.skipped
result.skipped

Out[8]:

	metric	site	reason
0	seven_day_min_mean	all	requires ['daily'] data, ensemble is monthly
1	seven_day_min_cv	all	requires ['daily'] data, ensemble is monthly

For custom analysis, the per-realization values are available as a tidy DataFrame with one row per (metric, site, component, realization):

In [9]:

Copied!

df = result.to_dataframe()
df.head()
df = result.to_dataframe()
df.head()

Out[9]:

	category	metric	kind	site	component	realization	value	observed	units
0	marginal	mean	scalar	USGS-01434000	<NA>	0	4521.710733	4511.235818	flow
1	marginal	mean	scalar	USGS-01434000	<NA>	1	4603.175684	4511.235818	flow
2	marginal	mean	scalar	USGS-01434000	<NA>	2	4348.589998	4511.235818	flow
3	marginal	mean	scalar	USGS-01434000	<NA>	3	4553.832153	4511.235818	flow
4	marginal	mean	scalar	USGS-01434000	<NA>	4	4626.539104	4511.235818	flow

Plot metric distributions¶

plot_metric_distributions shows each scalar metric as a boxplot across realizations with the observed value as a dashed line, the convention introduced by Stedinger and Taylor (1982):

In [10]:

Copied!

from synhydro.plotting import plot_metric_distributions, plot_metric_curve

fig, axes = plot_metric_distributions(result, metrics=["mean", "std", "skewness"])
fig.savefig(FIG_DIR / "metric_distributions.png", dpi=150, bbox_inches="tight")
from synhydro.plotting import plot_metric_distributions, plot_metric_curve

fig, axes = plot_metric_distributions(result, metrics=["mean", "std", "skewness"])
fig.savefig(FIG_DIR / "metric_distributions.png", dpi=150, bbox_inches="tight")

No description has been provided for this image

Curve metrics (flow duration curve, autocorrelation function, monthly statistics) show an ensemble band with the observed curve overlaid:

In [11]:

Copied!

fig, ax = plot_metric_curve(result, "fdc", site=Q_monthly.columns[0])
fig.savefig(FIG_DIR / "fdc_curve.png", dpi=150, bbox_inches="tight")
fig, ax = plot_metric_curve(result, "fdc", site=Q_monthly.columns[0])
fig.savefig(FIG_DIR / "fdc_curve.png", dpi=150, bbox_inches="tight")

The verification panel¶

plot_verification_panel produces a 5-panel figure comparing observed and synthetic distributions by month, including per-realization Wilcoxon rank-sum and Levene test p-values. Under a well-performing generator, roughly 5 percent of realizations fall below the dashed line at p = 0.05.

In [12]:

Copied!

from synhydro.plotting import plot_verification_panel

site = Q_monthly.columns[0]
fig, axes = plot_verification_panel(ensemble, observed=Q_monthly[site], site=site)
fig.savefig(FIG_DIR / "verification_panel.png", dpi=150, bbox_inches="tight")
from synhydro.plotting import plot_verification_panel

site = Q_monthly.columns[0]
fig, axes = plot_verification_panel(ensemble, observed=Q_monthly[site], site=site)
fig.savefig(FIG_DIR / "verification_panel.png", dpi=150, bbox_inches="tight")

Tip (Log-space comparison): Streamflow is often right-skewed. Comparing statistics of log-transformed flows can reveal differences in the lower tail:
fig, axes = plot_verification_panel(
    ensemble, observed=Q_monthly[site], log_space=True
)

Run validation (drought behavior)¶

synhydro.validate evaluates drought characteristics: run-theory events below a flow threshold (threshold_drought) and events on the Standardized Streamflow Index (ssi_drought). See the validation API reference.

In [13]:

Copied!

val = synhydro.validate(ensemble, Q_monthly, metrics="all")
val.summary()
val = synhydro.validate(ensemble, Q_monthly, metrics="all")
val.summary()

Out[13]:

	category	metric	kind	site	component	observed	syn_median	syn_mean	syn_q05	syn_q25	syn_q75	syn_q95	obs_percentile	in_90_band	relative_diff	reject_rate	n_realizations	units
0	threshold_drought	mean_drought_duration	scalar	USGS-01434000	NaN	2.412500	2.239828	2.277333	1.966855	2.122278	2.394542	2.668333	0.757426	True	-0.071574	NaN	100	timesteps
1	threshold_drought	mean_drought_severity	scalar	USGS-01434000	NaN	906.530925	855.846835	859.144677	656.193480	785.894525	948.420047	1049.088248	0.668317	True	-0.055910	NaN	100	flow_cumulative
2	threshold_drought	max_drought_duration	scalar	USGS-01434000	NaN	7.000000	6.000000	6.280000	5.000000	6.000000	7.000000	8.000000	0.816832	True	-0.142857	NaN	100	timesteps
3	threshold_drought	max_drought_severity	scalar	USGS-01434000	NaN	5041.842853	3866.064737	3950.655095	2801.379783	3347.226055	4479.273520	5240.889131	0.896040	True	-0.233204	NaN	100	flow_cumulative
4	threshold_drought	drought_frequency	scalar	USGS-01434000	NaN	0.082988	0.088333	0.087950	0.076583	0.081667	0.092083	0.100083	0.301980	True	0.064417	NaN	100	per_timestep
5	threshold_drought	mean_drought_duration	scalar	USGS-01438500	NaN	2.539474	2.298059	2.322117	2.017186	2.168476	2.474682	2.668889	0.836634	True	-0.095065	NaN	100	timesteps
6	threshold_drought	mean_drought_severity	scalar	USGS-01438500	NaN	1122.163097	1021.502840	1030.194067	822.579959	932.658116	1138.105012	1272.672884	0.727723	True	-0.089702	NaN	100	flow_cumulative
7	threshold_drought	max_drought_duration	scalar	USGS-01438500	NaN	8.000000	6.000000	6.380000	5.000000	6.000000	7.000000	8.000000	0.965347	True	-0.250000	NaN	100	timesteps
8	threshold_drought	max_drought_severity	scalar	USGS-01438500	NaN	5890.017359	4423.898633	4465.353586	3178.304156	3882.181106	5048.880592	5920.940612	0.915842	True	-0.248916	NaN	100	flow_cumulative
9	threshold_drought	drought_frequency	scalar	USGS-01438500	NaN	0.078838	0.085833	0.086350	0.073333	0.081250	0.091667	0.100083	0.193069	True	0.088728	NaN	100	per_timestep
10	threshold_drought	mean_drought_duration	scalar	USGS-01440000	NaN	2.680556	2.441154	2.470581	2.132430	2.338652	2.624275	2.823564	0.816832	True	-0.089310	NaN	100	timesteps
11	threshold_drought	mean_drought_severity	scalar	USGS-01440000	NaN	30.728183	28.165570	28.414398	22.605102	26.013088	30.722588	34.235812	0.747525	True	-0.083396	NaN	100	flow_cumulative
12	threshold_drought	max_drought_duration	scalar	USGS-01440000	NaN	8.000000	6.000000	6.060000	5.000000	6.000000	7.000000	7.000000	0.980198	False	-0.250000	NaN	100	timesteps
13	threshold_drought	max_drought_severity	scalar	USGS-01440000	NaN	98.536963	91.908800	92.908018	70.686386	81.777114	101.741143	117.944190	0.668317	True	-0.067266	NaN	100	flow_cumulative
14	threshold_drought	drought_frequency	scalar	USGS-01440000	NaN	0.074689	0.080000	0.080950	0.070000	0.076667	0.085417	0.091750	0.143564	True	0.071111	NaN	100	per_timestep
15	threshold_drought	mean_drought_duration	scalar	USGS-01463500	NaN	2.573333	2.417636	2.444668	2.019000	2.265017	2.618322	2.844730	0.668317	True	-0.060504	NaN	100	timesteps
16	threshold_drought	mean_drought_severity	scalar	USGS-01463500	NaN	2885.794755	2705.150516	2742.161004	2145.347451	2422.649621	3000.389553	3430.344943	0.648515	True	-0.062598	NaN	100	flow_cumulative
17	threshold_drought	max_drought_duration	scalar	USGS-01463500	NaN	8.000000	6.000000	6.440000	5.000000	6.000000	7.000000	8.000000	0.950495	True	-0.250000	NaN	100	timesteps
18	threshold_drought	max_drought_severity	scalar	USGS-01463500	NaN	13712.772764	10936.722342	11187.088257	8136.397423	9647.212370	12383.722460	14583.566608	0.866337	True	-0.202443	NaN	100	flow_cumulative
19	threshold_drought	drought_frequency	scalar	USGS-01463500	NaN	0.077801	0.081667	0.082750	0.071583	0.076667	0.087083	0.096667	0.262376	True	0.049689	NaN	100	per_timestep
20	ssi_drought	ssi_mean_drought_duration	scalar	USGS-01434000	NaN	22.727273	19.000000	20.154864	13.090179	16.625000	23.718750	29.502778	0.698020	True	-0.164000	NaN	100	timesteps
21	ssi_drought	ssi_max_drought_duration	scalar	USGS-01434000	NaN	94.000000	46.000000	47.150000	23.000000	34.000000	59.000000	76.100000	0.990099	False	-0.510638	NaN	100	timesteps
22	ssi_drought	ssi_mean_drought_severity	scalar	USGS-01434000	NaN	26.746027	19.033396	19.640510	11.981834	15.607476	23.348465	28.259859	0.886139	True	-0.288365	NaN	100	dimensionless
23	ssi_drought	ssi_max_drought_severity	scalar	USGS-01434000	NaN	115.877467	43.818215	48.396170	26.893895	35.304766	58.551747	77.652003	0.995050	False	-0.621857	NaN	100	dimensionless
24	ssi_drought	ssi_drought_frequency	scalar	USGS-01434000	NaN	0.138510	0.183362	0.183565	0.141596	0.162988	0.203735	0.244482	0.054455	False	0.323815	NaN	100	per_year
25	ssi_drought	ssi_mean_drought_duration	scalar	USGS-01438500	NaN	28.600000	19.916667	20.742221	12.768889	16.676136	24.031250	30.070000	0.935644	True	-0.303613	NaN	100	timesteps
26	ssi_drought	ssi_max_drought_duration	scalar	USGS-01438500	NaN	119.000000	48.000000	49.050000	23.950000	35.000000	59.250000	79.100000	0.985149	False	-0.596639	NaN	100	timesteps
27	ssi_drought	ssi_mean_drought_severity	scalar	USGS-01438500	NaN	30.713994	19.618107	20.195415	12.080874	15.829654	23.511931	30.578563	0.945545	False	-0.361265	NaN	100	dimensionless
28	ssi_drought	ssi_max_drought_severity	scalar	USGS-01438500	NaN	129.523979	46.435686	49.496396	26.850465	36.588384	59.496984	80.507288	0.995050	False	-0.641490	NaN	100	dimensionless
29	ssi_drought	ssi_drought_frequency	scalar	USGS-01438500	NaN	0.125918	0.183362	0.179491	0.122241	0.162988	0.203735	0.244482	0.123762	True	0.456197	NaN	100	per_year
30	ssi_drought	ssi_mean_drought_duration	scalar	USGS-01440000	NaN	17.687500	18.175000	19.105442	12.853472	15.665064	21.800000	27.421667	0.460396	True	0.027562	NaN	100	timesteps
31	ssi_drought	ssi_max_drought_duration	scalar	USGS-01440000	NaN	66.000000	42.000000	45.110000	25.900000	34.000000	52.250000	74.100000	0.856436	True	-0.363636	NaN	100	timesteps
32	ssi_drought	ssi_mean_drought_severity	scalar	USGS-01440000	NaN	18.287593	17.725193	18.070663	10.937021	14.543926	20.394255	26.522434	0.559406	True	-0.030753	NaN	100	dimensionless
33	ssi_drought	ssi_max_drought_severity	scalar	USGS-01440000	NaN	96.398416	42.884177	45.394764	22.304254	32.182237	54.441604	80.681722	0.985149	False	-0.555136	NaN	100	dimensionless
34	ssi_drought	ssi_drought_frequency	scalar	USGS-01440000	NaN	0.201469	0.203735	0.199253	0.141596	0.178268	0.224109	0.264856	0.391089	True	0.011248	NaN	100	per_year
35	ssi_drought	ssi_mean_drought_duration	scalar	USGS-01463500	NaN	21.307692	20.050000	20.986662	12.686458	16.452020	24.222527	31.891667	0.599010	True	-0.059025	NaN	100	timesteps
36	ssi_drought	ssi_max_drought_duration	scalar	USGS-01463500	NaN	95.000000	45.000000	48.070000	27.900000	34.750000	59.250000	78.000000	0.985149	False	-0.526316	NaN	100	timesteps
37	ssi_drought	ssi_mean_drought_severity	scalar	USGS-01463500	NaN	23.608564	19.847939	20.841609	12.178043	16.312751	24.438765	31.806155	0.698020	True	-0.159291	NaN	100	dimensionless
38	ssi_drought	ssi_max_drought_severity	scalar	USGS-01463500	NaN	128.768073	47.312461	50.859724	25.628891	37.599387	62.163918	85.033687	0.995050	False	-0.632576	NaN	100	dimensionless
39	ssi_drought	ssi_drought_frequency	scalar	USGS-01463500	NaN	0.163694	0.183362	0.183973	0.122241	0.142615	0.224109	0.245501	0.430693	True	0.120151	NaN	100	per_year

Because drought statistics were not fit by the generator, this is the stronger test: a generator can verify well and still understate multi-year drought severity.

Uncertainty and generator comparison¶

bootstrap_metric_ci puts confidence intervals on ensemble metric estimates by resampling realizations:

In [14]:

Copied!

ci = synhydro.bootstrap_metric_ci(result, n_bootstrap=1000, seed=1)
ci.head()
ci = synhydro.bootstrap_metric_ci(result, n_bootstrap=1000, seed=1)
ci.head()

Out[14]:

	category	metric	site	component	observed	estimate	ci_lower	ci_upper	relative_diff	rd_ci_lower	rd_ci_upper	n_realizations
0	marginal	mean	USGS-01434000	NaN	4511.235818	4514.525732	4464.884231	4553.394019	0.000729	-0.010275	0.009345	100
1	marginal	mean	USGS-01438500	NaN	5143.336687	5142.809205	5089.878374	5203.050524	-0.000103	-0.010394	0.011610	100
2	marginal	mean	USGS-01440000	NaN	100.506335	101.402675	99.954248	102.296805	0.008918	-0.005493	0.017814	100
3	marginal	mean	USGS-01463500	NaN	10593.393115	10609.626982	10492.505786	10770.708395	0.001532	-0.009524	0.016738	100
4	marginal	std	USGS-01434000	NaN	3269.603194	3245.814727	3222.687505	3310.553646	-0.007276	-0.014349	0.012525	100

compare_methods tests whether one generator reproduces observed statistics better than another, using a paired bootstrap over realizations:

In [15]:

Copied!





site = Q_monthly.columns[0]

gen_b = synhydro.ThomasFieringGenerator()
gen_b.fit(Q_monthly[[site]])
ensemble_b = gen_b.generate(n_realizations=100, n_years=50, seed=42)

result_b = synhydro.verify(ensemble_b, Q_monthly[[site]], metrics="all")
comparison = synhydro.compare_methods(result, result_b, seed=1)
comparison[comparison["significant"]]
site = Q_monthly.columns[0]

gen_b = synhydro.ThomasFieringGenerator()
gen_b.fit(Q_monthly[[site]])
ensemble_b = gen_b.generate(n_realizations=100, n_years=50, seed=42)

result_b = synhydro.verify(ensemble_b, Q_monthly[[site]], metrics="all")
comparison = synhydro.compare_methods(result, result_b, seed=1)
comparison[comparison["significant"]]

INFO:synhydro.ThomasFieringGenerator:Validated data: 964 timesteps, 1 sites, period 1945-01-01 00:00:00 to 2025-04-01 00:00:00

INFO:synhydro.ThomasFieringGenerator:Preprocessing complete: 964 months

INFO:synhydro.ThomasFieringGenerator:Fitting complete: 964 months

INFO:synhydro.ThomasFieringGenerator:Generated 100 realizations of 50 years each

Out[15]:

	category	metric	site	component	method_a_mae	method_b_mae	diff_estimate	diff_ci_lower	diff_ci_upper	significant	better_method	paired
1	marginal	std	USGS-01434000	<NA>	140.517129	239.630808	-99.113679	-145.175250	-57.187646	True	a	True
2	marginal	cv	USGS-01434000	<NA>	0.021717	0.043314	-0.021597	-0.028958	-0.014287	True	a	True
3	marginal	skewness	USGS-01434000	<NA>	0.151979	0.716057	-0.564078	-0.703603	-0.452419	True	a	True
4	marginal	kurtosis	USGS-01434000	<NA>	0.958836	5.999627	-5.040791	-7.018771	-3.541117	True	a	True
5	marginal	minimum	USGS-01434000	<NA>	87.695126	118.171770	-30.476644	-53.167077	-7.995104	True	a	True
...	...	...	...	...	...	...	...	...	...	...	...	...
152	extremes	annual_min_cv	USGS-01434000	<NA>	0.053238	0.081540	-0.028302	-0.043408	-0.013708	True	a	True
153	spectral	spectral_density	USGS-01434000	>8y	0.025619	0.035466	-0.009847	-0.013236	-0.006025	True	a	True
154	spectral	spectral_density	USGS-01434000	4-8y	0.013904	0.010310	0.003593	0.000674	0.006425	True	b	True
156	spectral	spectral_density	USGS-01434000	1-2y	0.149960	0.136863	0.013096	0.007478	0.018981	True	b	True
159	spectral	low_frequency_variance_fraction	USGS-01434000	<NA>	0.023748	0.034733	-0.010984	-0.016571	-0.005439	True	a	True

92 rows × 12 columns

Custom metrics¶

Any function of a series can join the suite:

In [16]:

Copied!





from synhydro.verification import register_metric

@register_metric(category="custom", units="flow")
def flow_q25(x):
    """25th percentile of flows."""
    return float(x.quantile(0.25))

result = synhydro.verify(ensemble, Q_monthly, metrics=["marginal", "flow_q25"])
from synhydro.verification import register_metric

@register_metric(category="custom", units="flow")
def flow_q25(x):
    """25th percentile of flows."""
    return float(x.quantile(0.25))

result = synhydro.verify(ensemble, Q_monthly, metrics=["marginal", "flow_q25"])

Next steps¶

Metric definitions and citations - Verification API and Validation API
Algorithm details can help diagnose which statistical properties a generator is designed to preserve - Algorithms

Previous: Working with Ensembles | Next: Plotting Walkthrough