2021 Update from HadISDH.land.v4.4.0.2021f

 1) What's New?

We use HadISD.3.2.0.2021f as the basis for HadISDH.land.4.4.0.2021f.

 

MAJOR CHANGES (X): 

• none

 

MINOR CHANGES (Y): 

• the climatological reference period has changed from 1981-2010 to 1991-2020, in line with WMO recommendations. All other processing steps remain the same.

 

BUG FIXES AND HISTORICAL DATA UPDATES (Z): 

• 9278 initial selection stations (same as last year)
• retrospective improvements (to correct data, add or remove data sections) to the historical data in NCEI's ISD archive are ongoing. These are not documented.
• Note that the missing April 2015 in previous versions was fixed from 4.3.1.2020f onwards.

 

2) Processing Steps

 

2.1) Number of HadISD stations: 

9278, same as last year

 

2.2) Number of stations passing through monthly averaging and climatology check: 

5581 where both T and Td are sufficiently present compared to 4520 last year.

 *** BELOW YET TO BE UPDATED   ***

2.3) Stations removed by PHA - too few neighbours: 

T = 4 (4493 remaining)8501099999 39.45 -31.13 34 PO FLORES                       
69146499999 40.87 141.38 5 JA DRAUGHON RANGE               
91066022701 28.20 -177.38 5 MQ HENDERSON FIELD AIRPORT      
91245041606 19.28 166.65 3 WQ WAKE ISLAND AIRFLD           
             
DPD = 7 (4490 remaining)68994099999 -46.88 37.87 22 SF MARION ISLAND                
85470099999 -27.30 -70.41 298 CI CHAMONATE                    
85488099999 -29.92 -71.20 146 CI LA FLORIDA                   
89002099999 -70.67 -8.25 50 AY NEUMAYER                     
89022099999 -75.61 -26.27 30 AY HALLEY                       
91610099999 1.38 173.15 2 KR BONRIKI INTL                 
61998099999 -49.35 70.25 30 FS PORT-AUX-FRANCAIS (ILES KERGU  

All variables (including T) are now based on 4486 stations with the 11 uniq station removals from T and DPD combined applied prior to running IDPHA.

 

2.4) Stations Removed by Indirect PHA in addition to those from T and DPD PHA - too few neighbours:

T = 2 (4484 remaining)   
68906099999 -40.35 -9.88 54 SH GOUGH ISLAND                  
85469099999 -27.16 -109.42 69 CI MATAVERI INTL                 

Td = 2 (4484 remaining)  (from Td PHA neighbour fails and T IDPHA neighbour fails) 68906099999 -40.35 -9.88 54 SH GOUGH ISLAND                  
85469099999 -27.16 -109.42 69 CI MATAVERI INTL                 

RH = 8 (4478 remaining)
61901099999 -15.9330 -5.6670 436.0 SH ST. HELENA IS.          
71917099999 79.9830 -85.9330 10.0 CA EUREKA                       
84203099999 -2.1570 -79.8840 5.8 EC SIMON BOLIVAR INTL  
84452099999 -6.7870 -79.8280 29.6 PE CAPT JOSE A  QUINONES GONZALE
84691099999 -13.7450 -76.2200 11.9 PE PISCO INTL                    
85469099999 -27.1650 -109.4220 69.2 CI MATAVERI INTL            
89564099999 -67.6000 62.8670 16.0 AY MAWSON                        
91958099999 -27.6170 -144.3330 2.0 FP RAPA 

q = 5 (4481 remaining)
61901099999 -15.9330 -5.6670 436.0 SH ST. HELENA IS.             
85469099999 -27.1650 -109.4220 69.2 CI MATAVERI INTL              
89571099999 -68.5830 77.9670 23.0 AY DAVIS                      
89642099999 -66.6670 140.0170 43.0 AY DUMONT D'URVILLE           
91366040604 8.7330 167.7330 2.1 RM BUCHOLZ AAF KWAJALEIN KMR ATO

e = 5 (4481 remaining)
61901099999 -15.9330 -5.6670 436.0 SH ST. HELENA IS.             
85469099999 -27.1650 -109.4220 69.2 CI MATAVERI INTL              
89571099999 -68.5830 77.9670 23.0 AY DAVIS                      
89642099999 -66.6670 140.0170 43.0 AY DUMONT D'URVILLE           
91366040604 8.7330 167.7330 2.1 RM BUCHOLZ AAF KWAJALEIN KMR ATO

Tw = 3 (4483 remaining)
08505099999 38.5200 -28.7160 36.0 PO HORTA       
68906099999 -40.3500 -9.8830 54.0 SH GOUGH ISLAND
85469099999 -27.1650 -109.4220 69.2 CI MATAVERI INTL

 

2.5) Stations Removed Because the Adjustments are TOO Large:

Stations are removed if an adjustment in T or Td > 5 deg C, and adjustment in q > 3 g/kg or an adjustment in RH > 15 %rh. This results in 55 stations removed in total

The largest adjustments for each variable are shown below:

q removals > 3 g/kg = 6 stations76577099999   3.99
76577099999   3.89
44347099999  -3.53
41128099999   3.52
16115099999   3.45
41240099999  -3.39
61498099999   3.29
41128099999  -3.14
44347099999   3.11
 
RH removals > 15 %rh = 28 stations44347099999 -37.65
36982099999 -26.38
04231099999  25.49
44347099999  24.64
44277099999 -22.46
71437099999  21.43
76577099999  20.17
71466099999  19.73
76577099999  19.72
72487099999 -19.66
01406099999 -18.92
37432099999 -18.63
71465099999  18.47
71800099999 -18.32
44298099999 -18.04
44284099999 -17.65
72487099999  17.52
47122099999  17.34
71261099999  17.28
71800099999  17.11
44287099999 -17.03
72355599999  17.01
59948099999  17.00
72570099999  16.65
44272099999  15.88
44347099999  15.77
38545099999 -15.76
72412799999  15.63
44213099999 -15.57
70333099999  15.55
60714099999 -15.19
85201099999  15.15
10935099999  15.13

Td removals > 5 deg C = stations 3171749099999  17.34
71749099999 -16.34
41128099999   9.47
44347099999  -8.63
71627899999   8.48
40841099999  -7.67
61498099999  -7.29
44277099999  -7.10
02095099999  -7.08
71627899999  -7.01
41128099999  -7.01
17330099999  -6.93
55664099999  -6.77
41128099999  -6.61
94332099999  -6.30
16115099999   6.24
41136099999  -6.20
72487099999   6.19
40420099999  -6.17
62103099999  -6.06
41128099999  -5.99
76471099999   5.94
41128099999   5.91
11993099999  -5.83
44284099999  -5.82
71836099999   5.81
94488099999  -5.79
40310099999  -5.69
72487099999  -5.66
40260099999   5.66
44358099999  -5.63
76382099999   5.55
71437099999   5.54
55279099999  -5.43
72365499999  -5.36
72365499999   5.31
44373099999   5.27
41136099999   5.22
41240099999  -5.21
44347099999   5.20
40340099999  -5.18
44347099999  -5.11
55578099999  -5.00

T removals > 5 deg C = 7 stations71749099999 -17.79
71749099999  17.34
71627899999   9.31
71627899999  -8.85
02095099999  -7.08
71836099999   6.42
16115099999   6.24
11993099999  -5.83
71836099999  -5.20
53588099999   5.04

This results in the following station counts before testing for data completeness (bads), supersaturation (sats) or subzero values (e and q only) :
e = 4426  
q = 4426  
RH = 4423
T = 4429 
Tw = 4428
Td = 4429
DPD = 4435

2.6) Stations Failing Post-PHA climatology check (data removed), Supersaturation and Subzeros:

Variable	Good Stations	Bad Stations	Subzero Stations	Supersaturated Stations
q	4332	25	44	65
RH	4333	25	0	65
T	4282	147	0	0
e	4332	25	44	65
Td	3693	508	0	220
Tw	3430	25	0	973
DPD	3868	343	0	224

2.7) Homogenisation Adjustment Uncertainty

The remaining stations are then gridded to create the HadISDH.4.2.0.2019f product which is a 5 by 5 degree gridbox average over all stations within each gridbox (no interpolation or weighting). Uncertainty fields are also gridded in addition to an estimate of uncertainty due to the uneven spatio-temporal sampling across each gridbox. Missed adjustment uncertainty is obtained from a global analysis of the distribution of adjustments for each variable subtracted from a fat-tailed Gaussian fit that infills the missing middle. The PHA/IDPHA adjustments and changepoint frequencies for q and RH are shown in Figures 1 to 2 below.

Variable	Mean Adjustment	Mean No. Adj per Station	Missed Adj Uncertainty
q	-0.004	4.5	0.221
RH	0.047	4.5	1.160
T	-0.021	4.4	0.299
e	-0.006	4.5	0.272
Td	-0.002	4.4	0.410
Tw	-0.011	4.5	0.233
DPD	-0.028	3.1	0.251

Figure 1 Specific Humidity adjustment and changepoint statistics. a) Distribution of adjustment sizes. The actual distribution is shown in black. The Gaussian best fit is shown in grey. A fat-tailed Gaussian merged fit made up of the real and Gaussian bins, taking the largest value at each bin, is shown in dashed red. The difference distribution between the fat-tailed Gaussian and the real distribution is shown in dotted blue. The mean and standard deviation of the difference distribution are shown. The standard deviation of the difference distribution is taken as the 1 sigma estimate for missed adjustment uncertainty.

Figure 2 Relative Humidity adjustment and changepoint statistics. See Figure 1 for details.

Figure 2 Relative Humidity adjustment and changepoint statistics. See Figure 1 for details.

These statistics are very similar to those from last year.

 

3) Differences in Gridbox and Regional Trends Between HadISDH.4.2.0.2019f and 4.1.0.2018f

Version 4.2.0.2019f adds an extra 12 months of data to most stations but additionally, there are some historical additions and amendments due to housekeeping done at the NOAA/NCEI ISD end. Even just adding an extra year of data can alter some of the changepoint locations detected and adjustment magnitudes applied. So, with this in mind, we would expect very small differences between the two versions. This year we have had several bug fixes affecting both the humidity values themselves and the number of stations. To look at the differences in more detail, Figures 3 to 4 explore differences in gridbox and regional trends between the two versions. Note that trends cover the full period of each version.

Figure 3 Specific Humidity gridbox and regional average trend differences between versions. a) Ratio of gridbox decadal trends. A ratio greater than 1 means that trends in v3.0.0.2016p are greater than in v4.0.0.2017f and vice versa. b) Scatter plot of gridbox decadal trends between the two version with percentage of gridboxes falling in each quadrant. c) Distribution of gridbox decadal trends. d) to g) Regional average time series and decadal trends (with 5th to 95 percentile confidence ranges in parentheses). Regional averages are calculated by weighting each gridbox by the cosine of its latitude. Trends are fitted using the median of pairwise slopes (Sen, 1968) and the range represents the 95% uncertainty.

 

Figure 4 Relative Humidity gridbox and regional trend differences between versions. Details are as for Figure 3.

Trends are identical for all regional averages in q although 6% of gridboxes have trends in a different direction to those last year. For RH, this year's trends are drier in all regions. 15% of gridboxes show trends of different direction to last year.

4) Gridbox decadal trends (1973-2019) for HadISDH.4.2.0.2019f

Figures 5 to 7 below show the decadal gridbox trends for q, RH and T. Gridbox trends are also shown as aggregates by latitude which demonstrates clear latitudinal differences in some cases and spread. The proportion of land gridboxes at each latitude (dark grey), and the proportion of those gridboxes with data (light grey), are also shown for each variable. All variables have near complete coverage in the Northern Hemisphere, with virtually no sampling at the high latitudes and poorest coverage in the Tropics.

Figure 5 Specific Humidity gridbox decadal trends over the 1973 to 2019 period. Decadal trends are also shown by latitude with the average for each latitude also plotted. Dark grey shading represents the proportion of area that is a land gridbox at that latitude. Light grey shading represents the proportion of those land gridboxes with data present. Trends are fitted using the median of pairwise slopes (Sen, 1968). Gridboxes with black borders are those considered to be significantly different from a zero trend (95% uncertainty).

Figure 6 Relative Humidity gridbox decadal trends over the 1973 to 2019 period. Details are as described for Figure 5.

Figure 7 Temperature gridbox decadal trends over the 1973 to 2019 period. Details are as described for Figure 5.

These figures appear to show very widespread and significant moistening in q, reduced saturation over the extra-tropics and mid-latitudes in RH and warming in T.

5) Global Average Annual Timeseries and Uncertainty for HadISDH.4.2.0.2019f

Station level uncertainties (climatology, measurement, homogeneity adjustment) are propagated to the gridbox and then regional average assuming that there is no correlation. These are then combined with estimates of the gridbox sampling uncertainty (accounting for uneven sampling in space and time) and regional coverage uncertainty (accounting for missing data and data gaps).

Figure 8. Global average annual time series for each variable with combined 2 sigma uncertainty estimates and decadal trends with 5th to 95th percentile confidence intervals (in parentheses). Station uncertainty is shown in blue, sampling uncertainty is shown in red and coverage uncertainty is shown in yellow.