2018 Update from HadISDH.land.v4.1.0.2018f

***WORK IN PROGRESS***

1) What's New? 

The annual update of HadISDH takes place every January. We continue to use the new HadISD2 station sub-daily dataset (version 2.0.2.2017p) to build HadISDH which provides a significant increase in station numbers of HadISD. This year we start with 8103 stations which reduces down considerably during processing but still gives 3500-4500 stations - ~700 more than HadISD versions.

The most significant change this year is a move from the 1976-2005 climatology period to the 1981-2010 climatology period. The previous period was chosen to maximise station coverage for HadISD. HadISD2 brings in improved station merges and the source database ISD has also had some improvements in spatial and temporal coverage. This allows us to use the more recent climatology period which now actually increases station coverage by ~300 stations. This major change warrants an increment to the 'x' element of the version number from 3 to 4.

We have added a minor code change to remove more stations with large adjustments following the PHA homogenisation. Previously we only removed stations which had adjustments greater than 5 deg C in temperature or dew point temperature. Now we also remove any stations with adjustments larger than 3 g/kg in specific humidity or 15 %rh in relative humidity. This results in 54 stations being removed. This would normally mean incrementing the 'y' element of the version number but this is unnecessary given the increment in 'x'.  

We have also fixed a minor bug in the gridding software which meant that very occasionally a station may be read in twice or a removed station included. Comparison of versions before and after the fix show only very tiny differences that do not affect multi-gridbox averages. The new scrape of data has included some historical changes in addition to the extra year of data (a standard state of affairs while the ISD database is still being developed). A few new stations have been added and some data will have been added/removed/changed within the existing stations. This would normally mean incrementing the 'z' element of the version number but this is unnecessary given the increment in 'y'. 

2) What's the Same?

The vast majority of the starting station list from ISD is the same as last year (first year of using HadISD2) and for the most part versions that used HadISD prior to that. Some stations  are merges of 2 or more stations where they have been established to be part of the same station as part of HadISD2 processing.

The underlying QC is the same in terms of methodology as last year. Due to the historical and annual updates this is now HadISD.2.0.2.2017p.

The HadISDH methodology is the same except for the small change to the large adjustment removals:

• equations for converting T and Td to DPD, Tw, e, q and RH are the same;
• methods for calculating monthly means and climate anomalies are the same;
• the PHA methodology is the same;
• IDPHA methodology is the same;
• gridding methodology is the same with a minor bug fix;
• uncertainty estimation framework is the same

 

3) Processing Steps

3.1) Number of HadISD2 stations: 

8103

3.2) Number of stations passing through climatology check: 

4576 where both T and Td are sufficiently present. This compares to 4216 in 2016 which used the 1976 to 2005 climatology and 4210 for a 2017 version using the 1976-2005 climatology.

We have not searched for stations that are no longer included compared to last year yet.

 

3.3) Stations removed by PHA: 

These are colour coded compared to v2.1.0 2015p this year.
blue = new,
black = same as last year,
red = last year (2015) and not this year

T = 4 (4572 remaining)
85469099999, -27.17S, -109.42W, 69m, CI, MATAVERI INTL
(was CH, ISLA DE PASCUA)           
91066022701, 28.20N, -177.38W, 5m, MQ, HENDERSON FIELD AIRPORT
(was US, MIDWAY ISLAND NAS)
91245041606, 19.28N, 166.65E, 3m, WQ, WAKE ISLAND AIRFLD    
91925099999, -9.80S,  -139.03W, 53m, FP, HIVA-OA 
(was PF, ATUONA)                 

DPD =  8 (4568 remaining)
61901099999, -15.93S, -5.67W, 436m, SH, ST. HELENA IS.         
61967070701, -7.30S, 72.40E, 2m, BT, DIEGO GARCIA NAF
68300099999, -26.68S, 15.25E, 139m, WA, LUDERITZ (DIAZ POINT)
68994099999, -46.88S, 37.87E, 22m, SF, MARION ISLAND (was 21m)      
85469099999, -27.17S, -109.42W, 69m, CI, MATAVERI INTL       
85488099999, -29.92S, -71.20W, 146m, CI, LA FLORIDA (was CH, LA SERENA)         
89002099999, -70.67S, -8.25W, 50m, AY, NEUMAYER            
89022099999, -75.61S, -26.27W, 30m, AY, HALLEY  (was -75.50S, -26.65W)              
89564099999, -67.60S, 62.87E, 16m, AY, MAWSON                
91610099999, 1.35N, 172.92E, 4m, KB, TARAWA

All variables (including T) are now based on 4565 station lists with the 11 uniq station removals from T and DPD combined applied prior to running IDPHA.

3.4) Stations Removed by Indirect PHA in addition to those from T and DPD PHA:

These are colour coded compared to v2.1.0 2015p this year.
blue = new,
black = same as last year,
red = last year (2015) and not this year

Td = 4 (4561 remaining)  (from Td PHA neighbour fails and T IDPHA neighbour fails)
08501099999,  39.4550S, -31.1310W, 34.1m, PO, FLORES (was 29m)
68906099999, -40.3500S, -9.8830W, 54.0m, SH, GOUGH ISLAND (was ZA)   
89532099999, -69.00S, 39.58E, 21m, AY, SYOWA                        
91190022516, 20.9000N, -156.4290W, 15.5m, US, KAHULUI AIRPORT

T= 4 (4561 remaining) 
08501099999,  39.4550S, -31.1310W, 34.1m, PO, FLORES (was 29m)
68906099999, -40.3500S, -9.8830W, 54.0m, SH, GOUGH ISLAND (was ZA)   
89532099999, -69.00S, 39.58E, 21m, AY, SYOWA                        
91190022516, 20.9000N, -156.4290W, 15.5m, US, KAHULUI AIRPORT 

RH = 9 (4556 remaining)
62721099999, 15.5890N, 32.5530E, 385.6m, SU, KHARTOUM                   
64400099999, -4.8170S, 11.9000E, 17.0m, CG, POINTE-NOIRE
64500099999, 0.4590N, 9.4120E, 11.9m, GB, LEON M BA                   
64501099999, -0.7120S, 8.7540E, 4.0m, GB, PORT GENTIL      
68104099999, -22.883S, 14.433N, 0.0m, NM, WALVIS BAY (PELICAN (not in start list)
78767099999, 9.9580N, -83.0220W, 2.1m, CS, LIMON INTL
(was 10.0N, -83.05W, 3.0m, PUERTO LIMON)
84117099999, -0.95S, -80.683W, 14.0m, EQ, MANTA
84782099999, -18.0500S, -70.2670W, 458.0m, PR, TACNA 
85470099999, -27.30S, -70.417W, 290.0m, CH, COPIAPO (not in start list)
85930099999, -52.40S, -75.10W, 52.0m, CH, FARO EVANGELISTAS (not in start list)
88963099999, -63.4000S,  -56.9830W, 24.0m, AY, BASE ESPERANZA          
89571099999, -68.5830S, 77.9500E, 13.0m, AY, DAVIS
89642099999, -66.6670S, 140.0170E, 43.0m, AY, DUMONT D'URVILLE
91218099999, 13.5670N, 144.9170E, 162.0m, GQ, ANDERSEN AFB  
91610099999, 1.3820N, 173.1470E, 2.7m, KR, BONRIKI INTL 
91943099999, -14.4830S, -145.0330W, 3.0m, FP, TAKAROA (was PF)         
93945099999, -52.55S, 169.167E, 19.0m, NZ, CAMPBELL ISLAND AWS (not in start list)

q = 9 (4556 remaining)
08501099999, 39.4550N, -31.1310W, 34.1m, PO, FLORES
(was -31133, 29.0m, FLORES (ACORES))        
08505099999, 38.5200N, -28.7160W, 36.0m, PO, HORTA          
47991099999, 24.2900N, 153.9790E, 6.7m, JA, MINAMI TORISHIMA
68906099999, -40.3500S, -9.8830W, 54.0m, SH, GOUGH ISLAND (was ZA)  
89532099999, -69.0000S, 39.5830E, 21.0m, AY, SYOWA             
89571099999, -68.583S, 77.95E, 13.0m, AY, DAVIS 
89611099999, -66.2830S, 110.5330E, 42.0m, AY, CASEY (was 110.517E, 41.0m)          
89642099999, -66.6670S, 140.0170E, 43.0m, AY, DUMONT D'URVILLE 
91165022536, 21.9840N, -159.3410W, 30.5m, US, LIHUE AIRPORT (was 21.9840, 45.1m)
91176022519, 21.4500N, -157.7680W, 7.3m, US, KANEOHE MCAS     

e = 9 (4556 remaining)
08501099999, 39.4550N, -31.1310W, 34.1m, PO, FLORES
(was -31133, 29.0m, FLORES (ACORES))        
08505099999, 38.5200N, -28.7160W, 36.0m, PO, HORTA          
47991099999, 24.2900N, 153.9790E, 6.7m, JA, MINAMI TORISHIMA
68906099999, -40.3500S, -9.8830W, 54.0m, SH, GOUGH ISLAND (was ZA)  
89532099999, -69.0000S, 39.5830E, 21.0m, AY, SYOWA             
89571099999, -68.583S, 77.95E, 13.0m, AY, DAVIS 
89611099999, -66.2830S, 110.5330E, 42.0m, AY, CASEY (was 110.517E, 41.0m)          
89642099999, -66.6670S, 140.0170E, 43.0m, AY, DUMONT D'URVILLE 
91165022536, 21.9840N, -159.3410W, 30.5m, US, LIHUE AIRPORT (was 21.9840, 45.1m)
91176022519, 21.4500N, -157.7680W, 7.3m, US, KANEOHE MCAS    

Tw = 6 (4559 remaining)
08501099999, 39.4550N, -31.1310W, 34.1m, PO, FLORES
(was -31133, 29.0m, FLORES (ACORES))           
 08505099999, 38.5200N, -28.7160W, 36.0m, PO, HORTA
(was 38.517N, -28.717W, 41.0m, HORTA/CASTELO BRANC)  
68906099999, -40.3500S, -9.8830W, 54.0m, SH, GOUGH ISLAND (was ZA)   
89532099999, -69.0000S, 39.5830E, 21.0m, AY, SYOWA 
91165022536, 21.9840N, -159.3410W, 30.5m, US, LIHUE AIRPORT (was 21.9840, 45.1m)
91176022519, 21.4500N, -157.7680W, 7.3m, US, KANEOHE MCAS 
(was 21.4510N, 5.5m, KANEOHE BAY MCAS)

 

3.5) Stations Removed Because the Adjustments are TOO Large:

Unlike 2015 and earlier, most of the stations removed due to large adjustments are not the result of station merges. However, all but one station with large adjustments in temperature are due to merges. In total 54 unique stations are removed compared to 9 in 2015 and 29 in 2016. 

01406099999 NO MERGE
02096099999 MERGE
04231099999 NO MERGE
11993099999 MERGE
13579099999 MERGE
13615099999 NO MERGE
15481099999 NO MERGE
16115099999 MERGE
17330099999 NO MERGE
36982099999 NO MERGE
37432099999 NO MERGE
38001099999 NO MERGE
38545099999 NO MERGE
40260099999 NO MERGE
40310099999 NO MERGE
40340099999 NO MERGE
40377099999 MERGE
40420099999 MERGE
40809099999 NO MERGE
41084099999 MERGE
41128099999 NO MERGE
41136099999 NO MERGE
41240099999 MERGE
41268099999 MERGE
44277099999 NO MERGE
44287099999 NO MERGE
44298099999 NO MERGE
44373099999 NO MERGE
47122099999 NO MERGE
55279099999 NO MERGE
55578099999 NO MERGE
55664099999 NO MERGE
59948099999 NO MERGE
60580099999 NO MERGE
60714099999 NO MERGE
61498099999 NO MERGE
71433099999 NO MERGE
71437099999 MERGE
71465099999 NO MERGE
71627899999 MERGE
71749099999 MERGE
71800099999 NO MERGE
72393099999 MERGE
72412799999 MERGE
72446713930 MERGE
72520799999 MERGE
76423099999 NO MERGE
76577099999 NO MERGE
76762099999 NO MERGE
78367011706 NO MERGE
78397099999 NO MERGE
78520199999 MERGE
82281099999 NO MERGE
94339099999 NO MERGE

This results in the following FINAL station counts before testing for data completeness (bads), supersaturation or subzero values (e and q only) :
e = 4502   
q = 4502   
RH = 4502
T = 4507  
Tw = 4505
Td = 4507
DPD = 4514

The largest adjustments for each variable are shown below:

T in deg C:
71749099999 -18.61
71749099999  17.13
71627899999  -9.01
71627899999   8.38
02096099999  -7.00
16115099999   6.25
11993099999  -5.87

q in g/kg:
76577099999   3.81
41240099999  -3.54
76577099999   3.52
16115099999   3.49
78367011706   3.40
41128099999   3.37
78397099999  -3.31
78397099999   3.25
41268099999  -3.16
82281099999  -3.14
76762099999  -3.13
78520199999   3.06
41128099999  -3.01

RH in %rh:
36982099999 -24.25
04231099999  23.54
44277099999 -21.58
71437099999  20.43
76577099999  20.41
71800099999  19.06
44298099999 -18.78
71465099999  17.66
78367011706  17.49
76577099999  17.28
01406099999 -17.16
38545099999 -16.89
60714099999 -16.69
44287099999 -16.53
76577099999 -16.36
71433099999  16.30
59948099999  16.15
72393099999  16.02
37432099999 -15.98
72412799999  15.85
41240099999 -15.84
15481099999  15.82
13615099999  15.80
72446713930  15.62
71800099999 -15.46
76577099999 -15.45
13579099999  15.43
47122099999  15.32
38001099999  15.29
76423099999 -15.11
72520799999  15.04
 
e in hPa:
41240099999  -5.60
78367011706   5.41
78397099999  -5.26
78397099999   5.21
41268099999  -5.04
82281099999  -4.97
76577099999   4.92
78520199999   4.89
41128099999   4.74
16115099999   4.48
76577099999   4.47
76762099999  -4.42
78990099999   4.38
61498099999   4.34
78388099999  -4.32
41128099999  -4.24
40340099999  -4.18
91670099999  -4.14
80421499999  -4.12
41218099999   4.09
41061099999   4.01

Td in deg C:
71749099999  17.13
71749099999 -15.76
41128099999   9.22
71627899999   7.42
41128099999  -7.24
76577099999   7.16
02096099999  -7.00
44277099999  -6.96
61498099999  -6.93
40377099999   6.93
55664099999  -6.88
71627899999  -6.61
41136099999  -6.57
17330099999  -6.32
40420099999  -6.28
41128099999  -6.27
16115099999   6.25
41128099999  -6.09
40260099999  -5.88
11993099999  -5.87
55578099999  -5.78
40310099999  -5.66
72520799999   5.63
41128099999   5.53
40340099999  -5.38
41240099999  -5.29
40809099999  -5.23
55279099999  -5.20
60580099999  -5.17
44373099999   5.16
44287099999  -5.15
94339099999   5.13
41084099999  -5.07
44298099999  -5.06

Tw in deg C:
71749099999 -16.56
71749099999  16.31
71627899999   7.73
71627899999  -7.50
02096099999  -6.48
16115099999   5.35

DPD in deg C:
41128099999  -8.88
41128099999   8.85
44277099999   6.98
76577099999  -6.85
55664099999   6.84
40377099999  -6.78
40357099999  -6.68
41128099999   6.60
41136099999   6.40
41128099999   6.39
40420099999   6.27
94346099999   6.17
41061099999  -6.17
17330099999   6.16
55578099999   6.12
41240099999   6.11
94332099999   6.07
40260099999   5.87
41128099999  -5.79
72520799999  -5.64
61498099999   5.50
55279099999   5.37
04231099999  -5.37
40310099999   5.33
44287099999   5.30
94238099999   5.29
95492099999   5.23
40809099999   5.19
60580099999   5.07
95482099999   5.03

 

3.6) Stations Failing Post-PHA climatology check (data removed), Supersaturation and Subzeros:

T:

Good Stations = 4364

Bad Stations = 143

RH:

Good Stations = 4438 (with supersaturated stations removed)
Bad Stations = 16
Supersaturated stations = 48

DPD:

Good Stations = 3865 (with supersaturated stations removed)

Bad Stations = 437

Supersaturated Stations = 212

q:
Good Stations = 4393 (with supersaturated and subzero stations removed)
Bad Stations = 17
Supersaturated Stations = 48
Subzero Stations = 44

e:
Good Stations = 4393 (with supersaturated and subzero stations removed)
Bad Stations = 17
Supersaturated Stations = 48
Subzero Stations = 44

Tw:  
Good Stations = 3592 (with supersaturated stations removed)
Bad Stations = 17
Supersaturated Stations = 896010080 Td:
Good Stations = 3724 (with supersaturated stations removed)
Bad Stations = 579
Supersaturated Stations = 204

 

3.7) Estimating uncertainty and gridding

The remaining stations are then gridded to create the HadISDH.4.0.0.2017f product which is a 5 by 5 degree gridbox average over all stations within each gridbox (no interpolation or weighting). Uncertainty fields are also gridded in addition to an estimate of uncertainty due to the uneven spatio-temporal sampling across each gridbox. Missed adjustment uncertainty is obtained from a global analysis of the distribution of adjustments for each variable subtracted from a fat-tailed Gaussian fit that infills the missing middle. The PHA/IDPHA adjustments and changepoint frequencies are shown in Figures 1 to 7 below.

Adjustment statistics are as follows:

q in g/kg: 
Mean adjustment size: -0.007
Standard Deviation of Adjustment size: 0.414
Missed adjustment uncertainty: 0.215
Mean number of adjustments per station: 4.358

e in hPa: 
Mean adjustment size: -0.010
Standard Deviation of Adjustment size: 0.641 
Missed adjustment uncertainty: 0.260 
Mean number of adjustments per station: 4.358 

RH in %rh:
Mean adjustment size: 0.029
Standard Deviation of Adjustment size: 3.655 
Missed adjustment uncertainty: 1.322 
Mean number of adjustments per station: 4.360 

Tw in deg C:
Mean adjustment size: -0.015 
Standard Deviation of Adjustment size: 0.483 
Missed adjustment uncertainty: 0.219 
Mean number of adjustments per station: 4.358 

T in deg C:
Mean adjustment size: -0.025 
Standard Deviation of Adjustment size: 0.605 
Missed adjustment uncertainty: 0.292 
Mean number of adjustments per station: 4.164
 
DPD in deg C:
Mean adjustment size: -0.012 
Standard Deviation of Adjustment size: 1.233
Missed adjustment uncertainty: 0.430 
Mean number of adjustments per station: 2.916

Td in deg C:
Mean adjustment size: -0.017
Standard Deviation of Adjustment size: 1.048
Missed adjustment uncertainty: 0.392
Mean number of adjustments per station: 4.244 

Figure 1 Specific Humidity adjustment and changepoint statistics. a) Distribution of adjustment sizes. The actual distribution is shown in black. The Gaussian best fit is shown in grey. A fat-tailed Gaussian merged fit made up of the real and Gaussian bins, taking the largest value at each bin, is shown in dashed red. The difference distribution between the fat-tailed Gaussian and the real distribution is shown in dotted blue. The mean and standard deviation of the difference distribution are shown. The standard deviation of the difference distribution is taken as the 1 sigma estimate for missed adjustment uncertainty.

Figure 2 Relative Humidity adjustment and changepoint statistics. See Figure 1 for details.

Figure 3 Vapour Pressure adjustment and changepoint statistics. See Figure 1 for details.

Figure 4 Dew Point Temperature adjustment and changepoint statistics. See Figure 1 for details.

Figure 5 Wet Bulb Temperature adjustment and changepoint statistics. See Figure 1 for details.

Figure 6 Dew Point Depression adjustment and changepoint statistics. See Figure 1 for details.

Figure 7 Temperature adjustment and changepoint statistics. See Figure 1 for details.

In all cases the changepoint frequency per station has increased as it did between 2013, 2014 and 2015 by about 0.5 and by about 0.1 compared to 2016. The larger number of stations compared to previous years, especially 2015 and earlier, should make it easier to detect inhomogeneities as does the longer temporal series from the added year of data. Missed adjustment uncertainties are also higher and the mean adjustment magnitude is slightly further away from zero. This implies that as our ability to detect inhomogeneity improves we learn that we have previously underestimated the size and frequency of missed adjustments. 

4) Differences in gridbox and regional average trends between HadISDH.4.0.0.2017f and 3.0.0.2016p

Version 4.0.0.2017f adds an extra 12 months of data to most stations but additionally, there are some historical additions and amendments due to housekeeping done at the NOAA/NCEI ISD end. Even just adding an extra year of data can alter some of the changepoint locations detected and adjustment magnitudes applied. So, with this in mind, we would expect very small differences between the two versions. This year we have moved to a 1981-2010 climatology compared to a 1976-2005 climatology which explains the offsets in the time series. To look at the differences in more detail, Figures 8 to 14 explore differences in gridbox and regional trends between the two versions. Note that trends cover the full period of each version.

Figure 8 Specific Humidity gridbox and regional average trend differences between versions. a) Ratio of gridbox decadal trends. A ratio greater than 1 means that trends in v3.0.0.2016p are greater than in v4.0.0.2017f and vice versa. b) Scatter plot of gridbox decadal trends between the two version with percentage of gridboxes falling in each quadrant. c) Distribution of gridbox decadal trends. d) to g) Regional average time series and decadal trends (with 5th to 95 percentile confidence ranges in parentheses). Regional averages are calculated by weighting each gridbox by the cosine of its latitude. Trends are fitted using the median of pairwise slopes (Sen, 1968) and the range represents the 95% uncertainty.

Figure 9 Relative Humidity gridbox and regional trend differences between versions. Details are as for Figure 8.

Figure 10 Vapour Pressure gridbox and regional trend differences between versions. Details are as for Figure 8.

Figure 11 Dew Point Temperature gridbox and regional trend differences between versions. Details are as for Figure 8.

Figure 12 Wet Bulb Temperature gridbox and regional trend differences between versions. Details are as for Figure 8.

Figure 13 Dew Point Depression gridbox and regional trend differences between versions. Details are as for Figure 8.

Figure 14 Temperature gridbox and regional trend differences between versions. Details are as for Figure 8.

Trend differences are generally very small. In most cases (except S. Hemisphere Td and DPD) the 1973-2017 trends show very slightly stronger warming and moistening (increasing water vapour)/less aridity (smaller decreasing trends in saturation [RH]). The 1973-2017 global RH drying trend is significantly different from zero but this is not the case for the Tropics or Northern Hemisphere and it is weaker than 1973-2016.

With the exception of RH and DPD very few gridboxes have a decadal trend in the opposite direction to thhe 1973-2016 trends from HadISDH.3.0.0.2016p. Those gridboxes where the trend has changed direction should be treated with caution.

5) Gridbox decadal trends (1973-2017) for HadISDH.4.0.0.2017f

As demonstrated in section 4, the addition of an extra 12 months of data and in a few cases, some historical amendments, has really made very little difference to large scale trends. Figures 15 to 21 below show the decadal gridbox trends. Gridbox trends are also shown as aggregates by latitude which demonstrates clear latitudinal differences in some cases and spread. The proportion of land gridboxes at each latitude (dark grey), and the proportion of those gridboxes with data (light grey), are also shown for each variable. All variables have near complete coverage in the Northern Hemisphere, with virtually no sampling at the high latitudes and poorest coverage in the Tropics.

Figure 15 Specific Humidity gridbox decadal trends over the 1973 to 2017 period. Decadal trends are also shown by latitude with the average for each latitude also plotted. Dark grey shading represents the proportion of area that is a land gridbox at that latitude. Light grey shading represents the proportion of those land gridboxes with data present. Trends are fitted using the median of pairwise slopes (Sen, 1968). Gridboxes with black borders are those considered to be significantly different from a zero trend (95% uncertainty).

Figure 16 Relative Humidity gridbox decadal trends over the 1973 to 2017 period. Details are as described for Figure 15.

Figure 17 Vapour Pressure gridbox decadal trends over the 1973 to 2017 period. Details are as described for Figure 15.

Figure 18 Dew Point Temperature gridbox decadal trends over the 1973 to 2017 period. Details are as described for Figure 15.

Figure 19 Wet Bulb Temperature gridbox decadal trends over the 1973 to 2017 period. Details are as described for Figure 15.

Figure 20 Dew Point Depression gridbox decadal trends over the 1973 to 2017 period. Details are as described for Figure 15.

Figure 21 Temperature gridbox decadal trends over the 1973 to 2017 period. Details are as described for Figure 15.

These figures appear to show more spatially consistent trends than HadISDH.2.1.0.2015p and, as expected given the greater number of stations, improved spatial coverage.