Drop off in HadISDH stations post-2005

The Problem

For a climate monitoring product, ideally, the number of ingested stations would stay constant over time. In reality, this would limit spatial coverage severely and so we compromise incomplete temporal coverage to maximise spatial coverage. For HadISDH, there are a odd features in the station count over time that cannot be explained by our data processing methods (QC and homogenisation result in the removal of data).  

Figures 1 and 2 show the temporal change in station density after QC and homogenisation for q in terms of station months present and station months missing respectively. The colours represent different regions in the WMO ID These are bounded as follows: Europe = 0-199999 (dark red), Russia/Eastern Europe = 200000-379999 (crimson), Central Asia/Middle East/India/Pakistan = 380000-439999 (dark orange), East Asia/China = 440000-599850 (orange), Africa = 600000-689999 (gold), USA/Canada = 690000-749999 (dark green), Central America = 760000-799999 (olive green), South America = 800000-879999 (royal blue), Antarctica = 880000-899999 (sky blue), Pacific Islands (Inc. Hawaii) = 911000-919999 (navy), Australia/New Zealand = 930000-949999 (lilac), Indonesia/Philippines/Borneo = 960000-999999 (purple).  

Figure 1 Station presence by month for the HadISDH.landq.2.0.0.2013p dataset shown by WMO ID number (left axis/colours) and total station count (right axis/black line).

 

Figure 2 Missing Stations by month for the HadISDH.landq.2.0.0.2013p dataset shown by WMO ID number (left axis/colours). Total stations present counts are also shown (right axis/black line).

Note that looking at either of these figures in isolation is deceptive in terms of the density of stations present and missing. While spatial coverage is reasonable it is clear that missing data is a problem throughout the time period and especially at the start and end periods of the record. Missing data can arise for three reasons. Firstly, and most likely, it exists but just has not made it through the GTS to ISD for some reason. Secondly, in some cases the data may not have been recorded in the first place due to technical, social or environmental reasons. Thirdly, these data may exist in ISD but have been removed by QC or homogenisation processing. In particular it is clear that the 2005 drop off is driven mostly by stations in the USA.
 

Why?

The station selection going into HadISDH was created in 2008. This was done by testing a few different climatology periods (1973-2002,1974-2003....1977-2006) - ISD station frequency is very small prior to 1973 and at this time we had only scraped data until 2007. The 1976-2005 period resulted in the largest number of stations that had sufficient coverage. 

In many cases there are lots of short periods of station reports from different ID numbers that actually belong to the same station. These would not be included in our selection but significant effort was made to find any other stations that may actually be the same as a station in our selection, reporting with a different ID. Where sufficient number of station match tests were passed, stations were merged together resulting in longer station records. This was a very time consuming process and so has not been revisited since 2008. This means that stations that have stopped reporting and started again with a new ID number towards the end of the period (e.g. 2005) are not recognised as stations that need merging. This is a significant contributing factor to the 2005 drop off in station count.

Below is a list of stations in our selection that do not have any data post-2005. It is clearly the USA stations that dominated this list, but there are others too.

128 US stations drop out from 2005 onwards:
701040, 701170, 701210, 701730, 702120, 702220, 702315, 702350, 702650,
702720, 702750, 702757, 702770, 702986, 703050, 703210, 703870, 722029,
722115, 722116, 722119, 722245, 722268, 722348, 722429, 722577, 722598,
722725, 722865, 722885, 722897, 722955, 723069, 723307, 723345, 723407,
723415, 723535, 723536, 723544, 723555, 723783, 724067, 724094, 724097,
724127, 724175, 724288, 724336, 724371, 724458, 724555, 724625, 724666,
724676, 724699, 724765, 724838, 724915, 724938, 725060, 725116, 725155,
725235, 725305, 725314, 725315, 725316, 725335, 725490, 725515, 725524,
725525, 725526, 725555, 725556, 725565, 725625, 725635, 725685, 725705,
725715, 725724, 725765, 725776, 725866, 726055, 726130, 726355, 726357,
726375, 726456, 726465, 726480, 726500, 726515, 726525, 726545, 726548,
726555, 726587, 726650, 726685, 726700, 726865, 726986, 727344, 727415,
727437, 727445, 727458, 727550, 727555, 727573, 727840, 727856, 727857,
727890, 727928, 727937, 743700, 747185, 747240, 747320, 747540, 747804,
747810, 747945

39 other stations drop out from 2005 onwards:
031740, 032450, 033140, 039280, 064080, 064520, 074280, 082100, 110280,
111720, 135620, 150040, 164594, 228370, 257440, 284190, 285370, 286150,
289250, 302530, 305370, 307260, 401760, 401910, 477640, 545870, 570160,
679750, 718360, 785140, 845640, 864300, 870780, 911780, 911860, 912320, 913660, 944020, 944480 

This can be seen more clearly in Figure 3 which shows the total count of stations for each month in each WMO block. There is a clear drop off in 2005 and also 2010 for USA stations (racing green). Other interesting features emerge; there is a common fall-out for 2000-2001 for European and Russian stations along with other short blips in data reporting.

Figure 3 Station counts per month for WMO blocks (coloured lines, left axis) and in total (black line, right axis) for HadISDH.landq.2.0.0.2013p. WMO blocks are as follows: Europe = 0-199999 (dark red), Russia/Eastern Europe = 200000-379999 (crimson), Central Asia/Middle East/India/Pakistan = 380000-439999 (dark orange), East Asia/China = 440000-599850 (orange), Africa = 600000-689999 (gold), USA/Canada = 690000-749999 (dark green), Central America = 760000-799999 (olive green), South America = 800000-879999 (royal blue), Antarctica = 880000-899999 (sky blue), Pacific Islands (Inc. Hawaii) = 911000-919999 (navy), Australia/New Zealand = 930000-949999 (lilac), Indonesia/Philippines/Borneo = 960000-999999 (purple). 

We can look at the USA stations that do not report post-2005 in more detail in Figure 4. They are reasonably evenly spread across the USA. This means that no particular region of the USA is completely lost, nor the regional average biased due to fall-out of an entire region.

Figure 4 Station locations of the 128 USA stations with no data post-2005.

What to do about it?

The station selection and merge process needs to be revisited for HadISD and HadISDH. As part of this we may consider other databanks to ISD although the format and accessibility of ISD is very convenient and there are ongoing developments to improve ISD. However, redoing the station selection and merge is a significant chunk of work and is unlikely to be tackled prior to the 2014 update of HadISD/HadISDH. Until such a time that it is done, users should be aware of this drop off in data (and in general the prevalence of missing data at start/end points. 

It is reasonable to question whether this change in station density contributes to any trends seen in the large scale averages of the humidity variables. We do not see matching patterns (at both ends of the time series) in any of the global average trends (see Figure 4). Furthermore, there is generally very good agreement with the alternative datasets shown in Figure 4. Finally, the 2005 drop off is mostly due to US stations (128 stations) so it should not have a great effect on the global average (~3500 stations). Users of solely HadISDH data over the USA region may wish to be careful although the USA has a relatively good station density (~740 stations) so the loss of 128 stations, reasonably evenly spread across the region, is less of an issue than for regions with worse spatial coverage.

Figure 5 Annual global average time series of all HadISDH variables alongside alternative datasets where available. All time series are shown as anomalies relative to 1981-2010.