Exploring Marine Data QC

HadISDH Marine QC analysis for TinyClim – March 10th 2016

The EUSTACEMDS has now been modified to pull through humidity from ICOADS.2.5.1 and associated variables. I have also modified the QC to include humidity which has also led to some changes being applied to temperature too.

The buddy checker (MDS only, not Bayesian – no appropriate data fields) now uses seasonally varying stdevs of air temperature  and dew point temperature from ERA Interim (as opposed to Jan 1-5th pentad climatological stdevs from SST for air temperature). It is only applied to those obs that pass the pos (real position), date (real date), blklst (not blacklisted), trk (track check) and clim (climatology check) tests.

The climatology check now works with seasonally and spatially varying limits related to the pentad stdev at that location (1by1 degree box). The limit is 4.5 * stdev but has a minimum threshold of 4.5 * 0.5 and a maximum threshold of 4.5 *3 for both air temperature and dew point temperature. The previous limit was set a 10 degrees for all times and locations for both air temperature and dew point temperature. 

Note that the climatology uses 1by1 degree fields averaged over pentads (5 days). We're initially using ERA-Interim 1981-2010. We plan to build an obs-based climatology infilled with ERA-Interim later on. As we are comparing hourly point values with a pentad mean gridbox values, this is not ideal. Therefore we need sufficiently large limits to test for outliers.

Both air temperature and dew point temperature are tested for repeated strings of variables within a voyage. There must be fewer than 70% matching values. 

Dew point temperature is tested for supersaturation – where it is greater than the air temperature. 

Dew point temperature is also tested for repeated saturation – where saturation persists for more than 48 hours and there are more than 4 obs within that time. 

Below are figures from two example months exploring the kick-out and retension from climatology and buddy checking with the latest code version (TinyClim, 10th March 2016). Figures 1 to 12 focus on December 1973 where 138196 obs are present that contain both air temperature and dew point temperature, and that pass the positional check (lat and long within -180-180, -90-90), date check (is a real hour, day, month, year) and are not blacklisted for any reason. Figures 13 to 24 focus on January 2010 where 200971 obs are present. These are terrible plots that I've just spat out of a quick play with Python. Y axes show latitude 90S to 90N. X-axes show the value of the variable.

Note the skew towards positive anomalies passing the QC for both dew point temperature and especially air temperature. This is particularly noticeable in the Northern Hemisphere. This could partly be the effect of the solar bias but this shouldn't really affect dew point temperature. This could be a bias relative to the ERA-Interim climatology. This could be a climate change signal but that is less likely given the climatology period is 1981-2010 and the positive skew is present in the 1973 figures. Ideally we should redo these figures with some way of looking at the density of points. Also we can plot the daytime vs nighttime obs to see if the positive bias is more prevalent in the daytime obs which would then suggest that the solar bias is playing a role.

Some other stats that might be interesting:

Total obs with air temperature and dewpoint temperature present, and that have a real location and real data and are not blacklisted
Dec 1973: 138196
Jan 2010: 200971

Anomalies within +/- 10 degrees of climatology:
Dec 1973 air temperature: 98.4% (135998)
Dec 1973 dew point temperature: 99.3% (137248)
Jan 2010 air temperature: 94.4% (189817)
Jan 2010 dew point temperature: 97.1% (194895)

Climatology test: 0=pass, 1=fail, 8=unable to test, 9=not assessed
Dec 1973 air temperature: 0=132123 (95.6%), 1=6073 (4.4%), 8=0, 9=0
Dec 1973 dew point temperature: 0=133753 (96.8%), 1=4443 (3.2%), 8=0, 9=0
Dec 1973 sea surface temperature: 0=130239 (94.2%), 1=1259 (0.9%), 8=6698 (4.8%), 9=0
Jan 2010 air temperature: 0=195919 (97.5%), 1=5052 (2.5%), 8=0, 9=0
Jan 2010 dew point temperature: 0=196149 (97.6%), 1=4822 (2.4%), 8=0, 9=0
Jan 2010 sea surface temperature:  0=141519 (70.4%), 1=1340 (0.7%), 8=58112 (28.9%), 9=0

Buddy test: 0=pass, 1=fail, 8=unable to test, 9=not assessed
(only applied to those that pass track and climatology check)
Dec 1973 air temperature: 0=119238 (86.3%), 1=9753 (7.1%), 8=9205 (6.7%), 9=0
Dec 1973 dew point temperature: 0=119618 (86.6%), 1=10930 (7.9%), 8=7648 (5.6%), 9=0
Jan 2010 air temperature: 0=185466 (92.3%), 1=10152 (5.1%), 8=5353 (2.7%), 9=0
Jan 2010 dew point temperature: 0=180787 (90.0%), 1=14580 (6.8%), 8=5604 (2.8%), 9=0

Buddy and climatology passes:Dec 1973 air and dewpoint temperature: 109102 (78.9%)
Dec 1973 air temperature: 119238 (86.3%) [18958, 13.7%]
Dec 1973 dew point temperature: 119618 (86.6%) [18578, 13.4%]
Jan 2010 air and dew point temperature: 171178 (85.2%)
Jan 2010 air temperature: 185466 (92.3%) [15505, 7.7%]
Jan 2010 dew point temperature: 180787 (90.0%) [20184, 10.0%]

Repeated values within a voyage test: 0=pass, 1=fail, 9=not assessed
Dec 1973 air temperature: 0=138191 (>99.999%), 1=5 (<0.001%), 8=0, 9=0
Dec 1973 dew point temperature: 0=138193 (>99.999%), 1=3 (<0.001%), 8=0, 9=0
Jan 2010 air temperature: 0=200925 (>99.9%), 1=46 (<0.1%), 8=0, 9=0
Jan 2010 dew point temperature: 0=200947 (>99.9%), 1=24 (<0.1%), 8=0, 9=0

Supersaturation test: 0=pass, 1=fail, 9=not assessed
Dec 1973 dew point temperature: 0=138094 (99.9%), 1=102 (0.1%), 8=0, 9=0
Jan 2010 dew point temperature: 0=199920 (99.5%), 1=1051 (0.5%), 8=0, 9=0

Prolongued saturation test: 0=pass, 1=fail, 9=not assessed
Dec 1973 dew point temperature: 0=138084 (99.9%), 1=112 (0.1%), 8=0, 9=0
Jan 2010 dew point temperature: 0=200676 (99.9%), 1=295 (0.1%), 8=0, 9=0

Track check: 0=pass, 1=fail, 9=not assessed
Dec 1973: 0=134655 (97.4%), 1=2303 (1.7%), 9=1238 (0.9%)
Jan 2010: 0=200329 (99.7%), 1=585 (0.3%), 9=57 (<0.1%)

Total QC passes: pass clim, buddy, trk, rep, repsat and ssat tests
Dec 1973 air and dew point temperature: 108986 (78.9%)Dec 1973 air tempeature: 119233 (86.3%)
Dec 1973 dew point temperature: 119485 (86.5%)
Jan 2010 air and dew point temperature: 170109 (84.6%)
Jan 2010 air temperature: 185431 (92.3%)
Jan 2010 dew point temperature: 179689 (89.4%)

Figure 1 Actual dew point temperature values that fail the climatology (blue) and buddy (red) check for December 1973 - 18578 (13.4%) by latitude. Yaxis = latitude, Xaxis = degrees C.

 

Figure 2 Actual dew point temperature values that pass the climatology and buddy check for December 1973 - 119618 (86.6%) by latitude. Yaxis = latitude, Xaxis = degrees C.

Figure 3 Anomaly dew point temperature values that fail the climatology (blue) and buddy (red) check for December 1973 - 18578 (13.4%) by latitude. Yaxis = latitude, Xaxis = degrees C.

Figure 4 Anomaly dew point temperature values that pass the climatology and buddy check for December 1973 - 119618 (86.6%) by latitude. Yaxis = latitude, Xaxis = degrees C.

Figure 5 Actual air temperature values that fail the climatology (blue) and buddy (red) check for December 1973 - 18958 (13.7%) by latitude. Yaxis = latitude, Xaxis = degrees C.

Figure 6 Actual air temperature values that pass the climatology and buddy check for December 1973 - 119238 (86.3%) by latitude. Yaxis = latitude, Xaxis = degrees C.

Figure 7 Anomaly air temperature values that fail the climatology (blue) and buddy (red) check for December 1973 - 18958 (13.7%) by latitude. Yaxis = latitude, Xaxis = degrees C.

Figure 8 Anomaly air temperature values that pass the climatology and buddy check for December 1973 - 119238 (86.3%) by latitude. Yaxis = latitude, Xaxis = degrees C.

Figure 9 Actual specific humidity values that pass the air temperature and dew point temperature climatology and buddy check for December 1973 - 109102 (78.9%) by latitude. Yaxis = latitude, Xaxis = g/kg.

Figure 10 Actual relative humidity values that pass the air temperature and dew point temperature climatology and buddy check for December 1973 - 109102 (78.9%) by latitude. Yaxis = latitude, Xaxis = %rh.

Figure 11 Anomaly specific humidity values that pass the air temperature and dew point temperature climatology and buddy check for December 1973 - 109102 (78.9%) by latitude. Yaxis = latitude, Xaxis = g/kg.

Figure 12 Anomaly relative humidity values that pass the air temperature and dew point temperature climatology and buddy check for December 1973 - 109102 (78.9%) by latitude. Yaxis = latitude, Xaxis = %rh.

Figure 13 Actual dew point temperature values that fail the climatology (green) and buddy (yellow) check for January 2010 - 20184 (10.0%) by latitude. Yaxis = latitude, Xaxis = degrees C.

Figure 14 Actual dew point temperature values that pass the climatology and buddy check for January 2010 - 180787 (90.0%) by latitude. Yaxis = latitude, Xaxis = degrees C.

Figure 15 Anomaly dew point temperature values that fail the climatology (green) and buddy (yellow) check for January 2010 - 20184 (10.0%) by latitude. Yaxis = latitude, Xaxis = degrees C.

Figure 16 Anomaly dew point temperature values that pass the climatology and buddy check for January 2010 - 180787 (90.0%) by latitude. Yaxis = latitude, Xaxis = degrees C.

Figure 17 Actual air temperature values that fail the climatology (green) and buddy (yellow) check for January 2010 - 15505 (7.7%) by latitude. Yaxis = latitude, Xaxis = degrees C.

Figure 18 Actual air temperature values that pass the climatology and buddy check for January 2010 - 185466 (92.3%) by latitude. Yaxis = latitude, Xaxis = degrees C.

Figure 19 Anomaly air temperature values that fail the climatology (green) and buddy (yellow) check for January 2010 - 15505 (7.7%) by latitude. Yaxis = latitude, Xaxis = degrees C.

Figure 20 Anomaly air temperature values that pass the climatology and buddy check for January 2010 - 185466 (92.3%) by latitude. Yaxis = latitude, Xaxis = degrees C.

Figure 21 Actual specific humidity values that pass the air temperature and dew point temperature climatology and buddy check for January 2010 - 171178 (85.2%) by latitude. Yaxis = latitude, Xaxis = g/kg.

Figure 22 Actual relative humidity values that pass the air temperature and dew point temperature climatology and buddy check for January 2010 - 171178 (85.2%) by latitude. Yaxis = latitude, Xaxis = %rh.

Figure 23 Anomaly specific humidity values that pass the air temperature and dew point temperature climatology and buddy check for January 2010 - 171178 (85.2%) by latitude. Yaxis = latitude, Xaxis = g/kg.

Figure 24 Actual relative humidity values that pass the air temperature and dew point temperature climatology and buddy check for January 2010 - 171178 (85.2%) by latitude. Yaxis = latitude, Xaxis = %rh.