Skip to main content

Cloud Mask Issues (35_L2)

Tracking of known problems and subsequent fixes is an important issue for MODIS data users. This page will act as a repository of all known MODIS Atmosphere Data Product problems, as well as how to determine the problematic version (and the fixed version) of the HDF data -- therefore data users should check this page for updates regularly. Data Users unfamiliar with how to properly track problems and fixes by determining the version of their downloaded HDF files should refer to the documentation in the Introduction sublink (the top link along the LHS of this page).   

Note that a checkmark (✔) in the tables below means Data Issue or Quality Statement applies.  A blank means it's been fixed or does not apply.

 




Cloud Mask Data Issues: Table of Contents & Anchor Links  

(Note: Anchor Links point to a specific place within this page.)

Collection/Platform where Issues Apply
Issue Impact

C6.1/Terra

C6.1/Aqua

C6.0/Terra

C6.0/Aqua

Link
#1. False Clouds over Ocean (Band 29)

H

 

 

 

Link
#2. Detection Prob. w. Antarctica Clouds

L

Link
#3. Band 27 anomalies impact CldMask

L

 

 

 

Link
#4. Spurious Clouds During Polar Night

L

 

 

Link
#5. GDAS Update causes slight changes

L

 

 

Link
#6. False Clears over colder Ocean Sunglint

M

 

 

Link
#7. PWA Failure in Oct 2020 affecting high lats.

M

 

 

 

Link

Cloud Mask Data Quality Statements: Table of Contents & Anchor Links  

(Note: Anchor Links point to a specific place within this page.)

Collection/Platform where Quality Statements Apply
Quality Statement Impact

C6.1/Terra

C6.1/Aqua

C6.0/Terra

C6.0/Aqua

Link
#1. Missing Granules Aug 2017

Q

✔ 

 

 ✔

 

Link
#2. GDAS Change Impacts

Q

Link

 




Issue #1: MOD35 (Terra Cloud Mask) False Clouds over Ocean (Impact of the MODIS Terra Band 29 (8.6 µm) radiometric calibration drift on cloud products)

Description (Updated Information): The Terra spacecraft and instruments went into safe hold from 2016-02-18 through 2016-02-22. On 2016-04-08, Terra MODIS Collection 5 (C5) and Collection 6 (C6) Atmosphere forward processing and distribution via LAADS was restarted from data day 2016 059 (Feb 28th 2016). The MODIS Characterization and Support Team (MCST) has made post-safe mode radiance calibration adjustments to the L1B (MOD02) code. However, degraded detector performance in Terra MODIS Band 27 (6.7 um) and Band 29 (8.6 um) necessitated algorithm changes to the L2 cloud mask (MOD35) and the L1B de-striper code used in the production of the L2 atmospheric profile (MOD07/05) and cloud-top (MOD06) products. Specifically, cloud mask thresholds were adjusted to remove false positives for the 6.7 um brightness temperature (BT) test for high clouds and the 8.6-11 um BTD test over the ocean, while reference detectors in the de-striper code were modified to avoid noisy detectors.  Users are advised that further testing is needed to re-validate the current forward stream for the above products (and also down stream products, e.g., MOD06 cloud optical properties and Level-3 Daily, Eight-Day, and Monthly) and to document the impact (if any) of these combined sensor and algorithm changes on the use of MODIS Terra atmosphere team products for climate studies. We previously noted that Band 29 detector crosstalk degradation resulted in artificially high cloud cover over tropical oceans in recent years. The post-safe mode adjusted threshold for the 8.6-11 um BTD test is expected to reduce the impact on cloud fraction. Therefore, the climate continuity record is broken, specifically cloud mask (i.e., fractions) and to a varying extent the downstream products such as cloud top, IR phase, and optical properties.

Description (Initial Detection): In early 2015, it was noticed that recent Terra Collection 6 (C6) cloud top pressure products (MOD06) contained ocean regions with an anonymously high incidence of low clouds. This was traced to false clouds being reported by the cloud mask (MOD35). In turn, this was found to be the result of errors in the b29-b31 brightness temperature difference (BTD) high (ice phase) cloud test. As a result, a substantial number of clear pixels were being flagged as cloudy by this test. The text that follows is a summary of the impact of the MODIS Terra Band 29 (8.6 µm) radiometric calibration drift on cloud products. MOD35 includes a spectral test using the brightness temperature difference (BTD) between band 29 (8.6 µm) and band 31 (11 µm), referred to here as the b29-b31 BTD test. This test is used only over water surfaces. The root problem is a gradual warming in the Terra MODIS b29 channel over the years, that is not being captured by the on-board calibration systems and apparently related to crosstalk issues. Monthly time series trend analysis of Terra MODIS cloud products indicates that the significance of the impact may extend back to 2010. In lieu of a successful fix to the L1B algorithm for b29, an adjustment to the 8.6-11 µm MOD35 test threshold was delivered to the production test system to quantify the impact. Testing of this potential fix in the data is ongoing as of September 2015. A fix should eventually be implemented, at which time affected data will be rerun (corrected) and made available. See the PDF document linked below for for detailed information and imagery.

MODIS Terra Band 29 (8.6 micron) impact of calibration drift on cloud products (PDF)

PGE Versions & Production Dates of Problematic & Corrected Data:

Problematic Data Corrected Data

PGE Versions

Production Dates

PGE Versions

Production Dates

All

≥ 2010 001 (1/1/2010)

Correction Testing Underway

Correction Testing Underway

Data Dates Affected: January 2010 and on (but worse 2014 and on)

Affected Platform: Terra Only

Collections Affected: Collection 6.0

Return to Menu at Top of Page

 


Issue #2: Cloud Mask not detecting high thick clouds over Weddell Sea (Antarctica) when ice covered

Description: In collection 006, the cloud mask is failing to detect high thick clouds over the Weddell Sea area in Antarctica when it is ice-covered. This represents a small area over very dry conditions. This problem might be fixed in a future PGE03 delivery, until then users should be aware of this shortcoming.

PGE Versions & Production Dates of Problematic & Corrected Data:

Problematic Data Corrected Data

PGE Versions

Production Dates

PGE Versions

Production Dates

≤ PGE03 v6.0.40

All

Not Corrected

Not Corrected

Data Dates Affected: July 2002 to present

Affected Platform: Aqua Only

Collections Affected: Collection 6.0 and Collection 6.1

Return to Menu at Top of Page

 


Issue #3: Occasional Band 27 Anomalies impact Cloud Mask Parameter

Description: An anomaly was noted sometime after the calibration change of Terra C6.1 (noted by Chris Moeller, UW). The cloud mask uses band 27 in a high cloud test that inadvertently flags many of the pixels at the end of scan (1349-1354) as being cloudy because of anomalous BTs.  (Note that intermittent edge-of-scan anomalies in Band 28 and Band 30 have also been recently noted. The anomalies in Band 28 match the location of anomalies in Band 27; however in band 30 the anomalies seem to switch to the first few pixels at the beginning of a scan 1-5 instead of at the end of a scan 1349-1354).  This band 27 issue propagates into the Cloud Mask and then into the Cloud Fraction from Cloud Mask and then into many Cloud Top Property parameters in L2.  Its difficult to see in L3 due to coarser resolution and orbit overlap.  It does not appear that this issue propagates into Cloud Optical Property parameters at L2.  Figure 9a below shows an unmapped granule image of b27 BTs and the b27 cloud mask test.  Figure 9b shows mapped impacts in downstream Cloud Top Property products (Date Imaged 2018 148).

PGE Versions & Production Dates of Problematic Data:

Data Dates Affected: Shows up intermittently in all time periods (from launch)

Affected Platform: Terra

Collections Affected: Collection 6.1 only (Cloud Mask and Cloud Top Property parameters at L2)

Additional Documentation:

Return to Menu at Top of Page

 


Issue #4: Spurious Clouds During Polar Night

Description: Spurious, noisy-looking (speckled) clouds appear in polar night conditions over Antarctica and Greenland in the Collection 6.1 cloud mask data. These false clouds are caused by the 11-3.9 micron BTD (brightness temperature difference) cloud test when pixel temperatures are very cold. In the absence of solar illumination, 3.9 micron observations are very noisy at temperatures below about 230K, as measured in the 11 micron band. This noise causes the test to lose sensitivity to cloudy vs. clear conditions (see Fig. 3). An application threshold of 230K was planned and tested for Collection 6 processing but due to an oversight was never implemented. Any reprocessing in the future will use the 230K threshold and remove most of these spurious clouds.

An example of the false clouds is shown here, where Fig. 1 shows 11 micron imagery of an Antarctic scene from July 1, 2018 at 14:30 UTC, Fig. 2 shows the cloud mask results with false clouds scattered throughout some clear areas, and Fig. 3 shows the offending 11-3.9 micron BTD test. Figs. 4 and 5 show the cloud mask and 11-3.9 micron BTD test results, respectively, when the test is not performed for pixels with 11 micron brightness temperatures < 230K.

PGE Versions & Production Dates of Problematic Data:

Data Dates Affected: Shows up intermittently in all C6.1 data (from launch)

Affected Platform: Terra & Aqua

Collections Affected: Collection 6.1 only (Cloud Mask parameters at L2)

Additional Documentation:

Return to Menu at Top of Page

 


Issue #5: New GDAS for data processed 12 June 2019 and forward

Description: A change has been made to the GDAS (Global Data Assimilation System) atmospheric profiles of temperature, moisture, and ozone profiles that are used as ancillary data in the cloud top properties algorithms (5-km and 1-km MOD06CT and MYD06CT products).  The greatest impact to these products is seen in polar regions where many cloud top heights are raised (cloud top pressures lowered).  A decrease is seen in the number of cloud height retrievals at surface levels due to errors in temperature profiles and uncertainties in the algorithms.  Changes in other parts of the world appear to be random and mostly less than 50 hPa.

The MODIS cloud masks (MOD35 and MYD35) are minimally impacted through changes to the GDAS surface temperature.  These cloud detection changes are small, random in nature, and are confined almost completely to nighttime data where surface temperatures are important for warmer surfaces.

Shown below are two difference images from Terra daily (MOD08_D3) for day 2019134. The test data used for these difference images were created using a new version of GDAS data and the baseline is the old version.

 

 

PGE Versions & Production Dates of Problematic Data:

Data Dates Affected: New GDAS Data was used to process starting on 12 June 2019 data.

Affected Platform: Terra & Aqua

Collections Affected: Collection 6.1 only

Additional Documentation:

Return to Menu at Top of Page

 


Issue #6: False Clears over colder Ocean Sunglint Regions

Description:  A problem has been seen in some colder ocean areas where sun-glint is expected -- in a few very limited areas low-level clouds and/or fog is mislabeled as "clear" or "probably clear" by the MODIS cloud mask (MOD35).  It rarely occurs in widespread regions and is usually found in colder ocean waters such as the North Pacific, and only where sunglint is mathematically expected.  Sometimes the problem is exacerbated by temperature inversions that make detection of these clouds even more difficult than usual via long wave infrared techniques that must be invoked in sun-glint conditions. Cloud detection in areas of expected sun-glint is difficult at best and the very small difference in temperature between clouds and the sea surface in these cases only adds to the uncertainty in the cloud mask algorithm.  It should be noted that this problem only affects very small regions in a few L2 granules each day (and only when conditions are right).

 

 

PGE Versions & Production Dates of Problematic Data:

Data Dates Affected: All from launch (but only in very small regions in a few L2 granules each day)

Affected Platform: Terra & Aqua

Collections Affected: Collection 6.1

 

Return to Menu at Top of Page

 


Issue #7: PWA Failure in Oct 2020 affecting high latitude data for CM and CTP

Description:  Cloud Mask (CM) and Cloud Top Properties (CTP) Issues After Printed Wire Assembly (PWA) Failure on 5 October 2020. On October 5, 2020 at 15:21:12, Terra MODIS experienced a Printed Wire Assembly (PWA) failure that has caused an ongoing reduction and irregularity in the overall Terra MODIS daytime coverage over high latitudes in both hemispheres. In many polar and some subpolar sunlit scenes, L1B visible and near infrared (VNIR) reflectances are missing.

Degraded results are seen in both the cloud mask and cloud top properties outputs. During daytime, the cloud mask (MOD35_L2 in PGE03) uses the Normalized Difference Snow Index (NDSI) to determine if snow or ice is present on the Earth’s surface. The lack of reflectance data means that this quantity cannot be computed and the cloud mask assumes no snow or ice is present, leading to many false cloudy pixels. A fix for this problem has been submitted to MODAPS where mapped National Snow and Ice Data Center (NSIDC) NISE (Near-Real-Time SSM/I-SSMIS EASE-Grid Daily Global Ice Concentration and Snow Extent) data is substituted for the missing NDSI. NISE data inputs are used in the cloud mask for nighttime applications. In daytime cases where there is no actual snow or ice cover but L1B is missing VNIR reflectances, the cloud mask is degraded and is essentially an infrared-only mask (1.38 µm data is always available).

An issue is also seen in the MOD_PR06CT (5-km Cloud Top Properties) algorithm contained in PGE06. Daytime L1B scans that are missing VNIR reflectances are marked as “night”. Currently, day/night decisions are based on both solar zenith angle (SZA) and scan mode from L1B data. If the SZA > 85° or scan mode is “night”, then the data is defined as nighttime and the outputs sorted accordingly. This leads to incorrect day vs. night designations in both L2 and L3 outputs, including 5-km cloud fractions (aggregated from 1-km cloud mask data) and 5-km cloud top pressures. The 1-km cloud top properties are not impacted. A fix has been submitted to MODAPS where the day/night decision is made according to SZA alone.

 

Figure.  The figure below shows resulting problems with L3 cloud fractions. The curves represent time series of daily mean 5-km cloud fractions taken from latitudes 70N-90N from 01 January to 21 May 2021. When daytime Terra L1B contains VNIR data, the agreement is good between Aqua and Terra cloud fractions (blue and green curves). Cloud fractions from sunlit regions but without Terra MODIS VNIR data are currently labeled as “night” and are included in the red trace. As the season advances, “nighttime” cloud fractions from Terra (red) and Aqua (black) diverge. Recall that the daytime Terra MODIS cloud mask is recording inappropriately large cloud amounts due to missing NDSI calculations and these large values are being aggregated into the nighttime L3 cloud fractions.

Figure. The figure below shows two difference images from Terra MODIS daily (MOD08_D3) for day 2021 137 (17 May 2021). The test data used for these difference images were created using a new version of PGE03 Cloud mask Algorithm and the baseline is the old version. As shown, the differences are primarily seen only over the higher latitudes.

 

PGE Versions & Production Dates of Problematic Data:

Data Dates Affected: Terra CM and CTP Data after 5 Oct 2020, but only in high latitude regions.

Affected Platform: Terra

Collections Affected: Collection 6.1

 

Return to Menu at Top of Page

 




Quality Statement #1: Missing Terra Data Granules between 5 Aug 2017 and 6 Aug 2017

Description: Following the Terra Deep Space Calibration on August 5th, 2017 (2017 217) there were some faulty C6 Terra MODIS Atmosphere data as the focal plane temperatures did not return to nominal values until the day after - August 6th, 2017 (2017 218) 1420 UTC. So the granules for the time period between 2017 217 23:00 UTC and 2017 218 14:15 UTC were removed before the downstream Atmosphere processing. Data users will notice missing MODIS Atmosphere Data during this period for all Level-2 (L2) and Level-3 (L3) Daily atmosphere products.

Data Dates Affected: Note there will be Missing Terra Data between 2017 217 (5 Aug 2017) at 23:00 UTC and 2017 218 (6 Aug 2017) at 14:15 UTC for all MODIS Atmosphere L2 and L3 Daily products.

Here is a sample L3 animation of Scattering Angle for 6 Aug 2017, which shows the missing data issue fairly clearly.  The animation loops between the C6.0 data and the C6.1 data. The C6.1 data is improved showing less missing data, however a gap still exists in C6.1


  C6.0 vs C6.1 comparison of missing data gap (slight improvement shown in C6.1)

Affected Platform: Terra Only

Collections Affected: Collection 6.0 and Collection 6.1 (some improvement noted in C6.1)

Return to Menu at Top of Page

 


Quality Statement #2: GDAS Change Impacts

Description: The GDAS Model changed in January 2015, so for both Aqua and Terra MODIS products users should note:

  1. For most daytime granules, there will be no changes in the cloud mask product (MOD35) due to the new GDAS model output. There will be more changes in nighttime data due to surface temperature and total precipitable water differences but we anticipate no biases due to the new GDAS inputs.
  2. We also anticipate no significant changes to the 5-km and 1-km cloud top pressure product (MOD_PR06CT). Most changes found in the Level 2 cloud top pressures are one atmospheric pressure level that seem to be randomly distributed.

Affected Platform: Aqua and Terra

Collections Affected: Collection 6.0 and Collection 6.1

Return to Menu at Top of Page