LAADS DAAC in the Cloud

Announcements

Beginning on July 31, 2024 LAADS DAAC data will be accessed from the Cloud when using Earthdata Search and other services using NASA's Common Metadata Repository (CMR). Data will continue to be available from LAADS DAAC’s on-premise data servers on LAADS DAAC's website, including LAADS DAAC's Data Archive, Find Data and View Data.

Introduction

As part of NASA’s open-science policy and related goals, the LAADS DAAC is migrating its Level-1 and Atmosphere product collections to the Earthdata Cloud. All these data collections and services remain freely available to the global user community.

This page, LAADS DAAC in the Cloud, is intended to help users discover and access all LAADS DAAC data collections in the Earthdata Cloud. It addresses the following four topics:

  1. What are the benefits of working with Earth science data in the Cloud?
  2. What is the migration status of the LAADS DAAC’s products in the Cloud?
  3. How would a user access and download any desired data product from the Cloud?
  4. What are the answers to frequently asked questions?

As the Cloud evolves with the addition of new datasets, services, and related functionalities, this page will be updated with new information.

Benefits

  • Easy access to data: Data users will be able to access data directly in the cloud, making the need to download volumes of data unnecessary. However, users will still be able to download data if they choose.
  • Rapid deployment: Users can bring their algorithms and processing software to the cloud and work directly with the data in the cloud, simplifying procurement and hardware support while expediting science discovery.
  • Scalability: The size and use of the archive can expand easily and rapidly as needed.
  • Flexibility: Mission needs can dictate options for selecting operating systems, programming languages, databases, and other criteria to enable the best use of mission data.
  • Reduced redundancy: The use of a common infrastructure with cloud native services will reduce redundant tools and services, enable sharing, and enforce the use of community standards as well as uniform policies and processes.
  • Cost effectiveness: NASA pays only for the storage and services used. Along with scalability benefits, this allows the amount of storage or services to be continually adjusted to ensure that data and services are effectively provided at the lowest possible cost to NASA. Under NASA’s full and open data policy, all NASA data will continue to be free to access and download. However, users who wish to store data in their own Amazon Web Services (AWS) cloud instance or cloud storage, are responsible for covering those cost. For more information on this topic, see “Understanding and Managing Costs in the AWS Cloud.”

Earthdata Cloud also benefits the scientific community that uses NASA Earth science data for research. By making NASA data, algorithmic code, and metadata available in the cloud, the scientific processes of NASA researchers will become more transparent and their results more reproducible, which in turn lends clarity and validity to the scientific process. In addition, the use of standardized software and code makes it easier for new users to learn to interact with the data and become more involved in the scientific process.

Migration Status

The LAADS DAAC is currently populating datasets in the Earthdata Cloud.

The following diagram and tables depict the proposed timelines for migrating data products, beginning with a historical selection that includes a set of the most frequently used products. The subsequent migration includes in-production datasets grouped by their popularity. Even as the LAADS DAAC Cloud grows and becomes operational, users may continue to search, access, and download all data products via the on-premises channel as well.

Architecture

LAADS DAAC Phased Dataset Migration

LAADS DAAC Phased Dataset Migration

Updated September 25, 2023

Total

5,021

Phase

Collection

Year

Total Volume (TB)

Migration Status

1.1

MODIS C61 L1, Geolocation and Cloud Mask (Cohort-2)

2022

1,260

Complete

1.2

LAADS DAAC product “Top 50 + 25” List

2022

355

Complete

1.3

MODIS C61 L2 and L3 Atmosphere Products

2023

35

Complete

2.1

C2 SNPP VIIRS L1B and Geolocation

2023

562

Complete

2.2

C2.1 J1 VIIRS L1B and Geolocation

2023

306

Complete

2.3

SNPP/J1 VIIRS C1/C1.1/C2 Atmosphere Products

2023

572

Complete

3.1

MODIS C61 L2 Land Surface Reflectance

2023

781

Complete

4.1

C6 Long Term Data Records from NOAA POES and ESA MetOps

2023

17

Complete

4.2

MERIS, Sentinel 3A, and Sentinel 3B L1 Products

2024

1,133

Complete

4.3

Airborne Data

2024

0.37

-

LAADS DAAC Phase 1

LAADS DAAC Phase 1 Datasets

Phase

Collection

Archive Set

Available

Data Products

Mission Period

1.1

MODIS Level 1B Calibrated Radiances

61

MOD021KM
MOD02HKM
MOD02QKM

Feb 2000 - Present

MYD021KM
MYD02HKM
MYD02QKM

April 2002 - present

MODIS Geolocation

61

MOD03

Feb 2000 - present

MYD03

April 2002 - present

MODIS Cloud Mask and Spectral Test Results

61

MOD35_L2

Feb 2000 - present

1.2

MODIS Cloud Properties

61

MOD06_L2

Feb 2000 - present

MYD06_L2

April 2002 - present

MODIS Temperature and Water Vapor Profiles

61

MOD07_L2

Feb 2000 - present

MYD07_L2

April 2002 - present

VIIRS/NPP Daily Gridded Day/Night Band

5000

VNP46A1

Jan 2012 - present

VIIRS/NPP Gap-Filled Lunar BRDF-Adjusted Nighttime Lights Daily L3 Global

5000

VNP46A2

Jan 2012 - present

1.3

MODIS C61 L2 and L3 Atmosphere Products

61

MOD08_D3
MOD08_E3
MOD08_M3
MODATML2

Feb 2000 - present

MYD08_D3
MYD08_E3
MYD08_M3
MYDATML2

April 2002 - present

LAADS DAAC Phase 2

LAADS DAAC Phase 2 Datasets

Phase

Collection

Archive Set

Available

Data Products

Mission Period

2.1

C2 SNPP VIIRS L1B and Geolocation

5200

VNP02MOD
VNP02IMG
VNP02DNB
VNP03MOD
VNP03IMG

Jan 2012 - Present

2.2

C2.1 J1 VIIRS L1B and Geolocation

5201

VJ102MOD
VJ102IMG
VJ102DNB
VJ103MOD
VJ103IMG

Jan 2018 - present

2.3

SNPP/J1 VIIRS C1.1/C2 Atmosphere Products

5110

AERDB_L2_VIIRS_SNPP
AERDB_D3_VIIRS_SNPP
AERDB_M3_VIIRS_SNPP AERDT_L2_VIIRS_SNPP CLDCR_L2_VIIRS_SNPP CLDMSK_L2_VIIRS_SNPP CLDPROP_L2_VIIRS_SNPP FSNRAD_L2_VIIRS_CRIS_SNPP

Mar 2012 - present

CLDMSK_L2_VIIRS_NOAA20
FSNRAD_L2_VIIRS_CRIS_NOAA20

Feb 2018 - present

CLDMSK_L2_MODIS_Aqua
CLDPROP_L2_MODIS_Aqua

July 2002 - present

LAADS DAAC Phase 3

LAADS DAAC Phase 3 Datasets

Phase

Collection

Archive Set

Available

Data Products

Mission Period

3.0

MODIS C61 L2 Land Surface Reflectance

61

MOD09

Feb 2000 - Present

MYD09

July 2002 - present

LAADS DAAC Phase 4

LAADS DAAC Phase 4 Datasets

Phase

Collection

Archive Set

Available

Data Products

Mission Period

4.1

C6 Long Term Data Records from NOAA POES and ESA MetOps

465

N07_AVH02C1
N07_AVH09C1
N07_AVH13C1

Jun 1981 - Feb 1985

N09_AVH02C1
N09_AVH09C1
N09_AVH13C1

Jan 1985 - Nov 1988

N11_AVH02C1
N11_AVH09C1
N11_AVH13C1

Nov 1988 - Dec 1994

N14_AVH02C1
N14_AVH09C1
N14_AVH13C1

Jan 1995 - Oct 2000

N16_AVH02C1
N16_AVH09C1
N16_AVH13C1

Nov 2000 - Dec 2007

N18_AVH02C1
N18_AVH09C1
N18_AVH13C1

July 2005 - Dec 2009

N19_AVH02C1
N19_AVH09C1
N19_AVH13C1

May 2009 - May 2022

M1_AVH02C1
M1_AVH09C1
M1_AVH13C1

Jan 2013 - Apr 2023

4.2

MERIS Envisat 4th reprocessing

489

EN1_MDSI_MER_FRS_1P

May 2002 - Apr 2012

EN1_MDSI_MER_FRS_2P

Jan 2003 - Apr 2012

EN1_MDSI_MER_RR__1P
EN1_MDSI_MER_RR__2P

Apr 2002 - Apr 2012

OLCI and SLSTR ESA-Copernicus-Sentinel3 A&B Level 1

450

S3A_OL_1_EFR
S3A_OL_1_ERR
S3A_SL_1_RBT

Apr 2016 - Present

S3B_OL_1_EFR
S3B_OL_1_ERR
S3B_SL_1_RBT

May 2018 - Present

S3B_SY_2_SYN
S3A_SY_2_SYN

Dec 2019 - Present

How-To

How do I access LAADS DAAC data files in the cloud?

Resources

  Openscapes

NASA Earthdata-Openscapes answers a NASA Earthdata call to support scientific researchers using data from NASA Distributed Active Archive Centers (DAACs) as they migrate workflows to the cloud. Openscapes has multiple resources such as the Earthdata Cloud Cookbook with material to learn how to work with NASA Earthdata in the cloud and documentation for how to contribute to these resources, Cheatsheets, Guides and Slides, tutorials and workshops.

  Transform to Open Science

NASA's Transform to Open Science (TOPS) initiative is designed to transform agencies, organizations, and communities to an inclusive culture of open science. TOPS's first priority is to develop the infrastructure to train scientists and researchers as part of a 5-year program. The open science curriculum introduces those beginning their open science journey to important definitions, tools, and resources; and provide participants at all levels recommendations on best practices.

  Understanding and Managing Costs in the AWS Cloud

NASA's Earth Observing System Data and Information System (EOSDIS) provides an introductory tutorial for understanding and estimating cost in the cloud. This resource 1) provides an overview of the cost model for NASA EOSDIS - cloud data access and use, 2) highlights several relevant user scenarios using the AWS Pricing Calculator to provide guiding examples and consideration for Earth Observing System (EOS) data users in the cloud, and 3) summarizes AWS resources for budgeting cost in the cloud.

Frequently Asked Questions

What are the advantages of using data from the Earthdata Cloud?

1) Improved performance: Quick access and subsetting of a large volume of data compared to data and services on-premises.

2) Multiple data access methods: In addition to existing LAADS DAAC data access methods, a new way to access data, S3 access, is available to users in the AWS us-west-2 region

3) Multidisciplinary data access: Find multidisciplinary NASA data from multiple DAACs in one place, the Earthdata Cloud.

4) Analysis next to data: Users in the AWS us-west-2 region can import their algorithms and processing software and work directly with NASA data in the cloud, expediting science discovery. Note that users are responsible for procuring their own cloud computing resources.

5) Expanded services: Expanded services, such as CMR-STAC, Harmony, and Zarr creation will be available to users in the future.*

What is S3 access?

Amazon's Simple Storage Service (S3) is an object storage service that offers scalability, data availability, security, and performance. S3 access is a way of accessing AWS cloud data from S3 storage using cloud native tools.

S3 access (also referred to as "direct S3 access") is only available to users in the same cloud region as LAADS DAAC data (AWS us-west-2).

Access to LAADS DAAC cloud data on S3 is described in the How-To section.

I am a user with a cloud account, how can I find out which LAADS DAAC datasets are available for direct S3 access in the Earthdata Cloud?

The LAADS DAAC Migration Status Section describes the LAADS DAAC phased dataset projected release dates.

You may also explore LAADS DAAC data through the Earthdata Search portal by filtering. To filter for LAADS DAAC Data in the Cloud, in the left panel under Organizations, select Level-1 and Atmosphere Archive & Distribution System (LAADS) and under Features, select Available from AWS Cloud.

Step-by-step instructions for obtaining the S3 bucket URL for a LAADS DAAC collection from Earthdata Search are found in the How To section.

I am a user with a cloud account, how can I access LAADS DAAC data in the Earthdata Cloud?

LAADS data stored in the Earthdata Cloud can be accessed via S3 buckets.

Users in the AWS us-west-2 region may use "S3 buckets" (bins of files, called objects) to access Earthdata Cloud data from S3 storage using cloud native tools. S3 access allows users to use Python and other language libraries to explicitly access files (or portions of files) in the LAADS DAAC cloud archive, list the contents of the bucket, and also interrogate and interact with cloud-native formats.

You will need a URL path to the specific LAADS DAAC collection you would like to access using S3. The structure for an S3 URL looks like:

s3://prod-lads/MOD021KM/MOD021KM.A2000055.0000.061.2017171194852.hdf

An example of an S3 URL for a LAADS DAAC collection is: s3://prod-lads/MOD021KM/

To learn more, please refer to What is S3 access?

Do I have to pay to access LAADS DAAC data stored in the Earthdata Cloud?

As part of NASA's commitment to open science, data access will continue to be free to users.

Do I have to pay for cloud computing resources?

Yes, the user maintains responsibility for their own cloud computing resources. Cost is variable depending on which cloud compute types you choose. See guidance on Understanding and Managing Costs in the AWS Cloud.

In which AWS region does LAADS DAAC data reside?

LAADS DAAC data are stored in the AWS region US West (Oregon), or us-west-2.

Can I access data in AWS regions outside of my geographic location? 

Yes. The AWS region refers to the location of servers, which are accessible via the internet regardless of a user's physical location.

See the How-To s3 direct access section.

Can I perform direct S3 access outside of my account's default region?

Yes, but first you will need to enable the region of interest in your AWS account.

For example, to acquire LAADS DAAC data using direct S3 access, you will need to enable the us-west-2 region in your account and then use an AWS service in that region (e.g. EC2 instance, AWS Command Line Interface, Python, Javascript, Java etc. program software development kits (SDK) or libraries) to access the data. 

How do I obtain credentials to access LAADS DAAC data in S3 storage?

You will need to pass credentials and configuration to AWS in order to interact with S3 objects. 

The How-To S3 Direct Access Section describes this process.

I want to perform direct S3 access in a cloud environment, do I still need my Earthdata Login?

Yes, you will still need to store your Earthdata Login credentials in your cloud environment. In order to access LAADS DAAC data, an Earthdata Login account is required. If you do not have an Earthdata Login, follow the steps listed here to obtain one.

A short tutorial on how to store your Earthdata Login credentials is found in How to S3 Direct Access.

With data hosting transitioning to the cloud, will I need to change the way I access data or services?

No, existing data access methods and services continue uninterrupted during LAADS DAAC's cloud migration.

With data hosting transitioning to the cloud, can I still download data from LAADS DAAC?

Yes, existing on-premises download methods and services are still available throughout LAADS DAAC's cloud migration.

Last updated: June 8, 2023