LAADS DAAC in the Cloud
Announcements
Beginning on July 31, 2024 LAADS DAAC data will be accessed from the Cloud when using Earthdata Search and other services using NASA's Common Metadata Repository (CMR). Data will continue to be available from LAADS DAAC’s on-premise data servers on LAADS DAAC's website, including LAADS DAAC's Data Archive, Find Data and View Data.
Introduction
As part of NASA’s open-science policy and related goals, the LAADS DAAC is migrating its Level-1 and Atmosphere product collections to the Earthdata Cloud. All these data collections and services remain freely available to the global user community.
This page, LAADS DAAC in the Cloud, is intended to help users discover and access all LAADS DAAC data collections in the Earthdata Cloud. It addresses the following four topics:
- What are the benefits of working with Earth science data in the Cloud?
- What is the migration status of the LAADS DAAC’s products in the Cloud?
- How would a user access and download any desired data product from the Cloud?
- What are the answers to frequently asked questions?
As the Cloud evolves with the addition of new datasets, services, and related functionalities, this page will be updated with new information.
Benefits
- Easy access to data: Data users will be able to access data directly in the cloud, making the need to download volumes of data unnecessary. However, users will still be able to download data if they choose.
- Rapid deployment: Users can bring their algorithms and processing software to the cloud and work directly with the data in the cloud, simplifying procurement and hardware support while expediting science discovery.
- Scalability: The size and use of the archive can expand easily and rapidly as needed.
- Flexibility: Mission needs can dictate options for selecting operating systems, programming languages, databases, and other criteria to enable the best use of mission data.
- Reduced redundancy: The use of a common infrastructure with cloud native services will reduce redundant tools and services, enable sharing, and enforce the use of community standards as well as uniform policies and processes.
- Cost effectiveness: NASA pays only for the storage and services used. Along with scalability benefits, this allows the amount of storage or services to be continually adjusted to ensure that data and services are effectively provided at the lowest possible cost to NASA. Under NASA’s full and open data policy, all NASA data will continue to be free to access and download. However, users who wish to store data in their own Amazon Web Services (AWS) cloud instance or cloud storage, are responsible for covering those cost. For more information on this topic, see “Understanding and Managing Costs in the AWS Cloud.”
Earthdata Cloud also benefits the scientific community that uses NASA Earth science data for research. By making NASA data, algorithmic code, and metadata available in the cloud, the scientific processes of NASA researchers will become more transparent and their results more reproducible, which in turn lends clarity and validity to the scientific process. In addition, the use of standardized software and code makes it easier for new users to learn to interact with the data and become more involved in the scientific process.
Migration Status
The LAADS DAAC is currently populating datasets in the Earthdata Cloud.
The following diagram and tables depict the proposed timelines for migrating data products, beginning with a historical selection that includes a set of the most frequently used products. The subsequent migration includes in-production datasets grouped by their popularity. Even as the LAADS DAAC Cloud grows and becomes operational, users may continue to search, access, and download all data products via the on-premises channel as well.
LAADS DAAC Phased Dataset Migration
LAADS DAAC Phased Dataset Migration Updated September 25, 2023 |
||||
Total |
5,021 |
|||
Phase |
Collection |
Year |
Total Volume (TB) |
Migration Status |
1.1 |
MODIS C61 L1, Geolocation and Cloud Mask (Cohort-2) |
2022 |
1,260 |
Complete |
1.2 |
LAADS DAAC product “Top 50 + 25” List |
2022 |
355 |
Complete |
1.3 |
MODIS C61 L2 and L3 Atmosphere Products |
2023 |
35 |
Complete |
2.1 |
C2 SNPP VIIRS L1B and Geolocation |
2023 |
562 |
Complete |
2.2 |
C2.1 J1 VIIRS L1B and Geolocation |
2023 |
306 |
Complete |
2.3 |
SNPP/J1 VIIRS C1/C1.1/C2 Atmosphere Products |
2023 |
572 |
Complete |
3.1 |
MODIS C61 L2 Land Surface Reflectance |
2023 |
781 |
Complete |
4.1 |
C6 Long Term Data Records from NOAA POES and ESA MetOps |
2023 |
17 |
Complete |
4.2 |
MERIS, Sentinel 3A, and Sentinel 3B L1 Products |
2024 |
1,133 |
Complete |
4.3 |
Airborne Data |
2024 |
0.37 |
- |
LAADS DAAC Phase 1
LAADS DAAC Phase 1 Datasets |
|||||
Phase |
Collection |
Archive Set |
Available |
Data Products |
Mission Period |
1.1 |
MODIS Level 1B Calibrated Radiances |
61 |
✔ |
Feb 2000 - Present |
|
April 2002 - present |
|||||
MODIS Geolocation |
61 |
✔ |
Feb 2000 - present |
||
April 2002 - present |
|||||
MODIS Cloud Mask and Spectral Test Results |
61 |
✔ |
Feb 2000 - present |
||
1.2 |
MODIS Cloud Properties |
61 |
✔ |
Feb 2000 - present |
|
April 2002 - present |
|||||
MODIS Temperature and Water Vapor Profiles |
61 |
✔ |
Feb 2000 - present |
||
April 2002 - present |
|||||
VIIRS/NPP Daily Gridded Day/Night Band |
5000 |
✔ |
Jan 2012 - present |
||
VIIRS/NPP Gap-Filled Lunar BRDF-Adjusted Nighttime Lights Daily L3 Global |
5000 |
✔ |
Jan 2012 - present |
||
1.3 |
MODIS C61 L2 and L3 Atmosphere Products |
61 |
✔ |
Feb 2000 - present |
|
April 2002 - present |
LAADS DAAC Phase 2
LAADS DAAC Phase 2 Datasets |
|||||
Phase |
Collection |
Archive Set |
Available |
Data Products |
Mission Period |
2.1 |
C2 SNPP VIIRS L1B and Geolocation |
5200 |
✔ |
Jan 2012 - Present |
|
2.2 |
C2.1 J1 VIIRS L1B and Geolocation |
5201 |
✔ |
Jan 2018 - present |
|
2.3 |
SNPP/J1 VIIRS C1.1/C2 Atmosphere Products |
5110 |
✔ |
AERDB_L2_VIIRS_SNPP |
Mar 2012 - present |
Feb 2018 - present |
|||||
July 2002 - present |
LAADS DAAC Phase 3
LAADS DAAC Phase 3 Datasets |
|||||
Phase |
Collection |
Archive Set |
Available |
Data Products |
Mission Period |
3.0 |
MODIS C61 L2 Land Surface Reflectance |
61 |
✔ |
Feb 2000 - Present | |
July 2002 - present |
LAADS DAAC Phase 4
LAADS DAAC Phase 4 Datasets |
|||||
Phase |
Collection |
Archive Set |
Available |
Data Products |
Mission Period |
4.1 |
C6 Long Term Data Records from NOAA POES and ESA MetOps |
465 |
✔ |
Jun 1981 - Feb 1985 |
|
Jan 1985 - Nov 1988 |
|||||
Nov 1988 - Dec 1994 |
|||||
Jan 1995 - Oct 2000 |
|||||
Nov 2000 - Dec 2007 |
|||||
July 2005 - Dec 2009 |
|||||
May 2009 - May 2022 |
|||||
Jan 2013 - Apr 2023 |
|||||
4.2 |
MERIS Envisat 4th reprocessing |
489 |
✔ |
May 2002 - Apr 2012 |
|
Jan 2003 - Apr 2012 |
|||||
Apr 2002 - Apr 2012 |
|||||
OLCI and SLSTR ESA-Copernicus-Sentinel3 A&B Level 1 |
450 |
✔ |
Apr 2016 - Present |
||
May 2018 - Present |
|||||
Dec 2019 - Present |
How-To
How do I access LAADS DAAC data files in the cloud?
Resources
Openscapes
NASA Earthdata-Openscapes answers a NASA Earthdata call to support scientific researchers using data from NASA Distributed Active Archive Centers (DAACs) as they migrate workflows to the cloud. Openscapes has multiple resources such as the Earthdata Cloud Cookbook with material to learn how to work with NASA Earthdata in the cloud and documentation for how to contribute to these resources, Cheatsheets, Guides and Slides, tutorials and workshops. |
||
Transform to Open
Science
NASA's Transform to Open Science (TOPS) initiative is designed to transform agencies, organizations, and communities to an inclusive culture of open science. TOPS's first priority is to develop the infrastructure to train scientists and researchers as part of a 5-year program. The open science curriculum introduces those beginning their open science journey to important definitions, tools, and resources; and provide participants at all levels recommendations on best practices. |
||
Understanding
and Managing Costs in the AWS Cloud
NASA's Earth Observing System Data and Information System (EOSDIS) provides an introductory tutorial for understanding and estimating cost in the cloud. This resource 1) provides an overview of the cost model for NASA EOSDIS - cloud data access and use, 2) highlights several relevant user scenarios using the AWS Pricing Calculator to provide guiding examples and consideration for Earth Observing System (EOS) data users in the cloud, and 3) summarizes AWS resources for budgeting cost in the cloud. |
Frequently Asked Questions
What are the advantages of using data from the Earthdata Cloud? 1) Improved performance: Quick access and subsetting of a large volume of data compared to data and services on-premises. 2) Multiple data access methods: In addition to existing LAADS DAAC data access methods, a new way to access data, S3 access, is available to users in the AWS us-west-2 region. 3) Multidisciplinary data access: Find multidisciplinary NASA data from multiple DAACs in one place, the Earthdata Cloud. 4) Analysis next to data: Users in the AWS us-west-2 region can import their algorithms and processing software and work directly with NASA data in the cloud, expediting science discovery. Note that users are responsible for procuring their own cloud computing resources. 5) Expanded services: Expanded services, such as CMR-STAC, Harmony, and Zarr creation will be available to users in the future.* |
What is S3 access? Amazon's Simple Storage Service (S3) is an object storage service that offers scalability, data availability, security, and performance. S3 access is a way of accessing AWS cloud data from S3 storage using cloud native tools. S3 access (also referred to as "direct S3 access") is only available to users in the same cloud region as LAADS DAAC data (AWS us-west-2). Access to LAADS DAAC cloud data on S3 is described in the How-To section. |
I am a user with a cloud account, how can I find out which LAADS DAAC datasets are available for direct S3 access in the Earthdata Cloud? The LAADS DAAC Migration Status Section describes the LAADS DAAC phased dataset projected release dates. You may also explore LAADS DAAC data through the Earthdata Search portal by filtering. To filter for LAADS DAAC Data in the Cloud, in the left panel under Organizations, select Level-1 and Atmosphere Archive & Distribution System (LAADS) and under Features, select Available from AWS Cloud. Step-by-step instructions for obtaining the S3 bucket URL for a LAADS DAAC collection from Earthdata Search are found in the How To section. |
I am a user with a cloud account, how can I access LAADS DAAC data in the Earthdata Cloud? LAADS data stored in the Earthdata Cloud can be accessed via S3 buckets. Users in the AWS us-west-2 region may use "S3 buckets" (bins of files, called objects) to access Earthdata Cloud data from S3 storage using cloud native tools. S3 access allows users to use Python and other language libraries to explicitly access files (or portions of files) in the LAADS DAAC cloud archive, list the contents of the bucket, and also interrogate and interact with cloud-native formats. You will need a URL path to the specific LAADS DAAC collection you would like to access using S3. The structure for an S3 URL looks like: s3://prod-lads/MOD021KM/MOD021KM.A2000055.0000.061.2017171194852.hdf An example of an S3 URL for a LAADS DAAC collection is: s3://prod-lads/MOD021KM/ To learn more, please refer to What is S3 access? |
Do I have to pay to access LAADS DAAC data stored in the Earthdata Cloud? As part of NASA's commitment to open science, data access will continue to be free to users. |
Do I have to pay for cloud computing resources? Yes, the user maintains responsibility for their own cloud computing resources. Cost is variable depending on which cloud compute types you choose. See guidance on Understanding and Managing Costs in the AWS Cloud. |
In which AWS region does LAADS DAAC data reside? LAADS DAAC data are stored in the AWS region US West (Oregon), or us-west-2. |
Can I access data in AWS regions outside of my geographic location? Yes. The AWS region refers to the location of servers, which are accessible via the internet regardless of a user's physical location. |
Can I perform direct S3 access outside of my account's default region? Yes, but first you will need to enable the region of interest in your AWS account. For example, to acquire LAADS DAAC data using direct S3 access, you will need to enable the us-west-2 region in your account and then use an AWS service in that region (e.g. EC2 instance, AWS Command Line Interface, Python, Javascript, Java etc. program software development kits (SDK) or libraries) to access the data. |
How do I obtain credentials to access LAADS DAAC data in S3 storage? You will need to pass credentials and configuration to AWS in order to interact with S3 objects. The How-To S3 Direct Access Section describes this process. |
I want to perform direct S3 access in a cloud environment, do I still need my Earthdata Login? Yes, you will still need to store your Earthdata Login credentials in your cloud environment. In order to access LAADS DAAC data, an Earthdata Login account is required. If you do not have an Earthdata Login, follow the steps listed here to obtain one. A short tutorial on how to store your Earthdata Login credentials is found in How to S3 Direct Access. |
With data hosting transitioning to the cloud, will I need to change the way I access data or services? No, existing data access methods and services continue uninterrupted during LAADS DAAC's cloud migration. |
With data hosting transitioning to the cloud, can I still download data from LAADS DAAC? Yes, existing on-premises download methods and services are still available throughout LAADS DAAC's cloud migration. |
Last updated: June 8, 2023