2. NEON Utilities Tools#
In this section, we’ll explore how to efficiently work with NEON data directly from the NEON API using the powerful tools provided by neonUtilities. This approach allows us to seamlessly download and organize data for further analysis.
2.1 Choosing Your Download Approach:#
The tutorial offers two approaches for working with NEON data:
Downloading data directly from the NEON API (current focus): This method retrieves data using Python and neonUtilities. You’ll provide specific details about the data you want.
Working with already downloaded data: If you have NEON data locally, there are alternative methods for processing it here.
2.2 Downloading Data with zipsByProduct():#
To begin, we utilize the zipsByProduct() function to download specific data products from NEON. This function requires essential details such as the Data Product ID (DPID), date range, site IDs, download package type, file save path, and whether to check download size.
Here’s a breakdown of its arguments:
dpID: The data product identifier (DPID) found on the NEON Explore Data page.startdate: The start date for data retrieval in YYYY-MM format (e.g., “2020-01”).enddate: The end date for data retrieval in YYYY-MM format (e.g., “2021-12”).site: A character vector containing 4-letter site IDs (e.g., ‘TALL’). You can use “all” for all sites.savepath: The local directory where downloaded data will be saved. Adjust the path as needed.package: Choose"basic"(default) or"expanded"depending on data needs.check_size: Set toTRUE(default) to verify download size before proceeding, orFALSEto skip this check.
from rpy2.rinterface_lib.callbacks import logger as rpy2_logger
import logging
import os
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from rpy2.robjects import r as base
from rpy2.robjects.packages import importr
# Set logging level to suppress R messages
rpy2_logger.setLevel(logging.ERROR)
# Load R's base, utils, and stats packages
base = importr('base')
utils = importr('utils')
stats = importr('stats')
# Importing R packages via rpy2
neonUtilities = importr('neonUtilities')
os.environ['R_HOME'] = '/Library/Frameworks/R.framework/Resources'
NOTE: Note: Ensure you have the correct DPID and site IDs for the data you intend to download. Adjust the savepath parameter to your desired location.
2.3 Downloading Specific Soil Data:#
The code examples demonstrate downloading three types of soil data:
Soil Moisture Content
neonUtilities.zipsByProduct(dpID='DP1.00094.001',
startdate="2017-01",
enddate="2023-12",
site=base.c('TALL'),
savepath='path', # Replace with your path
package='basic',
timeIndex = "1",
check_size='TRUE')
Finding available files
|======================================================================| 100%
Provisional data were excluded from available files list. To download provisional data, use input parameter include.provisional=TRUE.
neonUtilities.zipsByProduct(dpID='DP1.00094.001',
startdate="2017-01",
enddate="2023-12",
site=base.c('TALL'),
savepath='path', # Replace with your path
package='basic',
timeIndex = "30",
check_size='TRUE')
Finding available files
|======================================================================| 100%
Provisional data were excluded from available files list. To download provisional data, use input parameter include.provisional=TRUE.
Continuing will download files totaling approximately 494.151871 MB. Do you want to proceed y/n: y
Downloading 2577 files
|======================================================================| 100%
2577 files successfully downloaded to path/filesToStack00094
<rpy2.rinterface_lib.sexp.NULLType object at 0x14bcc10d0> [0]
Soil Temperature
neonUtilities.zipsByProduct(dpID='DP1.00041.001',
startdate="2017-01",
enddate="2023-12",
site=base.c('TALL'),
savepath='path', # Replace with your path
package='basic',
timeIndex = "30",
check_size='TRUE')
Finding available files
|======================================================================| 100%
Provisional data were excluded from available files list. To download provisional data, use input parameter include.provisional=TRUE.
Continuing will download files totaling approximately 450.134447 MB. Do you want to proceed y/n: y
Downloading 3207 files
|======================================================================| 100%
3207 files successfully downloaded to path/filesToStack00041
<rpy2.rinterface_lib.sexp.NULLType object at 0x14bcc10d0> [0]
Soil Carbon Dioxide (CO2) Concentration
neonUtilities.zipsByProduct(dpID='DP1.00095.001',
startdate="2017-01",
enddate="2023-12",
site=base.c('TALL'),
savepath='path', # Replace with your path
package='basic',
timeIndex = "30",
check_size='TRUE')
Finding available files
|======================================================================| 100%
Provisional data were excluded from available files list. To download provisional data, use input parameter include.provisional=TRUE.
Continuing will download files totaling approximately 145.83331 MB. Do you want to proceed y/n: y
Downloading 1079 files
|======================================================================| 100%
1079 files successfully downloaded to path/filesToStack00095
<rpy2.rinterface_lib.sexp.NULLType object at 0x14bcc10d0> [0]
2.4 Organizing Downloaded Data with stackByTable():#
Once data is downloaded, use stackByTable() to organize it into a more usable format within your specified save path. This function expects the path to the downloaded data directory as input (e.g., the output directory from zipsByProduct()):
# Stack Soil Temperature Data
neonUtilities.stackByTable(filepath='path/filesToStack00041')
# Stack Soil CO2 Data
neonUtilities.stackByTable(filepath='path/filesToStack00095')
# Stack Soil Water Content Data
neonUtilities.stackByTable(filepath='path/filesToStack00094')
| | 0 % ~calculating |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed=00s
Stacking operation across a single core.
Stacking table ST_30_minute
| | 0 % ~calculating |+ | 1 % ~10s |++ | 2 % ~10s |++ | 3 % ~13s |+++ | 4 % ~12s |+++ | 5 % ~11s |++++ | 6 % ~11s |++++ | 7 % ~11s |+++++ | 8 % ~11s |+++++ | 9 % ~12s |++++++ | 10% ~12s |++++++ | 11% ~11s |+++++++ | 12% ~12s |+++++++ | 13% ~11s |++++++++ | 14% ~11s |++++++++ | 15% ~11s |+++++++++ | 16% ~11s |+++++++++ | 17% ~11s |++++++++++ | 18% ~10s |++++++++++ | 19% ~11s |+++++++++++ | 20% ~10s |+++++++++++ | 21% ~10s |++++++++++++ | 22% ~10s |++++++++++++ | 23% ~09s |+++++++++++++ | 24% ~09s |+++++++++++++ | 26% ~09s |++++++++++++++ | 27% ~09s |++++++++++++++ | 28% ~09s |+++++++++++++++ | 29% ~08s |+++++++++++++++ | 30% ~08s |++++++++++++++++ | 31% ~08s |++++++++++++++++ | 32% ~08s |+++++++++++++++++ | 33% ~08s |+++++++++++++++++ | 34% ~08s |++++++++++++++++++ | 35% ~08s |++++++++++++++++++ | 36% ~08s |+++++++++++++++++++ | 37% ~07s |+++++++++++++++++++ | 38% ~07s |++++++++++++++++++++ | 39% ~07s |++++++++++++++++++++ | 40% ~07s |+++++++++++++++++++++ | 41% ~07s |+++++++++++++++++++++ | 42% ~07s |++++++++++++++++++++++ | 43% ~06s |++++++++++++++++++++++ | 44% ~06s |+++++++++++++++++++++++ | 45% ~06s |+++++++++++++++++++++++ | 46% ~06s |++++++++++++++++++++++++ | 47% ~06s |++++++++++++++++++++++++ | 48% ~06s |+++++++++++++++++++++++++ | 49% ~06s |+++++++++++++++++++++++++ | 50% ~06s |++++++++++++++++++++++++++ | 51% ~05s |+++++++++++++++++++++++++++ | 52% ~05s |+++++++++++++++++++++++++++ | 53% ~05s |++++++++++++++++++++++++++++ | 54% ~05s |++++++++++++++++++++++++++++ | 55% ~05s |+++++++++++++++++++++++++++++ | 56% ~05s |+++++++++++++++++++++++++++++ | 57% ~05s |++++++++++++++++++++++++++++++ | 58% ~05s |++++++++++++++++++++++++++++++ | 59% ~05s |+++++++++++++++++++++++++++++++ | 60% ~04s |+++++++++++++++++++++++++++++++ | 61% ~04s |++++++++++++++++++++++++++++++++ | 62% ~04s |++++++++++++++++++++++++++++++++ | 63% ~04s |+++++++++++++++++++++++++++++++++ | 64% ~04s |+++++++++++++++++++++++++++++++++ | 65% ~04s |++++++++++++++++++++++++++++++++++ | 66% ~04s |++++++++++++++++++++++++++++++++++ | 67% ~04s |+++++++++++++++++++++++++++++++++++ | 68% ~03s |+++++++++++++++++++++++++++++++++++ | 69% ~03s |++++++++++++++++++++++++++++++++++++ | 70% ~03s |++++++++++++++++++++++++++++++++++++ | 71% ~03s |+++++++++++++++++++++++++++++++++++++ | 72% ~03s |+++++++++++++++++++++++++++++++++++++ | 73% ~03s |++++++++++++++++++++++++++++++++++++++ | 74% ~03s |++++++++++++++++++++++++++++++++++++++ | 76% ~03s |+++++++++++++++++++++++++++++++++++++++ | 77% ~03s |+++++++++++++++++++++++++++++++++++++++ | 78% ~02s |++++++++++++++++++++++++++++++++++++++++ | 79% ~02s |++++++++++++++++++++++++++++++++++++++++ | 80% ~02s |+++++++++++++++++++++++++++++++++++++++++ | 81% ~02s |+++++++++++++++++++++++++++++++++++++++++ | 82% ~02s |++++++++++++++++++++++++++++++++++++++++++ | 83% ~02s |++++++++++++++++++++++++++++++++++++++++++ | 84% ~02s |+++++++++++++++++++++++++++++++++++++++++++ | 85% ~02s |+++++++++++++++++++++++++++++++++++++++++++ | 86% ~02s |++++++++++++++++++++++++++++++++++++++++++++ | 87% ~01s |++++++++++++++++++++++++++++++++++++++++++++ | 88% ~01s |+++++++++++++++++++++++++++++++++++++++++++++ | 89% ~01s |+++++++++++++++++++++++++++++++++++++++++++++ | 90% ~01s |++++++++++++++++++++++++++++++++++++++++++++++ | 91% ~01s |++++++++++++++++++++++++++++++++++++++++++++++ | 92% ~01s |+++++++++++++++++++++++++++++++++++++++++++++++ | 93% ~01s |+++++++++++++++++++++++++++++++++++++++++++++++ | 94% ~01s |++++++++++++++++++++++++++++++++++++++++++++++++ | 95% ~01s |++++++++++++++++++++++++++++++++++++++++++++++++ | 96% ~00s |+++++++++++++++++++++++++++++++++++++++++++++++++ | 97% ~00s |+++++++++++++++++++++++++++++++++++++++++++++++++ | 98% ~00s |++++++++++++++++++++++++++++++++++++++++++++++++++| 99% ~00s |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed=11s
Merged the most recent publication of sensor position files for each site and saved to /stackedFiles
Copied the most recent publication of variable definition file to /stackedFiles
Finished: Stacked 1 data tables and 3 metadata tables!
Stacking took 16.64224 secs
All unzipped monthly data folders have been removed.
| | 0 % ~calculating |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed=00s
Stacking operation across a single core.
Stacking table SCO2C_30_minute
| | 0 % ~calculating |+ | 1 % ~03s |++ | 2 % ~02s |++ | 3 % ~02s |+++ | 4 % ~03s |+++ | 5 % ~03s |++++ | 6 % ~03s |++++ | 7 % ~03s |+++++ | 8 % ~02s |+++++ | 9 % ~02s |++++++ | 10% ~02s |++++++ | 11% ~02s |+++++++ | 12% ~04s |+++++++ | 13% ~03s |++++++++ | 14% ~03s |++++++++ | 15% ~03s |+++++++++ | 16% ~03s |+++++++++ | 17% ~03s |++++++++++ | 18% ~03s |++++++++++ | 19% ~03s |+++++++++++ | 20% ~03s |+++++++++++ | 21% ~03s |++++++++++++ | 22% ~03s |++++++++++++ | 23% ~03s |+++++++++++++ | 24% ~03s |+++++++++++++ | 26% ~02s |++++++++++++++ | 27% ~02s |++++++++++++++ | 28% ~02s |+++++++++++++++ | 29% ~02s |+++++++++++++++ | 30% ~02s |++++++++++++++++ | 31% ~02s |++++++++++++++++ | 32% ~02s |+++++++++++++++++ | 33% ~02s |+++++++++++++++++ | 34% ~02s |++++++++++++++++++ | 35% ~02s |++++++++++++++++++ | 36% ~02s |+++++++++++++++++++ | 37% ~02s |+++++++++++++++++++ | 38% ~02s |++++++++++++++++++++ | 39% ~02s |++++++++++++++++++++ | 40% ~02s |+++++++++++++++++++++ | 41% ~02s |+++++++++++++++++++++ | 42% ~02s |++++++++++++++++++++++ | 43% ~02s |++++++++++++++++++++++ | 44% ~02s |+++++++++++++++++++++++ | 45% ~02s |+++++++++++++++++++++++ | 46% ~02s |++++++++++++++++++++++++ | 47% ~02s |++++++++++++++++++++++++ | 48% ~02s |+++++++++++++++++++++++++ | 49% ~01s |+++++++++++++++++++++++++ | 50% ~01s |++++++++++++++++++++++++++ | 51% ~01s |+++++++++++++++++++++++++++ | 52% ~01s |+++++++++++++++++++++++++++ | 53% ~01s |++++++++++++++++++++++++++++ | 54% ~01s |++++++++++++++++++++++++++++ | 55% ~01s |+++++++++++++++++++++++++++++ | 56% ~01s |+++++++++++++++++++++++++++++ | 57% ~01s |++++++++++++++++++++++++++++++ | 58% ~01s |++++++++++++++++++++++++++++++ | 59% ~01s |+++++++++++++++++++++++++++++++ | 60% ~01s |+++++++++++++++++++++++++++++++ | 61% ~01s |++++++++++++++++++++++++++++++++ | 62% ~01s |++++++++++++++++++++++++++++++++ | 63% ~01s |+++++++++++++++++++++++++++++++++ | 64% ~01s |+++++++++++++++++++++++++++++++++ | 65% ~01s |++++++++++++++++++++++++++++++++++ | 66% ~01s |++++++++++++++++++++++++++++++++++ | 67% ~01s |+++++++++++++++++++++++++++++++++++ | 68% ~01s |+++++++++++++++++++++++++++++++++++ | 69% ~01s |++++++++++++++++++++++++++++++++++++ | 70% ~01s |++++++++++++++++++++++++++++++++++++ | 71% ~01s |+++++++++++++++++++++++++++++++++++++ | 72% ~01s |+++++++++++++++++++++++++++++++++++++ | 73% ~01s |++++++++++++++++++++++++++++++++++++++ | 74% ~01s |++++++++++++++++++++++++++++++++++++++ | 76% ~01s |+++++++++++++++++++++++++++++++++++++++ | 77% ~01s |+++++++++++++++++++++++++++++++++++++++ | 78% ~01s |++++++++++++++++++++++++++++++++++++++++ | 79% ~01s |++++++++++++++++++++++++++++++++++++++++ | 80% ~01s |+++++++++++++++++++++++++++++++++++++++++ | 81% ~01s |+++++++++++++++++++++++++++++++++++++++++ | 82% ~01s |++++++++++++++++++++++++++++++++++++++++++ | 83% ~00s |++++++++++++++++++++++++++++++++++++++++++ | 84% ~00s |+++++++++++++++++++++++++++++++++++++++++++ | 85% ~00s |+++++++++++++++++++++++++++++++++++++++++++ | 86% ~00s |++++++++++++++++++++++++++++++++++++++++++++ | 87% ~00s |++++++++++++++++++++++++++++++++++++++++++++ | 88% ~00s |+++++++++++++++++++++++++++++++++++++++++++++ | 89% ~00s |+++++++++++++++++++++++++++++++++++++++++++++ | 90% ~00s |++++++++++++++++++++++++++++++++++++++++++++++ | 91% ~00s |++++++++++++++++++++++++++++++++++++++++++++++ | 92% ~00s |+++++++++++++++++++++++++++++++++++++++++++++++ | 93% ~00s |+++++++++++++++++++++++++++++++++++++++++++++++ | 94% ~00s |++++++++++++++++++++++++++++++++++++++++++++++++ | 95% ~00s |++++++++++++++++++++++++++++++++++++++++++++++++ | 96% ~00s |+++++++++++++++++++++++++++++++++++++++++++++++++ | 97% ~00s |+++++++++++++++++++++++++++++++++++++++++++++++++ | 98% ~00s |++++++++++++++++++++++++++++++++++++++++++++++++++| 99% ~00s |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed=03s
Merged the most recent publication of sensor position files for each site and saved to /stackedFiles
Copied the most recent publication of variable definition file to /stackedFiles
Finished: Stacked 1 data tables and 3 metadata tables!
Stacking took 4.80769 secs
All unzipped monthly data folders have been removed.
| | 0 % ~calculating |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed=00s
Stacking operation across a single core.
Stacking table SWS_30_minute
| | 0 % ~calculating |+ | 1 % ~08s |++ | 2 % ~08s |++ | 3 % ~08s |+++ | 4 % ~08s |+++ | 5 % ~08s |++++ | 6 % ~07s |++++ | 7 % ~07s |+++++ | 8 % ~07s |+++++ | 9 % ~07s |++++++ | 10% ~07s |++++++ | 11% ~07s |+++++++ | 12% ~07s |+++++++ | 13% ~07s |++++++++ | 14% ~07s |++++++++ | 15% ~07s |+++++++++ | 16% ~06s |+++++++++ | 17% ~06s |++++++++++ | 18% ~06s |++++++++++ | 19% ~06s |+++++++++++ | 20% ~06s |+++++++++++ | 21% ~06s |++++++++++++ | 22% ~06s |++++++++++++ | 23% ~06s |+++++++++++++ | 24% ~06s |+++++++++++++ | 25% ~06s |++++++++++++++ | 26% ~06s |++++++++++++++ | 27% ~06s |+++++++++++++++ | 28% ~06s |+++++++++++++++ | 29% ~06s |++++++++++++++++ | 30% ~05s |++++++++++++++++ | 31% ~05s |+++++++++++++++++ | 32% ~05s |+++++++++++++++++ | 33% ~05s |++++++++++++++++++ | 34% ~05s |++++++++++++++++++ | 35% ~05s |+++++++++++++++++++ | 36% ~05s |+++++++++++++++++++ | 37% ~05s |++++++++++++++++++++ | 38% ~05s |++++++++++++++++++++ | 39% ~05s |+++++++++++++++++++++ | 40% ~05s |+++++++++++++++++++++ | 41% ~05s |++++++++++++++++++++++ | 42% ~05s |++++++++++++++++++++++ | 43% ~05s |+++++++++++++++++++++++ | 44% ~04s |+++++++++++++++++++++++ | 45% ~04s |++++++++++++++++++++++++ | 46% ~04s |++++++++++++++++++++++++ | 47% ~04s |+++++++++++++++++++++++++ | 48% ~04s |+++++++++++++++++++++++++ | 49% ~04s |++++++++++++++++++++++++++ | 51% ~04s |++++++++++++++++++++++++++ | 52% ~04s |+++++++++++++++++++++++++++ | 53% ~04s |+++++++++++++++++++++++++++ | 54% ~04s |++++++++++++++++++++++++++++ | 55% ~04s |++++++++++++++++++++++++++++ | 56% ~04s |+++++++++++++++++++++++++++++ | 57% ~03s |+++++++++++++++++++++++++++++ | 58% ~03s |++++++++++++++++++++++++++++++ | 59% ~03s |++++++++++++++++++++++++++++++ | 60% ~03s |+++++++++++++++++++++++++++++++ | 61% ~03s |+++++++++++++++++++++++++++++++ | 62% ~03s |++++++++++++++++++++++++++++++++ | 63% ~03s |++++++++++++++++++++++++++++++++ | 64% ~03s |+++++++++++++++++++++++++++++++++ | 65% ~03s |+++++++++++++++++++++++++++++++++ | 66% ~03s |++++++++++++++++++++++++++++++++++ | 67% ~03s |++++++++++++++++++++++++++++++++++ | 68% ~03s |+++++++++++++++++++++++++++++++++++ | 69% ~03s |+++++++++++++++++++++++++++++++++++ | 70% ~02s |++++++++++++++++++++++++++++++++++++ | 71% ~02s |++++++++++++++++++++++++++++++++++++ | 72% ~02s |+++++++++++++++++++++++++++++++++++++ | 73% ~02s |+++++++++++++++++++++++++++++++++++++ | 74% ~02s |++++++++++++++++++++++++++++++++++++++ | 75% ~02s |++++++++++++++++++++++++++++++++++++++ | 76% ~02s |+++++++++++++++++++++++++++++++++++++++ | 77% ~02s |+++++++++++++++++++++++++++++++++++++++ | 78% ~02s |++++++++++++++++++++++++++++++++++++++++ | 79% ~02s |++++++++++++++++++++++++++++++++++++++++ | 80% ~02s |+++++++++++++++++++++++++++++++++++++++++ | 81% ~02s |+++++++++++++++++++++++++++++++++++++++++ | 82% ~01s |++++++++++++++++++++++++++++++++++++++++++ | 83% ~01s |++++++++++++++++++++++++++++++++++++++++++ | 84% ~01s |+++++++++++++++++++++++++++++++++++++++++++ | 85% ~01s |+++++++++++++++++++++++++++++++++++++++++++ | 86% ~01s |++++++++++++++++++++++++++++++++++++++++++++ | 87% ~01s |++++++++++++++++++++++++++++++++++++++++++++ | 88% ~01s |+++++++++++++++++++++++++++++++++++++++++++++ | 89% ~01s |+++++++++++++++++++++++++++++++++++++++++++++ | 90% ~01s |++++++++++++++++++++++++++++++++++++++++++++++ | 91% ~01s |++++++++++++++++++++++++++++++++++++++++++++++ | 92% ~01s |+++++++++++++++++++++++++++++++++++++++++++++++ | 93% ~01s |+++++++++++++++++++++++++++++++++++++++++++++++ | 94% ~01s |++++++++++++++++++++++++++++++++++++++++++++++++ | 95% ~00s |++++++++++++++++++++++++++++++++++++++++++++++++ | 96% ~00s |+++++++++++++++++++++++++++++++++++++++++++++++++ | 97% ~00s |+++++++++++++++++++++++++++++++++++++++++++++++++ | 98% ~00s |++++++++++++++++++++++++++++++++++++++++++++++++++| 99% ~00s |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed=08s
Merged the most recent publication of sensor position files for each site and saved to /stackedFiles
Copied the most recent publication of variable definition file to /stackedFiles
Finished: Stacked 1 data tables and 3 metadata tables!
Stacking took 14.45058 secs
All unzipped monthly data folders have been removed.
<rpy2.rinterface_lib.sexp.NULLType object at 0x14bcc10d0> [0]
Remember: Adjust the savepath argument in zipsByProduct() to reflect your desired download location. Similarly, replace the example directory names in stackByTable() with the actual names generated during download.
This section demonstrates the process of downloading NEON data directly using the NEON API and organizing it for further analysis. Ensure to execute these steps sequentially to effectively utilize the neonUtilities tools for working with NEON data.