Downloads and URLs

Search results usually contain several related URLs. EarthData.jl provides helpers for extracting HTTPS and S3 URLs from UMM records and for downloading individual files or batches of files.

URL helpers

Use urls to collect every related URL, or filter by scheme with https_urls, s3_urls, or download_url:

julia> struct ExampleItem <: EarthData.AbstractJSON
           RelatedUrls::Vector{NamedTuple{(:URL,), Tuple{String}}}
       end

julia> item = ExampleItem([(URL = "https://example.test/G1.h5",), (URL = "s3://example-bucket/G1.h5",)]);

julia> https_urls(item)
1-element Vector{String}:
 "https://example.test/G1.h5"

julia> s3_urls(item)
1-element Vector{String}:
 "s3://example-bucket/G1.h5"

julia> download_url(item)
"https://example.test/G1.h5"

The same helpers also accept vectors of UMM records:

gg = granules(short_name="GEDI02_A")
https_urls(gg)
s3_urls(gg)

Selecting the data file

A record's related URLs are not all data. A single MCD43A3 granule, for example, carries eight: the HTTPS file, its S3 copy, a DOI landing page, a metadata sidecar, browse imagery and a cloud-credentials endpoint. https_urls returns most of those, so filter on RelatedUrls[].Type instead — data_urls does exactly that, keeping only "GET DATA":

data_urls(first(gg))          # the file
download(gg, "data"; type="GET DATA")

The S3 copy of the same file is typed "GET DATA VIA DIRECT ACCESS":

urls(first(gg); scheme=:s3, type="GET DATA VIA DIRECT ACCESS")

Granule sizes

granule_size reports a granule's size in bytes:

granule_size(first(gg))

It prefers SizeInBytes, and otherwise converts Size using the record's SizeUnit. Size alone is unit-less — providers commonly report megabytes — so reading it without the unit understates the size by a factor of about a million.

Earthdata credentials

NASA Earthdata downloads require credentials. Store them in your .netrc file with netrc!:

EarthData.netrc!("earthdata_username", "earthdata_password")

The file is plaintext, so netrc! writes it at mode 600. Calling it again replaces the Earthdata stanza rather than adding a second one, which is what makes correcting a wrong password take effect: curl reads the first stanza matching a machine. Other machines in the file are left alone.

EarthData.netrc_path() is the file used: ~/.netrc, except on Windows, where the legacy ~/_netrc is used when it exists and ~/.netrc does not. aria2c is pointed at it explicitly, since it defaults to ~/.netrc with no _netrc fallback.

Bearer tokens

Earthdata Login also issues bearer tokens, which is the credential an HTTP client needs when it sets its own headers rather than delegating to curl's .netrc support. EarthData.token() resolves one from ENV["EARTHDATA_TOKEN"], else the first non-comment line of ~/.edl_token, and auth_headers builds the header:

headers = EarthData.auth_headers()          # Authorization: Bearer <token>
Downloads.download(data_urls(first(gg))[1], "granule.h5"; headers)

Resolution never touches the network. To obtain a token from your .netrc credentials instead, call EarthData.token_from_netrc() explicitly:

ENV["EARTHDATA_TOKEN"] = EarthData.token_from_netrc()

Downloading files

Download one URL to a path:

gg = granules(short_name="GEDI02_A")
download(download_url(first(gg)), "GEDI02_A_example.h5")

Download many HTTPS URLs into a folder. EarthData.jl uses aria2c by default for batch HTTPS downloads:

download(gg, "data")

To write a URL list for another tool:

write_urls("urls.txt", gg)

Errors and retries

Not every failure means the same thing, so EarthData.jl sorts them into two groups:

• Temporary — 5xx, 408, 429, and connection faults. The service said nothing about the request, so waiting and trying again can work. These throw TransientError.
• Permanent — 400, 401, 403, 404. These throw an ArgumentError saying what to fix. A 403 from a DAAC nearly always means the account has not accepted the collection's licence, and retrying cannot help.

Getting this wrong is costly either way: one brief 502 should not end an hours-long run, and a 403 should not be retried until the deadline.

The split is by status, not by exception type. Downloads.download raises Downloads.RequestError both for a transport fault and for any HTTP error status, so a 403 and a 503 arrive as the same type and only the status tells them apart.

with_retries applies the split, backing off exponentially up to a minute. A Retry-After from the server takes priority, capped at five minutes. Pass deadline (an absolute time()) to bound the whole thing — the attempt cap is set high enough to wait out a maintenance window, so deadline is what should normally stop a run.

EarthData.with_retries(context="my download") do
    download(url, path)
end

Searches are classified through the existing requester hook:

granules(short_name="MCD43A3", version="061", requester=classifying_requester())

Without it a CMR 503 arrives as a generic ErrorException from parse_cmr_error, which looks the same as a rejected query.

S3 downloads

S3 support is provided by the optional AWSS3 extension. After loading AWSS3, EarthData.jl can request temporary Earthdata Cloud S3 credentials and download S3 URLs:

using AWSS3

config = EarthData.create_aws_config("nsidc", "us-west-2")
EarthData.s3download("s3://bucket/path/file.h5", "file.h5", config)