High Rate GNSS Data Download

Overview


High rate GNSS data are available for download for the days surrounding major geologic events. These data are recorded at 1 Hz and 10 Hz sampling rates and are available in RINEX and Trimble proprietary raw formats. High rate GNSS data can be accessed from a Amazon Web Service (AWS) S3 bucket. This page provides a guide to downloading high rate GNSS data using AWS.

In the S3 bucket the data storage structure is:

s3://gnss.highrate.data/{1hz,10hz}/{raw,rinex}/yyyy/DOY/filename

And the filename format is:

Raw, 1 Hz = STATyyyyMMDDHH00b.TO{0,2} Raw, 10 Hz = STATyyyyMMDDHH00c.TO{0,2} RINEX, 1 Hz and 10 Hz = statDOY0.YYd.Z

Where STAT and stat are the station name in upper and lower case, yyyy is the CE year, MM is the zero padded month, DD is the zero padded day, HH is the zero padded hour, YY is the CE year without the century prefix, DOY is the day of year, and , denotes that either of the character sequences in curly braces is applicable.

!! Before proceeding ensure you have a local directory prepared for downloaded file storage !! This directory is referred to in command examples as download_directory, please replace this argument name with the name and location of your download directory.

FAST START FOR EXPERTS

To get details on all the geologic events contained within the available data, download the readme:

aws s3 sync s3://gnss.highrate.data/README.gnsshighrate /download_directory/ --no-sign-request

To download a specific file use the command:

aws s3 sync s3://gnss.highrate.data/[1hz/|10hz]/[raw/|rinex]/yyyy/DOY/filename /download_directory/ --no-sign-request

To download many files of a particular type only specify part of the file Key. A command to download all 10 Hz RINEX files in 2016 is:

aws s3 sync s3://gnss.highrate.data/10hz/rinex/2016/ /download_directory/ --no-sign-request

A simple command to download all the data available on AWS (~72 GB) and keep it up to date is:

aws s3 sync s3://gnss.highrate.data/ /download_directory/ --no-sign-request

AWS Installation


Data access through AWS requires the installation of the AWS Command Line Interface tool. If you have pip installed this is very easy, simply run the command:

pip install awscli --upgrade --user

to install the latest version. Run aws --version to test the installation. If this command fails check that the AWS Command Line Interface tool is discoverable via your PATH environmental variable.

Data Discovery


High rate GNSS data are stored in an AWS S3 bucket. Although S3 buckets do not have a storage hierarchy their files can be viewed like those in a directory tree using aws. The high rate GNSS data bucket is named gnss.highrate.data and currently no credentials are required for access if the --no-sign-request flag is set in each aws command. To list details about the bucket's "root directory" contents enter:

aws s3 ls s3://gnss.highrate.data/ --no-sign-request

The --recursive flag adds a lot of power to this command. By entering:

aws s3 ls s3://gnss.highrate.data/ --recursive --no-sign-request

every file Key in the bucket will be displayed following information about the file's creation time and size. If you are interested in what 1 Hz RINEX files exist in the bucket the command:

aws s3 ls s3://gnss.highrate.data/1hz/rinex/ --recursive --no-sign-request

will display details and Keys for every RINEX file with a 1 Hz sampling rate.

Data Download


!! Before proceeding ensure you have a local directory prepared for downloaded file storage !! This directory is referred to in command examples as download_directory, please replace this argument name with the name and location of your download directory.

While listing files it is only possible to look "down" file Keys, but when downloading files more specific file targetting is possible by using the aws s3 rsync command with the --exclude, and --include arguments.

File Keys act as pointers to files and each has a prefix reminiscent of a file path. In the bucket there is no structure between the files with "1hz/rinex/2016/324/kaik3240.16d.Z" and "10hz/rinex/2016/324/kaik3240.16d.Z" Keys, but each of the identically named files can be distinguished by providing information about them in their Keys. The naming convention for file Keys is "file sampling rate / file format / year of sampling / day of year of sampling / filename" so by specifying sampling rate information in an --include argument just one of the files can be downloaded, e.g. --exclude "*" --include "1hz/*kaik3240.16d.Z" will download the 1 Hz RINEX file and not the 10 Hz RINEX file.

To download only 1 Hz RINEX files for the PYGR GNSS receiver enter the command:

aws s3 sync s3://gnss.highrate.data/1hz/rinex/ /download_directory/ --exclude "*" --include "*pygr*" --no-sign-request

or the command:

aws s3 sync s3://gnss.highrate.data/ /download_directory/ --exclude "*" --include "1hz/rinex/*pygr*" --no-sign-request

In the first command the --exclude "*" --include "*pygr*" command segment says "only download files that contain the pygr string in their Key". If --exclude "*" was excluded then all the files with a "1hz/rinex/" Key prefix would be downloaded. If --include "*pygr*" was excluded then no files would be downloaded. In the second command specifying a more detailed Key prefix in the --include argument removes the need for the "extended" bucket path.

When aws downloads a file it creates directories in the download folder with directory names and hierarchies discerned from the sequence of strings found between "/" characters in the file Key. For example when the file with Key "1hz/rinex/2013/198/akto1980.13d.Z" is downloaded the directory /download_directory/1hz/rinex/2013/198 is created and within this directory the downloaded "akto1980.13d.Z" file is saved. Note that the aws s3 sync command will only download a file if the local version is older than the bucket version.

To discover which data in the high rate GNSS data bucket relates to a specific geologic event information beyond that contained in the file Keys is required. Such information is available in the "README.gnsshighrate" file in the bucket. Example download commands for some events is given in the next section of the guide. To download the readme file:

aws s3 sync s3://gnss.highrate.data/README.gnsshighrate /download_directory/ --no-sign-request

Data Sources

Raw data files are downloaded directly from the GNSS receivers in Trimble proprietary format (.T00 or .T02). Day-long RINEX files are generated by splicing together the available raw data. Due to data transmission constraints and limited internal receiver storage, in most cases it was not possible to download a full day of data so day-long RINEX files are often incomplete.

Data Download Examples


The current data available on AWS focuses on geologic events between 2013 and 2016. Below an example data download command is given for the four most recent geologic events. To see which stations recorded a displacement signature for any geologic event in the last seven years click their representative triangle on this map.

2016 Kaikoura Earthquake: /2016/318/ Download all data for the day of the event:

aws s3 sync s3://gnss.highrate.data/ /download_directory/ --exclude "*" --include "*/2016/318/*" --no-sign-request

2016 Kaikoura Earthquake and Early Aftershocks: /2016/318/ & /2016/319/ Download all data for the two days containing the events:

aws s3 sync s3://gnss.highrate.data/ /download_directory/ --exclude "*" --include "*/2016/31[8-9]/*" --no-sign-request

2016 White Island Eruption: /2016/256/ Download all data from the White Island GNSS receivers on the event day:

aws s3 sync s3://gnss.highrate.data/ /download_directory/ --exclude "*" --include "*/2016/256/RGW*" --no-sign-request

2016 East Cape Earthquake and M5.7 Foreshock: /2016/244/ & /2016/245/ Download all data from the three closest GNSS receivers for the two days containing the events:

aws s3 sync s3://gnss.highrate.data/ /download_directory/ --exclude "*" --include "*/2016/24[4-5]/HIKB*" --include "*/2016/24[4-5]/WMAT*" --include "*/2016/24[4-5]/PAKI*" --no-sign-request