Methods

Several methods exist to help combine multiple data sets and convert between equivalent indices.

ACE

Supports the ACE instrument by providing reference and acknowledgement information.

Provides general routines for the ACE space weather instruments.

pysatSpaceWeather.instruments.methods.ace.ace_swepam_hourly_omni_norm(as_inst, speed_key='sw_bulk_speed', dens_key='sw_proton_dens', temp_key='sw_ion_temp')[source]

Normalize ACE SWEPAM variables as described in the OMNI processing _[1].

Parameters:
as_instpysat.Instrument

pysat Instrument object with ACE SWEPAM data.

speed_keystr

Data key for bulk solar wind speed data in km/s (default=’sw_bulk_speed’)

dens_keystr

Data key for solar wind proton density data in P/cm^3 (default=’sw_proton_dens’)

temp_keystr

Data key for solar wind ion temperature data in K (default=’sw_ion_temp’)

References

[1] https://omniweb.gsfc.nasa.gov/html/omni_min_data.html

pysatSpaceWeather.instruments.methods.ace.acknowledgements()[source]

Define the acknowledgements for the specified ACE instrument.

Returns:
acknstr

Acknowledgements for the ACE instrument

pysatSpaceWeather.instruments.methods.ace.clean(inst)[source]

Clean the common aspects of the ACE space weather data.

Parameters:
instpysat.Instrument

ACE pysat.Instrument object

Returns:
max_statusint

Maximum allowed status

pysatSpaceWeather.instruments.methods.ace.common_metadata()[source]

Define the common metadata information for all ACE instruments.

Returns:
metapysat.Meta

pysat Meta class with ‘jd’ and ‘sec’ initiated

status_descstr

Description of the status flags

pysatSpaceWeather.instruments.methods.ace.download(date_array, name, tag='', inst_id='', data_path='', now=None, mock_download_dir=None)[source]

Download the requested ACE Space Weather data.

Parameters:
date_arrayarray-like

Array of datetime values

namestr

ACE Instrument name.

tagstr

ACE Instrument tag. (default=’’)

inst_idstr

ACE instrument ID. (default=’’)

data_pathstr

Path to data directory. (default=’’)

nowdt.datetime or NoneType

Current universal time, if None this is determined for each download. (default=None)

mock_download_dirstr or NoneType

Local directory with downloaded files or None. If not None, will process any files with the correct name and date as if they were downloaded (default=None)

Raises:
IOError

If an unknown mock download directory is supplied or the file format changes.

Warning

  • Only able to download current real-time data

  • File requested not available on server

pysatSpaceWeather.instruments.methods.ace.list_files(name, tag='', inst_id='', data_path='', format_str=None)[source]

List the local ACE data files.

Parameters:
namestr

ACE Instrument name.

tagstr

ACE Instrument tag. (default=’’)

inst_idstr

Specifies the ACE instrument ID. (default=’’)

data_pathstr

Path to data directory. (default=’’)

format_strstr or NoneType

User specified file format. If None is specified, the default formats associated with the supplied tags are used. (default=None)

Returns:
filespysat.Files

A class containing the verified available files

pysatSpaceWeather.instruments.methods.ace.references(name)[source]

Define the references for the specified ACE instrument.

Parameters:
namestr

Instrument name of the ACE instrument

Returns:
refstr

Reference for the ACE instrument paper

Dst

Supports the Dst ring current index by providing reference and acknowledgement information.

Provides default routines for Dst.

pysatSpaceWeather.instruments.methods.dst.acknowledgements(tag)[source]

Define the acknowledgements for the Dst data.

Parameters:
tagstr

Tag of the space weather index

Returns:
acknstr

Acknowledgements string associated with the appropriate Dst tag.

pysatSpaceWeather.instruments.methods.dst.references(tag)[source]

Define the references for the Dst data.

Parameters:
tagstring

Tag of the space weather index

Returns:
refsstr

Reference string associated with the appropriate Dst tag.

F 10.7

Supports the F 10.7 radio flux by providing reference and acknowledgement information as well as a routine to combine F 10.7 data obtained from multiple sources.

Routines for the F10.7 solar index.

pysatSpaceWeather.instruments.methods.f107.acknowledgements(tag)[source]

Define the acknowledgements for the F10.7 data.

Parameters:
tagstr

Tag of the space weather index

Returns:
acknstr

Acknowledgements string associated with the appropriate F10.7 tag.

pysatSpaceWeather.instruments.methods.f107.calc_f107a(f107_inst, f107_name='f107', f107a_name='f107a', min_pnts=41)[source]

Calculate the 81 day mean F10.7.

Parameters:
f107_instpysat.Instrument

pysat Instrument holding the F10.7 data

f107_namestr

Data column name for the F10.7 data (default=’f107’)

f107a_namestr

Data column name for the F10.7a data (default=’f107a’)

min_pntsint

Minimum number of points required to calculate an average (default=41)

pysatSpaceWeather.instruments.methods.f107.combine_f107(standard_inst, forecast_inst, start=None, stop=None)[source]

Combine the output from the measured and forecasted F10.7 sources.

Parameters:
standard_instpysat.Instrument or NoneType

Instrument object containing data for the ‘sw’ platform, ‘f107’ name, and ‘historic’, ‘prelim’, ‘now’, or ‘daily’ tag

forecast_instpysat.Instrument or NoneType

Instrument object containing data for the ‘sw’ platform, ‘f107’ name, and ‘now’, ‘prelim’, ‘45day’ or ‘forecast’ tag

startdt.datetime or NoneType

Starting time for combining data, or None to use earliest loaded date from the pysat Instruments (default=None)

stopdt.datetime or NoneType

Ending time for combining data, or None to use the latest loaded date from the pysat Instruments (default=None)

Returns:
f107_instpysat.Instrument

Instrument object containing F10.7 observations for the desired period of time, merging the standard, 45day, and forecasted values based on their reliability

Raises:
ValueError

If appropriate time data is not supplied, or if the date range is badly formed.

Notes

Merging prioritizes the standard data, then the 45day data, and finally the forecast data

Will not attempt to download any missing data, but will load data

pysatSpaceWeather.instruments.methods.f107.references(tag)[source]

Define the references for the F10.7 data.

Parameters:
tagstr

Instrument tag for the F10.7 data.

Returns:
refsstr

Reference string associated with the appropriate F10.7 tag.

General

General functions that are useful across platforms.

Provides routines that support general space weather instruments.

pysatSpaceWeather.instruments.methods.general.get_instrument_data_path(inst_mod_name, tag='', inst_id='', **kwargs)[source]

Get the data_path attribute from an Instrument sub-module.

Parameters:
inst_mod_namestr

pysatSpaceWeather Instrument module name

tagstr

String specifying the Instrument tag (default=’’)

inst_idstr

String specifying the instrument identification (default=’’)

kwargsdict

Optional additional kwargs that may be used to initialize an Instrument

Returns:
data_pathstr

Path where the Instrument data is stored

pysatSpaceWeather.instruments.methods.general.get_local_or_remote_text(url, mock_download_dir, filename)[source]

Retrieve text from a remote or local file.

Parameters:
filenamestr

Filename without any directory structure

urlstr

Remote URL where file is located

mock_download_dirstr or NoneType

Local directory with downloaded files or None. If not None, will process any files with the correct name and date as if they were downloaded (default=None)

Returns:
raw_txtstr or NoneType

All the text from the desired file or None if the file could not be retrieved

Raises:
IOError

If an unknown mock download directory is supplied.

pysatSpaceWeather.instruments.methods.general.preprocess(inst)[source]

Preprocess the meta data by replacing the file fill values with NaN.

Parameters:
instpysat.Instrument

pysat.Instrument object

GFZ

Supports the German Research Centre for Geosciences at Potsdam (GFZ) data products.

Provides routines that support GFZ space weather instruments.

pysatSpaceWeather.instruments.methods.gfz.json_downloads(date_array, data_path, local_file_prefix, local_date_fmt, gfz_data_name, freq, update_files=False, is_def=False, mock_download_dir=None)[source]

Download data from GFZ into CSV files at a specified cadence.

Parameters:
date_arrayarray-like or pandas.DatetimeIndex

Array-like or index of datetimes to be downloaded.

data_pathstr

Path to data directory.

local_file_prefixstr

Prefix for local files, e.g., ‘tag_’ or ‘tag_monthly_’

local_date_fmtstr

String format for the local filename, e.g., ‘%Y-%m-%d’ or ‘%Y-%m’

gfz_data_namestr

Name of the data index on the GFZ server, expects one of: ‘Kp’, ‘ap’, ‘Ap’, ‘Cp’, ‘C9’, ‘Hp30’, ‘Hp60’, ‘ap30’, ‘ap60’, ‘SN’, ‘Fobs’, ‘Fadj’ where SN is the international sunspot number and Fxxx is the observed and adjusted F10.7.

freqpds.DateOffset or dt.timedelta

Offset to add to the start date to ensure all data is downloaded (inclusive)

update_filesbool

Re-download data for files that already exist if True (default=False)

is_defbool

If true, selects only the definitive data, otherwise also includes nowcast data (default=False)

mock_download_dirstr or NoneType

Local directory with downloaded files or None. If not None, will process any files with the correct name and date (following the local file prefix and date format) as if they were downloaded (default=None)

Raises:
IOError

If there is a gateway timeout when downloading data or if an unknown mock download directory is supplied.

pysatSpaceWeather.instruments.methods.gfz.kp_ap_cp_download(platform, name, date_array, tag, inst_id, data_path, mock_download_dir=None)[source]

Download Kp, ap, and Cp data from GFZ.

Parameters:
platformstr

Instrument platform.

namestr

Instrument name.

date_arrayarray-like or pandas.DatetimeIndex

Array-like or index of datetimes to be downloaded.

tagstr

String specifying the database, expects ‘def’ (definitive) or ‘now’ (nowcast)

inst_idstr

Specifies the instrument identification, not used.

data_pathstr

Path to data directory.

mock_download_dirstr or NoneType

Local directory with downloaded files or None. If not None, will process any files with the correct name and date (following the local file prefix and date format) as if they were downloaded (default=None)

Raises:
ValueError

If an unknown instrument module is supplied.

IOError

If an unknown mock download directory is supplied.

pysatSpaceWeather.instruments.methods.gfz.kp_ap_cp_list_files(name, tag, inst_id, data_path, format_str=None)[source]

List local files for Kp, ap, or Cp data obtained from GFZ.

Parameters:
namestr

Instrument name.

tagstr

String specifying the database, expects ‘def’ (definitive) or ‘now’ (nowcast)

inst_idstr

Specifies the instrument identification, not used.

data_pathstr

Path to data directory.

format_strstr or NoneType

User specified file format. If None is specified, the default formats associated with the supplied tags are used. (default=None)

Returns:
filespysat._files.Files

A class containing the verified available files

pysatSpaceWeather.instruments.methods.gfz.load_def_now(fnames)[source]

Load GFZ yearly definitive or nowcast index data.

Parameters:
fnamespandas.Series

Series of filenames

Returns:
datapandas.DataFrame

Object containing satellite data

Kp and Ap

Supports the Kp instrument by providing reference and acknowledgement information, a routine to combine Kp from multiple sources, routines to convert between Kp and Ap, and a routine that uses Kp data as a geomagnetic activity filter for other data sets.

Provides routines to support the geomagnetic indices, Kp and Ap.

pysatSpaceWeather.instruments.methods.kp_ap.acknowledgements(name, tag)[source]

Define the acknowledgements for the geomagnetic data sets.

Parameters:
namestr

Instrument name of space weather index, accepts ‘kp’ or ‘ap’.

tagstr

Instrument tag.

pysatSpaceWeather.instruments.methods.kp_ap.calc_daily_Ap(ap_inst, ap_name='3hr_ap', daily_name='Ap', running_name=None, min_periods=8)[source]

Calculate the daily Ap index from the 3hr ap index.

Parameters:
ap_instpysat.Instrument

pysat instrument containing 3-hourly ap data

ap_namestr

Column name for 3-hourly ap data (default=’3hr_ap’)

daily_namestr

Column name for daily Ap data (default=’Ap’)

running_namestr or NoneType

Column name for daily running average of ap, not output if None (default=None)

min_periodsint

Mininmum number of observations needed to output an average value (default=8)

Raises:
ValueError

If ap_name or daily_name aren’t present in ap_inst

pysatSpaceWeather.instruments.methods.kp_ap.combine_kp(standard_inst=None, recent_inst=None, forecast_inst=None, start=None, stop=None, fill_val=nan)[source]

Combine the output from the different Kp sources for a range of dates.

Parameters:
standard_instpysat.Instrument or NoneType

Instrument object containing data for the ‘sw’ platform, ‘kp’ name, and ‘def’ tag or None to exclude (default=None)

recent_instpysat.Instrument or NoneType

Instrument object containing data for the ‘sw’ platform, ‘kp’ name, and ‘recent’ tag or None to exclude (default=None)

forecast_instpysat.Instrument or NoneType

Instrument object containing data for the ‘sw’ platform, ‘kp’ name, and ‘forecast’ tag or None to exclude (default=None)

startdt.datetime or NoneType

Starting time for combining data, or None to use earliest loaded date from the pysat Instruments (default=None)

stopdt.datetime

Ending time for combining data, or None to use the latest loaded date from the pysat Instruments (default=None)

fill_valint or float

Desired fill value (since the standard instrument fill value differs from the other sources) (default=np.nan)

Returns:
kp_instpysat.Instrument

Instrument object containing Kp observations for the desired period of time, merging the standard, recent, and forecasted values based on their reliability

Raises:
ValueError

If only one Kp instrument or bad times are provided

pysatSpaceWeather.instruments.methods.kp_ap.convert_3hr_kp_to_ap(kp_inst, var_name='Kp')[source]

Calculate 3 hour ap from 3 hour Kp index.

Parameters:
kp_instpysat.Instrument

Instrument containing Kp data

var_namestr

Variable name for the Kp data (default=’Kp’)

Raises:
ValueError

If var_name is not present in kp_inst.

pysatSpaceWeather.instruments.methods.kp_ap.convert_ap_to_kp(ap_data, fill_val=-1, ap_name='ap', kp_name='Kp')[source]

Convert Ap into Kp.

Parameters:
ap_dataarray-like

Array-like object containing Ap data

fill_valint, float, NoneType

Fill value for the data set (default=-1)

ap_namestr

Name of the input ap (default=’ap’)

kp_namestr

Name of the output Kp (default=’Kp’)

Returns:
kp_dataarray-like

Array-like object containing Kp data

metaMetadata

Metadata object containing information about transformed data

pysatSpaceWeather.instruments.methods.kp_ap.filter_geomag(inst, min_kp=0, max_kp=9, filter_time=24, kp_inst=None, var_name='Kp')[source]

Filter pysat.Instrument data for given time after Kp drops below gate.

Parameters:
instpysat.Instrument or NoneType

Instrument with non-Kp data to be filtered by geomagnetic activity

min_kpfloat

Minimum Kp value allowed. Kp values below this filter the data in inst (default=0)

max_kpfloat

Maximum Kp value allowed. Kp values above this filter the data in inst (default=9)

filter_timeint

Number of hours to filter data after Kp drops below max_kp (default=24)

kp_instpysat.Instrument or NoneType

Kp pysat.Instrument object with or without data already loaded. If None, will load GFZ definitive kp data for the instrument date (default=None)

var_namestr

String providing the variable name for the Kp data (default=’Kp’)

Raises:
IOError

If no Kp data is available to load

pysatSpaceWeather.instruments.methods.kp_ap.initialize_ap_metadata(meta, data_key, fill_val=-1)[source]

Initialize the ap meta data using our knowledge of the index.

Parameters:
metapysat._meta.Meta

Pysat Metadata

data_keystr

String denoting the data key

fill_valint or float

File-specific fill value (default=-1)

pysatSpaceWeather.instruments.methods.kp_ap.initialize_bartel_metadata(meta, data_key, fill_val=-1)[source]

Initialize the Bartel rotation meta data using our knowledge of the data.

Parameters:
metapysat._meta.Meta

Pysat Metadata

data_keystr

String denoting the data key

fill_valint or float

File-specific fill value (default=-1)

pysatSpaceWeather.instruments.methods.kp_ap.initialize_kp_metadata(meta, data_key, fill_val=-1)[source]

Initialize the Kp meta data using our knowledge of the index.

Parameters:
metapysat._meta.Meta

Pysat Metadata

data_keystr

String denoting the data key

fill_valint or float

File-specific fill value (default=-1)

pysatSpaceWeather.instruments.methods.kp_ap.references(name, tag)[source]

Define the references for the geomagnetic data sets.

Parameters:
namestr

Instrument name of space weather index, accepts kp or ap.

tagstr

Instrument tag.

pysatSpaceWeather.instruments.methods.kp_ap.round_ap(ap_in, fill_val=nan)[source]

Round an ap value to the nearest Kp value.

Parameters:
ap_infloat

Ap value as a floating point number.

fill_valfloat

Value for unassigned or bad indices. (default=np.nan)

Returns:
float

Fill value for infinite or out-of-range data, otherwise the best kp index is provided.

LISIRD

Contains functions to support LASP Interactive Solar IRradiance Datacenter (LISIRD) data sets and downloads.

Provides support functions for the LASP LISIRD data base.

pysatSpaceWeather.instruments.methods.lisird.build_lisird_url(lisird_data_name, start, stop)[source]

Build a LASP LISIRD direct download URL.

Parameters:
lisird_data_namestr

Name of the data set on the LISARD server

startdt.datetime

Start time

stopdt.datetime

Stop time

Returns:
urlstr

URL that will download the desired data

pysatSpaceWeather.instruments.methods.lisird.download(date_array, data_path, local_file_prefix, local_date_fmt, lisird_data_name, freq, update_files=False, fill_vals=None, mock_download_dir=None)[source]

Download routine for LISIRD data.

Parameters:
date_arrayarray-like

Sequence of dates for which files will be downloaded.

data_pathstr

Path to data directory.

local_file_prefixstr

Prefix for local files, e.g., ‘tag_’ or ‘tag_monthly_’

local_date_fmtstr

String format for the local filename, e.g., ‘%Y-%m-%d’ or ‘%Y-%m’

lisird_data_namestr

Name of the data set on the LISARD server

freqpds.DateOffset or dt.timedelta

Offset to add to the start date to ensure all data is downloaded (inclusive)

update_filesbool

Re-download data for files that already exist if True (default=False)

fill_valsdict or NoneType

Dict of fill values to replace with NaN by variable name or None to leave alone (default=None)

mock_download_dirstr or NoneType

Local directory with downloaded files or None. If not None, will process any files with the correct name and date (following the local file prefix and date format) as if they were downloaded (default=None)

Raises:
IOError

If there is a gateway timeout when downloading data or an unknown mock download directory is supplied.

KeyError

If the fill_vals input does not match the downloaded data.

pysatSpaceWeather.instruments.methods.lisird.references(platform, name, tag, inst_id)[source]

Provide references for different Instrument data products.

Parameters:
platformstr

Instrument platform

namestr

Instrument name

tagstr

Instrument tag

inst_idstr

Instrument ID

Returns:
refsstr

String of references

SWPC

Supports the Space Weather Prediction Center (SWPC) data products.

Provides routines that support SWPC space weather instruments.

pysatSpaceWeather.instruments.methods.swpc.daily_dsd_download(name, today, data_path, mock_download_dir=None)[source]

Download the daily NOAA Daily Solar Data indices.

Parameters:
namestr

Instrument name, expects one of ‘f107’, ‘flare’, ‘ssn’, or ‘sbfield’.

todaydt.datetime

Datetime for current day.

data_pathstr

Path to data directory.

mock_download_dirstr or NoneType

Local directory with downloaded files or None. If not None, will process any files with the correct name and date as if they were downloaded (default=None)

Raises:
IOError

If an unknown mock download directory is supplied or the desired file is missing.

pysatSpaceWeather.instruments.methods.swpc.geomag_forecast_download(name, date_array, data_path, mock_download_dir=None)[source]

Download the 3-day geomagnetic Kp, ap, and storm data from SWPC.

Parameters:
namestr

Instrument name, expects one of ‘kp’, ‘ap’, or ‘stormprob’.

date_arrayarray-like or pandas.DatetimeIndex

Array-like or index of datetimes to be downloaded.

data_pathstr

Path to data directory.

mock_download_dirstr or NoneType

Local directory with downloaded files or None. If not None, will process any files with the correct name and date as if they were downloaded (default=None)

Raises:
IOError

If an unknown mock download directory is supplied or the desired file is missing.

pysatSpaceWeather.instruments.methods.swpc.kp_ap_recent_download(name, date_array, data_path, mock_download_dir=None)[source]

Download recent Kp and ap data from SWPC.

Parameters:
namestr

Instrument name, expects ‘kp’ or ‘ap’.

date_arrayarray-like or pandas.DatetimeIndex

Array-like or index of datetimes to be downloaded.

data_pathstr

Path to data directory.

mock_download_dirstr or NoneType

Local directory with downloaded files or None. If not None, will process any files with the correct name and date as if they were downloaded (default=None)

Raises:
IOError

If an unknown mock download directory is supplied or the desired file is missing.

pysatSpaceWeather.instruments.methods.swpc.list_files(name, tag, inst_id, data_path, format_str=None)[source]

List local files for Kp or ap data obtained from SWPC.

Parameters:
namestr

Instrument name.

tagstr

String specifying the database, expects ‘def’ (definitive) or ‘now’ (nowcast)

inst_idstr

Specifies the instrument identification, not used.

data_pathstr

Path to data directory.

format_strstr or NoneType

User specified file format. If None is specified, the default formats associated with the supplied tags are used. (default=None)

Returns:
filespysat._files.Files

A class containing the verified available files

pysatSpaceWeather.instruments.methods.swpc.old_indices_dsd_download(name, date_array, data_path, local_files, today, mock_download_dir=None)[source]

Download the old NOAA Daily Solar Data indices.

Parameters:
namestr

Instrument name, expects one of ‘f107’, ‘flare’, ‘ssn’, or ‘sbfield’.

date_arrayarray-like or pandas.DatetimeIndex

Array-like or index of datetimes to be downloaded.

data_pathstr

Path to data directory.

local_filespds.Series

A Series containing the local filenames indexed by time.

todaydt.datetime

Datetime for current day

mock_download_dirstr or NoneType

Local directory with downloaded files or None. If not None, will process any files with the correct name and date as if they were downloaded (default=None)

Raises:
IOError

If an unknown mock download directory is supplied.

pysatSpaceWeather.instruments.methods.swpc.parse_45day_block(block_lines)[source]

Parse the data blocks used in the 45-day Ap and F10.7 Flux Forecast file.

Parameters:
block_lineslist

List of lines containing data in this data block

Returns:
dateslist

List of dates for each date/data pair in this block

valueslist

List of values for each date/data pair in this block

pysatSpaceWeather.instruments.methods.swpc.parse_daily_solar_data(data_lines, year, optical)[source]

Parse the data in the SWPC daily solar index file.

Parameters:
data_lineslist

List of lines containing data

yearlist

Year of file

opticalboolean

Flag denoting whether or not optical data is available

Returns:
dateslist

List of dates for each date/data pair in this block

valuesdict

Dict of lists of values, where each key is the value name

pysatSpaceWeather.instruments.methods.swpc.recent_ap_f107_download(name, date_array, data_path, mock_download_dir=None)[source]

Download 45-day ap and F10.7 data from SWPC.

Parameters:
namestr

Instrument name, expects ‘f107’ or ‘ap’.

date_arrayarray-like or pandas.DatetimeIndex

Array-like or index of datetimes to be downloaded.

data_pathstr

Path to data directory.

mock_download_dirstr or NoneType

Local directory with downloaded files or None. If not None, will process any files with the correct name and date as if they were downloaded (default=None)

Raises:
IOError

If an unknown mock download directory is supplied or the desored file is missing.

pysatSpaceWeather.instruments.methods.swpc.rewrite_daily_solar_data_file(year, outfiles, lines)[source]

Rewrite the SWPC Daily Solar Data files.

Parameters:
yearint

Year of data file (format changes based on date)

outfilesdict

Output filenames for all relevant Instruments

linesstr

String containing all output data (result of ‘read’)

pysatSpaceWeather.instruments.methods.swpc.solar_geomag_predictions_download(name, date_array, data_path, mock_download_dir=None)[source]

Download the 3-day solar-geomagnetic predictions from SWPC.

Parameters:
namestr

Instrument name, expects one of ‘kp’, ‘ap’, ‘stormprob’, ‘f107’, ‘flare’, or ‘polarcap’.

date_arrayarray-like or pandas.DatetimeIndex

Array-like or index of datetimes to be downloaded.

data_pathstr

Path to data directory.

mock_download_dirstr or NoneType

Local directory with downloaded files or None. If not None, will process any files with the correct name and date as if they were downloaded (default=None)

Raises:
IOError

If an unknown mock download directory is supplied or the desired file is missing.