Readers¶
Satpy supports reading and loading data from many input file formats and
schemes. The Scene
object provides a simple interface
around all the complexity of these various formats through its load
method. The following sections describe the different way data can be loaded,
requested, or added to a Scene object.
Available Readers¶
To get a list of available readers use the available_readers function:
>>> from satpy import available_readers
>>> available_readers()
Filter loaded files¶
Coming soon…
Load data¶
Datasets in Satpy are identified by certain pieces of metadata set during
data loading. These include name, wavelength, calibration,
resolution, polarization, and modifiers. Normally, once a Scene
is created requesting datasets by name or wavelength is all that is
needed:
>>> from satpy import Scene
>>> scn = Scene(reader="seviri_l1b_hrit", filenames=filenames)
>>> scn.load([0.6, 0.8, 10.8])
>>> scn.load(['IR_120', 'IR_134'])
However, in many cases datasets are available in multiple spatial resolutions,
multiple calibrations (brightness_temperature
, reflectance
,
radiance
, etc),
multiple polarizations, or have corrections or other modifiers already applied
to them. By default Satpy will provide the version of the dataset with the
highest resolution and the highest level of calibration (brightness
temperature or reflectance over radiance). It is also possible to request one
of these exact versions of a dataset by using the
DataQuery
class:
>>> from satpy import DataQuery
>>> my_channel_id = DataQuery(name='IR_016', calibration='radiance')
>>> scn.load([my_channel_id])
>>> print(scn['IR_016'])
Or request multiple datasets at a specific calibration, resolution, or polarization:
>>> scn.load([0.6, 0.8], resolution=1000)
Or multiple calibrations:
>>> scn.load([0.6, 10.8], calibrations=['brightness_temperature', 'radiance'])
In the above case Satpy will load whatever dataset is available and matches
the specified parameters. So the above load
call would load the 0.6
(a visible/reflectance band) radiance data and 10.8
(an IR band)
brightness temperature data.
For geostationary satellites that have the individual channel data
separated to several files (segments) the missing segments are padded
by default to full disk area. This is made to simplify caching of
resampling look-up tables (see Resampling for more information).
To disable this, the user can pass pad_data
keyword argument when
loading datasets:
>>> scn.load([0.6, 10.8], pad_data=False)
For geostationary products, where the imagery is stored in the files in a flipped orientation
(e.g. MSG SEVIRI L1.5 data which is flipped upside-down and left-right), the keyword argument
upper_right_corner
can be passed into the load call to automatically flip the datasets to the
wished orientation. Accepted argument values are 'NE'
, 'NW'
, 'SE'
, 'SW'
,
and 'native'
.
By default, no flipping is applied (corresponding to upper_right_corner='native'
) and
the data is delivered in the original format. To get the data in the common upright orientation,
load the datasets using e.g.:
>>> scn.load(['VIS008'], upper_right_corner='NE')
Note
If a dataset could not be loaded there is no exception raised. You must
check the
scn.missing_datasets
property for any DataID
that could not be loaded.
To find out what datasets are available from a reader from the files that were
provided to the Scene
use
available_dataset_ids()
:
>>> scn.available_dataset_ids()
Or available_dataset_names()
for just the string
names of Datasets:
>>> scn.available_dataset_names()
Search for local files¶
Satpy provides a utility
find_files_and_readers()
for searching for files in
a base directory matching various search parameters. This function discovers
files based on filename patterns. It returns a dictionary mapping reader name
to a list of filenames supported. This dictionary can be passed directly to
the Scene
initialization.
>>> from satpy import find_files_and_readers, Scene
>>> from datetime import datetime
>>> my_files = find_files_and_readers(base_dir='/data/viirs_sdrs',
... reader='viirs_sdr',
... start_time=datetime(2017, 5, 1, 18, 1, 0),
... end_time=datetime(2017, 5, 1, 18, 30, 0))
>>> scn = Scene(filenames=my_files)
See the find_files_and_readers()
documentation for
more information on the possible parameters.
Metadata¶
The datasets held by a scene also provide vital metadata such as dataset name, units, observation time etc. The following attributes are standardized across all readers:
name
, and other identifying metadata keys: See Satpy internal workings: having a look under the hood.start_time
: Left boundary of the time interval covered by the dataset.end_time
: Right boundary of the time interval covered by the dataset.area
:AreaDefinition
orSwathDefinition
if data is geolocated. Areas are used for gridded projected data and Swaths when data must be described by individual longitude/latitude coordinates. See the Coordinates section below.orbital_parameters
: Dictionary of orbital parameters describing the satellite’s position.For geostationary satellites it is described using the following scalar attributes:
satellite_actual_longitude/latitude/altitude
: Current position of the satellite at the time of observation in geodetic coordinates (i.e. altitude is relative and normal to the surface of the ellipsoid).satellite_nominal_longitude/latitude/altitude
: Center of the station keeping box (a confined area in which the satellite is actively maintained in using maneuvres). Inbetween major maneuvres, when the satellite is permanently moved, the nominal position is constant.nadir_longitude/latitude
: Intersection of the instrument’s Nadir with the surface of the earth. May differ from the actual satellite position, if the instrument is pointing slightly off the axis (satellite, earth-center). If available, this should be used to compute viewing angles etc. Otherwise, use the actual satellite position.projection_longitude/latitude/altitude
: Projection center of the re-projected data. This should be used to compute lat/lon coordinates. Note that the projection center can differ considerably from the actual satellite position. For example MSG-1 was at times positioned at 3.4 degrees west, while the image data was re-projected to 0 degrees.[DEPRECATED]
satellite_longitude/latitude/altitude
: Current position of the satellite at the time of observation in geodetic coordinates.
Note
Longitudes and latitudes are given in degrees, altitude in meters. For use in pyorbital, the altitude has to be converted to kilometers, see for example
pyorbital.orbital.get_observer_look()
.For polar orbiting satellites the readers usually provide coordinates and viewing angles of the swath as ancillary datasets. Additional metadata related to the satellite position include:
tle
: Two-Line Element (TLE) set used to compute the satellite’s orbit
raw_metadata
: Raw, unprocessed metadata from the reader.
Note that the above attributes are not necessarily available for each dataset.
Coordinates¶
Each DataArray
produced by Satpy has several Xarray
coordinate variables added to them.
x
andy
: Projection coordinates for gridded and projected data. By default y and x are the preferred dimensions for all 2D data, but these coordinates are only added for gridded (non-swath) data. For 1D data only they
dimension may be specified.crs
: ACRS
object defined the Coordinate Reference System for the data. Requires pyproj 2.0 or later to be installed. This is stored as a scalar array by Xarray so it must be accessed by doingcrs = my_data_arr.attrs['crs'].item()
. For swath data this defaults to alonglat
CRS using the WGS84 datum.longitude
: Array of longitude coordinates for swath data.latitude
: Array of latitude coordinates for swath data.
Readers are free to define any coordinates in addition to the ones above that are automatically added. Other possible coordinates you may see:
acq_time
: Instrument data acquisition time per scan or row of data.
Adding a Reader to Satpy¶
This is described in the developer guide, see Adding a Custom Reader to Satpy.
Implemented readers¶
xRIT-based readers¶
HRIT/LRIT format reader.
This module is the base module for all HRIT-based formats. Here, you will find the common building blocks for hrit reading.
One of the features here is the on-the-fly decompression of hrit files. It needs a path to the xRITDecompress binary to be provided through the environment variable called XRIT_DECOMPRESS_PATH. When compressed hrit files are then encountered (files finishing with .C_), they are decompressed to the system’s temporary directory for reading.
SEVIRI HRIT format reader¶
SEVIRI HRIT format reader.
Introduction¶
The seviri_l1b_hrit
reader reads and calibrates MSG-SEVIRI L1.5 image data in HRIT format. The format is explained
in the MSG Level 1.5 Image Format Description. The files are usually named as
follows:
H-000-MSG4__-MSG4________-_________-PRO______-201903011200-__
H-000-MSG4__-MSG4________-IR_108___-000001___-201903011200-__
H-000-MSG4__-MSG4________-IR_108___-000002___-201903011200-__
H-000-MSG4__-MSG4________-IR_108___-000003___-201903011200-__
H-000-MSG4__-MSG4________-IR_108___-000004___-201903011200-__
H-000-MSG4__-MSG4________-IR_108___-000005___-201903011200-__
H-000-MSG4__-MSG4________-IR_108___-000006___-201903011200-__
H-000-MSG4__-MSG4________-IR_108___-000007___-201903011200-__
H-000-MSG4__-MSG4________-IR_108___-000008___-201903011200-__
H-000-MSG4__-MSG4________-_________-EPI______-201903011200-__
Each image is decomposed into 24 segments (files) for the high-resolution-visible (HRV) channel and 8 segments for other visible (VIS) and infrared (IR) channels. Additionally there is one prologue and one epilogue file for the entire scan which contain global metadata valid for all channels.
Reader Arguments¶
Some arguments can be provided to the reader to change it’s behaviour. These are provided through the Scene instantiation, eg:
Scene(reader="seviri_l1b_hrit", filenames=fnames, reader_kwargs={'fill_hrv': False})
To see the full list of arguments that can be provided, look into the documentation
of HRITMSGFileHandler
.
Example
Here is an example how to read the data in satpy:
from satpy import Scene
import glob
filenames = glob.glob('data/H-000-MSG4__-MSG4________-*201903011200*')
scn = Scene(filenames=filenames, reader='seviri_l1b_hrit')
scn.load(['VIS006', 'IR_108'])
print(scn['IR_108'])
Output:
<xarray.DataArray (y: 3712, x: 3712)>
dask.array<shape=(3712, 3712), dtype=float32, chunksize=(464, 3712)>
Coordinates:
acq_time (y) datetime64[ns] NaT NaT NaT NaT NaT NaT ... NaT NaT NaT NaT NaT
* x (x) float64 5.566e+06 5.563e+06 5.56e+06 ... -5.566e+06 -5.569e+06
* y (y) float64 -5.566e+06 -5.563e+06 ... 5.566e+06 5.569e+06
Attributes:
satellite_longitude: 0.0
satellite_latitude: 0.0
satellite_altitude: 35785831.0
orbital_parameters: {'projection_longitude': 0.0, 'projection_latit...
platform_name: Meteosat-11
georef_offset_corrected: True
standard_name: brightness_temperature
raw_metadata: {'file_type': 0, 'total_header_length': 6198, '...
wavelength: (9.8, 10.8, 11.8)
units: K
sensor: seviri
platform_name: Meteosat-11
start_time: 2019-03-01 12:00:09.716000
end_time: 2019-03-01 12:12:42.946000
area: Area ID: some_area_name\\nDescription: On-the-fl...
name: IR_108
resolution: 3000.403165817
calibration: brightness_temperature
polarization: None
level: None
modifiers: ()
ancillary_variables: []
The
orbital_parameters
attribute provides the nominal and actual satellite position, as well as the projection centre.You can choose between nominal and GSICS calibration coefficients or even specify your own coefficients, see
HRITMSGFileHandler
.The
raw_metadata
attribute provides raw metadata from the prologue, epilogue and segment header. By default, arrays with more than 100 elements are excluded in order to limit memory usage. This threshold can be adjusted, seeHRITMSGFileHandler
.The
acq_time
coordinate provides the acquisition time for each scanline. Use aMultiIndex
to enable selection by acquisition time:import pandas as pd mi = pd.MultiIndex.from_arrays([scn['IR_108']['y'].data, scn['IR_108']['acq_time'].data], names=('y_coord', 'time')) scn['IR_108']['y'] = mi scn['IR_108'].sel(time=np.datetime64('2019-03-01T12:06:13.052000000'))
Notes
When loading solar channels, this reader applies a correction for the Sun-Earth distance variation throughout the year - as recommended by the EUMETSAT document:
‘Conversion from radiances to reflectances for SEVIRI warm channels’
In the unlikely situation that this correction is not required, it can be removed on a per-channel basis using the satpy.readers.utils.remove_earthsun_distance_correction(channel, utc_time) function.
References
JMA HRIT format reader¶
HRIT format reader for JMA data.
Introduction¶
The JMA HRIT format is described in the JMA HRIT - Mission Specific Implementation. There are three readers for this format in Satpy:
jami_hrit
: For data from the JAMI instrument on MTSAT-1Rmtsat2-imager_hrit
: For data from the Imager instrument on MTSAT-2ahi_hrit
: For data from the AHI instrument on Himawari-8/9
Although the data format is identical, the instruments have different characteristics, which is why there is a dedicated reader for each of them. Sample data is available here:
Example
Here is an example how to read Himwari-8 HRIT data with Satpy:
from satpy import Scene
import glob
filenames = glob.glob('data/IMG_DK01B14_2018011109*')
scn = Scene(filenames=filenames, reader='ahi_hrit')
scn.load(['B14'])
print(scn['B14'])
Output:
<xarray.DataArray (y: 5500, x: 5500)>
dask.array<concatenate, shape=(5500, 5500), dtype=float64, chunksize=(550, 4096), ...
Coordinates:
acq_time (y) datetime64[ns] 2018-01-11T09:00:20.995200 ... 2018-01-11T09:09:40.348800
crs object +proj=geos +lon_0=140.7 +h=35785831 +x_0=0 +y_0=0 +a=6378169 ...
* y (y) float64 5.5e+06 5.498e+06 5.496e+06 ... -5.496e+06 -5.498e+06
* x (x) float64 -5.498e+06 -5.496e+06 -5.494e+06 ... 5.498e+06 5.5e+06
Attributes:
satellite_longitude: 140.7
satellite_latitude: 0.0
satellite_altitude: 35785831.0
orbital_parameters: {'projection_longitude': 140.7, 'projection_latitud...
standard_name: toa_brightness_temperature
level: None
wavelength: (11.0, 11.2, 11.4)
units: K
calibration: brightness_temperature
file_type: ['hrit_b14_seg', 'hrit_b14_fd']
modifiers: ()
polarization: None
sensor: ahi
name: B14
platform_name: Himawari-8
resolution: 4000
start_time: 2018-01-11 09:00:20.995200
end_time: 2018-01-11 09:09:40.348800
area: Area ID: FLDK, Description: Full Disk, Projection I...
ancillary_variables: []
JMA HRIT data contain the scanline acquisition time for only a subset of scanlines. Timestamps of
the remaining scanlines are computed using linear interpolation. This is what you’ll find in the
acq_time
coordinate of the dataset.
GOES HRIT format reader¶
GOES HRIT format reader.
References
LRIT/HRIT Mission Specific Implementation, February 2012 GVARRDL98.pdf 05057_SPE_MSG_LRIT_HRI
Electro-L HRIT format reader¶
HRIT format reader.
References
- ELECTRO-L GROUND SEGMENT MSU-GS INSTRUMENT,
LRIT/HRIT Mission Specific Implementation, February 2012
hdf-eos based readers¶
Modis level 1b hdf-eos format reader.
Introduction¶
The modis_l1b
reader reads and calibrates Modis L1 image data in hdf-eos format. Files often have
a pattern similar to the following one:
M[O/Y]D02[1/H/Q]KM.A[date].[time].[collection].[processing_time].hdf
Other patterns where “collection” and/or “proccessing_time” are missing might also work (see the readers yaml file for details). Geolocation files (MOD03) are also supported.
Geolocation files¶
For the 1km data (mod021km) geolocation files (mod03) are optional. If not given to the reader 1km geolocations will be interpolated from the 5km geolocation contained within the file.
For the 500m and 250m data geolocation files are needed.
References
Modis gelocation description: http://www.icare.univ-lille1.fr/wiki/index.php/MODIS_geolocation
Modis level 2 hdf-eos format reader.
Introduction¶
The modis_l2
reader reads and calibrates Modis L2 image data in hdf-eos format.
Since there are a multitude of different level 2 datasets not all of theses are implemented (yet).
- Currently the reader supports:
m[o/y]d35_l2: cloud_mask dataset
some datasets in m[o/y]d06 files
To get a list of the available datasets for a given file refer to the “Load data” section in Readers.
Geolocation files¶
Similar to the modis_l1b
reader the geolocation files (mod03) for the 1km data are optional and if not
given 1km geolocations will be interpolated from the 5km geolocation contained within the file.
For the 500m and 250m data geolocation files are needed.
References
Documentation about the format: https://modis-atmos.gsfc.nasa.gov/products