:py:mod:`mlair.helpers.data_sources.data_loader`
================================================

.. py:module:: mlair.helpers.data_sources.data_loader


Module Contents
---------------


Functions
~~~~~~~~~

.. autoapisummary::

   mlair.helpers.data_sources.data_loader.download_data
   mlair.helpers.data_sources.data_loader.get_data_with_query
   mlair.helpers.data_sources.data_loader.get_data
   mlair.helpers.data_sources.data_loader.correct_stat_name
   mlair.helpers.data_sources.data_loader.create_url
   mlair.helpers.data_sources.data_loader.retries_session



Attributes
~~~~~~~~~~

.. autoapisummary::

   mlair.helpers.data_sources.data_loader.__author__
   mlair.helpers.data_sources.data_loader.__date__
   mlair.helpers.data_sources.data_loader.DEFAULT_TIME_DIM
   mlair.helpers.data_sources.data_loader.DEFAULT_TARGET_DIM
   mlair.helpers.data_sources.data_loader.DEFAULT_ITER_DIM
   mlair.helpers.data_sources.data_loader.DEFAULT_WINDOW_DIM


.. py:data:: __author__
   :annotation: = Lukas Leufen

   

.. py:data:: __date__
   :annotation: = 2023-06-01

   

.. py:data:: DEFAULT_TIME_DIM
   :annotation: = datetime

   

.. py:data:: DEFAULT_TARGET_DIM
   :annotation: = variables

   

.. py:data:: DEFAULT_ITER_DIM
   :annotation: = Stations

   

.. py:data:: DEFAULT_WINDOW_DIM
   :annotation: = window

   

.. py:function:: download_data(file_name: str, meta_file: str, station, statistics_per_var, sampling, store_data_locally=True, data_origin: Dict = None, time_dim=DEFAULT_TIME_DIM, target_dim=DEFAULT_TARGET_DIM, iter_dim=DEFAULT_ITER_DIM, window_dim=DEFAULT_WINDOW_DIM, era5_data_path=None, era5_file_names=None, ifs_data_path=None, ifs_file_names=None) -> [xarray.DataArray, pandas.DataFrame]

   Download data from TOAR database using the JOIN interface or load local era5 data.

   Data is transformed to a xarray dataset. If class attribute store_data_locally is true, data is additionally
   stored locally using given names for file and meta file.

   :param file_name: name of file to save data to (containing full path)
   :param meta_file: name of the meta data file (also containing full path)

   :return: downloaded data and its meta data


.. py:exception:: EmptyQueryResult

   Bases: :py:obj:`Exception`

   Exception that get raised if a query to JOIN returns empty results.


.. py:function:: get_data_with_query(opts: Dict, headers: Dict, as_json: bool = True, max_retries=5, timeout_base=60) -> bytes

   Download data from statistics rest api. This API is based on three steps: (1) post query and retrieve job id, (2)
   read status of id until finished, (3) download data with job id.


.. py:function:: get_data(opts: Dict, headers: Dict, as_json: bool = True, max_retries=5, timeout_base=60) -> Union[Dict, List, str]

   Download join data using requests framework.

   Data is returned as json like structure. Depending on the response structure, this can lead to a list or dictionary.

   :param opts: options to create the request url
   :param headers: additional headers information like authorization, can be empty
   :param as_json: extract response as json if true (default True)

   :return: requested data (either as list or dictionary)


.. py:function:: correct_stat_name(stat: str) -> str

   Map given statistic name to new namespace defined by mapping dict.

   Return given name stat if not element of mapping namespace.

   :param stat: namespace from JOIN server

   :return: stat mapped to local namespace


.. py:function:: create_url(base: str, service: str, param_id: Union[str, int, None] = None, **kwargs: Union[str, int, float, None]) -> str

   Create a request url with given base url, service type and arbitrarily many additional keyword arguments.

   :param base: basic url of the rest service
   :param service: service type, e.g. series, stats
   :param param_id: id for a distinct service, is added between ending / of service and ? of kwargs
   :param kwargs: keyword pairs for optional request specifications, e.g. 'statistics=maximum'

   :return: combined url as string


.. py:function:: retries_session(max_retries=5)