report.html

-------------------------------Captured log call--------------------------------
DEBUG root:path_config.py:125 Created path: data/test DEBUG root:path_config.py:132 Path already exists: data/test

------------------------------Captured stdout call------------------------------
Model: "model" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_1 (InputLayer) [(None, 14, 1, 2)] 0 _________________________________________________________________ flatten (Flatten) (None, 28) 0 _________________________________________________________________ dense (Dense) (None, 64) 1856 _________________________________________________________________ prelu_1 (PReLU) (None, 64) 64 _________________________________________________________________ dense_1 (Dense) (None, 32) 2080 _________________________________________________________________ prelu_2 (PReLU) (None, 32) 32 _________________________________________________________________ dense_2 (Dense) (None, 16) 528 _________________________________________________________________ prelu_3 (PReLU) (None, 16) 16 _________________________________________________________________ dense_3 (Dense) (None, 3) 51 _________________________________________________________________ linear_output (Activation) (None, 3) 0 ================================================================= Total params: 4,627 Trainable params: 4,627 Non-trainable params: 0 _________________________________________________________________ None
------------------------------Captured stderr call------------------------------
2023-02-15 15:55:29.078988: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory 2023-02-15 15:55:29.079031: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303) 2023-02-15 15:55:29.079072: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:163] no NVIDIA GPU device is present: /dev/nvidia0 does not exist 2023-02-15 15:55:29.079339: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

-------------------------------Captured log call--------------------------------
INFO root:time_tracking.py:137 undefined job finished after 0:00:01 (hh:mm:ss)

-------------------------------Captured log call--------------------------------
INFO root:time_tracking.py:137 my job finished after 0:00:01 (hh:mm:ss)

------------------------------Captured stderr call------------------------------
2023-02-15 16:00:00,794 - INFO: Could not find a valid match for variable no2 and networks []! Therefore, use first answer from JOIN: {'id': 16686, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'no2', 'parameter_label': 'NO2', 'parameter_attribute': ''} [join.py:_select_distinct_network:288] 2023-02-15 16:00:00,794 - INFO: Could not find a valid match for variable no2 and networks []! Therefore, use first answer from JOIN: {'id': 16686, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'no2', 'parameter_label': 'NO2', 'parameter_attribute': ''} [join.py:_select_distinct_network:288] 2023-02-15 16:00:00,794 - INFO: Could not find a valid match for variable no2 and networks []! Therefore, use first answer from JOIN: {'id': 16686, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'no2', 'parameter_label': 'NO2', 'parameter_attribute': ''} [join.py:_select_distinct_network:288] 2023-02-15 16:00:00,794 - INFO: Could not find a valid match for variable no2 and networks []! Therefore, use first answer from JOIN: {'id': 16686, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'no2', 'parameter_label': 'NO2', 'parameter_attribute': ''} [join.py:_select_distinct_network:288] 2023-02-15 16:00:00,794 - INFO: Could not find a valid match for variable o3 and networks []! Therefore, use first answer from JOIN: {'id': 16687, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'o3', 'parameter_label': 'O3', 'parameter_attribute': ''} [join.py:_select_distinct_network:288] 2023-02-15 16:00:00,794 - INFO: Could not find a valid match for variable o3 and networks []! Therefore, use first answer from JOIN: {'id': 16687, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'o3', 'parameter_label': 'O3', 'parameter_attribute': ''} [join.py:_select_distinct_network:288] 2023-02-15 16:00:00,794 - INFO: Could not find a valid match for variable o3 and networks []! Therefore, use first answer from JOIN: {'id': 16687, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'o3', 'parameter_label': 'O3', 'parameter_attribute': ''} [join.py:_select_distinct_network:288] 2023-02-15 16:00:00,794 - INFO: Could not find a valid match for variable o3 and networks []! Therefore, use first answer from JOIN: {'id': 16687, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'o3', 'parameter_label': 'O3', 'parameter_attribute': ''} [join.py:_select_distinct_network:288] 2023-02-15 16:00:00,795 - INFO: Could not find a valid match for variable cloudcover and networks []! Therefore, use first answer from JOIN: {'id': 54036, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'cloudcover', 'parameter_label': 'CLOUDCOVER', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288] 2023-02-15 16:00:00,795 - INFO: Could not find a valid match for variable cloudcover and networks []! Therefore, use first answer from JOIN: {'id': 54036, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'cloudcover', 'parameter_label': 'CLOUDCOVER', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288] 2023-02-15 16:00:00,795 - INFO: Could not find a valid match for variable cloudcover and networks []! Therefore, use first answer from JOIN: {'id': 54036, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'cloudcover', 'parameter_label': 'CLOUDCOVER', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288] 2023-02-15 16:00:00,795 - INFO: Could not find a valid match for variable cloudcover and networks []! Therefore, use first answer from JOIN: {'id': 54036, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'cloudcover', 'parameter_label': 'CLOUDCOVER', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288] 2023-02-15 16:00:00,795 - INFO: Could not find a valid match for variable temp and networks []! Therefore, use first answer from JOIN: {'id': 88491, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'temp', 'parameter_label': 'TEMP-REA-MIUB', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288] 2023-02-15 16:00:00,795 - INFO: Could not find a valid match for variable temp and networks []! Therefore, use first answer from JOIN: {'id': 88491, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'temp', 'parameter_label': 'TEMP-REA-MIUB', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288] 2023-02-15 16:00:00,795 - INFO: Could not find a valid match for variable temp and networks []! Therefore, use first answer from JOIN: {'id': 88491, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'temp', 'parameter_label': 'TEMP-REA-MIUB', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288] 2023-02-15 16:00:00,795 - INFO: Could not find a valid match for variable temp and networks []! Therefore, use first answer from JOIN: {'id': 88491, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'temp', 'parameter_label': 'TEMP-REA-MIUB', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288] 2023-02-15 16:00:00,795 - INFO: Could not find a valid match for variable press and networks []! Therefore, use first answer from JOIN: {'id': 102660, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'press', 'parameter_label': 'PRESS-REA-MIUB', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288] 2023-02-15 16:00:00,795 - INFO: Could not find a valid match for variable press and networks []! Therefore, use first answer from JOIN: {'id': 102660, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'press', 'parameter_label': 'PRESS-REA-MIUB', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288] 2023-02-15 16:00:00,795 - INFO: Could not find a valid match for variable press and networks []! Therefore, use first answer from JOIN: {'id': 102660, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'press', 'parameter_label': 'PRESS-REA-MIUB', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288] 2023-02-15 16:00:00,795 - INFO: Could not find a valid match for variable press and networks []! Therefore, use first answer from JOIN: {'id': 102660, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'press', 'parameter_label': 'PRESS-REA-MIUB', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288]
-------------------------------Captured log call--------------------------------
INFO root:join.py:288 Could not find a valid match for variable no2 and networks []! Therefore, use first answer from JOIN: {'id': 16686, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'no2', 'parameter_label': 'NO2', 'parameter_attribute': ''} INFO root:join.py:288 Could not find a valid match for variable o3 and networks []! Therefore, use first answer from JOIN: {'id': 16687, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'o3', 'parameter_label': 'O3', 'parameter_attribute': ''} INFO root:join.py:288 Could not find a valid match for variable cloudcover and networks []! Therefore, use first answer from JOIN: {'id': 54036, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'cloudcover', 'parameter_label': 'CLOUDCOVER', 'parameter_attribute': 'REA'} INFO root:join.py:288 Could not find a valid match for variable temp and networks []! Therefore, use first answer from JOIN: {'id': 88491, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'temp', 'parameter_label': 'TEMP-REA-MIUB', 'parameter_attribute': 'REA'} INFO root:join.py:288 Could not find a valid match for variable press and networks []! Therefore, use first answer from JOIN: {'id': 102660, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'press', 'parameter_label': 'PRESS-REA-MIUB', 'parameter_attribute': 'REA'}

------------------------------Captured stdout call------------------------------
In test_get_padding_for_same_negative_kernel_size

------------------------------Captured stdout call------------------------------
Epoch 1/5 1/1 [==============================] - ETA: 0s - loss: 4.8112 1/1 [==============================] - 0s 368ms/step - loss: 4.8112 Epoch 2/5 1/1 [==============================] - ETA: 0s - loss: 4.6374 1/1 [==============================] - 0s 2ms/step - loss: 4.6374 Epoch 3/5 1/1 [==============================] - ETA: 0s - loss: 4.4669 1/1 [==============================] - 0s 3ms/step - loss: 4.4669 Epoch 4/5 1/1 [==============================] - ETA: 0s - loss: 4.3080 1/1 [==============================] - 0s 3ms/step - loss: 4.3080 Epoch 5/5 1/1 [==============================] - ETA: 0s - loss: 4.1523 1/1 [==============================] - 0s 2ms/step - loss: 4.1523
------------------------------Captured stderr call------------------------------
2023-02-15 16:00:02.100022: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)

------------------------------Captured stdout call------------------------------
Epoch 00001: val_loss did not improve from 6.00000 Epoch 00010: val_loss did not improve from 6.00000 Epoch 00011: val_loss improved from 6.00000 to 4.00000, saving model to ckpt.test Epoch 00011: save to /builds/esde/machine-learning/mlair/test/test_model_modules/callback_lr Epoch 00011: save to /builds/esde/machine-learning/mlair/test/test_model_modules/callback_hist Epoch 00011: saving model to ckpt.test Epoch 00011: save to /builds/esde/machine-learning/mlair/test/test_model_modules/callback_lr Epoch 00011: save to /builds/esde/machine-learning/mlair/test/test_model_modules/callback_hist

------------------------------Captured stdout call------------------------------
1/1 [==============================] - ETA: 0s - loss: 1.2500 1/1 [==============================] - 0s 86ms/step - loss: 1.2500 1/1 [==============================] - ETA: 0s - loss: 2.2500 1/1 [==============================] - 0s 84ms/step - loss: 2.2500

------------------------------Captured stdout call------------------------------
1/1 [==============================] - ETA: 0s - loss: 0.1406 1/1 [==============================] - 0s 197ms/step - loss: 0.1406

------------------------------Captured stdout call------------------------------
1/1 [==============================] - ETA: 0s - loss: 0.6953 1/1 [==============================] - 0s 168ms/step - loss: 0.6953
------------------------------Captured stderr call------------------------------
2023-02-15 16:00:03,347 - WARNING: 5 out of the last 9 calls to <function Model.make_train_function.<locals>.train_function at 0x7fdcd607c3a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [def_function.py:called_with_tracing:147] 2023-02-15 16:00:03,347 - WARNING: 5 out of the last 9 calls to <function Model.make_train_function.<locals>.train_function at 0x7fdcd607c3a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [def_function.py:called_with_tracing:147] 2023-02-15 16:00:03,347 - WARNING: 5 out of the last 9 calls to <function Model.make_train_function.<locals>.train_function at 0x7fdcd607c3a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [def_function.py:called_with_tracing:147] 2023-02-15 16:00:03,347 - WARNING: 5 out of the last 9 calls to <function Model.make_train_function.<locals>.train_function at 0x7fdcd607c3a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [def_function.py:called_with_tracing:147]
-------------------------------Captured log call--------------------------------
WARNING tensorflow:def_function.py:147 5 out of the last 9 calls to <function Model.make_train_function.<locals>.train_function at 0x7fdcd607c3a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.

------------------------------Captured stdout call------------------------------
1/1 [==============================] - ETA: 0s - loss: 0.4734 1/1 [==============================] - 0s 133ms/step - loss: 0.4734
------------------------------Captured stderr call------------------------------
2023-02-15 16:00:03,544 - WARNING: 6 out of the last 10 calls to <function Model.make_train_function.<locals>.train_function at 0x7fdcd607ca60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [def_function.py:called_with_tracing:147] 2023-02-15 16:00:03,544 - WARNING: 6 out of the last 10 calls to <function Model.make_train_function.<locals>.train_function at 0x7fdcd607ca60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [def_function.py:called_with_tracing:147] 2023-02-15 16:00:03,544 - WARNING: 6 out of the last 10 calls to <function Model.make_train_function.<locals>.train_function at 0x7fdcd607ca60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [def_function.py:called_with_tracing:147] 2023-02-15 16:00:03,544 - WARNING: 6 out of the last 10 calls to <function Model.make_train_function.<locals>.train_function at 0x7fdcd607ca60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [def_function.py:called_with_tracing:147]
-------------------------------Captured log call--------------------------------
WARNING tensorflow:def_function.py:147 6 out of the last 10 calls to <function Model.make_train_function.<locals>.train_function at 0x7fdcd607ca60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.

------------------------------Captured stdout call------------------------------
1/1 [==============================] - ETA: 0s - loss: 0.6953 1/1 [==============================] - 0s 118ms/step - loss: 0.6953

------------------------------Captured stdout call------------------------------
1/1 [==============================] - ETA: 0s - loss: 0.6953 1/1 [==============================] - 0s 123ms/step - loss: 0.6953

------------------------------Captured stdout call------------------------------
1/1 [==============================] - ETA: 0s - loss: 1.0281 1/1 [==============================] - 0s 156ms/step - loss: 1.0281

-------------------------------Captured log call--------------------------------
DEBUG root:datastore.py:118 set: 23tester(general)=23 DEBUG root:experiment_setup.py:460 set experiment attribute: 23tester(general)=23 DEBUG root:datastore.py:118 get: 23tester(general)=23

-------------------------------Captured log call--------------------------------
DEBUG root:datastore.py:118 set: 109tester(general)=109 DEBUG root:experiment_setup.py:460 set experiment attribute: 109tester(general)=109 DEBUG root:datastore.py:118 get: 109tester(general.tester)=109

-------------------------------Captured log call--------------------------------
DEBUG root:datastore.py:118 set: NoneTester(general.testing)=notNone DEBUG root:experiment_setup.py:460 set experiment attribute: NoneTester(general.testing)=notNone DEBUG root:datastore.py:118 get: NoneTester(general.testing)=notNone DEBUG root:datastore.py:118 set: AnotherNoneTester(general)=None DEBUG root:experiment_setup.py:460 set experiment attribute: AnotherNoneTester(general)=None DEBUG root:datastore.py:118 get: AnotherNoneTester(general)=None

-------------------------------Captured log call--------------------------------
DEBUG root:datastore.py:118 set: NoneTester(general)=notNone DEBUG root:experiment_setup.py:460 set experiment attribute: NoneTester(general)=notNone DEBUG root:datastore.py:118 get: NoneTester(general)=notNone DEBUG root:datastore.py:118 set: NoneTester(general)=['notNone'] DEBUG root:experiment_setup.py:460 set experiment attribute: NoneTester(general)=['notNone'] DEBUG root:datastore.py:118 get: NoneTester(general)=['notNone'] DEBUG root:datastore.py:118 set: NoneTester(general)=[None] DEBUG root:experiment_setup.py:460 set experiment attribute: NoneTester(general)=[None] DEBUG root:datastore.py:118 get: NoneTester(general)=[None] DEBUG root:datastore.py:118 set: NoneTester(general)=2 DEBUG root:experiment_setup.py:460 set experiment attribute: NoneTester(general)=2 DEBUG root:datastore.py:118 get: NoneTester(general)=2

------------------------------Captured stdout call------------------------------
Model: "model_7" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_32 (InputLayer) [(None, 14, 1, 5)] 0 _________________________________________________________________ flatten_1 (Flatten) (None, 70) 0 _________________________________________________________________ dense_5 (Dense) (None, 64) 4544 _________________________________________________________________ prelu_1 (PReLU) (None, 64) 64 _________________________________________________________________ dense_6 (Dense) (None, 32) 2080 _________________________________________________________________ prelu_2 (PReLU) (None, 32) 32 _________________________________________________________________ dense_7 (Dense) (None, 16) 528 _________________________________________________________________ prelu_3 (PReLU) (None, 16) 16 _________________________________________________________________ dense_8 (Dense) (None, 5) 85 _________________________________________________________________ linear_output (Activation) (None, 5) 0 ================================================================= Total params: 7,349 Trainable params: 7,349 Non-trainable params: 0 _________________________________________________________________ None

------------------------------Captured stderr call------------------------------
2023-02-15 16:00:21,749 - INFO: PartitionCheck started [run_environment.py:__init__:103] 2023-02-15 16:00:21,749 - INFO: PartitionCheck started [run_environment.py:__init__:103] 2023-02-15 16:00:21,749 - INFO: PartitionCheck started [run_environment.py:__init__:103] 2023-02-15 16:00:21,749 - INFO: PartitionCheck started [run_environment.py:__init__:103] 2023-02-15 16:00:21,749 - INFO: PartitionCheck started [run_environment.py:__init__:103] 2023-02-15 16:00:21,750 - INFO: PartitionCheck finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:21,750 - INFO: PartitionCheck finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:21,750 - INFO: PartitionCheck finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:21,750 - INFO: PartitionCheck finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:21,750 - INFO: PartitionCheck finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:21,750 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:00:21,750 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:00:21,750 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:00:21,750 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:00:21,750 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:00:21,750 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:21,750 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:21,750 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:21,750 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:21,750 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:23,608 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json [run_environment.py:__save_tracking:159] 2023-02-15 16:00:23,608 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json [run_environment.py:__save_tracking:159] 2023-02-15 16:00:23,608 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json [run_environment.py:__save_tracking:159] 2023-02-15 16:00:23,608 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json [run_environment.py:__save_tracking:159] 2023-02-15 16:00:23,608 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json [run_environment.py:__save_tracking:159] 2023-02-15 16:00:23,627 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:00:23,627 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:00:23,627 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:00:23,627 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:00:23,627 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147]
-------------------------------Captured log call--------------------------------
INFO root:run_environment.py:103 PartitionCheck started INFO root:run_environment.py:120 PartitionCheck finished after 0:00:01 (hh:mm:ss) INFO root:run_environment.py:103 RunEnvironment started INFO root:run_environment.py:120 RunEnvironment finished after 0:00:01 (hh:mm:ss) INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log

------------------------------Captured stderr call------------------------------
2023-02-15 16:00:27,513 - INFO: PreProcessing started [run_environment.py:__init__:103] 2023-02-15 16:00:27,513 - INFO: PreProcessing started [run_environment.py:__init__:103] 2023-02-15 16:00:27,513 - INFO: PreProcessing started [run_environment.py:__init__:103] 2023-02-15 16:00:27,513 - INFO: PreProcessing started [run_environment.py:__init__:103] 2023-02-15 16:00:27,513 - INFO: PreProcessing started [run_environment.py:__init__:103] 2023-02-15 16:00:27,513 - INFO: check valid stations started (preprocessing) [pre_processing.py:validate_station:261] 2023-02-15 16:00:27,513 - INFO: check valid stations started (preprocessing) [pre_processing.py:validate_station:261] 2023-02-15 16:00:27,513 - INFO: check valid stations started (preprocessing) [pre_processing.py:validate_station:261] 2023-02-15 16:00:27,513 - INFO: check valid stations started (preprocessing) [pre_processing.py:validate_station:261] 2023-02-15 16:00:27,513 - INFO: check valid stations started (preprocessing) [pre_processing.py:validate_station:261] 2023-02-15 16:00:27,518 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:27,518 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:27,518 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:27,518 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:27,518 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:27,914 - INFO: load data for DEBW107 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2023-02-15 16:00:27,914 - INFO: load data for DEBW107 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2023-02-15 16:00:27,914 - INFO: load data for DEBW107 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2023-02-15 16:00:27,914 - INFO: load data for DEBW107 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2023-02-15 16:00:27,914 - INFO: load data for DEBW107 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2023-02-15 16:00:27,994 - INFO: load data for DEBW107 from JOIN [join.py:download_join:63] 2023-02-15 16:00:27,994 - INFO: load data for DEBW107 from JOIN [join.py:download_join:63] 2023-02-15 16:00:27,994 - INFO: load data for DEBW107 from JOIN [join.py:download_join:63] 2023-02-15 16:00:27,994 - INFO: load data for DEBW107 from JOIN [join.py:download_join:63] 2023-02-15 16:00:27,994 - INFO: load data for DEBW107 from JOIN [join.py:download_join:63] 2023-02-15 16:00:30,733 - INFO: setup_samples finished after 0:00:04 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:30,733 - INFO: setup_samples finished after 0:00:04 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:30,733 - INFO: setup_samples finished after 0:00:04 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:30,733 - INFO: setup_samples finished after 0:00:04 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:30,733 - INFO: setup_samples finished after 0:00:04 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:31,006 - INFO: PartitionCheck finished after 0:00:06 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:31,006 - INFO: PartitionCheck finished after 0:00:06 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:31,006 - INFO: PartitionCheck finished after 0:00:06 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:31,006 - INFO: PartitionCheck finished after 0:00:06 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:31,006 - INFO: PartitionCheck finished after 0:00:06 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:31,017 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:31,017 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:31,017 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:31,017 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:31,017 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:31,017 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:31,017 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:31,017 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:31,017 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:31,017 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:31,617 - INFO: load data for DEBW013 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2023-02-15 16:00:31,617 - INFO: load data for DEBW013 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2023-02-15 16:00:31,617 - INFO: load data for DEBW013 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2023-02-15 16:00:31,617 - INFO: load data for DEBW013 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2023-02-15 16:00:31,617 - INFO: load data for DEBW013 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2023-02-15 16:00:31,688 - INFO: load data for DEBW013 from JOIN [join.py:download_join:63] 2023-02-15 16:00:31,688 - INFO: load data for DEBW013 from JOIN [join.py:download_join:63] 2023-02-15 16:00:31,688 - INFO: load data for DEBW013 from JOIN [join.py:download_join:63] 2023-02-15 16:00:31,688 - INFO: load data for DEBW013 from JOIN [join.py:download_join:63] 2023-02-15 16:00:31,688 - INFO: load data for DEBW013 from JOIN [join.py:download_join:63] 2023-02-15 16:00:35,566 - INFO: setup_samples finished after 0:00:05 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:35,566 - INFO: setup_samples finished after 0:00:05 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:35,566 - INFO: setup_samples finished after 0:00:05 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:35,566 - INFO: setup_samples finished after 0:00:05 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:35,566 - INFO: setup_samples finished after 0:00:05 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:35,890 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:35,890 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:35,890 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:35,890 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:35,890 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:35,891 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:35,891 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:35,891 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:35,891 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:35,891 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:36,552 - INFO: load data for DEBW087 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2023-02-15 16:00:36,552 - INFO: load data for DEBW087 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2023-02-15 16:00:36,552 - INFO: load data for DEBW087 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2023-02-15 16:00:36,552 - INFO: load data for DEBW087 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2023-02-15 16:00:36,552 - INFO: load data for DEBW087 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2023-02-15 16:00:36,628 - INFO: load data for DEBW087 from JOIN [join.py:download_join:63] 2023-02-15 16:00:36,628 - INFO: load data for DEBW087 from JOIN [join.py:download_join:63] 2023-02-15 16:00:36,628 - INFO: load data for DEBW087 from JOIN [join.py:download_join:63] 2023-02-15 16:00:36,628 - INFO: load data for DEBW087 from JOIN [join.py:download_join:63] 2023-02-15 16:00:36,628 - INFO: load data for DEBW087 from JOIN [join.py:download_join:63] 2023-02-15 16:00:40,154 - INFO: setup_samples finished after 0:00:05 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:40,154 - INFO: setup_samples finished after 0:00:05 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:40,154 - INFO: setup_samples finished after 0:00:05 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:40,154 - INFO: setup_samples finished after 0:00:05 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:40,154 - INFO: setup_samples finished after 0:00:05 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:40,468 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:40,468 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:40,468 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:40,468 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:40,468 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:40,469 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:40,469 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:40,469 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:40,469 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:40,469 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:40,737 - INFO: run for 0:00:14 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (preprocessing). [pre_processing.py:validate_station:303] 2023-02-15 16:00:40,737 - INFO: run for 0:00:14 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (preprocessing). [pre_processing.py:validate_station:303] 2023-02-15 16:00:40,737 - INFO: run for 0:00:14 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (preprocessing). [pre_processing.py:validate_station:303] 2023-02-15 16:00:40,737 - INFO: run for 0:00:14 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (preprocessing). [pre_processing.py:validate_station:303] 2023-02-15 16:00:40,737 - INFO: run for 0:00:14 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (preprocessing). [pre_processing.py:validate_station:303] 2023-02-15 16:00:40,738 - INFO: check valid stations started (train) [pre_processing.py:validate_station:261] 2023-02-15 16:00:40,738 - INFO: check valid stations started (train) [pre_processing.py:validate_station:261] 2023-02-15 16:00:40,738 - INFO: check valid stations started (train) [pre_processing.py:validate_station:261] 2023-02-15 16:00:40,738 - INFO: check valid stations started (train) [pre_processing.py:validate_station:261] 2023-02-15 16:00:40,738 - INFO: check valid stations started (train) [pre_processing.py:validate_station:261] 2023-02-15 16:00:40,738 - INFO: setup transformation using train data exclusively [pre_processing.py:validate_station:264] 2023-02-15 16:00:40,738 - INFO: setup transformation using train data exclusively [pre_processing.py:validate_station:264] 2023-02-15 16:00:40,738 - INFO: setup transformation using train data exclusively [pre_processing.py:validate_station:264] 2023-02-15 16:00:40,738 - INFO: setup transformation using train data exclusively [pre_processing.py:validate_station:264] 2023-02-15 16:00:40,738 - INFO: setup transformation using train data exclusively [pre_processing.py:validate_station:264] 2023-02-15 16:00:40,738 - INFO: start to calculate transformation parameters. [pre_processing.py:transformation:327] 2023-02-15 16:00:40,738 - INFO: start to calculate transformation parameters. [pre_processing.py:transformation:327] 2023-02-15 16:00:40,738 - INFO: start to calculate transformation parameters. [pre_processing.py:transformation:327] 2023-02-15 16:00:40,738 - INFO: start to calculate transformation parameters. [pre_processing.py:transformation:327] 2023-02-15 16:00:40,738 - INFO: start to calculate transformation parameters. [pre_processing.py:transformation:327] 2023-02-15 16:00:40,743 - INFO: use serial transformation approach [default_data_handler.py:transformation:322] 2023-02-15 16:00:40,743 - INFO: use serial transformation approach [default_data_handler.py:transformation:322] 2023-02-15 16:00:40,743 - INFO: use serial transformation approach [default_data_handler.py:transformation:322] 2023-02-15 16:00:40,743 - INFO: use serial transformation approach [default_data_handler.py:transformation:322] 2023-02-15 16:00:40,743 - INFO: use serial transformation approach [default_data_handler.py:transformation:322] 2023-02-15 16:00:40,810 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:40,810 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:40,810 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:40,810 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:40,810 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:41,120 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:41,120 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:41,120 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:41,120 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:41,120 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:41,466 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:41,466 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:41,466 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:41,466 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:41,466 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:41,730 - INFO: Store transformation options locally for later use at: /builds/esde/machine-learning/mlair/TestExperiment_daily/data/transformation/transformation.pickle [pre_processing.py:_store_transformation:360] 2023-02-15 16:00:41,730 - INFO: Store transformation options locally for later use at: /builds/esde/machine-learning/mlair/TestExperiment_daily/data/transformation/transformation.pickle [pre_processing.py:_store_transformation:360] 2023-02-15 16:00:41,730 - INFO: Store transformation options locally for later use at: /builds/esde/machine-learning/mlair/TestExperiment_daily/data/transformation/transformation.pickle [pre_processing.py:_store_transformation:360] 2023-02-15 16:00:41,730 - INFO: Store transformation options locally for later use at: /builds/esde/machine-learning/mlair/TestExperiment_daily/data/transformation/transformation.pickle [pre_processing.py:_store_transformation:360] 2023-02-15 16:00:41,730 - INFO: Store transformation options locally for later use at: /builds/esde/machine-learning/mlair/TestExperiment_daily/data/transformation/transformation.pickle [pre_processing.py:_store_transformation:360] 2023-02-15 16:00:41,737 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:41,737 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:41,737 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:41,737 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:41,737 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:41,799 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:41,799 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:41,799 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:41,799 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:41,799 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,037 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,037 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,037 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,037 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,037 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,037 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,037 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,037 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,037 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,037 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,369 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,369 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,369 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,369 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,369 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,603 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,603 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,603 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,603 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,603 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,603 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,603 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,603 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,603 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,603 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,905 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,905 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,905 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,905 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:42,905 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,175 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,175 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,175 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,175 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,175 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,176 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,176 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,176 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,176 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,176 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,411 - INFO: run for 0:00:03 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train). [pre_processing.py:validate_station:303] 2023-02-15 16:00:43,411 - INFO: run for 0:00:03 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train). [pre_processing.py:validate_station:303] 2023-02-15 16:00:43,411 - INFO: run for 0:00:03 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train). [pre_processing.py:validate_station:303] 2023-02-15 16:00:43,411 - INFO: run for 0:00:03 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train). [pre_processing.py:validate_station:303] 2023-02-15 16:00:43,411 - INFO: run for 0:00:03 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train). [pre_processing.py:validate_station:303] 2023-02-15 16:00:43,412 - INFO: check valid stations started (val) [pre_processing.py:validate_station:261] 2023-02-15 16:00:43,412 - INFO: check valid stations started (val) [pre_processing.py:validate_station:261] 2023-02-15 16:00:43,412 - INFO: check valid stations started (val) [pre_processing.py:validate_station:261] 2023-02-15 16:00:43,412 - INFO: check valid stations started (val) [pre_processing.py:validate_station:261] 2023-02-15 16:00:43,412 - INFO: check valid stations started (val) [pre_processing.py:validate_station:261] 2023-02-15 16:00:43,419 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:43,419 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:43,419 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:43,419 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:43,419 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:43,477 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,477 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,477 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,477 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,477 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,477 - INFO: remove station DEBW107 because it raised an error: self.history consist of NaNs only - station ['DEBW107'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:462] 2023-02-15 16:00:43,477 - INFO: remove station DEBW107 because it raised an error: self.history consist of NaNs only - station ['DEBW107'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:462] 2023-02-15 16:00:43,477 - INFO: remove station DEBW107 because it raised an error: self.history consist of NaNs only - station ['DEBW107'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:462] 2023-02-15 16:00:43,477 - INFO: remove station DEBW107 because it raised an error: self.history consist of NaNs only - station ['DEBW107'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:462] 2023-02-15 16:00:43,477 - INFO: remove station DEBW107 because it raised an error: self.history consist of NaNs only - station ['DEBW107'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:462] 2023-02-15 16:00:43,536 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,536 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,536 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,536 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,536 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,536 - INFO: remove station DEBW013 because it raised an error: self.history consist of NaNs only - station ['DEBW013'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:462] 2023-02-15 16:00:43,536 - INFO: remove station DEBW013 because it raised an error: self.history consist of NaNs only - station ['DEBW013'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:462] 2023-02-15 16:00:43,536 - INFO: remove station DEBW013 because it raised an error: self.history consist of NaNs only - station ['DEBW013'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:462] 2023-02-15 16:00:43,536 - INFO: remove station DEBW013 because it raised an error: self.history consist of NaNs only - station ['DEBW013'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:462] 2023-02-15 16:00:43,536 - INFO: remove station DEBW013 because it raised an error: self.history consist of NaNs only - station ['DEBW013'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:462] 2023-02-15 16:00:43,593 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,593 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,593 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,593 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,593 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,593 - INFO: remove station DEBW087 because it raised an error: self.history consist of NaNs only - station ['DEBW087'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:462] 2023-02-15 16:00:43,593 - INFO: remove station DEBW087 because it raised an error: self.history consist of NaNs only - station ['DEBW087'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:462] 2023-02-15 16:00:43,593 - INFO: remove station DEBW087 because it raised an error: self.history consist of NaNs only - station ['DEBW087'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:462] 2023-02-15 16:00:43,593 - INFO: remove station DEBW087 because it raised an error: self.history consist of NaNs only - station ['DEBW087'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:462] 2023-02-15 16:00:43,593 - INFO: remove station DEBW087 because it raised an error: self.history consist of NaNs only - station ['DEBW087'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:462] 2023-02-15 16:00:43,593 - INFO: run for 0:00:01 (hh:mm:ss) to check 3 station(s). Found 0/3 valid stations (val). [pre_processing.py:validate_station:303] 2023-02-15 16:00:43,593 - INFO: run for 0:00:01 (hh:mm:ss) to check 3 station(s). Found 0/3 valid stations (val). [pre_processing.py:validate_station:303] 2023-02-15 16:00:43,593 - INFO: run for 0:00:01 (hh:mm:ss) to check 3 station(s). Found 0/3 valid stations (val). [pre_processing.py:validate_station:303] 2023-02-15 16:00:43,593 - INFO: run for 0:00:01 (hh:mm:ss) to check 3 station(s). Found 0/3 valid stations (val). [pre_processing.py:validate_station:303] 2023-02-15 16:00:43,593 - INFO: run for 0:00:01 (hh:mm:ss) to check 3 station(s). Found 0/3 valid stations (val). [pre_processing.py:validate_station:303] 2023-02-15 16:00:43,594 - INFO: check valid stations started (test) [pre_processing.py:validate_station:261] 2023-02-15 16:00:43,594 - INFO: check valid stations started (test) [pre_processing.py:validate_station:261] 2023-02-15 16:00:43,594 - INFO: check valid stations started (test) [pre_processing.py:validate_station:261] 2023-02-15 16:00:43,594 - INFO: check valid stations started (test) [pre_processing.py:validate_station:261] 2023-02-15 16:00:43,594 - INFO: check valid stations started (test) [pre_processing.py:validate_station:261] 2023-02-15 16:00:43,601 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:43,601 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:43,601 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:43,601 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:43,601 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:43,663 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,663 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,663 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,663 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,663 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,909 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,909 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,909 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,909 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,909 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,910 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,910 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,910 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,910 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:43,910 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:44,213 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:44,213 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:44,213 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:44,213 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:44,213 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:44,447 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:44,447 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:44,447 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:44,447 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:44,447 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:44,447 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:44,447 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:44,447 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:44,447 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:44,447 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:44,758 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:44,758 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:44,758 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:44,758 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:44,758 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,012 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,012 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,012 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,012 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,012 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,013 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,013 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,013 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,013 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,013 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,248 - INFO: run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (test). [pre_processing.py:validate_station:303] 2023-02-15 16:00:45,248 - INFO: run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (test). [pre_processing.py:validate_station:303] 2023-02-15 16:00:45,248 - INFO: run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (test). [pre_processing.py:validate_station:303] 2023-02-15 16:00:45,248 - INFO: run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (test). [pre_processing.py:validate_station:303] 2023-02-15 16:00:45,248 - INFO: run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (test). [pre_processing.py:validate_station:303] 2023-02-15 16:00:45,249 - INFO: check valid stations started (train_val) [pre_processing.py:validate_station:261] 2023-02-15 16:00:45,249 - INFO: check valid stations started (train_val) [pre_processing.py:validate_station:261] 2023-02-15 16:00:45,249 - INFO: check valid stations started (train_val) [pre_processing.py:validate_station:261] 2023-02-15 16:00:45,249 - INFO: check valid stations started (train_val) [pre_processing.py:validate_station:261] 2023-02-15 16:00:45,249 - INFO: check valid stations started (train_val) [pre_processing.py:validate_station:261] 2023-02-15 16:00:45,256 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:45,256 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:45,256 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:45,256 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:45,256 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:00:45,319 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,319 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,319 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,319 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,319 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,554 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,554 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,554 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,554 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,554 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,554 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,554 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,554 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,554 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,554 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,856 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,856 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,856 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,856 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:45,856 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,118 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,118 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,118 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,118 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,118 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,118 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,118 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,118 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,118 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,118 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,431 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,431 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,431 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,431 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,431 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,666 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,666 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,666 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,666 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,666 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,666 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,666 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,666 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,666 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,666 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:00:46,903 - INFO: run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train_val). [pre_processing.py:validate_station:303] 2023-02-15 16:00:46,903 - INFO: run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train_val). [pre_processing.py:validate_station:303] 2023-02-15 16:00:46,903 - INFO: run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train_val). [pre_processing.py:validate_station:303] 2023-02-15 16:00:46,903 - INFO: run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train_val). [pre_processing.py:validate_station:303] 2023-02-15 16:00:46,903 - INFO: run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train_val). [pre_processing.py:validate_station:303] 2023-02-15 16:00:46,906 - INFO: use serial create_info_df (train) [pre_processing.py:create_info_df:179] 2023-02-15 16:00:46,906 - INFO: use serial create_info_df (train) [pre_processing.py:create_info_df:179] 2023-02-15 16:00:46,906 - INFO: use serial create_info_df (train) [pre_processing.py:create_info_df:179] 2023-02-15 16:00:46,906 - INFO: use serial create_info_df (train) [pre_processing.py:create_info_df:179] 2023-02-15 16:00:46,906 - INFO: use serial create_info_df (train) [pre_processing.py:create_info_df:179] 2023-02-15 16:00:47,621 - INFO: use serial create_info_df (val) [pre_processing.py:create_info_df:179] 2023-02-15 16:00:47,621 - INFO: use serial create_info_df (val) [pre_processing.py:create_info_df:179] 2023-02-15 16:00:47,621 - INFO: use serial create_info_df (val) [pre_processing.py:create_info_df:179] 2023-02-15 16:00:47,621 - INFO: use serial create_info_df (val) [pre_processing.py:create_info_df:179] 2023-02-15 16:00:47,621 - INFO: use serial create_info_df (val) [pre_processing.py:create_info_df:179] 2023-02-15 16:00:47,622 - INFO: use serial create_info_df (test) [pre_processing.py:create_info_df:179] 2023-02-15 16:00:47,622 - INFO: use serial create_info_df (test) [pre_processing.py:create_info_df:179] 2023-02-15 16:00:47,622 - INFO: use serial create_info_df (test) [pre_processing.py:create_info_df:179] 2023-02-15 16:00:47,622 - INFO: use serial create_info_df (test) [pre_processing.py:create_info_df:179] 2023-02-15 16:00:47,622 - INFO: use serial create_info_df (test) [pre_processing.py:create_info_df:179] 2023-02-15 16:00:48,407 - INFO: Searching for competitors to be prepared for use. [pre_processing.py:prepare_competitors:369] 2023-02-15 16:00:48,407 - INFO: Searching for competitors to be prepared for use. [pre_processing.py:prepare_competitors:369] 2023-02-15 16:00:48,407 - INFO: Searching for competitors to be prepared for use. [pre_processing.py:prepare_competitors:369] 2023-02-15 16:00:48,407 - INFO: Searching for competitors to be prepared for use. [pre_processing.py:prepare_competitors:369] 2023-02-15 16:00:48,407 - INFO: Searching for competitors to be prepared for use. [pre_processing.py:prepare_competitors:369] 2023-02-15 16:00:48,407 - INFO: No preparation required for competitor ols as no specific instruction is provided. [pre_processing.py:prepare_competitors:399] 2023-02-15 16:00:48,407 - INFO: No preparation required for competitor ols as no specific instruction is provided. [pre_processing.py:prepare_competitors:399] 2023-02-15 16:00:48,407 - INFO: No preparation required for competitor ols as no specific instruction is provided. [pre_processing.py:prepare_competitors:399] 2023-02-15 16:00:48,407 - INFO: No preparation required for competitor ols as no specific instruction is provided. [pre_processing.py:prepare_competitors:399] 2023-02-15 16:00:48,407 - INFO: No preparation required for competitor ols as no specific instruction is provided. [pre_processing.py:prepare_competitors:399] 2023-02-15 16:00:48,407 - INFO: No preparation required for competitor persi as no specific instruction is provided. [pre_processing.py:prepare_competitors:399] 2023-02-15 16:00:48,407 - INFO: No preparation required for competitor persi as no specific instruction is provided. [pre_processing.py:prepare_competitors:399] 2023-02-15 16:00:48,407 - INFO: No preparation required for competitor persi as no specific instruction is provided. [pre_processing.py:prepare_competitors:399] 2023-02-15 16:00:48,407 - INFO: No preparation required for competitor persi as no specific instruction is provided. [pre_processing.py:prepare_competitors:399] 2023-02-15 16:00:48,407 - INFO: No preparation required for competitor persi as no specific instruction is provided. [pre_processing.py:prepare_competitors:399] 2023-02-15 16:00:48,409 - INFO: PreProcessing finished after 0:00:21 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:48,409 - INFO: PreProcessing finished after 0:00:21 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:48,409 - INFO: PreProcessing finished after 0:00:21 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:48,409 - INFO: PreProcessing finished after 0:00:21 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:48,409 - INFO: PreProcessing finished after 0:00:21 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:48,409 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:00:48,409 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:00:48,409 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:00:48,409 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:00:48,409 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:00:48,409 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:48,409 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:48,409 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:48,409 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:48,409 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:00:54,293 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json [run_environment.py:__save_tracking:159] 2023-02-15 16:00:54,293 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json [run_environment.py:__save_tracking:159] 2023-02-15 16:00:54,293 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json [run_environment.py:__save_tracking:159] 2023-02-15 16:00:54,293 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json [run_environment.py:__save_tracking:159] 2023-02-15 16:00:54,293 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json [run_environment.py:__save_tracking:159] 2023-02-15 16:00:54,317 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:00:54,317 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:00:54,317 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:00:54,317 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:00:54,317 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147]
-------------------------------Captured log call--------------------------------
INFO root:run_environment.py:103 PreProcessing started INFO root:pre_processing.py:261 check valid stations started (preprocessing) INFO root:pre_processing.py:295 use serial validate station approach INFO root:toar_data_v2.py:63 load data for DEBW107 from TOAR-DATA INFO root:join.py:63 load data for DEBW107 from JOIN INFO root:time_tracking.py:137 setup_samples finished after 0:00:04 (hh:mm:ss) INFO root:run_environment.py:120 PartitionCheck finished after 0:00:06 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:toar_data_v2.py:63 load data for DEBW013 from TOAR-DATA INFO root:join.py:63 load data for DEBW013 from JOIN INFO root:time_tracking.py:137 setup_samples finished after 0:00:05 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:toar_data_v2.py:63 load data for DEBW087 from TOAR-DATA INFO root:join.py:63 load data for DEBW087 from JOIN INFO root:time_tracking.py:137 setup_samples finished after 0:00:05 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:pre_processing.py:303 run for 0:00:14 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (preprocessing). INFO root:pre_processing.py:261 check valid stations started (train) INFO root:pre_processing.py:264 setup transformation using train data exclusively INFO root:pre_processing.py:327 start to calculate transformation parameters. INFO root:default_data_handler.py:322 use serial transformation approach INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:pre_processing.py:360 Store transformation options locally for later use at: /builds/esde/machine-learning/mlair/TestExperiment_daily/data/transformation/transformation.pickle INFO root:pre_processing.py:295 use serial validate station approach INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:pre_processing.py:303 run for 0:00:03 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train). INFO root:pre_processing.py:261 check valid stations started (val) INFO root:pre_processing.py:295 use serial validate station approach INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:pre_processing.py:462 remove station DEBW107 because it raised an error: self.history consist of NaNs only - station ['DEBW107'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:pre_processing.py:462 remove station DEBW013 because it raised an error: self.history consist of NaNs only - station ['DEBW013'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:pre_processing.py:462 remove station DEBW087 because it raised an error: self.history consist of NaNs only - station ['DEBW087'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') INFO root:pre_processing.py:303 run for 0:00:01 (hh:mm:ss) to check 3 station(s). Found 0/3 valid stations (val). INFO root:pre_processing.py:261 check valid stations started (test) INFO root:pre_processing.py:295 use serial validate station approach INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:pre_processing.py:303 run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (test). INFO root:pre_processing.py:261 check valid stations started (train_val) INFO root:pre_processing.py:295 use serial validate station approach INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:pre_processing.py:303 run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train_val). INFO root:pre_processing.py:179 use serial create_info_df (train) INFO root:pre_processing.py:179 use serial create_info_df (val) INFO root:pre_processing.py:179 use serial create_info_df (test) INFO root:pre_processing.py:369 Searching for competitors to be prepared for use. INFO root:pre_processing.py:399 No preparation required for competitor ols as no specific instruction is provided. INFO root:pre_processing.py:399 No preparation required for competitor persi as no specific instruction is provided. INFO root:run_environment.py:120 PreProcessing finished after 0:00:21 (hh:mm:ss) INFO root:run_environment.py:103 RunEnvironment started INFO root:run_environment.py:120 RunEnvironment finished after 0:00:01 (hh:mm:ss) INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log

------------------------------Captured stderr call------------------------------
2023-02-15 16:01:24,698 - INFO: check valid stations started (awesome) [pre_processing.py:validate_station:261] 2023-02-15 16:01:24,698 - INFO: check valid stations started (awesome) [pre_processing.py:validate_station:261] 2023-02-15 16:01:24,698 - INFO: check valid stations started (awesome) [pre_processing.py:validate_station:261] 2023-02-15 16:01:24,698 - INFO: check valid stations started (awesome) [pre_processing.py:validate_station:261] 2023-02-15 16:01:24,698 - INFO: check valid stations started (awesome) [pre_processing.py:validate_station:261] 2023-02-15 16:01:24,705 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:01:24,705 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:01:24,705 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:01:24,705 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:01:24,705 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:01:24,779 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:24,779 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:24,779 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:24,779 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:24,779 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:25,208 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:25,208 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:25,208 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:25,208 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:25,208 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:25,209 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:25,209 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:25,209 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:25,209 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:25,209 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:25,673 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:25,673 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:25,673 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:25,673 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:25,673 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:26,025 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:26,025 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:26,025 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:26,025 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:26,025 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:26,026 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:26,026 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:26,026 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:26,026 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:26,026 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:26,396 - INFO: run for 0:00:02 (hh:mm:ss) to check 2 station(s). Found 2/2 valid stations (awesome). [pre_processing.py:validate_station:303] 2023-02-15 16:01:26,396 - INFO: run for 0:00:02 (hh:mm:ss) to check 2 station(s). Found 2/2 valid stations (awesome). [pre_processing.py:validate_station:303] 2023-02-15 16:01:26,396 - INFO: run for 0:00:02 (hh:mm:ss) to check 2 station(s). Found 2/2 valid stations (awesome). [pre_processing.py:validate_station:303] 2023-02-15 16:01:26,396 - INFO: run for 0:00:02 (hh:mm:ss) to check 2 station(s). Found 2/2 valid stations (awesome). [pre_processing.py:validate_station:303] 2023-02-15 16:01:26,396 - INFO: run for 0:00:02 (hh:mm:ss) to check 2 station(s). Found 2/2 valid stations (awesome). [pre_processing.py:validate_station:303]
-------------------------------Captured log call--------------------------------
DEBUG root:datastore.py:118 set: use_all_stations_on_all_data_sets(general)=False DEBUG root:datastore.py:118 get: stations(general.awesome)=['DEBW107', 'DEBW013', 'DEBW087', 'DEBW99X'] DEBUG root:datastore.py:118 get: use_all_stations_on_all_data_sets(general)=False DEBUG root:pre_processing.py:243 Awesome stations (len=2): ['DEBW107', 'DEBW013'] DEBUG root:datastore.py:118 get: data_handler(general)=<class 'mlair.data_handler.default_data_handler.DefaultDataHandler'> INFO root:pre_processing.py:261 check valid stations started (awesome) DEBUG root:datastore.py:118 get: target_dim(general.awesome)=variables DEBUG root:datastore.py:118 get: max_number_multiprocessing(general.awesome)=16 DEBUG root:datastore.py:118 get: statistics_per_var(general.awesome)={'o3': 'dma8eu', 'temp': 'maximum'} DEBUG root:datastore.py:118 get: variables(general.awesome)=['o3', 'temp'] DEBUG root:datastore.py:118 get: time_dim(general.awesome)=datetime DEBUG root:datastore.py:118 get: start(general.awesome)=1997-01-01 DEBUG root:datastore.py:118 get: use_multiprocessing(general.awesome)=True DEBUG root:datastore.py:120 get: window_history_offset(general.awesome)=None DEBUG root:datastore.py:120 get: min_length(general.awesome)=None DEBUG root:datastore.py:118 get: data_origin(general.awesome)={'o3': 'UBA', 'temp': 'UBA'} DEBUG root:datastore.py:120 get: store_processed_data(general.awesome)=None DEBUG root:datastore.py:118 get: end(general.awesome)=2017-12-31 DEBUG root:datastore.py:120 get: store_data_locally(general.awesome)=None DEBUG root:datastore.py:118 get: interpolation_method(general.awesome)=linear DEBUG root:datastore.py:118 get: target_var(general.awesome)=o3 DEBUG root:datastore.py:118 get: iter_dim(general.awesome)=Stations DEBUG root:datastore.py:118 get: window_dim(general.awesome)=window DEBUG root:datastore.py:118 get: data_path(general.awesome)=/home/root/mlair/data/ DEBUG root:datastore.py:120 get: lazy_preprocessing(general.awesome)=None DEBUG root:datastore.py:118 get: transformation(general.awesome)={} DEBUG root:datastore.py:118 get: window_lead_time(general.awesome)=3 DEBUG root:datastore.py:118 get: sampling(general.awesome)=daily DEBUG root:datastore.py:120 get: extremes_on_right_tail_only(general.awesome)=None DEBUG root:datastore.py:120 get: window_history_end(general.awesome)=None DEBUG root:datastore.py:118 get: interpolation_limit(general.awesome)=1 DEBUG root:datastore.py:118 get: experiment_path(general.awesome)=/builds/esde/machine-learning/mlair/TestExperiment_daily DEBUG root:datastore.py:120 get: name_affix(general.awesome)=None DEBUG root:datastore.py:120 get: overwrite_lazy_data(general.awesome)=None DEBUG root:datastore.py:120 get: extreme_values(general.awesome)=None DEBUG root:datastore.py:120 get: overwrite_local_data(general.awesome)=None DEBUG root:datastore.py:118 get: window_history_size(general.awesome)=13 DEBUG root:datastore.py:118 get: use_multiprocessing(general)=True DEBUG root:datastore.py:118 get: tmp_path(general)=/builds/esde/machine-learning/mlair/TestExperiment_daily/tmp DEBUG root:datastore.py:118 get: max_number_multiprocessing(general)=16 INFO root:pre_processing.py:295 use serial validate station approach DEBUG root:path_config.py:132 Path already exists: /home/root/mlair/data/daily DEBUG root:data_handler_single_station.py:345 try to load local data from: /home/root/mlair/data/daily/DEBW107_o3_temp.nc DEBUG root:data_handler_single_station.py:349 loading finished INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) DEBUG root:default_data_handler.py:218 No extreme values given, skip multiply extremes INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) DEBUG root:default_data_handler.py:97 save pickle data to /builds/esde/machine-learning/mlair/TestExperiment_daily/data/DEBW107_awesome.pickle DEBUG root:path_config.py:132 Path already exists: /home/root/mlair/data/daily DEBUG root:data_handler_single_station.py:345 try to load local data from: /home/root/mlair/data/daily/DEBW013_o3_temp.nc DEBUG root:data_handler_single_station.py:349 loading finished INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) DEBUG root:default_data_handler.py:218 No extreme values given, skip multiply extremes INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) DEBUG root:default_data_handler.py:97 save pickle data to /builds/esde/machine-learning/mlair/TestExperiment_daily/data/DEBW013_awesome.pickle INFO root:pre_processing.py:303 run for 0:00:02 (hh:mm:ss) to check 2 station(s). Found 2/2 valid stations (awesome). DEBUG root:datastore.py:118 set: stations(general.awesome)=['DEBW107', 'DEBW013'] DEBUG root:datastore.py:118 set: data_collection(general.awesome)=<mlair.data_handler.iterator.DataCollection object at 0x7fda5c4fd3a0> DEBUG root:datastore.py:118 get: data_collection(general.awesome)=<mlair.data_handler.iterator.DataCollection object at 0x7fda5c4fd3a0> DEBUG root:datastore.py:120 get: data_collection(general)=None DEBUG root:datastore.py:118 get: stations(general.awesome)=['DEBW107', 'DEBW013']
----------------------------Captured stderr teardown----------------------------
2023-02-15 16:01:26,398 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:26,398 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:26,398 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:26,398 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:26,398 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:26,399 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:26,399 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:26,399 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:26,399 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:26,399 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:30,484 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_003.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:30,484 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_003.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:30,484 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_003.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:30,484 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_003.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:30,484 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_003.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:30,535 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:01:30,535 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:01:30,535 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:01:30,535 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:01:30,535 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147]
-----------------------------Captured log teardown------------------------------
INFO root:run_environment.py:103 RunEnvironment started INFO root:run_environment.py:120 RunEnvironment finished after 0:00:01 (hh:mm:ss) DEBUG root:datastore.py:118 get: logging_path(general)=/builds/esde/machine-learning/mlair/TestExperiment_daily/logging DEBUG root:datastore.py:118 get: logging_path(general)=/builds/esde/machine-learning/mlair/TestExperiment_daily/logging INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_003.json DEBUG root:datastore.py:118 get: logging_path(general)=/builds/esde/machine-learning/mlair/TestExperiment_daily/logging INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log

------------------------------Captured stderr call------------------------------
2023-02-15 16:01:30,549 - INFO: check valid stations started (awesome) [pre_processing.py:validate_station:261] 2023-02-15 16:01:30,549 - INFO: check valid stations started (awesome) [pre_processing.py:validate_station:261] 2023-02-15 16:01:30,549 - INFO: check valid stations started (awesome) [pre_processing.py:validate_station:261] 2023-02-15 16:01:30,549 - INFO: check valid stations started (awesome) [pre_processing.py:validate_station:261] 2023-02-15 16:01:30,549 - INFO: check valid stations started (awesome) [pre_processing.py:validate_station:261] 2023-02-15 16:01:30,558 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:01:30,558 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:01:30,558 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:01:30,558 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:01:30,558 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:01:30,639 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:30,639 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:30,639 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:30,639 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:30,639 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,030 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,030 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,030 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,030 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,030 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,030 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,030 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,030 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,030 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,030 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,496 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,496 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,496 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,496 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,496 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,865 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,865 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,865 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,865 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,865 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,866 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,866 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,866 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,866 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:31,866 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:32,329 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:32,329 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:32,329 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:32,329 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:32,329 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:32,703 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:32,703 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:32,703 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:32,703 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:32,703 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:32,703 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:32,703 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:32,703 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:32,703 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:32,703 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:33,224 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:33,224 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:33,224 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:33,224 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:33,224 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:33,227 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:462] 2023-02-15 16:01:33,227 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:462] 2023-02-15 16:01:33,227 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:462] 2023-02-15 16:01:33,227 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:462] 2023-02-15 16:01:33,227 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:462] 2023-02-15 16:01:33,229 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (awesome). [pre_processing.py:validate_station:303] 2023-02-15 16:01:33,229 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (awesome). [pre_processing.py:validate_station:303] 2023-02-15 16:01:33,229 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (awesome). [pre_processing.py:validate_station:303] 2023-02-15 16:01:33,229 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (awesome). [pre_processing.py:validate_station:303] 2023-02-15 16:01:33,229 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (awesome). [pre_processing.py:validate_station:303]
-------------------------------Captured log call--------------------------------
DEBUG root:datastore.py:118 get: stations(general.awesome)=['DEBW107', 'DEBW013', 'DEBW087', 'DEBW99X'] DEBUG root:datastore.py:118 get: use_all_stations_on_all_data_sets(general)=True DEBUG root:pre_processing.py:243 Awesome stations (len=4): ['DEBW107', 'DEBW013', 'DEBW087', 'DEBW99X'] DEBUG root:datastore.py:118 get: data_handler(general)=<class 'mlair.data_handler.default_data_handler.DefaultDataHandler'> INFO root:pre_processing.py:261 check valid stations started (awesome) DEBUG root:datastore.py:118 get: target_dim(general.awesome)=variables DEBUG root:datastore.py:118 get: max_number_multiprocessing(general.awesome)=16 DEBUG root:datastore.py:118 get: statistics_per_var(general.awesome)={'o3': 'dma8eu', 'temp': 'maximum'} DEBUG root:datastore.py:118 get: variables(general.awesome)=['o3', 'temp'] DEBUG root:datastore.py:118 get: time_dim(general.awesome)=datetime DEBUG root:datastore.py:118 get: start(general.awesome)=1997-01-01 DEBUG root:datastore.py:118 get: use_multiprocessing(general.awesome)=True DEBUG root:datastore.py:120 get: window_history_offset(general.awesome)=None DEBUG root:datastore.py:120 get: min_length(general.awesome)=None DEBUG root:datastore.py:118 get: data_origin(general.awesome)={'o3': 'UBA', 'temp': 'UBA'} DEBUG root:datastore.py:120 get: store_processed_data(general.awesome)=None DEBUG root:datastore.py:118 get: end(general.awesome)=2017-12-31 DEBUG root:datastore.py:120 get: store_data_locally(general.awesome)=None DEBUG root:datastore.py:118 get: interpolation_method(general.awesome)=linear DEBUG root:datastore.py:118 get: target_var(general.awesome)=o3 DEBUG root:datastore.py:118 get: iter_dim(general.awesome)=Stations DEBUG root:datastore.py:118 get: window_dim(general.awesome)=window DEBUG root:datastore.py:118 get: data_path(general.awesome)=/home/root/mlair/data/ DEBUG root:datastore.py:120 get: lazy_preprocessing(general.awesome)=None DEBUG root:datastore.py:118 get: transformation(general.awesome)={} DEBUG root:datastore.py:118 get: window_lead_time(general.awesome)=3 DEBUG root:datastore.py:118 get: sampling(general.awesome)=daily DEBUG root:datastore.py:120 get: extremes_on_right_tail_only(general.awesome)=None DEBUG root:datastore.py:120 get: window_history_end(general.awesome)=None DEBUG root:datastore.py:118 get: interpolation_limit(general.awesome)=1 DEBUG root:datastore.py:118 get: experiment_path(general.awesome)=/builds/esde/machine-learning/mlair/TestExperiment_daily DEBUG root:datastore.py:120 get: name_affix(general.awesome)=None DEBUG root:datastore.py:120 get: overwrite_lazy_data(general.awesome)=None DEBUG root:datastore.py:120 get: extreme_values(general.awesome)=None DEBUG root:datastore.py:120 get: overwrite_local_data(general.awesome)=None DEBUG root:datastore.py:118 get: window_history_size(general.awesome)=13 DEBUG root:datastore.py:118 get: use_multiprocessing(general)=True DEBUG root:datastore.py:118 get: tmp_path(general)=/builds/esde/machine-learning/mlair/TestExperiment_daily/tmp DEBUG root:datastore.py:118 get: max_number_multiprocessing(general)=16 INFO root:pre_processing.py:295 use serial validate station approach DEBUG root:path_config.py:132 Path already exists: /home/root/mlair/data/daily DEBUG root:data_handler_single_station.py:345 try to load local data from: /home/root/mlair/data/daily/DEBW107_o3_temp.nc DEBUG root:data_handler_single_station.py:349 loading finished INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) DEBUG root:default_data_handler.py:218 No extreme values given, skip multiply extremes INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) DEBUG root:default_data_handler.py:97 save pickle data to /builds/esde/machine-learning/mlair/TestExperiment_daily/data/DEBW107_awesome.pickle DEBUG root:path_config.py:132 Path already exists: /home/root/mlair/data/daily DEBUG root:data_handler_single_station.py:345 try to load local data from: /home/root/mlair/data/daily/DEBW013_o3_temp.nc DEBUG root:data_handler_single_station.py:349 loading finished INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) DEBUG root:default_data_handler.py:218 No extreme values given, skip multiply extremes INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) DEBUG root:default_data_handler.py:97 save pickle data to /builds/esde/machine-learning/mlair/TestExperiment_daily/data/DEBW013_awesome.pickle DEBUG root:path_config.py:132 Path already exists: /home/root/mlair/data/daily DEBUG root:data_handler_single_station.py:345 try to load local data from: /home/root/mlair/data/daily/DEBW087_o3_temp.nc DEBUG root:data_handler_single_station.py:349 loading finished INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) DEBUG root:default_data_handler.py:218 No extreme values given, skip multiply extremes INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) DEBUG root:default_data_handler.py:97 save pickle data to /builds/esde/machine-learning/mlair/TestExperiment_daily/data/DEBW087_awesome.pickle DEBUG root:path_config.py:132 Path already exists: /home/root/mlair/data/daily DEBUG root:data_handler_single_station.py:345 try to load local data from: /home/root/mlair/data/daily/DEBW99X_o3_temp.nc DEBUG root:data_handler_single_station.py:351 [Errno 2] No such file or directory: b'/home/root/mlair/data/daily/DEBW99X_o3_temp.nc' DEBUG root:data_handler_single_station.py:352 load new data DEBUG urllib3.connectionpool:connectionpool.py:1003 Starting new HTTPS connection (1): toar-data.fz-juelich.de:443 DEBUG urllib3.connectionpool:connectionpool.py:456 https://toar-data.fz-juelich.de:443 "GET /api/v2/variables/o3 HTTP/1.1" 200 156 DEBUG urllib3.connectionpool:connectionpool.py:1003 Starting new HTTPS connection (1): toar-data.fz-juelich.de:443 DEBUG urllib3.connectionpool:connectionpool.py:456 https://toar-data.fz-juelich.de:443 "GET /api/v2/variables/temp HTTP/1.1" 200 161 DEBUG urllib3.connectionpool:connectionpool.py:1003 Starting new HTTPS connection (1): toar-data.fz-juelich.de:443 DEBUG urllib3.connectionpool:connectionpool.py:456 https://toar-data.fz-juelich.de:443 "GET /api/v2/stationmeta/DEBW99X HTTP/1.1" 404 54 DEBUG urllib3.connectionpool:connectionpool.py:1003 Starting new HTTPS connection (1): join.fz-juelich.de:443 DEBUG urllib3.connectionpool:connectionpool.py:456 https://join.fz-juelich.de:443 "GET /services/rest/surfacedata/search/?station_id=DEBW99X&network_name=UBA&as_dict=true&parameter_name=o3,temp&columns=id,network_name,station_id,parameter_name,parameter_label,parameter_attribute HTTP/1.1" 200 2 DEBUG root:join.py:218 ['DEBW99X']: [] INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:pre_processing.py:462 remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") DEBUG root:pre_processing.py:464 detailed information for removal of station DEBW99X: Traceback (most recent call last): File "/opt/venv/lib64/python3.9/site-packages/xarray/backends/file_manager.py", line 199, in _acquire_with_cache_info file = self._cache[self._key] File "/opt/venv/lib64/python3.9/site-packages/xarray/backends/lru_cache.py", line 53, in __getitem__ value = self._cache[key] KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/home/root/mlair/data/daily/DEBW99X_o3_temp.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False))] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 346, in load_data data = xr.open_dataarray(file_name) File "/opt/venv/lib64/python3.9/site-packages/xarray/backends/api.py", line 701, in open_dataarray dataset = open_dataset( File "/opt/venv/lib64/python3.9/site-packages/xarray/backends/api.py", line 572, in open_dataset store = opener(filename_or_obj, **extra_kwargs, **backend_kwargs) File "/opt/venv/lib64/python3.9/site-packages/xarray/backends/netCDF4_.py", line 364, in open return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose) File "/opt/venv/lib64/python3.9/site-packages/xarray/backends/netCDF4_.py", line 314, in __init__ self.format = self.ds.data_model File "/opt/venv/lib64/python3.9/site-packages/xarray/backends/netCDF4_.py", line 373, in ds return self._acquire() File "/opt/venv/lib64/python3.9/site-packages/xarray/backends/netCDF4_.py", line 367, in _acquire with self._manager.acquire_context(needs_lock) as root: File "/usr/lib64/python3.9/contextlib.py", line 119, in __enter__ return next(self.gen) File "/opt/venv/lib64/python3.9/site-packages/xarray/backends/file_manager.py", line 187, in acquire_context file, cached = self._acquire_with_cache_info(needs_lock) File "/opt/venv/lib64/python3.9/site-packages/xarray/backends/file_manager.py", line 205, in _acquire_with_cache_info file = self._opener(*self._args, **kwargs) File "src/netCDF4/_netCDF4.pyx", line 2353, in netCDF4._netCDF4.Dataset.__init__ File "src/netCDF4/_netCDF4.pyx", line 1963, in netCDF4._netCDF4._ensure_nc_success FileNotFoundError: [Errno 2] No such file or directory: b'/home/root/mlair/data/daily/DEBW99X_o3_temp.nc' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/builds/esde/machine-learning/mlair/mlair/run_modules/pre_processing.py", line 459, in f_proc res = data_handler.build(station, name_affix=name_affix, store_processed_data=store, **kwargs) File "/builds/esde/machine-learning/mlair/mlair/data_handler/default_data_handler.py", line 72, in build sp = cls.data_handler(station, **sp_keys) File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 118, in __init__ self.setup_samples() File "/builds/esde/machine-learning/mlair/mlair/helpers/time_tracking.py", line 40, in __call__ return self.__wrapped__(*args, **kwargs) File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 264, in setup_samples self.make_input_target() File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 302, in make_input_target data, self.meta = self.load_data(self.path, self.station, self.statistics_per_var, self.sampling, File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 353, in load_data data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") mlair.helpers.data_sources.toar_data.EmptyQueryResult: No data available for era5 and toar-data INFO root:pre_processing.py:303 run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (awesome). DEBUG root:datastore.py:118 set: stations(general.awesome)=['DEBW107', 'DEBW013', 'DEBW087'] DEBUG root:datastore.py:118 set: data_collection(general.awesome)=<mlair.data_handler.iterator.DataCollection object at 0x7fd9fc7c2730> DEBUG root:datastore.py:118 get: data_collection(general.awesome)=<mlair.data_handler.iterator.DataCollection object at 0x7fd9fc7c2730> DEBUG root:datastore.py:120 get: data_collection(general)=None DEBUG root:datastore.py:118 get: stations(general.awesome)=['DEBW107', 'DEBW013', 'DEBW087']
----------------------------Captured stderr teardown----------------------------
2023-02-15 16:01:33,234 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:33,234 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:33,234 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:33,234 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:33,234 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:33,235 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:33,235 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:33,235 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:33,235 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:33,235 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:37,234 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_004.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:37,234 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_004.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:37,234 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_004.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:37,234 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_004.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:37,234 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_004.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:37,271 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:01:37,271 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:01:37,271 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:01:37,271 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:01:37,271 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147]
-----------------------------Captured log teardown------------------------------
INFO root:run_environment.py:103 RunEnvironment started INFO root:run_environment.py:120 RunEnvironment finished after 0:00:01 (hh:mm:ss) DEBUG root:datastore.py:118 get: logging_path(general)=/builds/esde/machine-learning/mlair/TestExperiment_daily/logging DEBUG root:datastore.py:118 get: logging_path(general)=/builds/esde/machine-learning/mlair/TestExperiment_daily/logging INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_004.json DEBUG root:datastore.py:118 get: logging_path(general)=/builds/esde/machine-learning/mlair/TestExperiment_daily/logging INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log

------------------------------Captured stderr call------------------------------
2023-02-15 16:01:37,284 - INFO: check valid stations started (all) [pre_processing.py:validate_station:261] 2023-02-15 16:01:37,284 - INFO: check valid stations started (all) [pre_processing.py:validate_station:261] 2023-02-15 16:01:37,284 - INFO: check valid stations started (all) [pre_processing.py:validate_station:261] 2023-02-15 16:01:37,284 - INFO: check valid stations started (all) [pre_processing.py:validate_station:261] 2023-02-15 16:01:37,284 - INFO: check valid stations started (all) [pre_processing.py:validate_station:261] 2023-02-15 16:01:37,287 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:01:37,287 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:01:37,287 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:01:37,287 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:01:37,287 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:01:37,364 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:37,364 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:37,364 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:37,364 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:37,364 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:37,827 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:37,827 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:37,827 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:37,827 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:37,827 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:37,828 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:37,828 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:37,828 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:37,828 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:37,828 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:38,294 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:38,294 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:38,294 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:38,294 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:38,294 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:38,692 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:38,692 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:38,692 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:38,692 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:38,692 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:38,693 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:38,693 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:38,693 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:38,693 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:38,693 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:39,170 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:39,170 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:39,170 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:39,170 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:39,170 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:39,591 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:39,591 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:39,591 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:39,591 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:39,591 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:39,591 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:39,591 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:39,591 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:39,591 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:39,591 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:40,140 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:40,140 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:40,140 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:40,140 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:40,140 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:40,143 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:462] 2023-02-15 16:01:40,143 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:462] 2023-02-15 16:01:40,143 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:462] 2023-02-15 16:01:40,143 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:462] 2023-02-15 16:01:40,143 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:462] 2023-02-15 16:01:40,145 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None). [pre_processing.py:validate_station:303] 2023-02-15 16:01:40,145 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None). [pre_processing.py:validate_station:303] 2023-02-15 16:01:40,145 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None). [pre_processing.py:validate_station:303] 2023-02-15 16:01:40,145 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None). [pre_processing.py:validate_station:303] 2023-02-15 16:01:40,145 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None). [pre_processing.py:validate_station:303]
-------------------------------Captured log call--------------------------------
INFO root:pre_processing.py:261 check valid stations started (all) INFO root:pre_processing.py:295 use serial validate station approach INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:pre_processing.py:462 remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") INFO root:pre_processing.py:303 run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None).
----------------------------Captured stderr teardown----------------------------
2023-02-15 16:01:40,149 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:40,149 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:40,149 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:40,149 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:40,149 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:40,150 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:40,150 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:40,150 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:40,150 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:40,150 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:44,114 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_005.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:44,114 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_005.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:44,114 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_005.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:44,114 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_005.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:44,114 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_005.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:44,152 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:01:44,152 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:01:44,152 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:01:44,152 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:01:44,152 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147]
-----------------------------Captured log teardown------------------------------
INFO root:run_environment.py:103 RunEnvironment started INFO root:run_environment.py:120 RunEnvironment finished after 0:00:01 (hh:mm:ss) INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_005.json INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log

------------------------------Captured stderr call------------------------------
2023-02-15 16:01:44,164 - INFO: check valid stations started (tester) [pre_processing.py:validate_station:261] 2023-02-15 16:01:44,164 - INFO: check valid stations started (tester) [pre_processing.py:validate_station:261] 2023-02-15 16:01:44,164 - INFO: check valid stations started (tester) [pre_processing.py:validate_station:261] 2023-02-15 16:01:44,164 - INFO: check valid stations started (tester) [pre_processing.py:validate_station:261] 2023-02-15 16:01:44,164 - INFO: check valid stations started (tester) [pre_processing.py:validate_station:261] 2023-02-15 16:01:44,169 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:01:44,169 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:01:44,169 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:01:44,169 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:01:44,169 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2023-02-15 16:01:44,247 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:44,247 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:44,247 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:44,247 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:44,247 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:44,742 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:44,742 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:44,742 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:44,742 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:44,742 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:44,743 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:44,743 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:44,743 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:44,743 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:44,743 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:45,227 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:45,227 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:45,227 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:45,227 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:45,227 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:45,659 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:45,659 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:45,659 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:45,659 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:45,659 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:45,660 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:45,660 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:45,660 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:45,660 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:45,660 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:46,140 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:46,140 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:46,140 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:46,140 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:46,140 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:46,574 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:46,574 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:46,574 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:46,574 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:46,574 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:46,574 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:46,574 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:46,574 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:46,574 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:46,574 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:47,140 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:47,140 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:47,140 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:47,140 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:47,140 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2023-02-15 16:01:47,143 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:462] 2023-02-15 16:01:47,143 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:462] 2023-02-15 16:01:47,143 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:462] 2023-02-15 16:01:47,143 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:462] 2023-02-15 16:01:47,143 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:462] 2023-02-15 16:01:47,144 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (tester). [pre_processing.py:validate_station:303] 2023-02-15 16:01:47,144 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (tester). [pre_processing.py:validate_station:303] 2023-02-15 16:01:47,144 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (tester). [pre_processing.py:validate_station:303] 2023-02-15 16:01:47,144 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (tester). [pre_processing.py:validate_station:303] 2023-02-15 16:01:47,144 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (tester). [pre_processing.py:validate_station:303]
-------------------------------Captured log call--------------------------------
INFO root:pre_processing.py:261 check valid stations started (tester) INFO root:pre_processing.py:295 use serial validate station approach INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:pre_processing.py:462 remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") INFO root:pre_processing.py:303 run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (tester).
----------------------------Captured stderr teardown----------------------------
2023-02-15 16:01:47,149 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:47,149 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:47,149 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:47,149 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:47,149 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:47,149 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:47,149 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:47,149 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:47,149 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:47,149 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:51,638 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_006.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:51,638 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_006.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:51,638 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_006.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:51,638 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_006.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:51,638 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_006.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:51,679 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:01:51,679 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:01:51,679 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:01:51,679 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:01:51,679 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147]
-----------------------------Captured log teardown------------------------------
INFO root:run_environment.py:103 RunEnvironment started INFO root:run_environment.py:120 RunEnvironment finished after 0:00:01 (hh:mm:ss) INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_006.json INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log

------------------------------Captured stderr call------------------------------
2023-02-15 16:01:51,691 - INFO: check valid stations started (all) [pre_processing.py:validate_station:261] 2023-02-15 16:01:51,691 - INFO: check valid stations started (all) [pre_processing.py:validate_station:261] 2023-02-15 16:01:51,691 - INFO: check valid stations started (all) [pre_processing.py:validate_station:261] 2023-02-15 16:01:51,691 - INFO: check valid stations started (all) [pre_processing.py:validate_station:261] 2023-02-15 16:01:51,691 - INFO: check valid stations started (all) [pre_processing.py:validate_station:261] 2023-02-15 16:01:51,695 - INFO: use parallel validate station approach [pre_processing.py:validate_station:276] 2023-02-15 16:01:51,695 - INFO: use parallel validate station approach [pre_processing.py:validate_station:276] 2023-02-15 16:01:51,695 - INFO: use parallel validate station approach [pre_processing.py:validate_station:276] 2023-02-15 16:01:51,695 - INFO: use parallel validate station approach [pre_processing.py:validate_station:276] 2023-02-15 16:01:51,695 - INFO: use parallel validate station approach [pre_processing.py:validate_station:276] 2023-02-15 16:01:51,695 - INFO: running 3 processes in parallel [pre_processing.py:validate_station:278] 2023-02-15 16:01:51,695 - INFO: running 3 processes in parallel [pre_processing.py:validate_station:278] 2023-02-15 16:01:51,695 - INFO: running 3 processes in parallel [pre_processing.py:validate_station:278] 2023-02-15 16:01:51,695 - INFO: running 3 processes in parallel [pre_processing.py:validate_station:278] 2023-02-15 16:01:51,695 - INFO: running 3 processes in parallel [pre_processing.py:validate_station:278] 2023-02-15 16:01:52,262 - INFO: ...finished: DEBW107 (25%) [pre_processing.py:validate_station:285] 2023-02-15 16:01:52,262 - INFO: ...finished: DEBW107 (25%) [pre_processing.py:validate_station:285] 2023-02-15 16:01:52,262 - INFO: ...finished: DEBW107 (25%) [pre_processing.py:validate_station:285] 2023-02-15 16:01:52,262 - INFO: ...finished: DEBW107 (25%) [pre_processing.py:validate_station:285] 2023-02-15 16:01:52,262 - INFO: ...finished: DEBW107 (25%) [pre_processing.py:validate_station:285] 2023-02-15 16:01:52,288 - INFO: ...finished: DEBW013 (50%) [pre_processing.py:validate_station:285] 2023-02-15 16:01:52,288 - INFO: ...finished: DEBW013 (50%) [pre_processing.py:validate_station:285] 2023-02-15 16:01:52,288 - INFO: ...finished: DEBW013 (50%) [pre_processing.py:validate_station:285] 2023-02-15 16:01:52,288 - INFO: ...finished: DEBW013 (50%) [pre_processing.py:validate_station:285] 2023-02-15 16:01:52,288 - INFO: ...finished: DEBW013 (50%) [pre_processing.py:validate_station:285] 2023-02-15 16:01:52,302 - INFO: ...finished: DEBW087 (75%) [pre_processing.py:validate_station:285] 2023-02-15 16:01:52,302 - INFO: ...finished: DEBW087 (75%) [pre_processing.py:validate_station:285] 2023-02-15 16:01:52,302 - INFO: ...finished: DEBW087 (75%) [pre_processing.py:validate_station:285] 2023-02-15 16:01:52,302 - INFO: ...finished: DEBW087 (75%) [pre_processing.py:validate_station:285] 2023-02-15 16:01:52,302 - INFO: ...finished: DEBW087 (75%) [pre_processing.py:validate_station:285] 2023-02-15 16:01:52,411 - INFO: ...finished: DEBW99X (100%) [pre_processing.py:validate_station:285] 2023-02-15 16:01:52,411 - INFO: ...finished: DEBW99X (100%) [pre_processing.py:validate_station:285] 2023-02-15 16:01:52,411 - INFO: ...finished: DEBW99X (100%) [pre_processing.py:validate_station:285] 2023-02-15 16:01:52,411 - INFO: ...finished: DEBW99X (100%) [pre_processing.py:validate_station:285] 2023-02-15 16:01:52,411 - INFO: ...finished: DEBW99X (100%) [pre_processing.py:validate_station:285] 2023-02-15 16:01:52,440 - INFO: run for 0:00:01 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None). [pre_processing.py:validate_station:303] 2023-02-15 16:01:52,440 - INFO: run for 0:00:01 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None). [pre_processing.py:validate_station:303] 2023-02-15 16:01:52,440 - INFO: run for 0:00:01 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None). [pre_processing.py:validate_station:303] 2023-02-15 16:01:52,440 - INFO: run for 0:00:01 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None). [pre_processing.py:validate_station:303] 2023-02-15 16:01:52,440 - INFO: run for 0:00:01 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None). [pre_processing.py:validate_station:303]
-------------------------------Captured log call--------------------------------
INFO root:pre_processing.py:261 check valid stations started (all) INFO root:pre_processing.py:276 use parallel validate station approach INFO root:pre_processing.py:278 running 3 processes in parallel INFO root:pre_processing.py:285 ...finished: DEBW107 (25%) INFO root:pre_processing.py:285 ...finished: DEBW013 (50%) INFO root:pre_processing.py:285 ...finished: DEBW087 (75%) INFO root:pre_processing.py:285 ...finished: DEBW99X (100%) INFO root:pre_processing.py:303 run for 0:00:01 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None).
----------------------------Captured stderr teardown----------------------------
2023-02-15 16:01:52,442 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:52,442 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:52,442 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:52,442 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:52,442 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:01:52,442 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:52,442 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:52,442 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:52,442 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:52,442 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:01:56,725 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_007.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:56,725 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_007.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:56,725 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_007.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:56,725 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_007.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:56,725 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_007.json [run_environment.py:__save_tracking:159] 2023-02-15 16:01:56,764 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:01:56,764 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:01:56,764 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:01:56,764 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:01:56,764 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147]
-----------------------------Captured log teardown------------------------------
INFO root:run_environment.py:103 RunEnvironment started INFO root:run_environment.py:120 RunEnvironment finished after 0:00:01 (hh:mm:ss) INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_007.json INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log

------------------------------Captured stderr call------------------------------
2023-02-15 16:02:01,127 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:02:01,127 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:02:01,127 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:02:01,127 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:02:01,127 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:02:01,127 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:01,127 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:01,127 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:01,127 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:01,127 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:05,973 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json [run_environment.py:__save_tracking:159] 2023-02-15 16:02:05,973 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json [run_environment.py:__save_tracking:159] 2023-02-15 16:02:05,973 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json [run_environment.py:__save_tracking:159] 2023-02-15 16:02:05,973 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json [run_environment.py:__save_tracking:159] 2023-02-15 16:02:05,973 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json [run_environment.py:__save_tracking:159] 2023-02-15 16:02:06,016 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:02:06,016 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:02:06,016 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:02:06,016 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:02:06,016 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147]
-------------------------------Captured log call--------------------------------
INFO root:run_environment.py:103 RunEnvironment started INFO root:run_environment.py:120 RunEnvironment finished after 0:00:01 (hh:mm:ss) INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/logging_000.log

------------------------------Captured stderr call------------------------------
2023-02-15 16:02:06,020 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:02:06,020 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:02:06,020 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:02:06,020 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:02:06,020 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:02:06,120 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:06,120 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:06,120 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:06,120 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:06,120 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:10,268 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_001.json [run_environment.py:__save_tracking:159] 2023-02-15 16:02:10,268 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_001.json [run_environment.py:__save_tracking:159] 2023-02-15 16:02:10,268 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_001.json [run_environment.py:__save_tracking:159] 2023-02-15 16:02:10,268 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_001.json [run_environment.py:__save_tracking:159] 2023-02-15 16:02:10,268 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_001.json [run_environment.py:__save_tracking:159] 2023-02-15 16:02:10,307 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:02:10,307 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:02:10,307 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:02:10,307 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:02:10,307 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147]
-------------------------------Captured log call--------------------------------
INFO root:run_environment.py:103 RunEnvironment started INFO root:run_environment.py:120 RunEnvironment finished after 0:00:01 (hh:mm:ss) INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/tracking_001.json INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/logging_000.log

------------------------------Captured stderr call------------------------------
2023-02-15 16:02:10,310 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:02:10,310 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:02:10,310 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:02:10,310 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:02:10,310 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:02:10,311 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:10,311 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:10,311 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:10,311 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:10,311 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:14,752 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_002.json [run_environment.py:__save_tracking:159] 2023-02-15 16:02:14,752 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_002.json [run_environment.py:__save_tracking:159] 2023-02-15 16:02:14,752 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_002.json [run_environment.py:__save_tracking:159] 2023-02-15 16:02:14,752 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_002.json [run_environment.py:__save_tracking:159] 2023-02-15 16:02:14,752 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_002.json [run_environment.py:__save_tracking:159] 2023-02-15 16:02:14,819 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:02:14,819 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:02:14,819 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:02:14,819 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:02:14,819 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147]
-------------------------------Captured log call--------------------------------
INFO root:run_environment.py:103 RunEnvironment started INFO root:run_environment.py:120 RunEnvironment finished after 0:00:01 (hh:mm:ss) INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/tracking_002.json INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/logging_000.log

------------------------------Captured stderr call------------------------------
2023-02-15 16:02:14,822 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:02:14,822 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:02:14,822 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:02:14,822 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:02:14,822 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2023-02-15 16:02:15,023 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:15,023 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:15,023 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:15,023 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:15,023 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:19,301 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_003.json [run_environment.py:__save_tracking:159] 2023-02-15 16:02:19,301 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_003.json [run_environment.py:__save_tracking:159] 2023-02-15 16:02:19,301 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_003.json [run_environment.py:__save_tracking:159] 2023-02-15 16:02:19,301 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_003.json [run_environment.py:__save_tracking:159] 2023-02-15 16:02:19,301 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_003.json [run_environment.py:__save_tracking:159] 2023-02-15 16:02:19,341 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:02:19,341 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:02:19,341 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:02:19,341 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:02:19,341 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147]
-------------------------------Captured log call--------------------------------
INFO root:run_environment.py:103 RunEnvironment started INFO root:run_environment.py:120 RunEnvironment finished after 0:00:01 (hh:mm:ss) INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/tracking_003.json INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/logging_000.log

-----------------------------Captured stdout setup------------------------------
Model: "model_8" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_33 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_2 (Flatten) (None, 16) 0 _________________________________________________________________ dense_9 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_10 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None
------------------------------Captured stdout call------------------------------
Epoch 1/2 13/13 - 1s - loss: 26.9985 - val_loss: 19.4394 Epoch 2/2 13/13 - 0s - loss: 15.3536 - val_loss: 10.5383
------------------------------Captured stderr call------------------------------
2023-02-15 16:02:25.839869: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.

-----------------------------Captured stdout setup------------------------------
Model: "model_9" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_34 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_3 (Flatten) (None, 16) 0 _________________________________________________________________ dense_11 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_12 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None
------------------------------Captured stderr call------------------------------
2023-02-15 16:02:38,334 - INFO: Training started [run_environment.py:__init__:103] 2023-02-15 16:02:38,334 - INFO: Training started [run_environment.py:__init__:103] 2023-02-15 16:02:38,334 - INFO: Training started [run_environment.py:__init__:103] 2023-02-15 16:02:38,334 - INFO: Training started [run_environment.py:__init__:103] 2023-02-15 16:02:38,334 - INFO: Training started [run_environment.py:__init__:103] 2023-02-15 16:02:38,335 - INFO: No training has started, because train_model parameter was false. [training.py:_run:92] 2023-02-15 16:02:38,335 - INFO: No training has started, because train_model parameter was false. [training.py:_run:92] 2023-02-15 16:02:38,335 - INFO: No training has started, because train_model parameter was false. [training.py:_run:92] 2023-02-15 16:02:38,335 - INFO: No training has started, because train_model parameter was false. [training.py:_run:92] 2023-02-15 16:02:38,335 - INFO: No training has started, because train_model parameter was false. [training.py:_run:92] 2023-02-15 16:02:38,336 - INFO: Training finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:38,336 - INFO: Training finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:38,336 - INFO: Training finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:38,336 - INFO: Training finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:02:38,336 - INFO: Training finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120]
-------------------------------Captured log call--------------------------------
INFO root:run_environment.py:103 Training started INFO root:training.py:92 No training has started, because train_model parameter was false. INFO root:run_environment.py:120 Training finished after 0:00:01 (hh:mm:ss)

-----------------------------Captured stdout setup------------------------------
Model: "model_10" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_35 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_4 (Flatten) (None, 16) 0 _________________________________________________________________ dense_13 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_14 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None
------------------------------Captured stdout call------------------------------
Epoch 1/2 13/13 - 1s - loss: 59914.9570 - mse: 737.3669 - mae: 20.9526 - var_loss: 119092.5469 - val_loss: 13137.2432 - val_mse: 420.8336 - val_mae: 16.3772 - val_var_loss: 25853.6523 Epoch 2/2 13/13 - 0s - loss: 8920.5713 - mse: 477.8673 - mae: 16.9797 - var_loss: 17363.2754 - val_loss: 1512.1428 - val_mse: 413.7453 - val_mae: 15.6545 - val_var_loss: 2610.5405

-----------------------------Captured stdout setup------------------------------
Model: "model_11" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_36 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_5 (Flatten) (None, 16) 0 _________________________________________________________________ dense_15 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_16 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None

-----------------------------Captured stdout setup------------------------------
Model: "model_12" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_37 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_6 (Flatten) (None, 16) 0 _________________________________________________________________ dense_17 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_18 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None

-----------------------------Captured stdout setup------------------------------
Model: "model_13" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_38 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_7 (Flatten) (None, 16) 0 _________________________________________________________________ dense_19 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_20 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None

-----------------------------Captured stdout setup------------------------------
Model: "model_14" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_39 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_8 (Flatten) (None, 16) 0 _________________________________________________________________ dense_21 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_22 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None
------------------------------Captured stdout call------------------------------
Epoch 1/2 13/13 - 0s - loss: 18.9927 - val_loss: 9.8644 Epoch 2/2 13/13 - 0s - loss: 16.4920 - val_loss: 23.5570

-----------------------------Captured stdout setup------------------------------
Model: "model_15" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_40 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_9 (Flatten) (None, 16) 0 _________________________________________________________________ dense_23 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_24 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None
------------------------------Captured stderr call------------------------------
2023-02-15 16:03:20,928 - WARNING: Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. [saving_utils.py:try_build_compiled_arguments:313] 2023-02-15 16:03:20,928 - WARNING: Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. [saving_utils.py:try_build_compiled_arguments:313] 2023-02-15 16:03:20,928 - WARNING: Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. [saving_utils.py:try_build_compiled_arguments:313] 2023-02-15 16:03:20,928 - WARNING: Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. [saving_utils.py:try_build_compiled_arguments:313] 2023-02-15 16:03:20,928 - WARNING: Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. [saving_utils.py:try_build_compiled_arguments:313] 2023-02-15 16:03:20,939 - WARNING: Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. [saving_utils.py:try_build_compiled_arguments:313] 2023-02-15 16:03:20,939 - WARNING: Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. [saving_utils.py:try_build_compiled_arguments:313] 2023-02-15 16:03:20,939 - WARNING: Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. [saving_utils.py:try_build_compiled_arguments:313] 2023-02-15 16:03:20,939 - WARNING: Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. [saving_utils.py:try_build_compiled_arguments:313] 2023-02-15 16:03:20,939 - WARNING: Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. [saving_utils.py:try_build_compiled_arguments:313]
-------------------------------Captured log call--------------------------------
DEBUG root:datastore.py:118 get: model_name(general.model)=/builds/esde/machine-learning/mlair/test/test_run_modules/TestExperiment/model/test_model.h5 DEBUG root:training.py:188 save model to /builds/esde/machine-learning/mlair/test/test_run_modules/TestExperiment/model/test_model.h5 WARNING tensorflow:saving_utils.py:313 Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. WARNING tensorflow:saving_utils.py:313 Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. DEBUG root:datastore.py:118 set: model(general)=<mlair.model_modules.fully_connected_networks.FCN object at 0x7fd9daf37e80>

-----------------------------Captured stdout setup------------------------------
Model: "model_16" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_41 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_10 (Flatten) (None, 16) 0 _________________________________________________________________ dense_25 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_26 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None

-----------------------------Captured stdout setup------------------------------
Model: "model_17" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_42 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_11 (Flatten) (None, 16) 0 _________________________________________________________________ dense_27 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_28 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None

-----------------------------Captured stdout setup------------------------------
Model: "model_18" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_43 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_12 (Flatten) (None, 16) 0 _________________________________________________________________ dense_29 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_30 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None

-----------------------------Captured stdout setup------------------------------
Model: "model_19" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_44 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_13 (Flatten) (None, 16) 0 _________________________________________________________________ dense_31 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_32 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None

-----------------------------Captured stdout setup------------------------------
Model: "model_20" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_45 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_14 (Flatten) (None, 16) 0 _________________________________________________________________ dense_33 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_34 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None

------------------------------Captured stderr call------------------------------
2023-02-15 16:03:59,186 - INFO: Workflow started [run_environment.py:__init__:103] 2023-02-15 16:03:59,186 - INFO: Workflow started [run_environment.py:__init__:103] 2023-02-15 16:03:59,186 - INFO: Workflow started [run_environment.py:__init__:103] 2023-02-15 16:03:59,186 - INFO: Workflow started [run_environment.py:__init__:103] 2023-02-15 16:03:59,186 - INFO: Workflow started [run_environment.py:__init__:103] 2023-02-15 16:03:59,186 - INFO: 6 [test_abstract_workflow.py:__init__:37] 2023-02-15 16:03:59,186 - INFO: 6 [test_abstract_workflow.py:__init__:37] 2023-02-15 16:03:59,186 - INFO: 6 [test_abstract_workflow.py:__init__:37] 2023-02-15 16:03:59,186 - INFO: 6 [test_abstract_workflow.py:__init__:37] 2023-02-15 16:03:59,186 - INFO: 6 [test_abstract_workflow.py:__init__:37] 2023-02-15 16:03:59,186 - INFO: 2 [test_abstract_workflow.py:__init__:42] 2023-02-15 16:03:59,186 - INFO: 2 [test_abstract_workflow.py:__init__:42] 2023-02-15 16:03:59,186 - INFO: 2 [test_abstract_workflow.py:__init__:42] 2023-02-15 16:03:59,186 - INFO: 2 [test_abstract_workflow.py:__init__:42] 2023-02-15 16:03:59,186 - INFO: 2 [test_abstract_workflow.py:__init__:42] 2023-02-15 16:03:59,187 - INFO: 3 [test_abstract_workflow.py:__init__:37] 2023-02-15 16:03:59,187 - INFO: 3 [test_abstract_workflow.py:__init__:37] 2023-02-15 16:03:59,187 - INFO: 3 [test_abstract_workflow.py:__init__:37] 2023-02-15 16:03:59,187 - INFO: 3 [test_abstract_workflow.py:__init__:37] 2023-02-15 16:03:59,187 - INFO: 3 [test_abstract_workflow.py:__init__:37] 2023-02-15 16:03:59,187 - INFO: Workflow finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:03:59,187 - INFO: Workflow finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:03:59,187 - INFO: Workflow finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:03:59,187 - INFO: Workflow finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:03:59,187 - INFO: Workflow finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2023-02-15 16:04:03,722 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json [run_environment.py:__save_tracking:159] 2023-02-15 16:04:03,722 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json [run_environment.py:__save_tracking:159] 2023-02-15 16:04:03,722 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json [run_environment.py:__save_tracking:159] 2023-02-15 16:04:03,722 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json [run_environment.py:__save_tracking:159] 2023-02-15 16:04:03,722 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json [run_environment.py:__save_tracking:159] 2023-02-15 16:04:03,765 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:04:03,765 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:04:03,765 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:04:03,765 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2023-02-15 16:04:03,765 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147]
-------------------------------Captured log call--------------------------------
INFO root:run_environment.py:103 Workflow started INFO root:test_abstract_workflow.py:37 6 INFO root:test_abstract_workflow.py:42 2 INFO root:test_abstract_workflow.py:37 3 INFO root:run_environment.py:120 Workflow finished after 0:00:01 (hh:mm:ss) INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/logging_000.log