report.html

-------------------------------Captured log call--------------------------------
DEBUG root:path_config.py:125 Created path: data/test DEBUG root:path_config.py:132 Path already exists: data/test

------------------------------Captured stdout call------------------------------
Model: "model" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_1 (InputLayer) [(None, 14, 1, 2)] 0 _________________________________________________________________ flatten (Flatten) (None, 28) 0 _________________________________________________________________ dense (Dense) (None, 64) 1856 _________________________________________________________________ prelu_1 (PReLU) (None, 64) 64 _________________________________________________________________ dense_1 (Dense) (None, 32) 2080 _________________________________________________________________ prelu_2 (PReLU) (None, 32) 32 _________________________________________________________________ dense_2 (Dense) (None, 16) 528 _________________________________________________________________ prelu_3 (PReLU) (None, 16) 16 _________________________________________________________________ dense_3 (Dense) (None, 3) 51 _________________________________________________________________ linear_output (Activation) (None, 3) 0 ================================================================= Total params: 4,627 Trainable params: 4,627 Non-trainable params: 0 _________________________________________________________________ None
------------------------------Captured stderr call------------------------------
2022-12-02 14:07:50.722578: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory 2022-12-02 14:07:50.722638: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303) 2022-12-02 14:07:50.722682: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:163] no NVIDIA GPU device is present: /dev/nvidia0 does not exist 2022-12-02 14:07:50.722900: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

-------------------------------Captured log call--------------------------------
INFO root:time_tracking.py:137 undefined job finished after 0:00:01 (hh:mm:ss)

-------------------------------Captured log call--------------------------------
INFO root:time_tracking.py:137 my job finished after 0:00:01 (hh:mm:ss)

------------------------------Captured stderr call------------------------------
2022-12-02 14:08:48,079 - INFO: Could not find a valid match for variable no2 and networks []! Therefore, use first answer from JOIN: {'id': 16686, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'no2', 'parameter_label': 'NO2', 'parameter_attribute': ''} [join.py:_select_distinct_network:288] 2022-12-02 14:08:48,079 - INFO: Could not find a valid match for variable no2 and networks []! Therefore, use first answer from JOIN: {'id': 16686, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'no2', 'parameter_label': 'NO2', 'parameter_attribute': ''} [join.py:_select_distinct_network:288] 2022-12-02 14:08:48,079 - INFO: Could not find a valid match for variable no2 and networks []! Therefore, use first answer from JOIN: {'id': 16686, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'no2', 'parameter_label': 'NO2', 'parameter_attribute': ''} [join.py:_select_distinct_network:288] 2022-12-02 14:08:48,079 - INFO: Could not find a valid match for variable no2 and networks []! Therefore, use first answer from JOIN: {'id': 16686, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'no2', 'parameter_label': 'NO2', 'parameter_attribute': ''} [join.py:_select_distinct_network:288] 2022-12-02 14:08:48,079 - INFO: Could not find a valid match for variable o3 and networks []! Therefore, use first answer from JOIN: {'id': 16687, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'o3', 'parameter_label': 'O3', 'parameter_attribute': ''} [join.py:_select_distinct_network:288] 2022-12-02 14:08:48,079 - INFO: Could not find a valid match for variable o3 and networks []! Therefore, use first answer from JOIN: {'id': 16687, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'o3', 'parameter_label': 'O3', 'parameter_attribute': ''} [join.py:_select_distinct_network:288] 2022-12-02 14:08:48,079 - INFO: Could not find a valid match for variable o3 and networks []! Therefore, use first answer from JOIN: {'id': 16687, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'o3', 'parameter_label': 'O3', 'parameter_attribute': ''} [join.py:_select_distinct_network:288] 2022-12-02 14:08:48,079 - INFO: Could not find a valid match for variable o3 and networks []! Therefore, use first answer from JOIN: {'id': 16687, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'o3', 'parameter_label': 'O3', 'parameter_attribute': ''} [join.py:_select_distinct_network:288] 2022-12-02 14:08:48,079 - INFO: Could not find a valid match for variable cloudcover and networks []! Therefore, use first answer from JOIN: {'id': 54036, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'cloudcover', 'parameter_label': 'CLOUDCOVER', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288] 2022-12-02 14:08:48,079 - INFO: Could not find a valid match for variable cloudcover and networks []! Therefore, use first answer from JOIN: {'id': 54036, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'cloudcover', 'parameter_label': 'CLOUDCOVER', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288] 2022-12-02 14:08:48,079 - INFO: Could not find a valid match for variable cloudcover and networks []! Therefore, use first answer from JOIN: {'id': 54036, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'cloudcover', 'parameter_label': 'CLOUDCOVER', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288] 2022-12-02 14:08:48,079 - INFO: Could not find a valid match for variable cloudcover and networks []! Therefore, use first answer from JOIN: {'id': 54036, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'cloudcover', 'parameter_label': 'CLOUDCOVER', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288] 2022-12-02 14:08:48,080 - INFO: Could not find a valid match for variable temp and networks []! Therefore, use first answer from JOIN: {'id': 88491, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'temp', 'parameter_label': 'TEMP-REA-MIUB', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288] 2022-12-02 14:08:48,080 - INFO: Could not find a valid match for variable temp and networks []! Therefore, use first answer from JOIN: {'id': 88491, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'temp', 'parameter_label': 'TEMP-REA-MIUB', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288] 2022-12-02 14:08:48,080 - INFO: Could not find a valid match for variable temp and networks []! Therefore, use first answer from JOIN: {'id': 88491, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'temp', 'parameter_label': 'TEMP-REA-MIUB', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288] 2022-12-02 14:08:48,080 - INFO: Could not find a valid match for variable temp and networks []! Therefore, use first answer from JOIN: {'id': 88491, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'temp', 'parameter_label': 'TEMP-REA-MIUB', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288] 2022-12-02 14:08:48,080 - INFO: Could not find a valid match for variable press and networks []! Therefore, use first answer from JOIN: {'id': 102660, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'press', 'parameter_label': 'PRESS-REA-MIUB', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288] 2022-12-02 14:08:48,080 - INFO: Could not find a valid match for variable press and networks []! Therefore, use first answer from JOIN: {'id': 102660, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'press', 'parameter_label': 'PRESS-REA-MIUB', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288] 2022-12-02 14:08:48,080 - INFO: Could not find a valid match for variable press and networks []! Therefore, use first answer from JOIN: {'id': 102660, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'press', 'parameter_label': 'PRESS-REA-MIUB', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288] 2022-12-02 14:08:48,080 - INFO: Could not find a valid match for variable press and networks []! Therefore, use first answer from JOIN: {'id': 102660, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'press', 'parameter_label': 'PRESS-REA-MIUB', 'parameter_attribute': 'REA'} [join.py:_select_distinct_network:288]
-------------------------------Captured log call--------------------------------
INFO root:join.py:288 Could not find a valid match for variable no2 and networks []! Therefore, use first answer from JOIN: {'id': 16686, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'no2', 'parameter_label': 'NO2', 'parameter_attribute': ''} INFO root:join.py:288 Could not find a valid match for variable o3 and networks []! Therefore, use first answer from JOIN: {'id': 16687, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'o3', 'parameter_label': 'O3', 'parameter_attribute': ''} INFO root:join.py:288 Could not find a valid match for variable cloudcover and networks []! Therefore, use first answer from JOIN: {'id': 54036, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'cloudcover', 'parameter_label': 'CLOUDCOVER', 'parameter_attribute': 'REA'} INFO root:join.py:288 Could not find a valid match for variable temp and networks []! Therefore, use first answer from JOIN: {'id': 88491, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'temp', 'parameter_label': 'TEMP-REA-MIUB', 'parameter_attribute': 'REA'} INFO root:join.py:288 Could not find a valid match for variable press and networks []! Therefore, use first answer from JOIN: {'id': 102660, 'network_name': 'UBA', 'station_id': 'DENW053', 'parameter_name': 'press', 'parameter_label': 'PRESS-REA-MIUB', 'parameter_attribute': 'REA'}

------------------------------Captured stdout call------------------------------
In test_get_padding_for_same_negative_kernel_size

------------------------------Captured stdout call------------------------------
Epoch 1/5 1/1 [==============================] - ETA: 0s - loss: 4.7404 1/1 [==============================] - 0s 372ms/step - loss: 4.7404 Epoch 2/5 1/1 [==============================] - ETA: 0s - loss: 4.5679 1/1 [==============================] - 0s 2ms/step - loss: 4.5679 Epoch 3/5 1/1 [==============================] - ETA: 0s - loss: 4.3987 1/1 [==============================] - 0s 2ms/step - loss: 4.3987 Epoch 4/5 1/1 [==============================] - ETA: 0s - loss: 4.2411 1/1 [==============================] - 0s 2ms/step - loss: 4.2411 Epoch 5/5 1/1 [==============================] - ETA: 0s - loss: 4.0866 1/1 [==============================] - 0s 2ms/step - loss: 4.0866
------------------------------Captured stderr call------------------------------
2022-12-02 14:08:49.401415: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)

------------------------------Captured stdout call------------------------------
Epoch 00001: val_loss did not improve from 6.00000 Epoch 00010: val_loss did not improve from 6.00000 Epoch 00011: val_loss improved from 6.00000 to 4.00000, saving model to ckpt.test Epoch 00011: save to /builds/esde/machine-learning/mlair/test/test_model_modules/callback_lr Epoch 00011: save to /builds/esde/machine-learning/mlair/test/test_model_modules/callback_hist Epoch 00011: saving model to ckpt.test Epoch 00011: save to /builds/esde/machine-learning/mlair/test/test_model_modules/callback_lr Epoch 00011: save to /builds/esde/machine-learning/mlair/test/test_model_modules/callback_hist

------------------------------Captured stdout call------------------------------
1/1 [==============================] - ETA: 0s - loss: 1.2500 1/1 [==============================] - 0s 84ms/step - loss: 1.2500 1/1 [==============================] - ETA: 0s - loss: 2.2500 1/1 [==============================] - 0s 83ms/step - loss: 2.2500

------------------------------Captured stdout call------------------------------
1/1 [==============================] - ETA: 0s - loss: 0.1406 1/1 [==============================] - 0s 154ms/step - loss: 0.1406

------------------------------Captured stdout call------------------------------
1/1 [==============================] - ETA: 0s - loss: 0.6953 1/1 [==============================] - 0s 174ms/step - loss: 0.6953
------------------------------Captured stderr call------------------------------
2022-12-02 14:08:50,582 - WARNING: 5 out of the last 9 calls to <function Model.make_train_function.<locals>.train_function at 0x7f83546f8280> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [def_function.py:called_with_tracing:147] 2022-12-02 14:08:50,582 - WARNING: 5 out of the last 9 calls to <function Model.make_train_function.<locals>.train_function at 0x7f83546f8280> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [def_function.py:called_with_tracing:147] 2022-12-02 14:08:50,582 - WARNING: 5 out of the last 9 calls to <function Model.make_train_function.<locals>.train_function at 0x7f83546f8280> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [def_function.py:called_with_tracing:147] 2022-12-02 14:08:50,582 - WARNING: 5 out of the last 9 calls to <function Model.make_train_function.<locals>.train_function at 0x7f83546f8280> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [def_function.py:called_with_tracing:147]
-------------------------------Captured log call--------------------------------
WARNING tensorflow:def_function.py:147 5 out of the last 9 calls to <function Model.make_train_function.<locals>.train_function at 0x7f83546f8280> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.

------------------------------Captured stdout call------------------------------
1/1 [==============================] - ETA: 0s - loss: 0.4734 1/1 [==============================] - 0s 177ms/step - loss: 0.4734
------------------------------Captured stderr call------------------------------
2022-12-02 14:08:50,835 - WARNING: 6 out of the last 10 calls to <function Model.make_train_function.<locals>.train_function at 0x7f853d28ea60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [def_function.py:called_with_tracing:147] 2022-12-02 14:08:50,835 - WARNING: 6 out of the last 10 calls to <function Model.make_train_function.<locals>.train_function at 0x7f853d28ea60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [def_function.py:called_with_tracing:147] 2022-12-02 14:08:50,835 - WARNING: 6 out of the last 10 calls to <function Model.make_train_function.<locals>.train_function at 0x7f853d28ea60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [def_function.py:called_with_tracing:147] 2022-12-02 14:08:50,835 - WARNING: 6 out of the last 10 calls to <function Model.make_train_function.<locals>.train_function at 0x7f853d28ea60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. [def_function.py:called_with_tracing:147]
-------------------------------Captured log call--------------------------------
WARNING tensorflow:def_function.py:147 6 out of the last 10 calls to <function Model.make_train_function.<locals>.train_function at 0x7f853d28ea60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.

------------------------------Captured stdout call------------------------------
1/1 [==============================] - ETA: 0s - loss: 0.6953 1/1 [==============================] - 0s 152ms/step - loss: 0.6953

------------------------------Captured stdout call------------------------------
1/1 [==============================] - ETA: 0s - loss: 0.6953 1/1 [==============================] - 0s 138ms/step - loss: 0.6953

------------------------------Captured stdout call------------------------------
1/1 [==============================] - ETA: 0s - loss: 1.0281 1/1 [==============================] - 0s 144ms/step - loss: 1.0281

-------------------------------Captured log call--------------------------------
DEBUG root:datastore.py:118 set: 23tester(general)=23 DEBUG root:experiment_setup.py:456 set experiment attribute: 23tester(general)=23 DEBUG root:datastore.py:118 get: 23tester(general)=23

-------------------------------Captured log call--------------------------------
DEBUG root:datastore.py:118 set: 109tester(general)=109 DEBUG root:experiment_setup.py:456 set experiment attribute: 109tester(general)=109 DEBUG root:datastore.py:118 get: 109tester(general.tester)=109

-------------------------------Captured log call--------------------------------
DEBUG root:datastore.py:118 set: NoneTester(general.testing)=notNone DEBUG root:experiment_setup.py:456 set experiment attribute: NoneTester(general.testing)=notNone DEBUG root:datastore.py:118 get: NoneTester(general.testing)=notNone DEBUG root:datastore.py:118 set: AnotherNoneTester(general)=None DEBUG root:experiment_setup.py:456 set experiment attribute: AnotherNoneTester(general)=None DEBUG root:datastore.py:118 get: AnotherNoneTester(general)=None

-------------------------------Captured log call--------------------------------
DEBUG root:datastore.py:118 set: NoneTester(general)=notNone DEBUG root:experiment_setup.py:456 set experiment attribute: NoneTester(general)=notNone DEBUG root:datastore.py:118 get: NoneTester(general)=notNone DEBUG root:datastore.py:118 set: NoneTester(general)=['notNone'] DEBUG root:experiment_setup.py:456 set experiment attribute: NoneTester(general)=['notNone'] DEBUG root:datastore.py:118 get: NoneTester(general)=['notNone'] DEBUG root:datastore.py:118 set: NoneTester(general)=[None] DEBUG root:experiment_setup.py:456 set experiment attribute: NoneTester(general)=[None] DEBUG root:datastore.py:118 get: NoneTester(general)=[None] DEBUG root:datastore.py:118 set: NoneTester(general)=2 DEBUG root:experiment_setup.py:456 set experiment attribute: NoneTester(general)=2 DEBUG root:datastore.py:118 get: NoneTester(general)=2

------------------------------Captured stdout call------------------------------
Model: "model_7" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_32 (InputLayer) [(None, 14, 1, 5)] 0 _________________________________________________________________ flatten_1 (Flatten) (None, 70) 0 _________________________________________________________________ dense_5 (Dense) (None, 64) 4544 _________________________________________________________________ prelu_1 (PReLU) (None, 64) 64 _________________________________________________________________ dense_6 (Dense) (None, 32) 2080 _________________________________________________________________ prelu_2 (PReLU) (None, 32) 32 _________________________________________________________________ dense_7 (Dense) (None, 16) 528 _________________________________________________________________ prelu_3 (PReLU) (None, 16) 16 _________________________________________________________________ dense_8 (Dense) (None, 5) 85 _________________________________________________________________ linear_output (Activation) (None, 5) 0 ================================================================= Total params: 7,349 Trainable params: 7,349 Non-trainable params: 0 _________________________________________________________________ None

------------------------------Captured stderr call------------------------------
2022-12-02 14:09:12,326 - INFO: PartitionCheck started [run_environment.py:__init__:103] 2022-12-02 14:09:12,326 - INFO: PartitionCheck started [run_environment.py:__init__:103] 2022-12-02 14:09:12,326 - INFO: PartitionCheck started [run_environment.py:__init__:103] 2022-12-02 14:09:12,326 - INFO: PartitionCheck started [run_environment.py:__init__:103] 2022-12-02 14:09:12,326 - INFO: PartitionCheck started [run_environment.py:__init__:103] 2022-12-02 14:09:12,326 - INFO: PartitionCheck finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:09:12,326 - INFO: PartitionCheck finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:09:12,326 - INFO: PartitionCheck finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:09:12,326 - INFO: PartitionCheck finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:09:12,326 - INFO: PartitionCheck finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:09:12,327 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:09:12,327 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:09:12,327 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:09:12,327 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:09:12,327 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:09:12,327 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:09:12,327 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:09:12,327 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:09:12,327 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:09:12,327 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:09:14,694 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json [run_environment.py:__save_tracking:159] 2022-12-02 14:09:14,694 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json [run_environment.py:__save_tracking:159] 2022-12-02 14:09:14,694 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json [run_environment.py:__save_tracking:159] 2022-12-02 14:09:14,694 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json [run_environment.py:__save_tracking:159] 2022-12-02 14:09:14,694 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json [run_environment.py:__save_tracking:159] 2022-12-02 14:09:14,714 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:09:14,714 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:09:14,714 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:09:14,714 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:09:14,714 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147]
-------------------------------Captured log call--------------------------------
INFO root:run_environment.py:103 PartitionCheck started INFO root:run_environment.py:120 PartitionCheck finished after 0:00:01 (hh:mm:ss) INFO root:run_environment.py:103 RunEnvironment started INFO root:run_environment.py:120 RunEnvironment finished after 0:00:01 (hh:mm:ss) INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log

------------------------------Captured stderr call------------------------------
2022-12-02 14:09:19,351 - INFO: PreProcessing started [run_environment.py:__init__:103] 2022-12-02 14:09:19,351 - INFO: PreProcessing started [run_environment.py:__init__:103] 2022-12-02 14:09:19,351 - INFO: PreProcessing started [run_environment.py:__init__:103] 2022-12-02 14:09:19,351 - INFO: PreProcessing started [run_environment.py:__init__:103] 2022-12-02 14:09:19,351 - INFO: PreProcessing started [run_environment.py:__init__:103] 2022-12-02 14:09:19,352 - INFO: check valid stations started (preprocessing) [pre_processing.py:validate_station:261] 2022-12-02 14:09:19,352 - INFO: check valid stations started (preprocessing) [pre_processing.py:validate_station:261] 2022-12-02 14:09:19,352 - INFO: check valid stations started (preprocessing) [pre_processing.py:validate_station:261] 2022-12-02 14:09:19,352 - INFO: check valid stations started (preprocessing) [pre_processing.py:validate_station:261] 2022-12-02 14:09:19,352 - INFO: check valid stations started (preprocessing) [pre_processing.py:validate_station:261] 2022-12-02 14:09:19,357 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:09:19,357 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:09:19,357 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:09:19,357 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:09:19,357 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:13:32,552 - INFO: load data for DEBW107 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2022-12-02 14:13:32,552 - INFO: load data for DEBW107 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2022-12-02 14:13:32,552 - INFO: load data for DEBW107 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2022-12-02 14:13:32,552 - INFO: load data for DEBW107 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2022-12-02 14:13:32,552 - INFO: load data for DEBW107 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2022-12-02 14:13:32,627 - INFO: load data for DEBW107 from JOIN [join.py:download_join:63] 2022-12-02 14:13:32,627 - INFO: load data for DEBW107 from JOIN [join.py:download_join:63] 2022-12-02 14:13:32,627 - INFO: load data for DEBW107 from JOIN [join.py:download_join:63] 2022-12-02 14:13:32,627 - INFO: load data for DEBW107 from JOIN [join.py:download_join:63] 2022-12-02 14:13:32,627 - INFO: load data for DEBW107 from JOIN [join.py:download_join:63] 2022-12-02 14:13:35,258 - INFO: setup_samples finished after 0:04:16 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:35,258 - INFO: setup_samples finished after 0:04:16 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:35,258 - INFO: setup_samples finished after 0:04:16 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:35,258 - INFO: setup_samples finished after 0:04:16 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:35,258 - INFO: setup_samples finished after 0:04:16 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:35,579 - INFO: PartitionCheck finished after 0:04:19 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:13:35,579 - INFO: PartitionCheck finished after 0:04:19 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:13:35,579 - INFO: PartitionCheck finished after 0:04:19 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:13:35,579 - INFO: PartitionCheck finished after 0:04:19 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:13:35,579 - INFO: PartitionCheck finished after 0:04:19 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:13:35,589 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:35,589 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:35,589 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:35,589 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:35,589 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:35,590 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:35,590 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:35,590 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:35,590 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:35,590 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:36,220 - INFO: load data for DEBW013 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2022-12-02 14:13:36,220 - INFO: load data for DEBW013 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2022-12-02 14:13:36,220 - INFO: load data for DEBW013 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2022-12-02 14:13:36,220 - INFO: load data for DEBW013 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2022-12-02 14:13:36,220 - INFO: load data for DEBW013 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2022-12-02 14:13:36,297 - INFO: load data for DEBW013 from JOIN [join.py:download_join:63] 2022-12-02 14:13:36,297 - INFO: load data for DEBW013 from JOIN [join.py:download_join:63] 2022-12-02 14:13:36,297 - INFO: load data for DEBW013 from JOIN [join.py:download_join:63] 2022-12-02 14:13:36,297 - INFO: load data for DEBW013 from JOIN [join.py:download_join:63] 2022-12-02 14:13:36,297 - INFO: load data for DEBW013 from JOIN [join.py:download_join:63] 2022-12-02 14:13:40,056 - INFO: setup_samples finished after 0:00:05 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:40,056 - INFO: setup_samples finished after 0:00:05 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:40,056 - INFO: setup_samples finished after 0:00:05 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:40,056 - INFO: setup_samples finished after 0:00:05 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:40,056 - INFO: setup_samples finished after 0:00:05 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:40,342 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:40,342 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:40,342 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:40,342 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:40,342 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:40,343 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:40,343 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:40,343 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:40,343 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:40,343 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:40,961 - INFO: load data for DEBW087 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2022-12-02 14:13:40,961 - INFO: load data for DEBW087 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2022-12-02 14:13:40,961 - INFO: load data for DEBW087 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2022-12-02 14:13:40,961 - INFO: load data for DEBW087 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2022-12-02 14:13:40,961 - INFO: load data for DEBW087 from TOAR-DATA [toar_data_v2.py:download_toar:63] 2022-12-02 14:13:41,047 - INFO: load data for DEBW087 from JOIN [join.py:download_join:63] 2022-12-02 14:13:41,047 - INFO: load data for DEBW087 from JOIN [join.py:download_join:63] 2022-12-02 14:13:41,047 - INFO: load data for DEBW087 from JOIN [join.py:download_join:63] 2022-12-02 14:13:41,047 - INFO: load data for DEBW087 from JOIN [join.py:download_join:63] 2022-12-02 14:13:41,047 - INFO: load data for DEBW087 from JOIN [join.py:download_join:63] 2022-12-02 14:13:44,691 - INFO: setup_samples finished after 0:00:05 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:44,691 - INFO: setup_samples finished after 0:00:05 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:44,691 - INFO: setup_samples finished after 0:00:05 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:44,691 - INFO: setup_samples finished after 0:00:05 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:44,691 - INFO: setup_samples finished after 0:00:05 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:45,003 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:45,003 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:45,003 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:45,003 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:45,003 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:45,004 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:45,004 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:45,004 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:45,004 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:45,004 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:45,276 - INFO: run for 0:04:26 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (preprocessing). [pre_processing.py:validate_station:303] 2022-12-02 14:13:45,276 - INFO: run for 0:04:26 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (preprocessing). [pre_processing.py:validate_station:303] 2022-12-02 14:13:45,276 - INFO: run for 0:04:26 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (preprocessing). [pre_processing.py:validate_station:303] 2022-12-02 14:13:45,276 - INFO: run for 0:04:26 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (preprocessing). [pre_processing.py:validate_station:303] 2022-12-02 14:13:45,276 - INFO: run for 0:04:26 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (preprocessing). [pre_processing.py:validate_station:303] 2022-12-02 14:13:45,277 - INFO: check valid stations started (train) [pre_processing.py:validate_station:261] 2022-12-02 14:13:45,277 - INFO: check valid stations started (train) [pre_processing.py:validate_station:261] 2022-12-02 14:13:45,277 - INFO: check valid stations started (train) [pre_processing.py:validate_station:261] 2022-12-02 14:13:45,277 - INFO: check valid stations started (train) [pre_processing.py:validate_station:261] 2022-12-02 14:13:45,277 - INFO: check valid stations started (train) [pre_processing.py:validate_station:261] 2022-12-02 14:13:45,277 - INFO: setup transformation using train data exclusively [pre_processing.py:validate_station:264] 2022-12-02 14:13:45,277 - INFO: setup transformation using train data exclusively [pre_processing.py:validate_station:264] 2022-12-02 14:13:45,277 - INFO: setup transformation using train data exclusively [pre_processing.py:validate_station:264] 2022-12-02 14:13:45,277 - INFO: setup transformation using train data exclusively [pre_processing.py:validate_station:264] 2022-12-02 14:13:45,277 - INFO: setup transformation using train data exclusively [pre_processing.py:validate_station:264] 2022-12-02 14:13:45,277 - INFO: start to calculate transformation parameters. [pre_processing.py:transformation:327] 2022-12-02 14:13:45,277 - INFO: start to calculate transformation parameters. [pre_processing.py:transformation:327] 2022-12-02 14:13:45,277 - INFO: start to calculate transformation parameters. [pre_processing.py:transformation:327] 2022-12-02 14:13:45,277 - INFO: start to calculate transformation parameters. [pre_processing.py:transformation:327] 2022-12-02 14:13:45,277 - INFO: start to calculate transformation parameters. [pre_processing.py:transformation:327] 2022-12-02 14:13:45,282 - INFO: use serial transformation approach [default_data_handler.py:transformation:322] 2022-12-02 14:13:45,282 - INFO: use serial transformation approach [default_data_handler.py:transformation:322] 2022-12-02 14:13:45,282 - INFO: use serial transformation approach [default_data_handler.py:transformation:322] 2022-12-02 14:13:45,282 - INFO: use serial transformation approach [default_data_handler.py:transformation:322] 2022-12-02 14:13:45,282 - INFO: use serial transformation approach [default_data_handler.py:transformation:322] 2022-12-02 14:13:45,351 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:45,351 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:45,351 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:45,351 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:45,351 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:45,657 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:45,657 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:45,657 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:45,657 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:45,657 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,000 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,000 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,000 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,000 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,000 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,265 - INFO: Store transformation options locally for later use at: /builds/esde/machine-learning/mlair/TestExperiment_daily/data/transformation/transformation.pickle [pre_processing.py:_store_transformation:360] 2022-12-02 14:13:46,265 - INFO: Store transformation options locally for later use at: /builds/esde/machine-learning/mlair/TestExperiment_daily/data/transformation/transformation.pickle [pre_processing.py:_store_transformation:360] 2022-12-02 14:13:46,265 - INFO: Store transformation options locally for later use at: /builds/esde/machine-learning/mlair/TestExperiment_daily/data/transformation/transformation.pickle [pre_processing.py:_store_transformation:360] 2022-12-02 14:13:46,265 - INFO: Store transformation options locally for later use at: /builds/esde/machine-learning/mlair/TestExperiment_daily/data/transformation/transformation.pickle [pre_processing.py:_store_transformation:360] 2022-12-02 14:13:46,265 - INFO: Store transformation options locally for later use at: /builds/esde/machine-learning/mlair/TestExperiment_daily/data/transformation/transformation.pickle [pre_processing.py:_store_transformation:360] 2022-12-02 14:13:46,272 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:13:46,272 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:13:46,272 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:13:46,272 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:13:46,272 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:13:46,336 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,336 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,336 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,336 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,336 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,576 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,576 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,576 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,576 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,576 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,576 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,576 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,576 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,576 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,576 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,889 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,889 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,889 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,889 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:46,889 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,125 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,125 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,125 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,125 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,125 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,125 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,125 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,125 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,125 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,125 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,464 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,464 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,464 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,464 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,464 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,699 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,699 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,699 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,699 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,699 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,700 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,700 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,700 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,700 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,700 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:47,938 - INFO: run for 0:00:03 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train). [pre_processing.py:validate_station:303] 2022-12-02 14:13:47,938 - INFO: run for 0:00:03 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train). [pre_processing.py:validate_station:303] 2022-12-02 14:13:47,938 - INFO: run for 0:00:03 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train). [pre_processing.py:validate_station:303] 2022-12-02 14:13:47,938 - INFO: run for 0:00:03 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train). [pre_processing.py:validate_station:303] 2022-12-02 14:13:47,938 - INFO: run for 0:00:03 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train). [pre_processing.py:validate_station:303] 2022-12-02 14:13:47,939 - INFO: check valid stations started (val) [pre_processing.py:validate_station:261] 2022-12-02 14:13:47,939 - INFO: check valid stations started (val) [pre_processing.py:validate_station:261] 2022-12-02 14:13:47,939 - INFO: check valid stations started (val) [pre_processing.py:validate_station:261] 2022-12-02 14:13:47,939 - INFO: check valid stations started (val) [pre_processing.py:validate_station:261] 2022-12-02 14:13:47,939 - INFO: check valid stations started (val) [pre_processing.py:validate_station:261] 2022-12-02 14:13:47,947 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:13:47,947 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:13:47,947 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:13:47,947 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:13:47,947 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:13:48,013 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,013 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,013 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,013 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,013 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,014 - INFO: remove station DEBW107 because it raised an error: self.history consist of NaNs only - station ['DEBW107'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:452] 2022-12-02 14:13:48,014 - INFO: remove station DEBW107 because it raised an error: self.history consist of NaNs only - station ['DEBW107'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:452] 2022-12-02 14:13:48,014 - INFO: remove station DEBW107 because it raised an error: self.history consist of NaNs only - station ['DEBW107'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:452] 2022-12-02 14:13:48,014 - INFO: remove station DEBW107 because it raised an error: self.history consist of NaNs only - station ['DEBW107'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:452] 2022-12-02 14:13:48,014 - INFO: remove station DEBW107 because it raised an error: self.history consist of NaNs only - station ['DEBW107'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:452] 2022-12-02 14:13:48,078 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,078 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,078 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,078 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,078 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,078 - INFO: remove station DEBW013 because it raised an error: self.history consist of NaNs only - station ['DEBW013'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:452] 2022-12-02 14:13:48,078 - INFO: remove station DEBW013 because it raised an error: self.history consist of NaNs only - station ['DEBW013'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:452] 2022-12-02 14:13:48,078 - INFO: remove station DEBW013 because it raised an error: self.history consist of NaNs only - station ['DEBW013'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:452] 2022-12-02 14:13:48,078 - INFO: remove station DEBW013 because it raised an error: self.history consist of NaNs only - station ['DEBW013'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:452] 2022-12-02 14:13:48,078 - INFO: remove station DEBW013 because it raised an error: self.history consist of NaNs only - station ['DEBW013'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:452] 2022-12-02 14:13:48,140 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,140 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,140 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,140 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,140 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,140 - INFO: remove station DEBW087 because it raised an error: self.history consist of NaNs only - station ['DEBW087'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:452] 2022-12-02 14:13:48,140 - INFO: remove station DEBW087 because it raised an error: self.history consist of NaNs only - station ['DEBW087'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:452] 2022-12-02 14:13:48,140 - INFO: remove station DEBW087 because it raised an error: self.history consist of NaNs only - station ['DEBW087'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:452] 2022-12-02 14:13:48,140 - INFO: remove station DEBW087 because it raised an error: self.history consist of NaNs only - station ['DEBW087'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:452] 2022-12-02 14:13:48,140 - INFO: remove station DEBW087 because it raised an error: self.history consist of NaNs only - station ['DEBW087'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') [pre_processing.py:f_proc:452] 2022-12-02 14:13:48,141 - INFO: run for 0:00:01 (hh:mm:ss) to check 3 station(s). Found 0/3 valid stations (val). [pre_processing.py:validate_station:303] 2022-12-02 14:13:48,141 - INFO: run for 0:00:01 (hh:mm:ss) to check 3 station(s). Found 0/3 valid stations (val). [pre_processing.py:validate_station:303] 2022-12-02 14:13:48,141 - INFO: run for 0:00:01 (hh:mm:ss) to check 3 station(s). Found 0/3 valid stations (val). [pre_processing.py:validate_station:303] 2022-12-02 14:13:48,141 - INFO: run for 0:00:01 (hh:mm:ss) to check 3 station(s). Found 0/3 valid stations (val). [pre_processing.py:validate_station:303] 2022-12-02 14:13:48,141 - INFO: run for 0:00:01 (hh:mm:ss) to check 3 station(s). Found 0/3 valid stations (val). [pre_processing.py:validate_station:303] 2022-12-02 14:13:48,142 - INFO: check valid stations started (test) [pre_processing.py:validate_station:261] 2022-12-02 14:13:48,142 - INFO: check valid stations started (test) [pre_processing.py:validate_station:261] 2022-12-02 14:13:48,142 - INFO: check valid stations started (test) [pre_processing.py:validate_station:261] 2022-12-02 14:13:48,142 - INFO: check valid stations started (test) [pre_processing.py:validate_station:261] 2022-12-02 14:13:48,142 - INFO: check valid stations started (test) [pre_processing.py:validate_station:261] 2022-12-02 14:13:48,149 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:13:48,149 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:13:48,149 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:13:48,149 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:13:48,149 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:13:48,217 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,217 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,217 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,217 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,217 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,486 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,486 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,486 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,486 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,486 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,486 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,486 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,486 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,486 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,486 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,810 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,810 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,810 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,810 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:48,810 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,054 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,054 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,054 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,054 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,054 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,054 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,054 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,054 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,054 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,054 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,362 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,362 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,362 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,362 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,362 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,601 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,601 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,601 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,601 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,601 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,601 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,601 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,601 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,601 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,601 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,839 - INFO: run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (test). [pre_processing.py:validate_station:303] 2022-12-02 14:13:49,839 - INFO: run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (test). [pre_processing.py:validate_station:303] 2022-12-02 14:13:49,839 - INFO: run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (test). [pre_processing.py:validate_station:303] 2022-12-02 14:13:49,839 - INFO: run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (test). [pre_processing.py:validate_station:303] 2022-12-02 14:13:49,839 - INFO: run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (test). [pre_processing.py:validate_station:303] 2022-12-02 14:13:49,840 - INFO: check valid stations started (train_val) [pre_processing.py:validate_station:261] 2022-12-02 14:13:49,840 - INFO: check valid stations started (train_val) [pre_processing.py:validate_station:261] 2022-12-02 14:13:49,840 - INFO: check valid stations started (train_val) [pre_processing.py:validate_station:261] 2022-12-02 14:13:49,840 - INFO: check valid stations started (train_val) [pre_processing.py:validate_station:261] 2022-12-02 14:13:49,840 - INFO: check valid stations started (train_val) [pre_processing.py:validate_station:261] 2022-12-02 14:13:49,848 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:13:49,848 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:13:49,848 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:13:49,848 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:13:49,848 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:13:49,917 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,917 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,917 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,917 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:49,917 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,157 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,157 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,157 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,157 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,157 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,157 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,157 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,157 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,157 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,157 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,459 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,459 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,459 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,459 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,459 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,695 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,695 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,695 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,695 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,695 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,695 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,695 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,695 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,695 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:50,695 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:51,040 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:51,040 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:51,040 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:51,040 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:51,040 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:51,277 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:51,277 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:51,277 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:51,277 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:51,277 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:51,277 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:51,277 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:51,277 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:51,277 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:51,277 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:13:51,516 - INFO: run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train_val). [pre_processing.py:validate_station:303] 2022-12-02 14:13:51,516 - INFO: run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train_val). [pre_processing.py:validate_station:303] 2022-12-02 14:13:51,516 - INFO: run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train_val). [pre_processing.py:validate_station:303] 2022-12-02 14:13:51,516 - INFO: run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train_val). [pre_processing.py:validate_station:303] 2022-12-02 14:13:51,516 - INFO: run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train_val). [pre_processing.py:validate_station:303] 2022-12-02 14:13:51,520 - INFO: use serial create_info_df (train) [pre_processing.py:create_info_df:179] 2022-12-02 14:13:51,520 - INFO: use serial create_info_df (train) [pre_processing.py:create_info_df:179] 2022-12-02 14:13:51,520 - INFO: use serial create_info_df (train) [pre_processing.py:create_info_df:179] 2022-12-02 14:13:51,520 - INFO: use serial create_info_df (train) [pre_processing.py:create_info_df:179] 2022-12-02 14:13:51,520 - INFO: use serial create_info_df (train) [pre_processing.py:create_info_df:179] 2022-12-02 14:13:52,278 - INFO: use serial create_info_df (val) [pre_processing.py:create_info_df:179] 2022-12-02 14:13:52,278 - INFO: use serial create_info_df (val) [pre_processing.py:create_info_df:179] 2022-12-02 14:13:52,278 - INFO: use serial create_info_df (val) [pre_processing.py:create_info_df:179] 2022-12-02 14:13:52,278 - INFO: use serial create_info_df (val) [pre_processing.py:create_info_df:179] 2022-12-02 14:13:52,278 - INFO: use serial create_info_df (val) [pre_processing.py:create_info_df:179] 2022-12-02 14:13:52,279 - INFO: use serial create_info_df (test) [pre_processing.py:create_info_df:179] 2022-12-02 14:13:52,279 - INFO: use serial create_info_df (test) [pre_processing.py:create_info_df:179] 2022-12-02 14:13:52,279 - INFO: use serial create_info_df (test) [pre_processing.py:create_info_df:179] 2022-12-02 14:13:52,279 - INFO: use serial create_info_df (test) [pre_processing.py:create_info_df:179] 2022-12-02 14:13:52,279 - INFO: use serial create_info_df (test) [pre_processing.py:create_info_df:179] 2022-12-02 14:13:53,042 - INFO: Searching for competitors to be prepared for use. [pre_processing.py:prepare_competitors:369] 2022-12-02 14:13:53,042 - INFO: Searching for competitors to be prepared for use. [pre_processing.py:prepare_competitors:369] 2022-12-02 14:13:53,042 - INFO: Searching for competitors to be prepared for use. [pre_processing.py:prepare_competitors:369] 2022-12-02 14:13:53,042 - INFO: Searching for competitors to be prepared for use. [pre_processing.py:prepare_competitors:369] 2022-12-02 14:13:53,042 - INFO: Searching for competitors to be prepared for use. [pre_processing.py:prepare_competitors:369] 2022-12-02 14:13:53,042 - INFO: No preparation required for competitor ols as no specific instruction is provided. [pre_processing.py:prepare_competitors:389] 2022-12-02 14:13:53,042 - INFO: No preparation required for competitor ols as no specific instruction is provided. [pre_processing.py:prepare_competitors:389] 2022-12-02 14:13:53,042 - INFO: No preparation required for competitor ols as no specific instruction is provided. [pre_processing.py:prepare_competitors:389] 2022-12-02 14:13:53,042 - INFO: No preparation required for competitor ols as no specific instruction is provided. [pre_processing.py:prepare_competitors:389] 2022-12-02 14:13:53,042 - INFO: No preparation required for competitor ols as no specific instruction is provided. [pre_processing.py:prepare_competitors:389] 2022-12-02 14:13:53,043 - INFO: No preparation required for competitor persi as no specific instruction is provided. [pre_processing.py:prepare_competitors:389] 2022-12-02 14:13:53,043 - INFO: No preparation required for competitor persi as no specific instruction is provided. [pre_processing.py:prepare_competitors:389] 2022-12-02 14:13:53,043 - INFO: No preparation required for competitor persi as no specific instruction is provided. [pre_processing.py:prepare_competitors:389] 2022-12-02 14:13:53,043 - INFO: No preparation required for competitor persi as no specific instruction is provided. [pre_processing.py:prepare_competitors:389] 2022-12-02 14:13:53,043 - INFO: No preparation required for competitor persi as no specific instruction is provided. [pre_processing.py:prepare_competitors:389] 2022-12-02 14:13:53,044 - INFO: PreProcessing finished after 0:04:34 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:13:53,044 - INFO: PreProcessing finished after 0:04:34 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:13:53,044 - INFO: PreProcessing finished after 0:04:34 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:13:53,044 - INFO: PreProcessing finished after 0:04:34 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:13:53,044 - INFO: PreProcessing finished after 0:04:34 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:13:53,044 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:13:53,044 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:13:53,044 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:13:53,044 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:13:53,044 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:13:53,044 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:13:53,044 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:13:53,044 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:13:53,044 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:13:53,044 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:13:58,896 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json [run_environment.py:__save_tracking:159] 2022-12-02 14:13:58,896 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json [run_environment.py:__save_tracking:159] 2022-12-02 14:13:58,896 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json [run_environment.py:__save_tracking:159] 2022-12-02 14:13:58,896 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json [run_environment.py:__save_tracking:159] 2022-12-02 14:13:58,896 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json [run_environment.py:__save_tracking:159] 2022-12-02 14:13:58,920 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:13:58,920 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:13:58,920 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:13:58,920 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:13:58,920 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147]
-------------------------------Captured log call--------------------------------
INFO root:run_environment.py:103 PreProcessing started INFO root:pre_processing.py:261 check valid stations started (preprocessing) INFO root:pre_processing.py:295 use serial validate station approach INFO root:toar_data_v2.py:63 load data for DEBW107 from TOAR-DATA INFO root:join.py:63 load data for DEBW107 from JOIN INFO root:time_tracking.py:137 setup_samples finished after 0:04:16 (hh:mm:ss) INFO root:run_environment.py:120 PartitionCheck finished after 0:04:19 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:toar_data_v2.py:63 load data for DEBW013 from TOAR-DATA INFO root:join.py:63 load data for DEBW013 from JOIN INFO root:time_tracking.py:137 setup_samples finished after 0:00:05 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:toar_data_v2.py:63 load data for DEBW087 from TOAR-DATA INFO root:join.py:63 load data for DEBW087 from JOIN INFO root:time_tracking.py:137 setup_samples finished after 0:00:05 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:pre_processing.py:303 run for 0:04:26 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (preprocessing). INFO root:pre_processing.py:261 check valid stations started (train) INFO root:pre_processing.py:264 setup transformation using train data exclusively INFO root:pre_processing.py:327 start to calculate transformation parameters. INFO root:default_data_handler.py:322 use serial transformation approach INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:pre_processing.py:360 Store transformation options locally for later use at: /builds/esde/machine-learning/mlair/TestExperiment_daily/data/transformation/transformation.pickle INFO root:pre_processing.py:295 use serial validate station approach INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:pre_processing.py:303 run for 0:00:03 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train). INFO root:pre_processing.py:261 check valid stations started (val) INFO root:pre_processing.py:295 use serial validate station approach INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:pre_processing.py:452 remove station DEBW107 because it raised an error: self.history consist of NaNs only - station ['DEBW107'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:pre_processing.py:452 remove station DEBW013 because it raised an error: self.history consist of NaNs only - station ['DEBW013'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:pre_processing.py:452 remove station DEBW087 because it raised an error: self.history consist of NaNs only - station ['DEBW087'] is therefore dropped -> self.remove_nan(self.time_dim) | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 632, in remove_nan | raise ValueError(f'self.history consist of NaNs only - station {self.station} is therefore dropped') INFO root:pre_processing.py:303 run for 0:00:01 (hh:mm:ss) to check 3 station(s). Found 0/3 valid stations (val). INFO root:pre_processing.py:261 check valid stations started (test) INFO root:pre_processing.py:295 use serial validate station approach INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:pre_processing.py:303 run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (test). INFO root:pre_processing.py:261 check valid stations started (train_val) INFO root:pre_processing.py:295 use serial validate station approach INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:pre_processing.py:303 run for 0:00:02 (hh:mm:ss) to check 3 station(s). Found 3/3 valid stations (train_val). INFO root:pre_processing.py:179 use serial create_info_df (train) INFO root:pre_processing.py:179 use serial create_info_df (val) INFO root:pre_processing.py:179 use serial create_info_df (test) INFO root:pre_processing.py:369 Searching for competitors to be prepared for use. INFO root:pre_processing.py:389 No preparation required for competitor ols as no specific instruction is provided. INFO root:pre_processing.py:389 No preparation required for competitor persi as no specific instruction is provided. INFO root:run_environment.py:120 PreProcessing finished after 0:04:34 (hh:mm:ss) INFO root:run_environment.py:103 RunEnvironment started INFO root:run_environment.py:120 RunEnvironment finished after 0:00:01 (hh:mm:ss) INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_000.json INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log

------------------------------Captured stderr call------------------------------
2022-12-02 14:14:29,399 - INFO: check valid stations started (awesome) [pre_processing.py:validate_station:261] 2022-12-02 14:14:29,399 - INFO: check valid stations started (awesome) [pre_processing.py:validate_station:261] 2022-12-02 14:14:29,399 - INFO: check valid stations started (awesome) [pre_processing.py:validate_station:261] 2022-12-02 14:14:29,399 - INFO: check valid stations started (awesome) [pre_processing.py:validate_station:261] 2022-12-02 14:14:29,399 - INFO: check valid stations started (awesome) [pre_processing.py:validate_station:261] 2022-12-02 14:14:29,406 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:14:29,406 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:14:29,406 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:14:29,406 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:14:29,406 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:14:29,485 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:29,485 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:29,485 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:29,485 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:29,485 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:29,934 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:29,934 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:29,934 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:29,934 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:29,934 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:29,934 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:29,934 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:29,934 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:29,934 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:29,934 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:30,379 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:30,379 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:30,379 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:30,379 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:30,379 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:30,733 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:30,733 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:30,733 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:30,733 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:30,733 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:30,734 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:30,734 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:30,734 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:30,734 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:30,734 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:31,107 - INFO: run for 0:00:02 (hh:mm:ss) to check 2 station(s). Found 2/2 valid stations (awesome). [pre_processing.py:validate_station:303] 2022-12-02 14:14:31,107 - INFO: run for 0:00:02 (hh:mm:ss) to check 2 station(s). Found 2/2 valid stations (awesome). [pre_processing.py:validate_station:303] 2022-12-02 14:14:31,107 - INFO: run for 0:00:02 (hh:mm:ss) to check 2 station(s). Found 2/2 valid stations (awesome). [pre_processing.py:validate_station:303] 2022-12-02 14:14:31,107 - INFO: run for 0:00:02 (hh:mm:ss) to check 2 station(s). Found 2/2 valid stations (awesome). [pre_processing.py:validate_station:303] 2022-12-02 14:14:31,107 - INFO: run for 0:00:02 (hh:mm:ss) to check 2 station(s). Found 2/2 valid stations (awesome). [pre_processing.py:validate_station:303]
-------------------------------Captured log call--------------------------------
DEBUG root:datastore.py:118 set: use_all_stations_on_all_data_sets(general)=False DEBUG root:datastore.py:118 get: stations(general.awesome)=['DEBW107', 'DEBW013', 'DEBW087', 'DEBW99X'] DEBUG root:datastore.py:118 get: use_all_stations_on_all_data_sets(general)=False DEBUG root:pre_processing.py:243 Awesome stations (len=2): ['DEBW107', 'DEBW013'] DEBUG root:datastore.py:118 get: data_handler(general)=<class 'mlair.data_handler.default_data_handler.DefaultDataHandler'> INFO root:pre_processing.py:261 check valid stations started (awesome) DEBUG root:datastore.py:118 get: sampling(general.awesome)=daily DEBUG root:datastore.py:118 get: variables(general.awesome)=['o3', 'temp'] DEBUG root:datastore.py:118 get: window_dim(general.awesome)=window DEBUG root:datastore.py:118 get: window_lead_time(general.awesome)=3 DEBUG root:datastore.py:120 get: overwrite_lazy_data(general.awesome)=None DEBUG root:datastore.py:118 get: time_dim(general.awesome)=datetime DEBUG root:datastore.py:118 get: start(general.awesome)=1997-01-01 DEBUG root:datastore.py:120 get: extreme_values(general.awesome)=None DEBUG root:datastore.py:120 get: overwrite_local_data(general.awesome)=None DEBUG root:datastore.py:118 get: use_multiprocessing(general.awesome)=True DEBUG root:datastore.py:120 get: lazy_preprocessing(general.awesome)=None DEBUG root:datastore.py:118 get: target_dim(general.awesome)=variables DEBUG root:datastore.py:118 get: end(general.awesome)=2017-12-31 DEBUG root:datastore.py:118 get: window_history_size(general.awesome)=13 DEBUG root:datastore.py:120 get: name_affix(general.awesome)=None DEBUG root:datastore.py:118 get: experiment_path(general.awesome)=/builds/esde/machine-learning/mlair/TestExperiment_daily DEBUG root:datastore.py:118 get: target_var(general.awesome)=o3 DEBUG root:datastore.py:120 get: store_data_locally(general.awesome)=None DEBUG root:datastore.py:120 get: window_history_offset(general.awesome)=None DEBUG root:datastore.py:118 get: data_origin(general.awesome)={'o3': 'UBA', 'temp': 'UBA'} DEBUG root:datastore.py:118 get: transformation(general.awesome)={} DEBUG root:datastore.py:118 get: interpolation_limit(general.awesome)=1 DEBUG root:datastore.py:118 get: interpolation_method(general.awesome)=linear DEBUG root:datastore.py:118 get: statistics_per_var(general.awesome)={'o3': 'dma8eu', 'temp': 'maximum'} DEBUG root:datastore.py:120 get: extremes_on_right_tail_only(general.awesome)=None DEBUG root:datastore.py:120 get: min_length(general.awesome)=None DEBUG root:datastore.py:118 get: data_path(general.awesome)=/home/root/mlair/data/ DEBUG root:datastore.py:118 get: iter_dim(general.awesome)=Stations DEBUG root:datastore.py:118 get: max_number_multiprocessing(general.awesome)=16 DEBUG root:datastore.py:120 get: window_history_end(general.awesome)=None DEBUG root:datastore.py:120 get: store_processed_data(general.awesome)=None DEBUG root:datastore.py:118 get: use_multiprocessing(general)=True DEBUG root:datastore.py:118 get: tmp_path(general)=/builds/esde/machine-learning/mlair/TestExperiment_daily/tmp DEBUG root:datastore.py:118 get: max_number_multiprocessing(general)=16 INFO root:pre_processing.py:295 use serial validate station approach DEBUG root:path_config.py:132 Path already exists: /home/root/mlair/data/daily DEBUG root:data_handler_single_station.py:345 try to load local data from: /home/root/mlair/data/daily/DEBW107_o3_temp.nc DEBUG root:data_handler_single_station.py:349 loading finished INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) DEBUG root:default_data_handler.py:218 No extreme values given, skip multiply extremes INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) DEBUG root:default_data_handler.py:97 save pickle data to /builds/esde/machine-learning/mlair/TestExperiment_daily/data/DEBW107_awesome.pickle DEBUG root:path_config.py:132 Path already exists: /home/root/mlair/data/daily DEBUG root:data_handler_single_station.py:345 try to load local data from: /home/root/mlair/data/daily/DEBW013_o3_temp.nc DEBUG root:data_handler_single_station.py:349 loading finished INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) DEBUG root:default_data_handler.py:218 No extreme values given, skip multiply extremes INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) DEBUG root:default_data_handler.py:97 save pickle data to /builds/esde/machine-learning/mlair/TestExperiment_daily/data/DEBW013_awesome.pickle INFO root:pre_processing.py:303 run for 0:00:02 (hh:mm:ss) to check 2 station(s). Found 2/2 valid stations (awesome). DEBUG root:datastore.py:118 set: stations(general.awesome)=['DEBW107', 'DEBW013'] DEBUG root:datastore.py:118 set: data_collection(general.awesome)=<mlair.data_handler.iterator.DataCollection object at 0x7f82b43bf250> DEBUG root:datastore.py:118 get: data_collection(general.awesome)=<mlair.data_handler.iterator.DataCollection object at 0x7f82b43bf250> DEBUG root:datastore.py:120 get: data_collection(general)=None DEBUG root:datastore.py:118 get: stations(general.awesome)=['DEBW107', 'DEBW013']
----------------------------Captured stderr teardown----------------------------
2022-12-02 14:14:31,109 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:31,109 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:31,109 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:31,109 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:31,109 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:31,109 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:31,109 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:31,109 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:31,109 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:31,109 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:35,079 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_003.json [run_environment.py:__save_tracking:159] 2022-12-02 14:14:35,079 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_003.json [run_environment.py:__save_tracking:159] 2022-12-02 14:14:35,079 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_003.json [run_environment.py:__save_tracking:159] 2022-12-02 14:14:35,079 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_003.json [run_environment.py:__save_tracking:159] 2022-12-02 14:14:35,079 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_003.json [run_environment.py:__save_tracking:159] 2022-12-02 14:14:35,113 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:14:35,113 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:14:35,113 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:14:35,113 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:14:35,113 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147]
-----------------------------Captured log teardown------------------------------
INFO root:run_environment.py:103 RunEnvironment started INFO root:run_environment.py:120 RunEnvironment finished after 0:00:01 (hh:mm:ss) DEBUG root:datastore.py:118 get: logging_path(general)=/builds/esde/machine-learning/mlair/TestExperiment_daily/logging DEBUG root:datastore.py:118 get: logging_path(general)=/builds/esde/machine-learning/mlair/TestExperiment_daily/logging INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_003.json DEBUG root:datastore.py:118 get: logging_path(general)=/builds/esde/machine-learning/mlair/TestExperiment_daily/logging INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log

------------------------------Captured stderr call------------------------------
2022-12-02 14:14:35,125 - INFO: check valid stations started (awesome) [pre_processing.py:validate_station:261] 2022-12-02 14:14:35,125 - INFO: check valid stations started (awesome) [pre_processing.py:validate_station:261] 2022-12-02 14:14:35,125 - INFO: check valid stations started (awesome) [pre_processing.py:validate_station:261] 2022-12-02 14:14:35,125 - INFO: check valid stations started (awesome) [pre_processing.py:validate_station:261] 2022-12-02 14:14:35,125 - INFO: check valid stations started (awesome) [pre_processing.py:validate_station:261] 2022-12-02 14:14:35,133 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:14:35,133 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:14:35,133 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:14:35,133 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:14:35,133 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:14:35,212 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:35,212 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:35,212 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:35,212 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:35,212 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:35,647 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:35,647 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:35,647 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:35,647 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:35,647 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:35,648 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:35,648 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:35,648 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:35,648 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:35,648 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:36,097 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:36,097 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:36,097 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:36,097 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:36,097 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:36,510 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:36,510 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:36,510 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:36,510 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:36,510 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:36,510 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:36,510 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:36,510 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:36,510 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:36,510 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:36,959 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:36,959 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:36,959 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:36,959 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:36,959 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:37,329 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:37,329 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:37,329 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:37,329 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:37,329 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:37,330 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:37,330 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:37,330 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:37,330 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:37,330 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:38,042 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:38,042 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:38,042 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:38,042 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:38,042 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:38,045 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:452] 2022-12-02 14:14:38,045 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:452] 2022-12-02 14:14:38,045 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:452] 2022-12-02 14:14:38,045 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:452] 2022-12-02 14:14:38,045 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:452] 2022-12-02 14:14:38,047 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (awesome). [pre_processing.py:validate_station:303] 2022-12-02 14:14:38,047 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (awesome). [pre_processing.py:validate_station:303] 2022-12-02 14:14:38,047 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (awesome). [pre_processing.py:validate_station:303] 2022-12-02 14:14:38,047 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (awesome). [pre_processing.py:validate_station:303] 2022-12-02 14:14:38,047 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (awesome). [pre_processing.py:validate_station:303]
-------------------------------Captured log call--------------------------------
DEBUG root:datastore.py:118 get: stations(general.awesome)=['DEBW107', 'DEBW013', 'DEBW087', 'DEBW99X'] DEBUG root:datastore.py:118 get: use_all_stations_on_all_data_sets(general)=True DEBUG root:pre_processing.py:243 Awesome stations (len=4): ['DEBW107', 'DEBW013', 'DEBW087', 'DEBW99X'] DEBUG root:datastore.py:118 get: data_handler(general)=<class 'mlair.data_handler.default_data_handler.DefaultDataHandler'> INFO root:pre_processing.py:261 check valid stations started (awesome) DEBUG root:datastore.py:118 get: sampling(general.awesome)=daily DEBUG root:datastore.py:118 get: variables(general.awesome)=['o3', 'temp'] DEBUG root:datastore.py:118 get: window_dim(general.awesome)=window DEBUG root:datastore.py:118 get: window_lead_time(general.awesome)=3 DEBUG root:datastore.py:120 get: overwrite_lazy_data(general.awesome)=None DEBUG root:datastore.py:118 get: time_dim(general.awesome)=datetime DEBUG root:datastore.py:118 get: start(general.awesome)=1997-01-01 DEBUG root:datastore.py:120 get: extreme_values(general.awesome)=None DEBUG root:datastore.py:120 get: overwrite_local_data(general.awesome)=None DEBUG root:datastore.py:118 get: use_multiprocessing(general.awesome)=True DEBUG root:datastore.py:120 get: lazy_preprocessing(general.awesome)=None DEBUG root:datastore.py:118 get: target_dim(general.awesome)=variables DEBUG root:datastore.py:118 get: end(general.awesome)=2017-12-31 DEBUG root:datastore.py:118 get: window_history_size(general.awesome)=13 DEBUG root:datastore.py:120 get: name_affix(general.awesome)=None DEBUG root:datastore.py:118 get: experiment_path(general.awesome)=/builds/esde/machine-learning/mlair/TestExperiment_daily DEBUG root:datastore.py:118 get: target_var(general.awesome)=o3 DEBUG root:datastore.py:120 get: store_data_locally(general.awesome)=None DEBUG root:datastore.py:120 get: window_history_offset(general.awesome)=None DEBUG root:datastore.py:118 get: data_origin(general.awesome)={'o3': 'UBA', 'temp': 'UBA'} DEBUG root:datastore.py:118 get: transformation(general.awesome)={} DEBUG root:datastore.py:118 get: interpolation_limit(general.awesome)=1 DEBUG root:datastore.py:118 get: interpolation_method(general.awesome)=linear DEBUG root:datastore.py:118 get: statistics_per_var(general.awesome)={'o3': 'dma8eu', 'temp': 'maximum'} DEBUG root:datastore.py:120 get: extremes_on_right_tail_only(general.awesome)=None DEBUG root:datastore.py:120 get: min_length(general.awesome)=None DEBUG root:datastore.py:118 get: data_path(general.awesome)=/home/root/mlair/data/ DEBUG root:datastore.py:118 get: iter_dim(general.awesome)=Stations DEBUG root:datastore.py:118 get: max_number_multiprocessing(general.awesome)=16 DEBUG root:datastore.py:120 get: window_history_end(general.awesome)=None DEBUG root:datastore.py:120 get: store_processed_data(general.awesome)=None DEBUG root:datastore.py:118 get: use_multiprocessing(general)=True DEBUG root:datastore.py:118 get: tmp_path(general)=/builds/esde/machine-learning/mlair/TestExperiment_daily/tmp DEBUG root:datastore.py:118 get: max_number_multiprocessing(general)=16 INFO root:pre_processing.py:295 use serial validate station approach DEBUG root:path_config.py:132 Path already exists: /home/root/mlair/data/daily DEBUG root:data_handler_single_station.py:345 try to load local data from: /home/root/mlair/data/daily/DEBW107_o3_temp.nc DEBUG root:data_handler_single_station.py:349 loading finished INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) DEBUG root:default_data_handler.py:218 No extreme values given, skip multiply extremes INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) DEBUG root:default_data_handler.py:97 save pickle data to /builds/esde/machine-learning/mlair/TestExperiment_daily/data/DEBW107_awesome.pickle DEBUG root:path_config.py:132 Path already exists: /home/root/mlair/data/daily DEBUG root:data_handler_single_station.py:345 try to load local data from: /home/root/mlair/data/daily/DEBW013_o3_temp.nc DEBUG root:data_handler_single_station.py:349 loading finished INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) DEBUG root:default_data_handler.py:218 No extreme values given, skip multiply extremes INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) DEBUG root:default_data_handler.py:97 save pickle data to /builds/esde/machine-learning/mlair/TestExperiment_daily/data/DEBW013_awesome.pickle DEBUG root:path_config.py:132 Path already exists: /home/root/mlair/data/daily DEBUG root:data_handler_single_station.py:345 try to load local data from: /home/root/mlair/data/daily/DEBW087_o3_temp.nc DEBUG root:data_handler_single_station.py:349 loading finished INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) DEBUG root:default_data_handler.py:218 No extreme values given, skip multiply extremes INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) DEBUG root:default_data_handler.py:97 save pickle data to /builds/esde/machine-learning/mlair/TestExperiment_daily/data/DEBW087_awesome.pickle DEBUG root:path_config.py:132 Path already exists: /home/root/mlair/data/daily DEBUG root:data_handler_single_station.py:345 try to load local data from: /home/root/mlair/data/daily/DEBW99X_o3_temp.nc DEBUG root:data_handler_single_station.py:351 [Errno 2] No such file or directory: b'/home/root/mlair/data/daily/DEBW99X_o3_temp.nc' DEBUG root:data_handler_single_station.py:352 load new data DEBUG urllib3.connectionpool:connectionpool.py:1003 Starting new HTTPS connection (1): toar-data.fz-juelich.de:443 DEBUG urllib3.connectionpool:connectionpool.py:456 https://toar-data.fz-juelich.de:443 "GET /api/v2/variables/o3 HTTP/1.1" 200 156 DEBUG urllib3.connectionpool:connectionpool.py:1003 Starting new HTTPS connection (1): toar-data.fz-juelich.de:443 DEBUG urllib3.connectionpool:connectionpool.py:456 https://toar-data.fz-juelich.de:443 "GET /api/v2/variables/temp HTTP/1.1" 200 161 DEBUG urllib3.connectionpool:connectionpool.py:1003 Starting new HTTPS connection (1): toar-data.fz-juelich.de:443 DEBUG urllib3.connectionpool:connectionpool.py:456 https://toar-data.fz-juelich.de:443 "GET /api/v2/stationmeta/DEBW99X HTTP/1.1" 404 54 DEBUG urllib3.connectionpool:connectionpool.py:1003 Starting new HTTPS connection (1): join.fz-juelich.de:443 DEBUG urllib3.connectionpool:connectionpool.py:456 https://join.fz-juelich.de:443 "GET /services/rest/surfacedata/search/?station_id=DEBW99X&network_name=UBA&as_dict=true&parameter_name=o3,temp&columns=id,network_name,station_id,parameter_name,parameter_label,parameter_attribute HTTP/1.1" 200 2 DEBUG root:join.py:218 ['DEBW99X']: [] INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:pre_processing.py:452 remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") DEBUG root:pre_processing.py:454 detailed information for removal of station DEBW99X: Traceback (most recent call last): File "/opt/venv/lib64/python3.9/site-packages/xarray/backends/file_manager.py", line 199, in _acquire_with_cache_info file = self._cache[self._key] File "/opt/venv/lib64/python3.9/site-packages/xarray/backends/lru_cache.py", line 53, in __getitem__ value = self._cache[key] KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/home/root/mlair/data/daily/DEBW99X_o3_temp.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False))] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 346, in load_data data = xr.open_dataarray(file_name) File "/opt/venv/lib64/python3.9/site-packages/xarray/backends/api.py", line 701, in open_dataarray dataset = open_dataset( File "/opt/venv/lib64/python3.9/site-packages/xarray/backends/api.py", line 572, in open_dataset store = opener(filename_or_obj, **extra_kwargs, **backend_kwargs) File "/opt/venv/lib64/python3.9/site-packages/xarray/backends/netCDF4_.py", line 364, in open return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose) File "/opt/venv/lib64/python3.9/site-packages/xarray/backends/netCDF4_.py", line 314, in __init__ self.format = self.ds.data_model File "/opt/venv/lib64/python3.9/site-packages/xarray/backends/netCDF4_.py", line 373, in ds return self._acquire() File "/opt/venv/lib64/python3.9/site-packages/xarray/backends/netCDF4_.py", line 367, in _acquire with self._manager.acquire_context(needs_lock) as root: File "/usr/lib64/python3.9/contextlib.py", line 119, in __enter__ return next(self.gen) File "/opt/venv/lib64/python3.9/site-packages/xarray/backends/file_manager.py", line 187, in acquire_context file, cached = self._acquire_with_cache_info(needs_lock) File "/opt/venv/lib64/python3.9/site-packages/xarray/backends/file_manager.py", line 205, in _acquire_with_cache_info file = self._opener(*self._args, **kwargs) File "src/netCDF4/_netCDF4.pyx", line 2353, in netCDF4._netCDF4.Dataset.__init__ File "src/netCDF4/_netCDF4.pyx", line 1963, in netCDF4._netCDF4._ensure_nc_success FileNotFoundError: [Errno 2] No such file or directory: b'/home/root/mlair/data/daily/DEBW99X_o3_temp.nc' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/builds/esde/machine-learning/mlair/mlair/run_modules/pre_processing.py", line 449, in f_proc res = data_handler.build(station, name_affix=name_affix, store_processed_data=store, **kwargs) File "/builds/esde/machine-learning/mlair/mlair/data_handler/default_data_handler.py", line 72, in build sp = cls.data_handler(station, **sp_keys) File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 118, in __init__ self.setup_samples() File "/builds/esde/machine-learning/mlair/mlair/helpers/time_tracking.py", line 40, in __call__ return self.__wrapped__(*args, **kwargs) File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 264, in setup_samples self.make_input_target() File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 302, in make_input_target data, self.meta = self.load_data(self.path, self.station, self.statistics_per_var, self.sampling, File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 353, in load_data data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") mlair.helpers.data_sources.toar_data.EmptyQueryResult: No data available for era5 and toar-data INFO root:pre_processing.py:303 run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (awesome). DEBUG root:datastore.py:118 set: stations(general.awesome)=['DEBW107', 'DEBW013', 'DEBW087'] DEBUG root:datastore.py:118 set: data_collection(general.awesome)=<mlair.data_handler.iterator.DataCollection object at 0x7f82b46f2e20> DEBUG root:datastore.py:118 get: data_collection(general.awesome)=<mlair.data_handler.iterator.DataCollection object at 0x7f82b46f2e20> DEBUG root:datastore.py:120 get: data_collection(general)=None DEBUG root:datastore.py:118 get: stations(general.awesome)=['DEBW107', 'DEBW013', 'DEBW087']
----------------------------Captured stderr teardown----------------------------
2022-12-02 14:14:38,052 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:38,052 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:38,052 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:38,052 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:38,052 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:38,053 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:38,053 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:38,053 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:38,053 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:38,053 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:41,993 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_004.json [run_environment.py:__save_tracking:159] 2022-12-02 14:14:41,993 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_004.json [run_environment.py:__save_tracking:159] 2022-12-02 14:14:41,993 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_004.json [run_environment.py:__save_tracking:159] 2022-12-02 14:14:41,993 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_004.json [run_environment.py:__save_tracking:159] 2022-12-02 14:14:41,993 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_004.json [run_environment.py:__save_tracking:159] 2022-12-02 14:14:42,031 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:14:42,031 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:14:42,031 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:14:42,031 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:14:42,031 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147]
-----------------------------Captured log teardown------------------------------
INFO root:run_environment.py:103 RunEnvironment started INFO root:run_environment.py:120 RunEnvironment finished after 0:00:01 (hh:mm:ss) DEBUG root:datastore.py:118 get: logging_path(general)=/builds/esde/machine-learning/mlair/TestExperiment_daily/logging DEBUG root:datastore.py:118 get: logging_path(general)=/builds/esde/machine-learning/mlair/TestExperiment_daily/logging INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_004.json DEBUG root:datastore.py:118 get: logging_path(general)=/builds/esde/machine-learning/mlair/TestExperiment_daily/logging INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log

------------------------------Captured stderr call------------------------------
2022-12-02 14:14:42,045 - INFO: check valid stations started (all) [pre_processing.py:validate_station:261] 2022-12-02 14:14:42,045 - INFO: check valid stations started (all) [pre_processing.py:validate_station:261] 2022-12-02 14:14:42,045 - INFO: check valid stations started (all) [pre_processing.py:validate_station:261] 2022-12-02 14:14:42,045 - INFO: check valid stations started (all) [pre_processing.py:validate_station:261] 2022-12-02 14:14:42,045 - INFO: check valid stations started (all) [pre_processing.py:validate_station:261] 2022-12-02 14:14:42,049 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:14:42,049 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:14:42,049 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:14:42,049 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:14:42,049 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:14:42,130 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:42,130 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:42,130 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:42,130 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:42,130 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:42,616 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:42,616 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:42,616 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:42,616 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:42,616 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:42,616 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:42,616 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:42,616 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:42,616 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:42,616 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:43,082 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:43,082 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:43,082 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:43,082 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:43,082 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:43,483 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:43,483 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:43,483 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:43,483 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:43,483 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:43,484 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:43,484 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:43,484 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:43,484 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:43,484 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:43,950 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:43,950 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:43,950 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:43,950 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:43,950 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:44,348 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:44,348 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:44,348 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:44,348 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:44,348 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:44,348 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:44,348 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:44,348 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:44,348 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:44,348 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:44,957 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:44,957 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:44,957 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:44,957 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:44,957 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:44,960 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:452] 2022-12-02 14:14:44,960 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:452] 2022-12-02 14:14:44,960 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:452] 2022-12-02 14:14:44,960 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:452] 2022-12-02 14:14:44,960 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:452] 2022-12-02 14:14:44,961 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None). [pre_processing.py:validate_station:303] 2022-12-02 14:14:44,961 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None). [pre_processing.py:validate_station:303] 2022-12-02 14:14:44,961 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None). [pre_processing.py:validate_station:303] 2022-12-02 14:14:44,961 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None). [pre_processing.py:validate_station:303] 2022-12-02 14:14:44,961 - INFO: run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None). [pre_processing.py:validate_station:303]
-------------------------------Captured log call--------------------------------
INFO root:pre_processing.py:261 check valid stations started (all) INFO root:pre_processing.py:295 use serial validate station approach INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:pre_processing.py:452 remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") INFO root:pre_processing.py:303 run for 0:00:03 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None).
----------------------------Captured stderr teardown----------------------------
2022-12-02 14:14:44,966 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:44,966 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:44,966 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:44,966 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:44,966 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:44,966 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:44,966 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:44,966 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:44,966 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:44,966 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:48,982 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_005.json [run_environment.py:__save_tracking:159] 2022-12-02 14:14:48,982 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_005.json [run_environment.py:__save_tracking:159] 2022-12-02 14:14:48,982 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_005.json [run_environment.py:__save_tracking:159] 2022-12-02 14:14:48,982 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_005.json [run_environment.py:__save_tracking:159] 2022-12-02 14:14:48,982 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_005.json [run_environment.py:__save_tracking:159] 2022-12-02 14:14:49,020 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:14:49,020 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:14:49,020 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:14:49,020 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:14:49,020 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147]
-----------------------------Captured log teardown------------------------------
INFO root:run_environment.py:103 RunEnvironment started INFO root:run_environment.py:120 RunEnvironment finished after 0:00:01 (hh:mm:ss) INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_005.json INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log

------------------------------Captured stderr call------------------------------
2022-12-02 14:14:49,031 - INFO: check valid stations started (tester) [pre_processing.py:validate_station:261] 2022-12-02 14:14:49,031 - INFO: check valid stations started (tester) [pre_processing.py:validate_station:261] 2022-12-02 14:14:49,031 - INFO: check valid stations started (tester) [pre_processing.py:validate_station:261] 2022-12-02 14:14:49,031 - INFO: check valid stations started (tester) [pre_processing.py:validate_station:261] 2022-12-02 14:14:49,031 - INFO: check valid stations started (tester) [pre_processing.py:validate_station:261] 2022-12-02 14:14:49,037 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:14:49,037 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:14:49,037 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:14:49,037 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:14:49,037 - INFO: use serial validate station approach [pre_processing.py:validate_station:295] 2022-12-02 14:14:49,114 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:49,114 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:49,114 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:49,114 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:49,114 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:49,600 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:49,600 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:49,600 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:49,600 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:49,600 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:49,601 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:49,601 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:49,601 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:49,601 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:49,601 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:50,102 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:50,102 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:50,102 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:50,102 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:50,102 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:50,515 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:50,515 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:50,515 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:50,515 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:50,515 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:50,515 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:50,515 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:50,515 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:50,515 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:50,515 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:51,017 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:51,017 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:51,017 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:51,017 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:51,017 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:51,473 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:51,473 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:51,473 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:51,473 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:51,473 - INFO: harmonise_X finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:51,473 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:51,473 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:51,473 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:51,473 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:51,473 - INFO: multiply_extremes finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:52,049 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:52,049 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:52,049 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:52,049 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:52,049 - INFO: setup_samples finished after 0:00:01 (hh:mm:ss) [time_tracking.py:__exit__:137] 2022-12-02 14:14:52,052 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:452] 2022-12-02 14:14:52,052 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:452] 2022-12-02 14:14:52,052 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:452] 2022-12-02 14:14:52,052 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:452] 2022-12-02 14:14:52,052 - INFO: remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") [pre_processing.py:f_proc:452] 2022-12-02 14:14:52,053 - INFO: run for 0:00:04 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (tester). [pre_processing.py:validate_station:303] 2022-12-02 14:14:52,053 - INFO: run for 0:00:04 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (tester). [pre_processing.py:validate_station:303] 2022-12-02 14:14:52,053 - INFO: run for 0:00:04 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (tester). [pre_processing.py:validate_station:303] 2022-12-02 14:14:52,053 - INFO: run for 0:00:04 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (tester). [pre_processing.py:validate_station:303] 2022-12-02 14:14:52,053 - INFO: run for 0:00:04 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (tester). [pre_processing.py:validate_station:303]
-------------------------------Captured log call--------------------------------
INFO root:pre_processing.py:261 check valid stations started (tester) INFO root:pre_processing.py:295 use serial validate station approach INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 harmonise_X finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 multiply_extremes finished after 0:00:01 (hh:mm:ss) INFO root:time_tracking.py:137 setup_samples finished after 0:00:01 (hh:mm:ss) INFO root:pre_processing.py:452 remove station DEBW99X because it raised an error: No data available for era5 and toar-data -> data, meta = self.download_data(file_name, meta_file, station, statistics_per_var, sampling, | File "/builds/esde/machine-learning/mlair/mlair/data_handler/data_handler_single_station.py", line 402, in download_data | raise data_sources.toar_data.EmptyQueryResult(f"No data available for era5 and toar-data") INFO root:pre_processing.py:303 run for 0:00:04 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (tester).
----------------------------Captured stderr teardown----------------------------
2022-12-02 14:14:52,056 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:52,056 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:52,056 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:52,056 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:52,056 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:52,057 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:52,057 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:52,057 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:52,057 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:52,057 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:56,916 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_006.json [run_environment.py:__save_tracking:159] 2022-12-02 14:14:56,916 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_006.json [run_environment.py:__save_tracking:159] 2022-12-02 14:14:56,916 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_006.json [run_environment.py:__save_tracking:159] 2022-12-02 14:14:56,916 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_006.json [run_environment.py:__save_tracking:159] 2022-12-02 14:14:56,916 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_006.json [run_environment.py:__save_tracking:159] 2022-12-02 14:14:56,958 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:14:56,958 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:14:56,958 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:14:56,958 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:14:56,958 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147]
-----------------------------Captured log teardown------------------------------
INFO root:run_environment.py:103 RunEnvironment started INFO root:run_environment.py:120 RunEnvironment finished after 0:00:01 (hh:mm:ss) INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_006.json INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log

------------------------------Captured stderr call------------------------------
2022-12-02 14:14:56,972 - INFO: check valid stations started (all) [pre_processing.py:validate_station:261] 2022-12-02 14:14:56,972 - INFO: check valid stations started (all) [pre_processing.py:validate_station:261] 2022-12-02 14:14:56,972 - INFO: check valid stations started (all) [pre_processing.py:validate_station:261] 2022-12-02 14:14:56,972 - INFO: check valid stations started (all) [pre_processing.py:validate_station:261] 2022-12-02 14:14:56,972 - INFO: check valid stations started (all) [pre_processing.py:validate_station:261] 2022-12-02 14:14:56,976 - INFO: use parallel validate station approach [pre_processing.py:validate_station:276] 2022-12-02 14:14:56,976 - INFO: use parallel validate station approach [pre_processing.py:validate_station:276] 2022-12-02 14:14:56,976 - INFO: use parallel validate station approach [pre_processing.py:validate_station:276] 2022-12-02 14:14:56,976 - INFO: use parallel validate station approach [pre_processing.py:validate_station:276] 2022-12-02 14:14:56,976 - INFO: use parallel validate station approach [pre_processing.py:validate_station:276] 2022-12-02 14:14:56,977 - INFO: running 3 processes in parallel [pre_processing.py:validate_station:278] 2022-12-02 14:14:56,977 - INFO: running 3 processes in parallel [pre_processing.py:validate_station:278] 2022-12-02 14:14:56,977 - INFO: running 3 processes in parallel [pre_processing.py:validate_station:278] 2022-12-02 14:14:56,977 - INFO: running 3 processes in parallel [pre_processing.py:validate_station:278] 2022-12-02 14:14:56,977 - INFO: running 3 processes in parallel [pre_processing.py:validate_station:278] 2022-12-02 14:14:57,482 - INFO: ...finished: DEBW107 (25%) [pre_processing.py:validate_station:285] 2022-12-02 14:14:57,482 - INFO: ...finished: DEBW107 (25%) [pre_processing.py:validate_station:285] 2022-12-02 14:14:57,482 - INFO: ...finished: DEBW107 (25%) [pre_processing.py:validate_station:285] 2022-12-02 14:14:57,482 - INFO: ...finished: DEBW107 (25%) [pre_processing.py:validate_station:285] 2022-12-02 14:14:57,482 - INFO: ...finished: DEBW107 (25%) [pre_processing.py:validate_station:285] 2022-12-02 14:14:57,510 - INFO: ...finished: DEBW013 (50%) [pre_processing.py:validate_station:285] 2022-12-02 14:14:57,510 - INFO: ...finished: DEBW013 (50%) [pre_processing.py:validate_station:285] 2022-12-02 14:14:57,510 - INFO: ...finished: DEBW013 (50%) [pre_processing.py:validate_station:285] 2022-12-02 14:14:57,510 - INFO: ...finished: DEBW013 (50%) [pre_processing.py:validate_station:285] 2022-12-02 14:14:57,510 - INFO: ...finished: DEBW013 (50%) [pre_processing.py:validate_station:285] 2022-12-02 14:14:57,520 - INFO: ...finished: DEBW087 (75%) [pre_processing.py:validate_station:285] 2022-12-02 14:14:57,520 - INFO: ...finished: DEBW087 (75%) [pre_processing.py:validate_station:285] 2022-12-02 14:14:57,520 - INFO: ...finished: DEBW087 (75%) [pre_processing.py:validate_station:285] 2022-12-02 14:14:57,520 - INFO: ...finished: DEBW087 (75%) [pre_processing.py:validate_station:285] 2022-12-02 14:14:57,520 - INFO: ...finished: DEBW087 (75%) [pre_processing.py:validate_station:285] 2022-12-02 14:14:57,648 - INFO: ...finished: DEBW99X (100%) [pre_processing.py:validate_station:285] 2022-12-02 14:14:57,648 - INFO: ...finished: DEBW99X (100%) [pre_processing.py:validate_station:285] 2022-12-02 14:14:57,648 - INFO: ...finished: DEBW99X (100%) [pre_processing.py:validate_station:285] 2022-12-02 14:14:57,648 - INFO: ...finished: DEBW99X (100%) [pre_processing.py:validate_station:285] 2022-12-02 14:14:57,648 - INFO: ...finished: DEBW99X (100%) [pre_processing.py:validate_station:285] 2022-12-02 14:14:57,679 - INFO: run for 0:00:01 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None). [pre_processing.py:validate_station:303] 2022-12-02 14:14:57,679 - INFO: run for 0:00:01 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None). [pre_processing.py:validate_station:303] 2022-12-02 14:14:57,679 - INFO: run for 0:00:01 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None). [pre_processing.py:validate_station:303] 2022-12-02 14:14:57,679 - INFO: run for 0:00:01 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None). [pre_processing.py:validate_station:303] 2022-12-02 14:14:57,679 - INFO: run for 0:00:01 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None). [pre_processing.py:validate_station:303]
-------------------------------Captured log call--------------------------------
INFO root:pre_processing.py:261 check valid stations started (all) INFO root:pre_processing.py:276 use parallel validate station approach INFO root:pre_processing.py:278 running 3 processes in parallel INFO root:pre_processing.py:285 ...finished: DEBW107 (25%) INFO root:pre_processing.py:285 ...finished: DEBW013 (50%) INFO root:pre_processing.py:285 ...finished: DEBW087 (75%) INFO root:pre_processing.py:285 ...finished: DEBW99X (100%) INFO root:pre_processing.py:303 run for 0:00:01 (hh:mm:ss) to check 4 station(s). Found 3/4 valid stations (None).
----------------------------Captured stderr teardown----------------------------
2022-12-02 14:14:57,682 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:57,682 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:57,682 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:57,682 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:57,682 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:14:57,683 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:57,683 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:57,683 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:57,683 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:14:57,683 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:01,958 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_007.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:01,958 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_007.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:01,958 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_007.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:01,958 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_007.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:01,958 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_007.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:02,001 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:15:02,001 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:15:02,001 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:15:02,001 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:15:02,001 - INFO: Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log [run_environment.py:__move_log_file:147]
-----------------------------Captured log teardown------------------------------
INFO root:run_environment.py:103 RunEnvironment started INFO root:run_environment.py:120 RunEnvironment finished after 0:00:01 (hh:mm:ss) INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/tracking_007.json INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/TestExperiment_daily/logging/logging_000.log

------------------------------Captured stderr call------------------------------
2022-12-02 14:15:06,476 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:15:06,476 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:15:06,476 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:15:06,476 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:15:06,476 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:15:06,477 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:06,477 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:06,477 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:06,477 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:06,477 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:11,326 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:11,326 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:11,326 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:11,326 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:11,326 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:11,371 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:15:11,371 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:15:11,371 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:15:11,371 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:15:11,371 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147]
-------------------------------Captured log call--------------------------------
INFO root:run_environment.py:103 RunEnvironment started INFO root:run_environment.py:120 RunEnvironment finished after 0:00:01 (hh:mm:ss) INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/logging_000.log

------------------------------Captured stderr call------------------------------
2022-12-02 14:15:11,374 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:15:11,374 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:15:11,374 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:15:11,374 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:15:11,374 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:15:11,475 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:11,475 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:11,475 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:11,475 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:11,475 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:15,857 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_001.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:15,857 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_001.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:15,857 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_001.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:15,857 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_001.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:15,857 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_001.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:15,897 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:15:15,897 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:15:15,897 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:15:15,897 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:15:15,897 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147]
-------------------------------Captured log call--------------------------------
INFO root:run_environment.py:103 RunEnvironment started INFO root:run_environment.py:120 RunEnvironment finished after 0:00:01 (hh:mm:ss) INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/tracking_001.json INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/logging_000.log

------------------------------Captured stderr call------------------------------
2022-12-02 14:15:15,901 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:15:15,901 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:15:15,901 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:15:15,901 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:15:15,901 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:15:15,901 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:15,901 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:15,901 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:15,901 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:15,901 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:20,015 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_002.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:20,015 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_002.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:20,015 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_002.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:20,015 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_002.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:20,015 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_002.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:20,054 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:15:20,054 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:15:20,054 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:15:20,054 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:15:20,054 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147]
-------------------------------Captured log call--------------------------------
INFO root:run_environment.py:103 RunEnvironment started INFO root:run_environment.py:120 RunEnvironment finished after 0:00:01 (hh:mm:ss) INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/tracking_002.json INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/logging_000.log

------------------------------Captured stderr call------------------------------
2022-12-02 14:15:20,057 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:15:20,057 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:15:20,057 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:15:20,057 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:15:20,057 - INFO: RunEnvironment started [run_environment.py:__init__:103] 2022-12-02 14:15:20,257 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:20,257 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:20,257 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:20,257 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:20,257 - INFO: RunEnvironment finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:24,532 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_003.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:24,532 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_003.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:24,532 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_003.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:24,532 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_003.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:24,532 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_003.json [run_environment.py:__save_tracking:159] 2022-12-02 14:15:24,573 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:15:24,573 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:15:24,573 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:15:24,573 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:15:24,573 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147]
-------------------------------Captured log call--------------------------------
INFO root:run_environment.py:103 RunEnvironment started INFO root:run_environment.py:120 RunEnvironment finished after 0:00:01 (hh:mm:ss) INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/tracking_003.json INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/logging_000.log

-----------------------------Captured stdout setup------------------------------
Model: "model_8" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_33 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_2 (Flatten) (None, 16) 0 _________________________________________________________________ dense_9 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_10 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None
------------------------------Captured stdout call------------------------------
Epoch 1/2 13/13 - 1s - loss: 20.2893 - val_loss: 10.8208 Epoch 2/2 13/13 - 0s - loss: 16.4509 - val_loss: 10.9605
------------------------------Captured stderr call------------------------------
2022-12-02 14:15:30.812574: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.

-----------------------------Captured stdout setup------------------------------
Model: "model_9" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_34 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_3 (Flatten) (None, 16) 0 _________________________________________________________________ dense_11 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_12 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None
------------------------------Captured stderr call------------------------------
2022-12-02 14:15:42,225 - INFO: Training started [run_environment.py:__init__:103] 2022-12-02 14:15:42,225 - INFO: Training started [run_environment.py:__init__:103] 2022-12-02 14:15:42,225 - INFO: Training started [run_environment.py:__init__:103] 2022-12-02 14:15:42,225 - INFO: Training started [run_environment.py:__init__:103] 2022-12-02 14:15:42,225 - INFO: Training started [run_environment.py:__init__:103] 2022-12-02 14:15:42,226 - INFO: No training has started, because train_model parameter was false. [training.py:_run:92] 2022-12-02 14:15:42,226 - INFO: No training has started, because train_model parameter was false. [training.py:_run:92] 2022-12-02 14:15:42,226 - INFO: No training has started, because train_model parameter was false. [training.py:_run:92] 2022-12-02 14:15:42,226 - INFO: No training has started, because train_model parameter was false. [training.py:_run:92] 2022-12-02 14:15:42,226 - INFO: No training has started, because train_model parameter was false. [training.py:_run:92] 2022-12-02 14:15:42,226 - INFO: Training finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:42,226 - INFO: Training finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:42,226 - INFO: Training finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:42,226 - INFO: Training finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:15:42,226 - INFO: Training finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120]
-------------------------------Captured log call--------------------------------
INFO root:run_environment.py:103 Training started INFO root:training.py:92 No training has started, because train_model parameter was false. INFO root:run_environment.py:120 Training finished after 0:00:01 (hh:mm:ss)

-----------------------------Captured stdout setup------------------------------
Model: "model_10" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_35 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_4 (Flatten) (None, 16) 0 _________________________________________________________________ dense_13 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_14 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None
------------------------------Captured stdout call------------------------------
Epoch 1/2 13/13 - 1s - loss: 12665.8867 - mse: 539.0455 - mae: 18.1283 - var_loss: 24792.7285 - val_loss: 8959.4746 - val_mse: 414.6833 - val_mae: 16.1960 - val_var_loss: 17504.2656 Epoch 2/2 13/13 - 0s - loss: 4774.0361 - mse: 341.8710 - mae: 14.6364 - var_loss: 9206.2021 - val_loss: 1050.7948 - val_mse: 295.0338 - val_mae: 13.5856 - val_var_loss: 1806.5555

-----------------------------Captured stdout setup------------------------------
Model: "model_11" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_36 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_5 (Flatten) (None, 16) 0 _________________________________________________________________ dense_15 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_16 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None

-----------------------------Captured stdout setup------------------------------
Model: "model_12" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_37 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_6 (Flatten) (None, 16) 0 _________________________________________________________________ dense_17 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_18 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None

-----------------------------Captured stdout setup------------------------------
Model: "model_13" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_38 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_7 (Flatten) (None, 16) 0 _________________________________________________________________ dense_19 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_20 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None

-----------------------------Captured stdout setup------------------------------
Model: "model_14" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_39 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_8 (Flatten) (None, 16) 0 _________________________________________________________________ dense_21 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_22 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None
------------------------------Captured stdout call------------------------------
Epoch 1/2 13/13 - 1s - loss: 17.6535 - val_loss: 24.4831 Epoch 2/2 13/13 - 0s - loss: 21.3837 - val_loss: 10.1362

-----------------------------Captured stdout setup------------------------------
Model: "model_15" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_40 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_9 (Flatten) (None, 16) 0 _________________________________________________________________ dense_23 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_24 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None
------------------------------Captured stderr call------------------------------
2022-12-02 14:16:26,201 - WARNING: Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. [saving_utils.py:try_build_compiled_arguments:313] 2022-12-02 14:16:26,201 - WARNING: Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. [saving_utils.py:try_build_compiled_arguments:313] 2022-12-02 14:16:26,201 - WARNING: Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. [saving_utils.py:try_build_compiled_arguments:313] 2022-12-02 14:16:26,201 - WARNING: Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. [saving_utils.py:try_build_compiled_arguments:313] 2022-12-02 14:16:26,201 - WARNING: Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. [saving_utils.py:try_build_compiled_arguments:313] 2022-12-02 14:16:26,212 - WARNING: Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. [saving_utils.py:try_build_compiled_arguments:313] 2022-12-02 14:16:26,212 - WARNING: Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. [saving_utils.py:try_build_compiled_arguments:313] 2022-12-02 14:16:26,212 - WARNING: Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. [saving_utils.py:try_build_compiled_arguments:313] 2022-12-02 14:16:26,212 - WARNING: Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. [saving_utils.py:try_build_compiled_arguments:313] 2022-12-02 14:16:26,212 - WARNING: Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. [saving_utils.py:try_build_compiled_arguments:313]
-------------------------------Captured log call--------------------------------
DEBUG root:datastore.py:118 get: model_name(general.model)=/builds/esde/machine-learning/mlair/test/test_run_modules/TestExperiment/model/test_model.h5 DEBUG root:training.py:188 save model to /builds/esde/machine-learning/mlair/test/test_run_modules/TestExperiment/model/test_model.h5 WARNING tensorflow:saving_utils.py:313 Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. WARNING tensorflow:saving_utils.py:313 Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model. DEBUG root:datastore.py:118 set: model(general)=<mlair.model_modules.fully_connected_networks.FCN object at 0x7f8506263b20>

-----------------------------Captured stdout setup------------------------------
Model: "model_16" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_41 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_10 (Flatten) (None, 16) 0 _________________________________________________________________ dense_25 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_26 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None

-----------------------------Captured stdout setup------------------------------
Model: "model_17" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_42 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_11 (Flatten) (None, 16) 0 _________________________________________________________________ dense_27 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_28 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None

-----------------------------Captured stdout setup------------------------------
Model: "model_18" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_43 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_12 (Flatten) (None, 16) 0 _________________________________________________________________ dense_29 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_30 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None

-----------------------------Captured stdout setup------------------------------
Model: "model_19" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_44 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_13 (Flatten) (None, 16) 0 _________________________________________________________________ dense_31 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_32 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None

-----------------------------Captured stdout setup------------------------------
Model: "model_20" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_45 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_14 (Flatten) (None, 16) 0 _________________________________________________________________ dense_33 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_34 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None

------------------------------Captured stdout call------------------------------
Model: "model_21" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_46 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_15 (Flatten) (None, 16) 0 _________________________________________________________________ dense_35 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_36 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None Epoch 1/4 13/13 - 1s - loss: 22534.6133 - mse: 1589.0845 - mae: 32.8977 - var_loss: 43480.1484 - val_loss: 15841.4570 - val_mse: 1449.6302 - val_mae: 32.0471 - val_var_loss: 30233.2832 Epoch 2/4 13/13 - 0s - loss: 6880.6729 - mse: 1320.6576 - mae: 29.1475 - var_loss: 12440.6924 - val_loss: 9969.6289 - val_mse: 1223.2863 - val_mae: 26.6917 - val_var_loss: 18715.9707 Epoch 3/4 13/13 - 0s - loss: 10609.7129 - mse: 1168.1818 - mae: 26.9065 - var_loss: 20051.2383 - val_loss: 4382.9307 - val_mse: 1107.1632 - val_mae: 26.3912 - val_var_loss: 7658.6973 Epoch 4/4 13/13 - 0s - loss: 5212.3369 - mse: 1077.3201 - mae: 25.4961 - var_loss: 9347.3555 - val_loss: 3739.9001 - val_mse: 1046.4552 - val_mae: 25.0522 - val_var_loss: 6433.3457 Model: "model_22" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_47 (InputLayer) [(None, 8, 1, 2)] 0 _________________________________________________________________ flatten_16 (Flatten) (None, 16) 0 _________________________________________________________________ dense_37 (Dense) (None, 10) 170 _________________________________________________________________ relu_1 (ReLU) (None, 10) 0 _________________________________________________________________ dense_38 (Dense) (None, 2) 22 _________________________________________________________________ linear_output (Activation) (None, 2) 0 ================================================================= Total params: 192 Trainable params: 192 Non-trainable params: 0 _________________________________________________________________ None Epoch 5/8 13/13 - 1s - loss: 4517.4683 - mse: 1030.0127 - mae: 25.1768 - var_loss: 8004.9229 - val_loss: 4267.0732 - val_mse: 1016.7682 - val_mae: 25.2271 - val_var_loss: 7517.3784 Epoch 6/8 13/13 - 0s - loss: 2926.0154 - mse: 1017.7377 - mae: 24.9686 - var_loss: 4834.2935 - val_loss: 4110.5698 - val_mse: 1043.1619 - val_mae: 25.0585 - val_var_loss: 7177.9775 Epoch 7/8 13/13 - 0s - loss: 3374.6968 - mse: 1019.4047 - mae: 24.8248 - var_loss: 5729.9888 - val_loss: 3125.1057 - val_mse: 990.9284 - val_mae: 24.7217 - val_var_loss: 5259.2832 Epoch 8/8 13/13 - 0s - loss: 3282.2046 - mse: 990.1358 - mae: 24.8492 - var_loss: 5574.2729 - val_loss: 2724.8208 - val_mse: 1023.5216 - val_mae: 25.3608 - val_var_loss: 4426.1201
------------------------------Captured stderr call------------------------------
2022-12-02 14:17:07.101769: W tensorflow/core/util/tensor_slice_reader.cc:95] Could not open /builds/esde/machine-learning/mlair/test/test_run_modules/TestExperiment/model_checkpoint: Failed precondition: /builds/esde/machine-learning/mlair/test/test_run_modules/TestExperiment/model_checkpoint; Is a directory: perhaps your file is in a different file format and you need to use a different restore operator?

------------------------------Captured stderr call------------------------------
2022-12-02 14:17:11,267 - INFO: Workflow started [run_environment.py:__init__:103] 2022-12-02 14:17:11,267 - INFO: Workflow started [run_environment.py:__init__:103] 2022-12-02 14:17:11,267 - INFO: Workflow started [run_environment.py:__init__:103] 2022-12-02 14:17:11,267 - INFO: Workflow started [run_environment.py:__init__:103] 2022-12-02 14:17:11,267 - INFO: Workflow started [run_environment.py:__init__:103] 2022-12-02 14:17:11,268 - INFO: 6 [test_abstract_workflow.py:__init__:37] 2022-12-02 14:17:11,268 - INFO: 6 [test_abstract_workflow.py:__init__:37] 2022-12-02 14:17:11,268 - INFO: 6 [test_abstract_workflow.py:__init__:37] 2022-12-02 14:17:11,268 - INFO: 6 [test_abstract_workflow.py:__init__:37] 2022-12-02 14:17:11,268 - INFO: 6 [test_abstract_workflow.py:__init__:37] 2022-12-02 14:17:11,268 - INFO: 2 [test_abstract_workflow.py:__init__:42] 2022-12-02 14:17:11,268 - INFO: 2 [test_abstract_workflow.py:__init__:42] 2022-12-02 14:17:11,268 - INFO: 2 [test_abstract_workflow.py:__init__:42] 2022-12-02 14:17:11,268 - INFO: 2 [test_abstract_workflow.py:__init__:42] 2022-12-02 14:17:11,268 - INFO: 2 [test_abstract_workflow.py:__init__:42] 2022-12-02 14:17:11,268 - INFO: 3 [test_abstract_workflow.py:__init__:37] 2022-12-02 14:17:11,268 - INFO: 3 [test_abstract_workflow.py:__init__:37] 2022-12-02 14:17:11,268 - INFO: 3 [test_abstract_workflow.py:__init__:37] 2022-12-02 14:17:11,268 - INFO: 3 [test_abstract_workflow.py:__init__:37] 2022-12-02 14:17:11,268 - INFO: 3 [test_abstract_workflow.py:__init__:37] 2022-12-02 14:17:11,268 - INFO: Workflow finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:17:11,268 - INFO: Workflow finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:17:11,268 - INFO: Workflow finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:17:11,268 - INFO: Workflow finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:17:11,268 - INFO: Workflow finished after 0:00:01 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:17:13,037 - INFO: Training finished after 0:00:13 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:17:13,037 - INFO: Training finished after 0:00:13 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:17:13,037 - INFO: Training finished after 0:00:13 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:17:13,037 - INFO: Training finished after 0:00:13 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:17:13,037 - INFO: Training finished after 0:00:13 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:17:13,038 - INFO: Training finished after 0:00:07 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:17:13,038 - INFO: Training finished after 0:00:07 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:17:13,038 - INFO: Training finished after 0:00:07 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:17:13,038 - INFO: Training finished after 0:00:07 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:17:13,038 - INFO: Training finished after 0:00:07 (hh:mm:ss) [run_environment.py:__del__:120] 2022-12-02 14:17:16,446 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json [run_environment.py:__save_tracking:159] 2022-12-02 14:17:16,446 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json [run_environment.py:__save_tracking:159] 2022-12-02 14:17:16,446 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json [run_environment.py:__save_tracking:159] 2022-12-02 14:17:16,446 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json [run_environment.py:__save_tracking:159] 2022-12-02 14:17:16,446 - INFO: Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json [run_environment.py:__save_tracking:159] 2022-12-02 14:17:16,491 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:17:16,491 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:17:16,491 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:17:16,491 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147] 2022-12-02 14:17:16,491 - INFO: Move log file to /builds/esde/machine-learning/mlair/logging_000.log [run_environment.py:__move_log_file:147]
-------------------------------Captured log call--------------------------------
INFO root:run_environment.py:103 Workflow started INFO root:test_abstract_workflow.py:37 6 INFO root:test_abstract_workflow.py:42 2 INFO root:test_abstract_workflow.py:37 3 INFO root:run_environment.py:120 Workflow finished after 0:00:01 (hh:mm:ss) INFO root:run_environment.py:120 Training finished after 0:00:13 (hh:mm:ss) INFO root:run_environment.py:120 Training finished after 0:00:07 (hh:mm:ss) INFO root:run_environment.py:159 Copy tracker file to /builds/esde/machine-learning/mlair/tracking_000.json INFO root:run_environment.py:147 Move log file to /builds/esde/machine-learning/mlair/logging_000.log