Pipeline tasks
-
10h - automate past meteo, NDVI, enogis download and conversion -
? depends on source data, setup meeting with enogis - find and automate future meteo -
optimize ram usage of stack and marge (marare) (upper bound 32gb of RAM) -
4h - check the amount of memory used by one year for one cellar (worst case scenario with the biggest cellar) -
12h (common A) - write a function that adds one 2d array to the 3d ndvi array contained in the stack, in this way the ndvi addition is incremental (and should be less ram greedy)
-
-
optimize train and inference procedures of shortfuture -
12h (common A) - write a function that adds one 2d array to the 3d ndvi array contained in the stack, in this way the ndvi addition is incremental (and should be less ram greedy) -
3h (secondary task) - be able to load a subsequence of the dataset during inference time to avoid computation from dayofyear 1 -
12h (common A) - incremental inference -
3h - test train loop with all new data (multi cellar and year)
-
-
? - deferring computation to cluster, only for tasks that require too much RAM -
3h - setup airflow -
16h - deploy a container with config for model inferencing and data processing rethinking args passing -
2h - deploy a container with config for model training -
4h (secondary task) - review docs -
3h - add support for normalization with precomputed scalers and offsets
primary tasks estimation: 50h~70h + ?
secondary tasks estimation: 7h