Develop CMs

1. Calculation Modules

A Calculation Module is a plug-in application that expends the Visualization Tool. They are used to perform calculations on georeferenced data.
In this documentation, we refer to a Calculation Module as a CM. List of existing calculation module can be found on the main Readme

This documentation is meant for the CM developers, who write the calculation logic.

Go To Top

2. Getting Started

Run VisualizationTool locally

CMs have no graphical user interface (GUI). CMs are meant to be used via the Visualization Tool (citiwatts, COOLlife, Openmod4africa).
Meaning, to work on the development of a CM, the Visualization Tool is needed for testing.

Follow the steps from the main Readme to run it locally on your machine.

To access to the logs, see chapter logs

Adding a new CM

1. Contact citiwatts@hevs.ch to create a new git for your CM. Please specify a name, a repo name (no space, lower case) and which vlhtuleap accounts need access to it.
2. HEVS team will duplicate base_calculation module, and adapt configuration to display it on citiwattsdev.hevs.ch

3. After validation le HEVS team, run in your Visualization_tool folder:

   sh $ (sudo) git submodule update --init --recursive

   or Go in calculation_modules folder and run

   sh (sudo) git clone <git_link_of_your_cm>

   All link of CM can be found on the main Readme

Visual Checkpoint: What You Should See

At this point, you should verify that your local development environment is running correctly.

Open Docker Desktop: You should see running containers for the Visualization Tool and its components:

Open your browser to http://localhost:4200 : The Visualization Tool UI should load.

Navigate to the layer/CM list: Your newly added CM should be visible as an option:

Start working on your CM

All new CMs are based on the Scale heat and cool density maps, also known as the Base CM. The Base CM has two purpose: 1. An actual CM, it takes a heat demand layer and a multiplication factor, and it returns a heat demand layer multiplied by the factor 2. A basic and documented example to help in the development of other CMs

Change the inputs

Since your CM starts as a duplicate of the Base CM, the first element you will want to change is the inputs. See the chapter Signature & Inputs to learn what type of inputs are available and how to change them. After modifying the code of the CM, save the files and restart the dockers as explained in the VisualizationTool README.

Modify the calculation logic

The calculation happens in calculation_modules\base_calculation_module\cm\app\api_v1\calculation_module.py in the def calculation() method. See the Base CM, or your own CM documentation, for a detailed documentation of the process. No other files in the CMs should be modified to implement the calculation logic.

If you need additional files, or if you wish to split the logic into dedicated functions in dedicated files, you can add them to the calculation_modules\base_calculation_module\cm\app\api_v1\my_calculation_module_directory folder.

For the generation of layers that will be returned by the CM, you will find helper functions in the calculation_modules\base_calculation_module\cm\app\helper.py file.

Change the outputs

The outputs are all the information that the CM will return to the VisualizationTool. Those can be key-values (indicators), csv and zip files, and all the layers to be displayed. The outputs are returned by the return statement of the def calculation() method. See the chapter Outputs for more informations on what data can be returned as output and how to format them.

Update the tests for the pipeline

Unit testing is used to ensure code quality in the CM, and to avoid regression during development. Tests must be set-up and kept up-to-date for the CM to be accepted for development and production deployment.
See the chapter Tests for more informations on what data can be returned as output and how to format them.

Git and version control

The main branch is managed by the HES-SO, so all development work should be done on the develop branch, or parallel branches that get merged into develop.

Go To Top

3. Architecture

CMs are Python applications built with Flask. They run in a docker container and they extends the functionalities of the Visualization Tool. CMs are run in docker containers and they communicate with the Visualisation Tool backend via a docker network. Once a CM is started, it will automatically register itself to the Backend (API). After that, the CM wait for calls to perform their computation. Those calls start from the Frontend, go the the Backend(API) and are transmitted to the CM via the intermediary of tasks and queues manager (RabbitMQ, Redis and Flower).

Most of the files in a CM are there to ensure the auto-registration and the proper interactions between the CM and the rest of the Visualization Tool project. The following sequence is the relevant part for CM development:

1. CM registers to backen and is identified with the SIGNATURE
2. A CM takes in inputs, mostly georeferenced data (or layers) and more. More details in the signature and inputs.
3. Then it performs a calculation based on those data.
4. And finally, it outputs the new data so that it can be displayed in the frontend of the Visualization Tool.

Important files for CM development:

• calculation_module.py: Contains the calculation() method. calculation() is called when the CM is run. It is in the body of this method that the calculation happens. You have to define the outputs inside. Path: cm/app/api_v1/calculation_module.py

• my_calculation_module_directory: All additional files needed for running the calculation module must be added in this directory. Path: cm/app/api_v1/my_calculation_module_directory

• transactions.py: Contains all the routes used for communication between the CM and the Visualization Tool backend. Normally you don't need to edit this file. Path: cm/app/api_v1/transactions.py

• constant.py: Contains the constants used by the CM that defines all the inputs used to call the CM. More details in the signature and inputs
  Path: cm/app/constant.py

File structure

The base_calculation_module is a simple CM, used as the base for all other CMs.
Known as the "CM - Scale heat and cool density maps" in the Visualizatin Tool, it takes a layer of type heat and a multiplication factor and returns a new layer with its data multiplied by the multiplication factor. It is available at Base CM.
Below is the file structure of the base_calculation_module:

│   .gitignore
│   docker-compose.tests.yml
│   Jenkinsfile
│   LICENSE
│   README.md
│
└───cm
    │   consumer_cm_alive.py
    │   consumer_cm_compute.py
    │   Dockerfile
    │   gunicorn-config.py
    │   register_cm.py
    │   requirements.txt
    │   run.py
    │   run_all_local_terminal.sh
    │   supervisord.conf
    │   test.py
    │   wait-for-it.sh
    │   __init__.py
    │
    ├───app
    │   │   constant.py
    │   │   exceptions.py
    │   │   helper.py
    │   │   logging.conf
    │   │   utils.py
    │   │   __init__.py
    │   │
    │   ├───api_v1
    │   │   │   calculation_module.py
    │   │   │   errors.py
    │   │   │   transactions.py
    │   │   │   __init__.py
    │   │   │
    │   │   └───my_calculation_module_directory
    │   │           __init__.py
    │   │
    │   └───decorators
    │           caching.py
    │           json.py
    │           paginate.py
    │           rate_limit.py
    │           __init__.py
    │
    ├───config
    │       development.py
    │       production.py
    │       testing.py
    │       __init__.py
    │
    └───tests
        │   tests.py
        │   test_client.py
        │   __init__.py
        │
        └───data
                raster_for_test.tif

Go To Top

4. Signature & Inputs

The SIGNATURE is an object that contains important information about the CM, such as its name and the inputs it requires. It can be found in cm/app/constant.py. All fields must be included in the SIGNATURE, unless they are specified as optional.

INPUTS_CALCULATION_MODULE = [
    {
        'input_name': 'Multiplication factor',
        'input_type': 'input',
        'input_description': 'This factor will multiply the layer values',
        'input_parameter_name': 'multiplication_factor',
        'input_value': '1',
        'input_priority': 0,
        'input_unit': '',
        'input_min': 0,
        'input_max': 10, 'cm_id': CM_ID  # Do no change this value
    },
]

SIGNATURE = {
    "cm_name": CM_NAME,
    "category": "Demand",
    "wiki_url": "https://wiki.hotmaps.hevs.ch/en/CM-Scale-heat-and-cool-density-maps",
    "cm_url": "Do not add something",
    "cm_description": "This calculation module allows to scale the heat demand density layer up or down.",
    "cm_id": CM_ID, # Do no change this value
    "authorized_scale":[], # Details below
    "inputs_calculation_module": INPUTS_CALCULATION_MODULE,
    "layers_needed": [],
    "type_layer_needed": [], # Details below
    "type_vectors_needed": [], # Details below
}

Signature Fields:

• cm_name: name of the calculation module that will be displayed in the Frontend
• category: category of the calculation module. It is used to group CMs in the left panel of the Frontend
• wiki_url: url of the page in the wiki
• cm_url: (deprecated) must be left empty
• cm_description: description shown in the frontend, must be completed by CM developer
• cm_id: ID used to make the CM accessible, must not be changed. Same as CM repository name
• authorized_scale: Array that indicates what level of region selection can be used for this CM. If the array is empty, the CM can be used without selected regions. Different scales allowed: NUTS 0, NUTS 1, NUTS 2, NUTS 3, LAU 2, Hectare. Example: "authorized_scale":["NUTS 3", "LAU 2", "Hectare"],

• layers_needed: Leave empty. used for communication with other part of the code.

• type_layer_needed: raster layers needed to run the calculation module.

• name: The name is shown above the input field in the frontend. If empty, description is taken instead as name.
• type: raster layers required by the CM, this will appear as a dropdown selection in the frontend. If you put a type, all layers of this tye will be in the list. The available layers and type can be found in the Layers page

• description : The description is displayed in the frontend as a tooltip. It should help the user to choose a layer.

  Example:

  json "type_layer_needed": [ { "name":"Heat demand density", "type": "heat", // all layers of type heat "description": "Choose a heat demand density layer." }, { "name":"Heat demand residential density", "type": "heat_res_curr_density", // only layer heat density residential sector "description": "Choose a heat demand density layer." } ]

  

• type_vectors_needed: vector layers needed to run the calculation module.

• name: the name is shown above the input field in the frontend.

• type: layers required by the CM; this will appear as a dropdown selection in the frontend. If you put a type, all layers of this type will be in the list. The available layers and types can be found in the Layers section.

• description: description displayed in the frontend as a tooltip. It should help the user choose the correct layer.

Example:

json "type_vectors_needed": [ { "name": "Industrial emissions", "type": "industrial_database_emissions", "description": "You can choose the layer of type 'industrial_database_emissions'." } ]

4.1 Inputs

A CM inputs are defined in the INPUTS_CALCULATION_MODULE constant. INPUTS_CALCULATION_MODULE is an array of input objects. Each input object is defined as a python dict and must contain all of the following fields:

Input Fields:

• input_name: name of the input that will be displayed on the frontend GUI

• input_type: the input is the graphical control element that the user needs in order to enter data. There are five possible inputs, see https://getuikit.com/docs/form for more information about the implementation of the frontend GUI.

  • `input: this is a textbox in which the user can enter a value
    

  • select: this is a drop down menu that allows the user to choose one value from a list
    

  • radio: this allows the user to select only one of the predefined choices
    

  • checkbox: this allows the user to choose between two choices mutually exclusive options /!\ @todo checkbox currently not available

  • range: this allows the user to set a value by moving an indicator
    

• input_description: description put in the tooltip, can be empty
  

• input_parameter_name: used as an identifying key to access the value of an input during the computation

• input_value: default value for the input that will be displayed on the user interface

• input_options_array: only for radio and select inputs, array that contains all the possible options the user can choose from. The input_value must correspond to one of the item in this array.

• input_priority: this parameter allows to categorize input in the user interface. The needed value is an integer from 0 to 4:

  • 0: input always visible
  • 1 : Basic inputs
  • 2 : Advanced inputs (level 1)
  • 3 : Advanced inputs (level 2)
    

• cm_id: do no change this value

• input_unit: unit used for the input value

• input_min & input_max: range of the input values needed, used to limit that minimum and maximum possible values for an input. If not applicable to the input, they can be defined as ''

Find below examples of inputs:

INPUTS_CALCULATION_MODULE = [
     {  'input_name': 'Write your input',
        'input_type': 'input',
        'input_description': 'Description of the input',
        'input_parameter_name': 'input_item_name',
        'input_value': 1,
        'input_priority': 0,
        'input_unit': '',
        'input_min': 1,
        'input_max': 10,
        'cm_id': CM_ID
      },
      { 'input_name': 'Select your input',
        'input_type': 'select',
        'input_description': 'Description of the input',
        'input_parameter_name':'input_select_name' ,
        'input_value': 'List item 1',
        'input_options_array': ["List item 1",
                       "List item 2",
                       "List item 3",
                       "List item 4"],
        'input_priority': 0,
        'input_unit': '',
        'input_min': '',
        'input_max': '',
        'cm_id': CM_ID
      },
      {
        'input_name': 'Select the radio button',
        'input_type': 'radio',
        'input_description': 'Description of the input',
        'input_parameter_name': 'input_radio_name',
        'input_value': 'Yes',
        'input_options_array': ['Yes', 'No'],
        'input_priority': 0,
        'input_unit': '',
        'input_min': '',
        'input_max': '',
        'cm_id': CM_ID
      },
      {
         #<b style="color: red;">@todo checkbox example </b>
      },
      {
         'input_name': 'Move the slider to select your value',
         'input_type': 'range',
         'input_parameter_name': 'input_range_name',
         'input_value': 23.5, 
         'input_priority': 0,
         'input_unit': '%' + ' of area',
         'input_min': 0,
         'input_max': 100,
         'cm_id': CM_ID
      },
]

4.2 Dynamic Inputs (beta)

Special interactive inputs that have more functionalities than regular inputs. Dynamic Inputs are objects that can be added to the INPUTS_CALCULATION_MODULE constant.

For usage examples of dynamic inputs, checkout the base_calculation_module branch demo/dynamic-inputs

Dynamic Input Array

Used when a single input can occur multiple times. For example, a CM might ask for multiple coefficients or thresholds, and the user decides how many to provide (within defined min/max).

This type allows the frontend to generate a repeatable input block based on a template.

Dynamic Input Array Fields:

• default_inputs_amount: occurences of the template input shown by default.

• minimum_inputs_amount: minimum number of inputs the user must provide.

• maximum_inputs_amount: maximum number of inputs the user can provide.

• input_parameter_name: used as the base key to retrieve the resulting array of values. All repeated inputs will share this key.

• input_priority: controls visibility category in the frontend (same as for regular inputs).

• input_template: CM input object used as the template for each item in the array.

Example

{
  "default_inputs_amount": 1,
  "minimum_inputs_amount": 0,
  "maximum_inputs_amount": 5,
  "input_parameter_name": "custom_multiplier",
  "input_priority": 0,
  "input_template": {
    "input_name": "Custom multiplier entry",
    "input_type": "input",
    "input_description": "User-defined multiplier from dynamic array.",
    "input_parameter_name": "custom_multiplier",
    "input_value": 1,
    "input_priority": 0,
    "input_unit": "",
    "input_min": 0,
    "input_max": 10,
    "cm_id": 1
  }
}

Dynamic Input Options

Used when the input tree changes based on the user's selection in a select or radio input.

• A parent input (e.g., Processing Path) defines multiple options (e.g., Path A, Path B).

• Each option is linked to a set of child inputs, which are shown only if the option is selected.

• These children can themselves include regular inputs, arrays, or nested options, creating hierarchical dynamic structures.

Dynamic Input Options Fields:

• input_name: name of the input that will be displayed on the frontend GUI

• input_priority: controls visibility category in the frontend (same as for regular inputs).

• parent_input: a regular input of type select or radio

• ~path_defined_in_parent_input~: An array of inputs that is displayed only when the corresponding value is selected in the parent input. There can be as many ~path_defined_in_parent_input~ as there are options in the parent input. The key must be an exact match to the parent input options. The inputs in this array can be regular inputs, dynamic input array or dynamic input options.

/!\ Caution: when nesting dynamic input options into dynamic input options, you should: - make sure a regular input cannot be displayed multiple time simultaneously. If the hierarchy of dynamic input options can have different paths, leading to displaying the same regular input, make sure those paths cannot be used at the same time. Only the most recently changed value of the regular input will be received in the CM computation method. Ideally, avoid regular inputs being used in multiple paths. - avoid too many levels of nesting with dynamic input options

    {
        "input_name": "Path-based dynamic children",
        "input_priority": 0,
        "parent_input": {
            "input_name": "Path selection",
            "input_type": "radio",
            "input_description": "Choose a processing path",
            "input_parameter_name": "path_selector",
            "input_value": "Path Alpha",
            "input_options_array": ["Path Alpha", "Path Beta"],
            "input_priority": 0,
            "input_unit": "",
            "input_min": "",
            "input_max": "",
            "cm_id": 1,
        },
        "Path Alpha": [
            {
                "input_name": "Alpha Input A",
                "input_type": "input",
                "input_description": "First input for Path Alpha",
                "input_parameter_name": "alpha_input_a",
                "input_value": 10,
                "input_priority": 0,
                "input_unit": "units",
                "input_min": 0,
                "input_max": 100,
                "cm_id": 1,
            },
            {
                "input_name": "Alpha Input B",
                "input_type": "range",
                "input_description": "Second input for Path Alpha",
                "input_parameter_name": "alpha_input_b",
                "input_value": 50,
                "input_priority": 0,
                "input_unit": "%",
                "input_min": 0,
                "input_max": 100,
                "cm_id": 1,
            },
        ],
        "Path Beta": [
            {
                "input_name": "Beta Input",
                "input_type": "input",
                "input_description": "Only input for Path Beta",
                "input_parameter_name": "beta_input",
                "input_value": 5,
                "input_priority": 0,
                "input_unit": "kWh",
                "input_min": 0,
                "input_max": 1000,
                "cm_id": 1,
            }
        ],
    },

Go To Top

5. Outputs

The Outputs are all the data that the CM will return to the be displayed in the Visualization Tool. Standards have been defined to ensure the frontend can display the data correctly. It can be found in cm/app/api_v1/calculation_module.py All the outputs should be returned in the form of a dictionary named result. Find below an example:

"result":{
    "name":"CM - Scale heat and cool density maps",
    "indicator":[...],
    "graphics":[...],
    "vector_layers":[...], 
    "raster_layers":[...], 
    "extra_zip_files":[...],
    "csv_files":[...]
}

name

Display name of the CM, shown in the frontend

indicator

Array of indicators shown in the result panel. Each item in the array must have:

• unit (string): unit of the indicator
• name (string): name of the indicator
• value (number): value of the indicator

"indicator":[
    {
        "unit":"GWh/yr",
        "name":"Heat density total multiplied by 1.0",
        "value":"398.82725"
    }, 
    {
        "unit": "", 
        "name": "Successful intervention strategy", 
        "value": 'Feedback & gamification through an App'
    }
],

Indicators result:

graphics

Array of graphs, each item in the array is contains the neccessary data to produce a chartjs graph. Find below the most common item to declare for a graph, for more detail refer to chartjs documentation:

• type (string) : type of the graphic that will be displayed. The possible types are bar, line, radar, pie, polarArea, bubble:

  • line: a line chart or line graph is a type of chart which displays information as a series of data points called 'markers' connected by a straight line segment:
  • bar: a bar chart or bar graph is a chart or graph that presents categorical data with rectangular bars with heights or lengths proportional to the values that they represent:
  • radar: a radar chart is a way of showing multiple data points and the variation between them:
  • pie: a pie chart is divided into segments, the arc of each segment shows the proportional value of each piece of data:
  • polarArea: polar area charts are similar to pie charts, but each segment has the same angle - the radius of the segment differs depending on the value:

• deprecated xLabel: This attribute is not supported by chartjs anymore, to define the text displayed on x-axis, use the new syntax

• deprecated yLabel: This attribute is not supported by chartjs anymore, to define the text displayed on y-axis, use the new syntax New syntax to specify axis names. (do not add to pie diagrams or it will add axis lines): New axis label syntax:

    "options": {
        "scales": {
            "x": {
                "title": {
                    "display": True,
                    "text":"x axis name"
                }
            },
            "y": {
                "title": {
                "display": True,
                "text":"y axis name"
            }
        }
    }

• data: contains labels and datasets

  • labels (string[]) : x axis labels only x axis.
  • datasets (array): set of data with its configuration
  • label (string) : serie's label
  • backgroundColor (string[]) : background color of each value to display
  • data (number[]) : values of the serie

• options: options available to handle the charts, learn more at https://www.chartjs.org/docs/latest/charts/line.html, /!\ chartjs example are in JavaScript and should be adapted to Python syntax

All chart types and code example in page Example of Charts
Basic implementation :

  graphics = []
  graphics.append(
    {
      "type": "bar",
      "data": {
        "labels": ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"],
        "datasets": [
          {
            "label": "Temperature (°C)",
            "data": [12, 15, 14, 17, 20, 22, 19],
            "backgroundColor": "#4169E1", #blue
            "borderColor": "#ff0000", #red
            "borderWidth": 1
          }
        ]
      },
      "options": {
        "responsive": True,
        "scales": {
          "y": {
            "beginAtZero": True
          }
        }
      }
    }
  )
  result = dict()
  result['graphics'] = graphics

vector_layers

• name (string): Should include "shapefile", name to be displayed on the frontend
• path (string): path generated for the geotif file
• type (string): type of the layer generated for symbology (legend). If set as custom, symbology must be defined. This case is useful for dynamic legends.
• symbology (array): legend used for a layer of type custom, the symbology defines how the layer will be displayed in the frontend. It only is applied if the type is custom. It should not be defined if the type is not custom. The symbology should contain at least two elements including a case for the value 0 (or the lowest value that should not show on the map), and an opacity of 0, to avoid unwanted visual artifacts.

[
    {
        "name": "shapefile of coherent areas with their potential", 
        "path": output_shp1 
    },
    {
        "name": "District Cooling areas and there potentials - shapefile", 
        "path": output_shp2, 
        "type": "custom",
        "symbology": [
            {"red": 254, "green": 237, "blue": 222, "opacity": 0.5, "value": symbol_vals_str[0], "label": symbol_vals_str[0] + " GWh"},
            {"red": 253, "green": 208, "blue": 162, "opacity": 0.5, "value": symbol_vals_str[1], "label": symbol_vals_str[1] + " GWh"},
            {"red": 253, "green": 174, "blue": 107, "opacity": 0.5, "value": symbol_vals_str[2], "label": symbol_vals_str[2] + " GWh"},
            {"red": 253, "green": 141, "blue": 60, "opacity": 0.5, "value": symbol_vals_str[3], "label": symbol_vals_str[3] + " GWh"},
            {"red": 230, "green": 85, "blue": 13, "opacity": 0.5, "value": symbol_vals_str[4], "label": symbol_vals_str[4] + " GWh"},
            {"red": 166, "green": 54, "blue": 3, "opacity": 0.5, "value": str(float(symbol_vals_str[4]) + step), "label": ">" + symbol_vals_str[4] + " GWh"}
        ]
    }
]

• vector layer default result (no defined type:):

• vector layer custom result:

raster_layers

array of raster layers

• name (string): name to be displayed on the frontend
• path (string): path generated for the geotif file
• type (string): type of the layer generated for symbology (legend). If set as custom, symbology must be defined. This case is useful for dynamic legends.
• symbology (array): legend used for a layer of type custom, the symbology defines how the layer will be displayed in the frontend. It only is applied if the type is custom. The symbology should contain at least two elements including a case for the value 0 (or the lowest value that should not show on the map), and an opacity of 0, to avoid unwanted visual artifacts.

"type": "custom",
"symbology": [
  {"red": 254, "green": 237, "blue": 222, "opacity": 0, "value": "1", "label": "1 kWh/ha"},
  {"red": 253, "green": 208, "blue": 162, "opacity": 0.8, "value": "100", "label": "100 kWh/ha"},
  {"red": 253, "green": 174, "blue": 107, "opacity": 0.8, "value": "200", "label": "200 kWh/ha"},
  {"red": 253, "green": 141, "blue": 60, "opacity": 0.8, "value": "400", "label": "400 kWh/ha"},
  {"red": 230, "green": 85, "blue": 13, "opacity": 0.8, "value": "600", "label": "600 kWh/ha"},
  {"red": 166, "green": 54, "blue": 3, "opacity": 0.8, "value": "800", "label": "800 kWh/ha"}
]

• layer (string): type of the layer generated (for toolchaining in CM)

In order to generate a path, developers should use the function generate_output_file_tif(), which needs the output directory as an argument. This function should be imported to the calculation_module.py. The path must be generated on the first lines of calculation() function found in calculation_module.py. This function For example:

    output_tif_1 = generate_output_file_tif(output_directory)

All the layers outputs must be retrieved and added to the raster_layers array after they have been generated by the calculation module provider functions.

[
    {
        "name":"layers of heat_density 1.0",
        "path": output_raster1 
        "type":"heat",
        "layer":"heat_tot_curr_density"
    },
    {
        "name": "layer of charging needs",
        "path": output_raster2,
        "type": "custom",
        "symbology": [
            {"red": 254, "green": 237, "blue": 222, "opacity": 0, "value": "1", "label": "<1 kWh/ha"},
            {"red": 253, "green": 208, "blue": 162, "opacity": 0.8, "value": "100", "label": "100 kWh/ha"},
            {"red": 253, "green": 174, "blue": 107, "opacity": 0.8, "value": "200", "label": "200 kWh/ha"},
            {"red": 253, "green": 141, "blue": 60, "opacity": 0.8, "value": "400", "label": "400 kWh/ha"},
            {"red": 230, "green": 85, "blue": 13, "opacity": 0.8, "value": "600", "label": "600 kWh/ha"},
            {"red": 166, "green": 54, "blue": 3, "opacity": 0.8, "value": "800", "label": "800 kWh/ha"}]
     }
]

• raster heat result:

• raster custom result:

extra_zip_files

How to return a .zip file:

1. Import the create_extra_zip_file() helper function:

     # in calculation_module.py
     from ..helper import create_extra_zip_file

1. Use the new helper function create_extra_zip_file() to create the .zip file to return:

   The first parameter must be the output directory path. It is recommended to use output_directory that is accessible in the calculation() method of calculation_module.py.

   Any other parameter should be a path to a file that must be added to the zip. There can be as many file path passed as parameters as needed.

In calculation_module.py:

    def calculation(output_directory, inputs_raster_selection, inputs_parameter_selection):
        extra_output_files = create_extra_zip_file(output_directory, path_to_extra_file1, path_to_extra_file2)

1. Add the .zip to the result dict:

   use "extra_zip_files" as the key

   The .zip must be added as an object in an array, meaning you could return multiple .zip

   [{"name": "name-shown-when-downloaded", "path": path-returned-by-create_extra_zip_file}]

    # in calculation_module.py
    def calculation(output_directory, inputs_raster_selection, inputs_parameter_selection):
        extra_output_files = create_extra_zip_file(output_directory, path_to_extra_file1, path_to_extra_file2)
        result["extra_zip_files"] = [{"name": "ComplementaryCSVandLogs", "path": extra_output_files }]

csv_files

See below how to add .csv files to the outputs:

The .csv must be added as an object in an array, meaning you can return multiple .csv

[{"name": "name-shown-when-downloaded", "path": path-to-csv-file}]

    # in calculation_module.py
    def calculation(output_directory, inputs_raster_selection, inputs_parameter_selection):
        result["csv_files"] = [{"name": "ComplementaryCSV", "path": path_to_complementary_csv }]

Go To Top

6. Logs

Logs can help in the development process and for monitoring purposes. To log from a CM, avoid using print(). The print statement is not always flushed and it may not appear in the log files. Instead use :

• logging.info("log") (with import logging)
• print("log", flush=True)

examples:

    import logging
    logging.info("Log message")       #-> 2025-01-02 08:59:55 [97] [INFO] Log message
    print("Log message", flush=True)  #-> Log message

Access Logs from local machine

At Visualization_tool\.dockerVolumes\logs\calculation_modules\base_calculation_module, you can find the all the log files generated by the instance of the CM running on you local machine.
At .dockerVolumes\logs\calculation_modules\base_calculation_module\calculation_module_service.out.log, you can find your logs defined as explained in this documentation.

Python syntax errors should also be logged in .dockerVolumes\logs\calculation_modules\base_calculation_module\calculation_module_service.out.log.

Access Logs from development and production

To obtain access and information relating to the VPN, an email must be sent to citiwatts@hevs.ch. The HES-SO Valais-Wallis IT department will then send an email to the requester with the information needed to download the VPN software and connection access. You will also receive a login from citiwatts@hevs.ch to access the SFTP server containing the logs. We recommend using the filezilla client to access the contents of the SFTP server.

Once connected to the SFTP server, the user can view a folder by module calculation. Inside, there are 8 log files (calculation_module_service.err.log, calculation_module_service.out.log, consumer_cm_alive.err.log, consumer_cm_alive.out.log, consumer_cm_compute.err.log, consumer_cm_compute.out.log, register_cm.err.log, register_cm.out.log). It is within these files that the developers of calculation module will be able to find all the system logs.

Go To Top

7. Tests

CM developers are encouraged to utilize Unit Tests to enhance the quality of their CM code. By identifying bugs early, ensuring expected functionality, and automating testing, Unit Tests streamline development. They also make refactoring safer, improve code structure, and simplify debugging.

Reminder

A unit test checks the smallest testable parts of an application (units) in isolation to ensure they work correctly. It is typically automated and focuses on individual functions, methods, or classes.

Writing Unit Tests

The base_calculation_module already contains a /tests folder with a tests.py file that must be completed for each CM.

    class TestAPI(unittest.TestCase):
        def setUp(self):
            ...

        def tearDown(self):
            ...

        def test_compute(self):
            raster_file_path = 'tests/data/raster_for_test.tif'
            # simulate copy from HTAPI to CM
            save_path = UPLOAD_DIRECTORY+"/raster_for_test.tif"
            copyfile(raster_file_path, save_path)

            inputs_raster_selection = {}
            inputs_parameter_selection = {}
            inputs_vector_selection = {}
            inputs_raster_selection["heat"]  = save_path
            inputs_vector_selection["industrial_database_emissions"] = ''
            inputs_parameter_selection["multiplication_factor"] = 2

            # register the calculation module a
            payload = {"inputs_raster_selection": inputs_raster_selection,
                   "inputs_parameter_selection": inputs_parameter_selection,
                   "inputs_vector_selection": inputs_vector_selection}

            rv, json = self.client.post('computation-module/compute/', data=payload)

            # Assert CM API has been called successfully
            self.assertTrue(rv.status_code == 200)
            ...

This file defines a class that subclasses unittest.TestCase and will be therefore discovered as Unit Test by the Python Unit Test Framework (unittest).

The test_compute(self) method triggers the communication route of the CM API in order to call the calculation() method explicitly defined per CM in /cm/app/api_v1/calculation_module.py. Depending on the calculation implemented, appropriate inputs must be set accordingly: inputs_raster_selection, inputs_parameter_selection, inputs_parameter_selection, inputs_vector_selection.

self.assertTrue(rv.status_code == 200 asserts that the CM API has been called successfully.

Enhance Unit Tests

More than only testing if the CM can be successfully called, CM developers are advised to deeply test their code.

This can be first realized by verifying the results of the calculation() method.

The last lines in test_compute(self) method assert that the calculated values correspond to the expected values. Each section of the results are compared individually:

    # Assert result is correct
    expected_result = {
        'name': 'CM - Scale heat and cool density maps',
    'indicator': [
        {"unit": "GWh/yr", "name": "Heat density total multiplied by  2.0", "value": '14070.01'}
    ],
    'graphics': [],
    'vector_layers': [],
    'raster_layers': [{"name": "layers of heat_density 2.0", "path": 'dummy_output.tif', "type": "heat", "layer": "heat_tot_curr_density"}]
    }
    result = json['result']
    self.assertEqual(result['indicator'], expected_result['indicator'])
    self.assertEqual(result['name'], expected_result['name'])
    self.assertEqual(result['indicator'][0]['unit'], expected_result['indicator'][0]['unit'])
    self.assertEqual(result['indicator'][0]['name'], expected_result['indicator'][0]['name'])
    self.assertEqual(result['indicator'][0]['value'], expected_result['indicator'][0]['value'])
    self.assertEqual(result['raster_layers'][0]['name'], expected_result['raster_layers'][0]['name'])
    self.assertEqual(result['raster_layers'][0]['type'], expected_result['raster_layers'][0]['type'])
    self.assertEqual(result['raster_layers'][0]['layer'], expected_result['raster_layers'][0]['layer'])

Comparisons rely on the given inputs, so testing multiple inputs is essential to cover a wide range of scenarios.

Cover different scenarios

CM developers are encouraged to create multiple test_* methods in tests.py file to cover individually different scenarios. These scenarios can be identified as the edge cases that the CM must support.

Additional test_* methods should verify that: * Results vary accordingly to the inputs provided * Outputs files correspond to the expectations (quantity, name, path, content, ...) * Exceptions should be raised when necessary (type, message, ...)

Run Unit Tests

Unit Tests run inside a container.

From the CM root folder (containing docker-compose.tests.yml file), CM developers can run Unit Test with following command: docker-compose -f docker-compose.tests.yml up --build --force-recreate

In this case, 1 test has been run and OK statement confirm that everything ran correctly.

In case of errors, CM developers are quickly informed that some tests have failed.

Stack trace is provided to indicate in which file (i.e. calculation_module.py) and where (i.e. line 46) error happened.

Automated testing

Unit Tests are run by our CI/CD infrastructure. The Jenkins file located in every CM root folder calls the docker-compose -f docker-compose.tests.yml command on every commit made in CM repository.

Go To Top

References

Go To Top

How To Cite

Go To Top

Authors And Reviewers

This page was written by the citiwatts Team and namely by:

EASILab - HES-SO: David Gianadda, Gwenaëlle Gustin, Jérémie Vanin, Alain Duc

Go To Top

License

Go To Top

Acknowledgement

Go To Top