Skip to content

Develop CMs

1. Calculation Modules

A Calculation Module is a plug-in application that expends the Visualization Tool. They are used to perform calculations on georeferenced data.
In this documentation, we refer to a Calculation Module as a CM. List of existing calculation module can be found on the main Readme

This documentation is meant for the CM developers, who write the calculation logic.

Go To Top

2. Getting Started

Run VisualizationTool locally

CMs have no graphical user interface (GUI). CMs are meant to be used via the Visualization Tool (citiwatts, COOLlife, Openmod4africa).
Meaning, to work on the development of a CM, the Visualization Tool is needed for testing.

Follow the steps from the main Readme to run it locally on your machine.

To access to the logs, see chapter logs

Adding a new CM

  1. Contact citiwatts@hevs.ch to create a new git for your CM. Please specify a name, a repo name (no space, lower case) and which vlhtuleap accounts need access to it.
  2. HEVS team will duplicate base_calculation module, and adapt configuration to display it on citiwattsdev.hevs.ch

  3. After validation le HEVS team, run in your Visualization_tool folder:

    sh $ (sudo) git submodule update --init --recursive

    or Go in calculation_modules folder and run

    sh (sudo) git clone <git_link_of_your_cm>

    All link of CM can be found on the main Readme

Visual Checkpoint: What You Should See

At this point, you should verify that your local development environment is running correctly.

Open Docker Desktop: You should see running containers for the Visualization Tool and its components:
image

Open your browser to http://localhost:4200 : The Visualization Tool UI should load.

Navigate to the layer/CM list: Your newly added CM should be visible as an option:
image

Start working on your CM

All new CMs are based on the Scale heat and cool density maps, also known as the Base CM. The Base CM has two purpose: 1. An actual CM, it takes a heat demand layer and a multiplication factor, and it returns a heat demand layer multiplied by the factor 2. A basic and documented example to help in the development of other CMs

Change the inputs

Since your CM starts as a duplicate of the Base CM, the first element you will want to change is the inputs. See the chapter Signature & Inputs to learn what type of inputs are available and how to change them. After modifying the code of the CM, save the files and restart the dockers as explained in the VisualizationTool README.

Modify the calculation logic

The calculation happens in calculation_modules\base_calculation_module\cm\app\api_v1\calculation_module.py in the def calculation() method. See the Base CM, or your own CM documentation, for a detailed documentation of the process. No other files in the CMs should be modified to implement the calculation logic.

If you need additional files, or if you wish to split the logic into dedicated functions in dedicated files, you can add them to the calculation_modules\base_calculation_module\cm\app\api_v1\my_calculation_module_directory folder.

For the generation of layers that will be returned by the CM, you will find helper functions in the calculation_modules\base_calculation_module\cm\app\helper.py file.

Change the outputs

The outputs are all the information that the CM will return to the VisualizationTool. Those can be key-values (indicators), csv and zip files, and all the layers to be displayed. The outputs are returned by the return statement of the def calculation() method. See the chapter Outputs for more informations on what data can be returned as output and how to format them.

Update the tests for the pipeline

Unit testing is used to ensure code quality in the CM, and to avoid regression during development. Tests must be set-up and kept up-to-date for the CM to be accepted for development and production deployment.
See the chapter Tests for more informations on what data can be returned as output and how to format them.

Git and version control

The main branch is managed by the HES-SO, so all development work should be done on the develop branch, or parallel branches that get merged into develop.

Go To Top

3. Architecture

image

CMs are Python applications built with Flask. They run in a docker container and they extends the functionalities of the Visualization Tool. CMs are run in docker containers and they communicate with the Visualisation Tool backend via a docker network. Once a CM is started, it will automatically register itself to the Backend (API). After that, the CM wait for calls to perform their computation. Those calls start from the Frontend, go the the Backend(API) and are transmitted to the CM via the intermediary of tasks and queues manager (RabbitMQ, Redis and Flower).

Most of the files in a CM are there to ensure the auto-registration and the proper interactions between the CM and the rest of the Visualization Tool project. The following sequence is the relevant part for CM development:

  1. CM registers to backen and is identified with the SIGNATURE
  2. A CM takes in inputs, mostly georeferenced data (or layers) and more. More details in the signature and inputs.
  3. Then it performs a calculation based on those data.
  4. And finally, it outputs the new data so that it can be displayed in the frontend of the Visualization Tool.

Important files for CM development:

  • calculation_module.py: Contains the calculation() method. calculation() is called when the CM is run. It is in the body of this method that the calculation happens. You have to define the outputs inside. Path: cm/app/api_v1/calculation_module.py

  • my_calculation_module_directory: All additional files needed for running the calculation module must be added in this directory. Path: cm/app/api_v1/my_calculation_module_directory

  • transactions.py: Contains all the routes used for communication between the CM and the Visualization Tool backend. Normally you don't need to edit this file. Path: cm/app/api_v1/transactions.py

  • constant.py: Contains the constants used by the CM that defines all the inputs used to call the CM. More details in the signature and inputs
    Path: cm/app/constant.py

File structure

The base_calculation_module is a simple CM, used as the base for all other CMs.
Known as the "CM - Scale heat and cool density maps" in the Visualizatin Tool, it takes a layer of type heat and a multiplication factor and returns a new layer with its data multiplied by the multiplication factor. It is available at Base CM.
Below is the file structure of the base_calculation_module:

│   .gitignore
│   docker-compose.tests.yml
│   Jenkinsfile
│   LICENSE
│   README.md
│
└───cm
    │   consumer_cm_alive.py
    │   consumer_cm_compute.py
    │   Dockerfile
    │   gunicorn-config.py
    │   register_cm.py
    │   requirements.txt
    │   run.py
    │   run_all_local_terminal.sh
    │   supervisord.conf
    │   test.py
    │   wait-for-it.sh
    │   __init__.py
    │
    ├───app
    │   │   constant.py
    │   │   exceptions.py
    │   │   helper.py
    │   │   logging.conf
    │   │   utils.py
    │   │   __init__.py
    │   │
    │   ├───api_v1
    │   │   │   calculation_module.py
    │   │   │   errors.py
    │   │   │   transactions.py
    │   │   │   __init__.py
    │   │   │
    │   │   └───my_calculation_module_directory
    │   │           __init__.py
    │   │
    │   └───decorators
    │           caching.py
    │           json.py
    │           paginate.py
    │           rate_limit.py
    │           __init__.py
    │
    ├───config
    │       development.py
    │       production.py
    │       testing.py
    │       __init__.py
    │
    └───tests
        │   tests.py
        │   test_client.py
        │   __init__.py
        │
        └───data
                raster_for_test.tif

Go To Top

4. Signature & Inputs

The SIGNATURE is an object that contains important information about the CM, such as its name and the inputs it requires. It can be found in cm/app/constant.py. All fields must be included in the SIGNATURE, unless they are specified as optional.

base_calculation_module example: 

INPUTS_CALCULATION_MODULE = [
    {
        "input_name": "Multiplication factor",
        "input_type": "input",
        "input_description": "This factor will multiply the layer values",
        "input_parameter_name": "multiplication_factor",
        "input_value": "1",
        "input_priority": 0,
        "input_unit": "",
        "input_min": 0,
        "input_max": 10,
        "cm_id": CM_ID,  # Do no change this value
    },
    ### Layer Inputs
    {
        "input_name": "Heat demand density",
        "input_type": "layer",
        "input_description": "Choose a heat demand density layer.",
        "input_parameter_name": "layer_to_processs",
        "cm_id": CM_ID,  # Do no change this value
        "input_priority": 5,
        "input_value": "",
        "input_options_array": [
            "heat_tot_curr_density",
            "heat_res_curr_density",
            "heat_nonres_curr_density",
            "cool_tot_curr_density",
        ],  # workspacenames only
        "layer_format": "raster",
    },
]

SIGNATURE = {
    "cm_id": CM_ID,
    "cm_name": CM_NAME,
    "category": "Demand",
    "wiki_url": "https://citiwatts.github.io/wiki/heat-cm-scale-heat-and-cool-density-maps/",
    "cm_url": "Do not add something",
    "cm_description": "This calculation module allows to scale the heat demand density layer up or down.",
    "authorized_scale": [
        "NUTS 1",
        "NUTS 2",
        "NUTS 3",
        "LAU 2",
        "Hectare",
    ],  # if empty, allow all
    "inputs_calculation_module": INPUTS_CALCULATION_MODULE,
}

Signature Fields:

  • cm_id: ID used to make the CM accessible, must not be changed. Same as CM repository name
  • cm_name: name of the calculation module that will be displayed in the Frontend
  • category: category of the calculation module. It is used to group CMs in the left panel of the Frontend
  • wiki_url: url of the page in the wiki
  • cm_url: (deprecated) must be left empty
  • cm_description: description shown in the frontend, must be completed by CM developer
  • authorized_scale: Array that indicates what level of region selection can be used for this CM. If the array is empty, the CM can be used without selected regions. Different scales allowed: NUTS 0, NUTS 1, NUTS 2, NUTS 3, LAU 2, Hectare. Example: "authorized_scale":["NUTS 3", "LAU 2", "Hectare"]
  • inputs_calculation_module: Array of input objects. Contains all the user defined inputs for the same. See next section for details

4.1 Inputs

CM inputs are defined in the INPUTS_CALCULATION_MODULE constant. INPUTS_CALCULATION_MODULE is an array of input objects. Each input object is defined as a python dict and must contain all of the following fields:

Input Fields:

  • input_name: name of the input that will be displayed on the frontend GUI

  • input_type: specifies the type of graphical control element that the user interacts with to provide data. The available input types are:

    • input: a single-line text box where the user can type in a value.
      image

    • select: a drop-down menu that lets the user choose one value from a predefined list.
      image

    • radio: a set of radio buttons that allow the user to select exactly one option from multiple choices.
      image

    • checkbox: a group of checkboxes that allow the user to select one or more options.
      image

    • range: a slider control that allows the user to adjust a value by moving an indicator along a scale.
      image

    • datetime-local: a calendar and time picker that allows the user to select both a date and a time using a graphical interface.
      image

    • file: a file upload that allows the user to upload a file (CSV or JSON) in the CM using a graphical interface. The file must be less than 10 MB and must not be empty. It must comply with the codes of its json/csv format. If you need to accept larger files, please contact the developers (citiwatts@hevs.ch).
      exampleJsonInputFile

    • layer: a layer selector that lets the user choose a map layer, which will then be processed by the CM for calculation.
      See more in Section 4.2: Layer Inputs.
      image

  • input_description: description put in the tooltip, can be empty
    image

  • input_parameter_name: Identifier for the input, this field must be unique to the CM signature as it is used to access the input value in the CM's calculation() method.

  • input_value: default value for the input that will be displayed on the user interface. Not applicable for layer type inputs. For checkbox inputs, the input_value can be an array.

  • input_options_array: only for radio, select and checkbox inputs, array that contains all the possible options the user can choose from. The input_value must correspond to one of the item in this array.

  • input_priority: this parameter allows to categorize input in the user interface. The needed value is an integer from 0 to 4:

    • 0 : Inputs, open by default
    • 1 : Basic inputs, closed by default
    • 2 : Advanced inputs (level 1), closed by default
    • 3 : Advanced inputs (level 2), closed by default
    • 4 : Advanced inputs (level 3), closed by default
    • 5 : Layer inputs, open by default
      image

  • input_format: only for file. The input_format must correspond to one of the two file type format accepted by the backend (csv or json) in lowercase.

  • cm_id: do no change this value

  • input_unit: unit used for the input value

  • input_min & input_max: range of the input values needed, used to limit that minimum and maximum possible values for an input. If not applicable to the input, they can be defined as ''

Find below examples of inputs:

INPUTS_CALCULATION_MODULE = [
     {  'input_name': 'Write your input',
        'input_type': 'input',
        'input_description': 'Description of the input',
        'input_parameter_name': 'input_item_name',
        'input_value': 1,
        'input_priority': 0,
        'input_unit': '',
        'input_min': 1,
        'input_max': 10,
        'cm_id': CM_ID
      },
      { 'input_name': 'Select your input',
        'input_type': 'select',
        'input_description': 'Description of the input',
        'input_parameter_name':'input_select_name' ,
        'input_value': 'List item 1',
        'input_options_array': ["List item 1",
                       "List item 2",
                       "List item 3",
                       "List item 4"],
        'input_priority': 0,
        'input_unit': '',
        'input_min': '',
        'input_max': '',
        'cm_id': CM_ID
      },
      {
        'input_name': 'Select the radio button',
        'input_type': 'radio',
        'input_description': 'Description of the input',
        'input_parameter_name': 'input_radio_name',
        'input_value': 'Yes',
        'input_options_array': ['Yes', 'No'],
        'input_priority': 0,
        'input_unit': '',
        'input_min': '',
        'input_max': '',
        'cm_id': CM_ID
      },
      {
        "input_name": "Select the checkbox",
        "input_type": "checkbox",
        "input_description": "Description of the input",
        "input_parameter_name": "input_checkbox_name",
        "input_value": ["Option1", "Option2"],
        "input_options_array": ["Option1", "Option2", "Option3", "Option4"],
        "input_priority": 0,
        "input_unit": "",
        "input_min": "",
        "input_max": "",
        "cm_id": CM_ID,
      },
      {
         'input_name': 'Move the slider to select your value',
         'input_type': 'range',
         'input_parameter_name': 'input_range_name',
         'input_value': 23.5, 
         'input_priority': 0,
         'input_unit': '%' + ' of area',
         'input_min': 0,
         'input_max': 100,
         'cm_id': CM_ID
      },
      {
          "input_name": "Select a file",
          "input_type": "file",
          "input_parameter_name": "file_name_json",
          "input_value": "",
          "input_priority": 4,
          "input_unit": "",
          "input_min": "",
          "input_max": "",
          "input_format": "json", # It can be csv or json
          "cm_id": CM_ID,
      },
]

4.2 Layer Inputs:

Layer inputs are a special type of input field designed for selecting geospatial layers that the Calculation Module (CM) will use in its computation.
When a user selects a layer, the CM receives the data clipped to the user’s selected area, ensuring that only the relevant portion of the dataset is processed inside the calculation() method.

Layer Input Fields:

  • input_name: name of the input that will be displayed on the frontend GUI.
  • input_type: must be set to "layer".
  • input_description: a description displayed as a tooltip in the user interface.
  • input_parameter_name: unique identifier for the input, used to access the layer in the CM’s calculation() method.
  • In the calculation() method, the value of input_parameter_name is a dict containing the keys "path" and "workspacename".
  • Example usage:
    python input_raster_selection_path = inputs_raster_selection["layer_to_process"]["path"] input_raster_selection_workspacename = inputs_raster_selection["layer_to_process"]["workspacename"]
  • See the base_calculation_module calculation() method for usage example calculation_modules\base_calculation_module\cm\app\api_v1\calculation_module.py
  • cm_id: do not change this value.
  • input_priority: preferably set to 5 for layer inputs. This ensures that the input is categorized correctly in the GUI.
  • input_value: not applicable for layer inputs (set to ""). The default value is the first option specified in input_options_array.
  • input_options_array: array of valid workspacenames that can be selected as the layer source.
  • List of layers with their workspacename can be found at https://citiwatts.github.io/wiki/layers-available/
  • layer_format: specifies whether the selected layer is a "raster" or "vector".

Example 1: Heat demand density layer (raster)

{
    "input_name": "Heat demand density",
    "input_type": "layer",
    "input_description": "Choose a heat demand density layer.",
    "input_parameter_name": "layer_to_process",
    "cm_id": CM_ID,  # Do not change this value
    "input_priority": 5,
    "input_value": "",
    "input_options_array": [
        "heat_tot_curr_density",
        "heat_res_curr_density",
        "heat_nonres_curr_density",
        "cool_tot_curr_density"
    ],  # workspacenames only
    "layer_format": "raster"
}

Example 2: Motorization rate layer (vector)

{
    "input_name": "Motorization rate",
    "input_type": "layer",
    "input_description": "Choose a motorization rate layer.",
    "input_parameter_name": "motorization_rate_nuts2_view_to_process",
    "cm_id": CM_ID,  # Do not change this value
    "input_priority": 5,
    "input_value": "",
    "input_options_array": [
        "motorization_rate_nuts2_view"
    ],  # workspacenames only
    "layer_format": "vector"
}

4.3 Dynamic Inputs (beta)

Special interactive inputs that have more functionalities than regular inputs. Dynamic Inputs are objects that can be added to the INPUTS_CALCULATION_MODULE constant.

For usage examples of dynamic inputs, checkout the base_calculation_module branch demo/dynamic-inputs

Dynamic Input Array

Used when a single input can occur multiple times. For example, a CM might ask for multiple coefficients or thresholds, and the user decides how many to provide (within defined min/max).

This type allows the frontend to generate a repeatable input block based on a template.

[!NOTE]
Inputs of type layer are not yet compatible with Dynamic Input Array

Dynamic Input Array Fields:

  • input_name: name of the input that will be displayed on the frontend GUI

  • default_inputs_amount: occurences of the template input shown by default.

  • minimum_inputs_amount: minimum number of inputs the user must provide.

  • maximum_inputs_amount: maximum number of inputs the user can provide.

  • input_parameter_name: used as the base key to retrieve the resulting array of values. All repeated inputs will share this key.

  • input_priority: controls visibility category in the frontend (same as for regular inputs).

  • input_template: CM input object used as the template for each item in the array.

Example

{
  "input_name": "Number of input occurrences",
  "default_inputs_amount": 1,
  "minimum_inputs_amount": 0,
  "maximum_inputs_amount": 5,
  "input_parameter_name": "custom_multiplier",
  "input_priority": 0,
  "input_template": {
    "input_name": "Custom multiplier entry",
    "input_type": "input",
    "input_description": "User-defined multiplier from dynamic array.",
    "input_parameter_name": "custom_multiplier",
    "input_value": 1,
    "input_priority": 0,
    "input_unit": "",
    "input_min": 0,
    "input_max": 10,
    "cm_id": 1
  }
}

image

Dynamic Input Options

Used when the input tree changes based on the user's selection in a select or radio input.

  • A parent input (e.g., Processing Path) defines multiple options (e.g., Path A, Path B).

  • Each option is linked to a set of child inputs, which are shown only if the option is selected.

  • These children can themselves include regular inputs, arrays, or nested options, creating hierarchical dynamic structures.

Dynamic Input Options Fields:

  • input_name: name of the input that will be displayed on the frontend GUI

  • input_priority: controls visibility category in the frontend (same as for regular inputs).

  • parent_input: a regular input of type select or radio

  • ~path_defined_in_parent_input~: An array of inputs that is displayed only when the corresponding value is selected in the parent input. There can be as many ~path_defined_in_parent_input~ as there are options in the parent input. The key must be an exact match to the parent input options. The inputs in this array can be regular inputs, dynamic input array or dynamic input options.

[!WARNING]
When nesting dynamic input options into dynamic input options, you should:
- Make sure a regular input cannot be displayed multiple times simultaneously.
If the hierarchy of dynamic input options can have different paths leading to the same regular input, ensure those paths cannot be used at the same time. Only the most recently changed value of the regular input will be received in the CM computation method. Ideally, avoid reusing regular inputs in multiple paths.
- Avoid too many levels of nesting with dynamic input options.

    {
        "input_name": "Path-based dynamic children",
        "input_priority": 0,
        "parent_input": {
            "input_name": "Path selection",
            "input_type": "radio",
            "input_description": "Choose a processing path",
            "input_parameter_name": "path_selector",
            "input_value": "Path Alpha",
            "input_options_array": ["Path Alpha", "Path Beta"],
            "input_priority": 0,
            "input_unit": "",
            "input_min": "",
            "input_max": "",
            "cm_id": 1,
        },
        "Path Alpha": [
            {
                "input_name": "Alpha Input A",
                "input_type": "input",
                "input_description": "First input for Path Alpha",
                "input_parameter_name": "alpha_input_a",
                "input_value": 10,
                "input_priority": 0,
                "input_unit": "units",
                "input_min": 0,
                "input_max": 100,
                "cm_id": 1,
            },
            {
                "input_name": "Alpha Input B",
                "input_type": "range",
                "input_description": "Second input for Path Alpha",
                "input_parameter_name": "alpha_input_b",
                "input_value": 50,
                "input_priority": 0,
                "input_unit": "%",
                "input_min": 0,
                "input_max": 100,
                "cm_id": 1,
            },
        ],
        "Path Beta": [
            {
                "input_name": "Beta Input",
                "input_type": "input",
                "input_description": "Only input for Path Beta",
                "input_parameter_name": "beta_input",
                "input_value": 5,
                "input_priority": 0,
                "input_unit": "kWh",
                "input_min": 0,
                "input_max": 1000,
                "cm_id": 1,
            }
        ],
    },

image image

[!NOTE]
Inputs of type layer can be used as the parent_input of a Dynamic Input Options. In this case, use the workspacenames of layers as the input_options_array and define the inputs needed for each layer by using the layer workspacename as the ~path_defined_in_parent_input~ key.

Go To Top

5. Outputs

The Outputs are all the data that the CM will return to the be displayed in the Visualization Tool. Standards have been defined to ensure the frontend can display the data correctly. It can be found in cm/app/api_v1/calculation_module.py All the outputs should be returned in the form of a dictionary named result. Find below an example:

"result":{
    "name":"CM - Scale heat and cool density maps",
    "indicator":[...],
    "graphics":[...],
    "vector_layers":[...], 
    "raster_layers":[...], 
    "extra_zip_files":[...],
    "csv_files":[...]
}

name

Display name of the CM, shown in the frontend

indicator

Array of indicators shown in the result panel. Each item in the array must have:

  • unit (string): unit of the indicator
  • name (string): name of the indicator
  • value (number): value of the indicator
"indicator":[
    {
        "unit":"GWh/yr",
        "name":"Heat density total multiplied by 1.0",
        "value":"398.82725"
    }, 
    {
        "unit": "", 
        "name": "Successful intervention strategy", 
        "value": 'Feedback & gamification through an App'
    }
],

Indicators result:

image image

graphics

Array of graphs, each item in the array is contains the neccessary data to produce a chartjs graph. Find below the most common item to declare for a graph, for more detail refer to chartjs documentation:

  • type (string) : type of the graphic that will be displayed. The possible types are bar, line, radar, pie, polarArea, bubble:

    • line: a line chart or line graph is a type of chart which displays information as a series of data points called 'markers' connected by a straight line segment:
    • bar: a bar chart or bar graph is a chart or graph that presents categorical data with rectangular bars with heights or lengths proportional to the values that they represent:
    • radar: a radar chart is a way of showing multiple data points and the variation between them:
    • pie: a pie chart is divided into segments, the arc of each segment shows the proportional value of each piece of data:
    • polarArea: polar area charts are similar to pie charts, but each segment has the same angle - the radius of the segment differs depending on the value:

  • deprecated xLabel: This attribute is not supported by chartjs anymore, to define the text displayed on x-axis, use the new syntax

  • deprecated yLabel: This attribute is not supported by chartjs anymore, to define the text displayed on y-axis, use the new syntax New syntax to specify axis names. (do not add to pie diagrams or it will add axis lines): New axis label syntax: 
    "options": {
        "scales": {
            "x": {
                "title": {
                    "display": True,
                    "text":"x axis name"
                }
            },
            "y": {
                "title": {
                "display": True,
                "text":"y axis name"
            }
        }
    }

  • data: contains labels and datasets

    • labels (string[]) : x axis labels only x axis.
    • datasets (array): set of data with its configuration
    • label (string) : serie's label
    • backgroundColor (string[]) : background color of each value to display
    • data (number[]) : values of the serie

  • options: options available to handle the charts, learn more at https://www.chartjs.org/docs/latest/charts/line.html, /!\ chartjs example are in JavaScript and should be adapted to Python syntax

All chart types and code example in page Example of Charts
Basic implementation :

  graphics = []
  graphics.append(
    {
      "type": "bar",
      "data": {
        "labels": ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"],
        "datasets": [
          {
            "label": "Temperature (°C)",
            "data": [12, 15, 14, 17, 20, 22, 19],
            "backgroundColor": "#4169E1", #blue
            "borderColor": "#ff0000", #red
            "borderWidth": 1
          }
        ]
      },
      "options": {
        "responsive": True,
        "scales": {
          "y": {
            "beginAtZero": True
          }
        }
      }
    }
  )
  result = dict()
  result['graphics'] = graphics

image

vector_layers

  • name (string): Should include "shapefile", name to be displayed on the frontend
  • path (string): path generated for the geotif file
  • type (string): type of the layer generated for symbology (legend). If set as custom, symbology must be defined. This case is useful for dynamic legends.
  • symbology (array): legend used for a layer of type custom, the symbology defines how the layer will be displayed in the frontend. It only is applied if the type is custom. It should not be defined if the type is not custom. The symbology should contain at least two elements including a case for the value 0 (or the lowest value that should not show on the map), and an opacity of 0, to avoid unwanted visual artifacts.
[
    {
        "name": "shapefile of coherent areas with their potential", 
        "path": output_shp1 
    },
    {
        "name": "District Cooling areas and there potentials - shapefile", 
        "path": output_shp2, 
        "type": "custom",
        "symbology": [
            {"red": 254, "green": 237, "blue": 222, "opacity": 0.5, "value": symbol_vals_str[0], "label": symbol_vals_str[0] + " GWh"},
            {"red": 253, "green": 208, "blue": 162, "opacity": 0.5, "value": symbol_vals_str[1], "label": symbol_vals_str[1] + " GWh"},
            {"red": 253, "green": 174, "blue": 107, "opacity": 0.5, "value": symbol_vals_str[2], "label": symbol_vals_str[2] + " GWh"},
            {"red": 253, "green": 141, "blue": 60, "opacity": 0.5, "value": symbol_vals_str[3], "label": symbol_vals_str[3] + " GWh"},
            {"red": 230, "green": 85, "blue": 13, "opacity": 0.5, "value": symbol_vals_str[4], "label": symbol_vals_str[4] + " GWh"},
            {"red": 166, "green": 54, "blue": 3, "opacity": 0.5, "value": str(float(symbol_vals_str[4]) + step), "label": ">" + symbol_vals_str[4] + " GWh"}
        ]
    }
]
  • vector layer default result (no defined type:):

image

  • vector layer custom result:

image

raster_layers

array of raster layers

  • name (string): name to be displayed on the frontend
  • path (string): path generated for the geotif file
  • type (string): type of the layer generated for symbology (legend). If set as custom, symbology must be defined. This case is useful for dynamic legends.
  • symbology (array): legend used for a layer of type custom, the symbology defines how the layer will be displayed in the frontend. It only is applied if the type is custom. The symbology should contain at least two elements including a case for the value 0 (or the lowest value that should not show on the map), and an opacity of 0, to avoid unwanted visual artifacts.
"type": "custom",
"symbology": [
  {"red": 254, "green": 237, "blue": 222, "opacity": 0, "value": "1", "label": "1 kWh/ha"},
  {"red": 253, "green": 208, "blue": 162, "opacity": 0.8, "value": "100", "label": "100 kWh/ha"},
  {"red": 253, "green": 174, "blue": 107, "opacity": 0.8, "value": "200", "label": "200 kWh/ha"},
  {"red": 253, "green": 141, "blue": 60, "opacity": 0.8, "value": "400", "label": "400 kWh/ha"},
  {"red": 230, "green": 85, "blue": 13, "opacity": 0.8, "value": "600", "label": "600 kWh/ha"},
  {"red": 166, "green": 54, "blue": 3, "opacity": 0.8, "value": "800", "label": "800 kWh/ha"}
]
  • layer (string): type of the layer generated (for toolchaining in CM)

In order to generate a path, developers should use the function generate_output_file_tif(), which needs the output directory as an argument. This function should be imported to the calculation_module.py. The path must be generated on the first lines of calculation() function found in calculation_module.py. This function For example:

    output_tif_1 = generate_output_file_tif(output_directory)

All the layers outputs must be retrieved and added to the raster_layers array after they have been generated by the calculation module provider functions.

[
    {
        "name":"layers of heat_density 1.0",
        "path": output_raster1 
        "type":"heat",
        "layer":"heat_tot_curr_density"
    },
    {
        "name": "layer of charging needs",
        "path": output_raster2,
        "type": "custom",
        "symbology": [
            {"red": 254, "green": 237, "blue": 222, "opacity": 0, "value": "1", "label": "<1 kWh/ha"},
            {"red": 253, "green": 208, "blue": 162, "opacity": 0.8, "value": "100", "label": "100 kWh/ha"},
            {"red": 253, "green": 174, "blue": 107, "opacity": 0.8, "value": "200", "label": "200 kWh/ha"},
            {"red": 253, "green": 141, "blue": 60, "opacity": 0.8, "value": "400", "label": "400 kWh/ha"},
            {"red": 230, "green": 85, "blue": 13, "opacity": 0.8, "value": "600", "label": "600 kWh/ha"},
            {"red": 166, "green": 54, "blue": 3, "opacity": 0.8, "value": "800", "label": "800 kWh/ha"}]
     }
]
  • raster heat result:

image

  • raster custom result:

image

extra_zip_files

How to return a .zip file:

  1. Import the create_extra_zip_file() helper function:
     # in calculation_module.py
     from ..helper import create_extra_zip_file

  1. Use the new helper function create_extra_zip_file() to create the .zip file to return:

    The first parameter must be the output directory path. It is recommended to use output_directory that is accessible in the calculation() method of calculation_module.py.

    Any other parameter should be a path to a file that must be added to the zip. There can be as many file path passed as parameters as needed.

In calculation_module.py:

    def calculation(output_directory, inputs_raster_selection, inputs_parameter_selection):
        extra_output_files = create_extra_zip_file(output_directory, path_to_extra_file1, path_to_extra_file2)

  1. Add the .zip to the result dict:

    use "extra_zip_files" as the key

    The .zip must be added as an object in an array, meaning you could return multiple .zip

    [{"name": "name-shown-when-downloaded", "path": path-returned-by-create_extra_zip_file}]

    # in calculation_module.py
    def calculation(output_directory, inputs_raster_selection, inputs_parameter_selection):
        extra_output_files = create_extra_zip_file(output_directory, path_to_extra_file1, path_to_extra_file2)
        result["extra_zip_files"] = [{"name": "ComplementaryCSVandLogs", "path": extra_output_files }]

csv_files

See below how to add .csv files to the outputs:

The .csv must be added as an object in an array, meaning you can return multiple .csv

[{"name": "name-shown-when-downloaded", "path": path-to-csv-file}]

    # in calculation_module.py
    def calculation(output_directory, inputs_raster_selection, inputs_parameter_selection):
        result["csv_files"] = [{"name": "ComplementaryCSV", "path": path_to_complementary_csv }]

Go To Top

6. Logs

Logs can help in the development process and for monitoring purposes. To log from a CM, avoid using print(). The print statement is not always flushed and it may not appear in the log files. Instead use :

  • logging.info("log") (with import logging)
  • print("log", flush=True)

examples:

    import logging
    logging.info("Log message")       #-> 2025-01-02 08:59:55 [97] [INFO] Log message
    print("Log message", flush=True)  #-> Log message

Access Logs from local machine

At Visualization_tool\.dockerVolumes\logs\calculation_modules\base_calculation_module, you can find the all the log files generated by the instance of the CM running on you local machine.
At .dockerVolumes\logs\calculation_modules\base_calculation_module\calculation_module_service.out.log, you can find your logs defined as explained in this documentation.

Python syntax errors should also be logged in .dockerVolumes\logs\calculation_modules\base_calculation_module\calculation_module_service.out.log.

Access Logs from development and production

To obtain access and information relating to the VPN, an email must be sent to citiwatts@hevs.ch. The HES-SO Valais-Wallis IT department will then send an email to the requester with the information needed to download the VPN software and connection access. You will also receive a login from citiwatts@hevs.ch to access the SFTP server containing the logs. We recommend using the filezilla client to access the contents of the SFTP server.

Once connected to the SFTP server, the user can view a folder by module calculation. Inside, there are 8 log files (calculation_module_service.err.log, calculation_module_service.out.log, consumer_cm_alive.err.log, consumer_cm_alive.out.log, consumer_cm_compute.err.log, consumer_cm_compute.out.log, register_cm.err.log, register_cm.out.log). It is within these files that the developers of calculation module will be able to find all the system logs.

Go To Top

7. Tests

CM developers are encouraged to utilize Unit Tests to enhance the quality of their CM code. By identifying bugs early, ensuring expected functionality, and automating testing, Unit Tests streamline development. They also make refactoring safer, improve code structure, and simplify debugging.

Reminder

A unit test checks the smallest testable parts of an application (units) in isolation to ensure they work correctly. It is typically automated and focuses on individual functions, methods, or classes.

Writing Unit Tests

The base_calculation_module already contains a /tests folder with a tests.py file that must be completed for each CM.

    class TestAPI(unittest.TestCase):
        def setUp(self):
            ...

        def tearDown(self):
            ...

        def test_compute(self):
            raster_file_path = 'tests/data/raster_for_test.tif'
            # simulate copy from HTAPI to CM
            save_path = UPLOAD_DIRECTORY+"/raster_for_test.tif"
            copyfile(raster_file_path, save_path)

            inputs_raster_selection = {}
            inputs_parameter_selection = {}
            inputs_vector_selection = {}
            inputs_raster_selection["layer_to_processs"] = {
                "path": save_path,
                "workspacename": "heat_tot_curr_density",
            }
            inputs_vector_selection["industrial_database_emissions"] = ''
            inputs_parameter_selection["multiplication_factor"] = 2

            # register the calculation module a
            payload = {"inputs_raster_selection": inputs_raster_selection,
                   "inputs_parameter_selection": inputs_parameter_selection,
                   "inputs_vector_selection": inputs_vector_selection}

            rv, json = self.client.post('computation-module/compute/', data=payload)

            # Assert CM API has been called successfully
            self.assertTrue(rv.status_code == 200)
            ...

This file defines a class that subclasses unittest.TestCase and will be therefore discovered as Unit Test by the Python Unit Test Framework (unittest).

The test_compute(self) method triggers the communication route of the CM API in order to call the calculation() method explicitly defined per CM in /cm/app/api_v1/calculation_module.py. Depending on the calculation implemented, appropriate inputs must be set accordingly: inputs_raster_selection, inputs_parameter_selection, inputs_parameter_selection, inputs_vector_selection.

self.assertTrue(rv.status_code == 200 asserts that the CM API has been called successfully.

Enhance Unit Tests

More than only testing if the CM can be successfully called, CM developers are advised to deeply test their code.

This can be first realized by verifying the results of the calculation() method.

The last lines in test_compute(self) method assert that the calculated values correspond to the expected values. Each section of the results are compared individually:

    # Assert result is correct
    expected_result = {
        'name': 'CM - Scale heat and cool density maps',
    'indicator': [
        {"unit": "GWh/yr", "name": "Heat density total multiplied by  2.0", "value": '14070.01'}
    ],
    'graphics': [],
    'vector_layers': [],
    'raster_layers': [{"name": "layers of heat_density 2.0", "path": 'dummy_output.tif', "type": "heat", "layer": "heat_tot_curr_density"}]
    }
    result = json['result']
    self.assertEqual(result['indicator'], expected_result['indicator'])
    self.assertEqual(result['name'], expected_result['name'])
    self.assertEqual(result['indicator'][0]['unit'], expected_result['indicator'][0]['unit'])
    self.assertEqual(result['indicator'][0]['name'], expected_result['indicator'][0]['name'])
    self.assertEqual(result['indicator'][0]['value'], expected_result['indicator'][0]['value'])
    self.assertEqual(result['raster_layers'][0]['name'], expected_result['raster_layers'][0]['name'])
    self.assertEqual(result['raster_layers'][0]['type'], expected_result['raster_layers'][0]['type'])
    self.assertEqual(result['raster_layers'][0]['layer'], expected_result['raster_layers'][0]['layer'])

Comparisons rely on the given inputs, so testing multiple inputs is essential to cover a wide range of scenarios.

Cover different scenarios

CM developers are encouraged to create multiple test_* methods in tests.py file to cover individually different scenarios. These scenarios can be identified as the edge cases that the CM must support.

Additional test_* methods should verify that: * Results vary accordingly to the inputs provided * Outputs files correspond to the expectations (quantity, name, path, content, ...) * Exceptions should be raised when necessary (type, message, ...)

Run Unit Tests

Unit Tests run inside a container.

From the CM root folder (containing docker-compose.tests.yml file), CM developers can run Unit Test with following command: docker-compose -f docker-compose.tests.yml up --build --force-recreate

OK

In this case, 1 test has been run and OK statement confirm that everything ran correctly.

In case of errors, CM developers are quickly informed that some tests have failed.

KO

Stack trace is provided to indicate in which file (i.e. calculation_module.py) and where (i.e. line 46) error happened.

Automated testing

Unit Tests are run by our CI/CD infrastructure. The Jenkins file located in every CM root folder calls the docker-compose -f docker-compose.tests.yml command on every commit made in CM repository.

Go To Top

References

Go To Top

How To Cite

Go To Top

Authors And Reviewers

This page was written by the citiwatts Team and namely by:

EASILab - HES-SO: David Gianadda, Gwenaëlle Gustin, Jérémie Vanin, Alain Duc

Go To Top

License

Go To Top

Acknowledgement

Go To Top