yannforget

Python tooling & style

Public API and abstractions

• Give public methods explicit, annotated parameters. Avoid forwarding **kwargs in public APIs when it hides available options from readers or IDE completion. Repeating a few parameters is preferable to adding an options type solely to avoid that repetition.
• Share substantial behavior, such as retries and error handling, so it stays consistent across methods. But do not introduce type aliases or wrapper types

yannforget

Typing tool deadends

What we mean by "deadends"

The typing tool is based on a CART tree trained on survey data. Each categorical split in that tree only knows about the values that were observed during training: at every node, some values go left, some go right, and some were never seen at all. A deadend is a form path that lands on one of those unseen values - the CART has no rule to apply, so it can neither pick a branch nor assign a segment. This happens when a respondent picks a choice

yannforget

Standardization of custom logic in typing tools

We report here all the custom logic we implemented in previous typing tools, and propose a standardization that fits the new web app we are building.

What we are currently doing with the "options" tab

In the current implementation, the options sheet controls node-level mutations of the CART tree before the XLSForm is generated. Each row in the sheet says: *when you reach the tree node whose variable is src_question, replace it

yannforget

de = DataElementCreate(
    name="New data element",
    aggregation_type=AggregationType.SUM,
    domain_type=DataElementDomain.AGGREGATE,
    value_type=ValueType.NUMBER,
    short_name="new de",
)

client.data_elements.post(de)

yannforget

Explicit task ordering

Current implementation and the problem with it

Task dependencies are currently inferred implicitly through data flow. When a task's output is passed as input to another task, a dependency is created:

@pipeline("example")
def my_pipeline():

yannforget

#' Regression analysis for vulnerability factors.
#'
#' This script performs logistic regression analysis on vulnerability factors
#' using survey-weighted data to identify significant predictors of health outcomes.

library(dplyr)
library(duckdb)
library(survey)
library(stringr)
library(parallel)

yannforget

read_parquet <- function(fp) {
  con <- dbConnect(duckdb::duckdb())
  query <- sprintf("CREATE TABLE data AS SELECT * FROM read_parquet('%s')", fp)
  dbExecute(con, query)
  df <- dbGetQuery(con, "SELECT * FROM data")
  dbDisconnect(con)
  return(df)
}

df <- read_parquet("Senegal_2019DHS8_1.0/data/input/SEN_2019DHS8.parquet")

yannforget

library("rpart")
library("dplyr")
library("tibble")
library("jsonlite")

to_json <- function(tree, fp) {

  # Convert rownames to columns as they contain important information: binary
  # node index for tree$frame and split variable for tree$splits
  #   - tree: rpart object

yannforget

from pathlib import Path

import requests


BASE_URL = "https://storage.googleapis.com/gcp-public-data-arco-era5/raw/date-variable-single_level"
DATA_DIR = Path("data")

VARIABLE = "total_precipitation"
YEAR = 2010

yannforget

import sys
from pathlib import Path
from time import sleep

import docker
import psycopg2
import requests


def run(dbdump_dir: Path, conf_dir: Path, version: str = "2.40"):