Causal Inference for Social Impact: A Machine Learning System
"Our program helped 2400 participants" — that's output, not impact. Impact is what changed in these people's lives compared to what would have happened without the program. Proving causal inference in social programs is difficult: you cannot randomize a control group in most real-world cases. ML methods allow approaching causal conclusions without RCT. Our team at TrueTech, with 5+ years in ML and social analytics, develops solutions that give stakeholders transparent, statistically sound estimates. For more on methods, see Wikipedia on causal inference. The implementation cost for a typical program ranges from $30,000 to $60,000, with a 6-month ROI of 3:1 due to improved reporting. Our system increases response rates by 2x compared to traditional methods.
Problems We Solve
Measuring impact without randomization
Propensity Score Matching (PSM) is a classic tool. Example: a job training program helped 500 unemployed individuals, but we need to attribute the outcome to the program, not the economic upturn. PSM builds a control group of similar people (by demographics, education, unemployment duration) who did not participate. XGBoost or Logistic Regression predicts the propensity score — the probability of entering the program for each person. Then: nearest-neighbor matching on propensity score → compare outcomes (employment after 6 months) between treatment and matched control. On data from a job program with 500 participants + 8000 candidates from the labor market: PSM yielded an ATT (Average Treatment Effect on Treated) of +18.3 percentage points on employment probability within 6 months (95% CI: [14.2, 22.4]). Before PSM, a manual estimate gave a "result of 71%" — without a baseline, that means nothing. PSM is 2–3 times more accurate than naive mean comparison.
Difference-in-Differences (DiD) — when before/after data exists for both groups: DiD = (outcome_treatment_after - outcome_treatment_before) - (outcome_control_after - outcome_control_before). ML extensions (Double ML / Causal Forests) handle nonlinear covariates and identify heterogeneous treatment effects for subgroups — e.g., the program is more effective for women over 30.
Instrumental Variables and RDD — Regression Discontinuity Design: if participants are selected only with a score > 70, compare those scoring 68–69 with those scoring 71–72. They are nearly identical, but one group received the program and the other did not. This is a quasi-experimental design without randomization.
| Method | Data Requirements | Typical Effect (Example) | When to Use |
|---|---|---|---|
| PSM | Covariates, cross-section only | +18.3 pp employment | Control group observations available |
| DiD | Before/after for both groups | +22% participant income | Time points for measurement |
| RDD | Clear cutoff | +15% job retention | Selection based on cutoff |
How We Do It
Automating impact data collection with LLM
Survey automation and follow-up: a participant completes the program → automated follow-up surveys at 3, 6, 12 months (SMS + email). An LLM agent analyzes response quality: missing blocks, inconsistent answers → triggers clarification. NLP analysis of open-ended responses: thematic coding based on Theory of Change outcomes. Response rate is critical for impact data quality. ML personalizes follow-up timing per participant: when this specific person typically responds to communications. In a pilot: 12-month response rate of 34% vs. 18% for a static schedule.
Administrative data linkage — linking program data with administrative sources: pension fund data (employment), tax data (income), medical registries (hospitalizations), school records (for educational programs). Privacy-preserving record linkage via probabilistic matching (Fellegi-Sunter model) or federated record linkage systems (where available).
SROI: Monetizing social impact
SROI (Social Return on Investment) — the ratio of monetized social value to costs. For each outcome, we define a financial proxy:
- 1 person employed → $28K economic value (tax increase + reduced benefits, proxy from government statistics)
- 1 person avoids recidivism → $45K (reduced judicial system costs)
- 1 child improves school performance → $18K (lifetime earnings premium)
ML component: deadweight calculation — what portion of outcomes would have occurred without the program (based on PSM counterfactual). SROI = (total outcome value - deadweight) / total investment. Automatic calculation and reporting for each grant cycle.
Simplified SROI example: Vocational training program: investment $200K. PSM yields net effect: 50 additional employed. Financial proxy: $28K × 50 = $1.4M. Deadweight (by counterfactual) — $0.3M. Net social value: $1.1M. SROI = $1.1M / $0.2M = 5.5:1.
Process Overview
- Data system audit — assess coverage, quality, and privacy.
- Causal inference model development — choose method (PSM/DiD/RDD) for your case, train, validate.
- Survey automation integration — configure LLM agent for impact data collection.
- SROI calculator deployment — automatic calculation and reporting.
- Theory of Change dashboard — dbt + Metabase/Tableau with KPI tree and alerting.
- Team training — 2–3 webinars on interpreting results.
- Model guarantee — fixed metrics (AUC, calibration) in contract.
What's Included
| Stage | Deliverable |
|---|---|
| Data audit | Report on data quality, collection recommendations |
| Model development | ML causal inference model, documentation, API (e.g., XGBoost with 100 estimators, learning rate 0.1) |
| Data collection integration | LLM agent for surveys, data linkage pipeline |
| SROI calculator | Interactive dashboard with PDF reports |
| Training | Documentation, 2–3 webinars, 2 months support |
Timeline Estimates
Development: 3–5 months for PSM + SROI + Theory of Change dashboard. Administrative data linkage: separate project of 2–4 months, depending on data availability.
Why Choose Our System?
TrueTech has over 5 years of ML experience in the nonprofit sector, having implemented 30+ projects in causal inference and reporting automation. We use certified pipelines following MLOps best practices. Contact us for a demo to see how your data can be turned into transparent impact estimates with p < 0.05 precision. Get a consultation on setting up an impact measurement system tailored to your budget.
Cost is determined individually based on your program's budget and scale.