ETL_Datamanagement/README.md

# Hetionet Drug Analysis Pipeline

A comprehensive ETL pipeline and interactive dashboard for analyzing biomedical knowledge graph data from Hetionet v1.0. This project processes complex relationships between diseases, genes, drugs, and symptoms to generate actionable insights for drug repurposing, polypharmacy risk assessment, and biomedical research.

## Table of Contents

- [Overview](#overview)
- [Features](#features)
- [Prerequisites](#prerequisites)
- [Installation](#installation)
- [Project Structure](#project-structure)
- [Data Analyses](#data-analyses)
- [Dashboard Features](#dashboard-features)
- [Output Files](#output-files)
- [Technical Details](#technical-details)
- [Neo4j ETL & Analysis Pipeline](#neo4j-etl--analysis-pipeline)
  
## Overview

This project implements a complete data pipeline for the Hetionet knowledge graph, consisting of:

1. **ETL Pipeline**: Extracts, transforms, and loads Hetionet data into structured CSV files
2. **Analytics Engine**: Performs 8 different biomedical analyses
3. **Interactive Dashboard**: Streamlit-based web interface for data exploration and visualization

### What is Hetionet?

Hetionet is a biomedical knowledge graph containing 47,031 nodes (genes, diseases, drugs, etc.) and 2,250,197 relationships. This project analyzes these connections to identify:

- Genes with the most disease associations ("hotspot genes")
- Drug repurposing opportunities
- Polypharmacy risks
- Disease-symptom relationships
- Drug-drug interaction conflicts

## Features

### ETL Pipeline

- Processes 2.2M+ edges and 47K+ nodes
- Generates analysis-ready CSV files
- Optimized for performance with pre-filtering and indexing
- Handles complex data transformations

### Analyses

1. **Hotspot Genes**: Identifies genes associated with multiple diseases
2. **Drug Repurposing**: Finds existing drugs that could treat new diseases
3. **Polypharmacy Risk**: Calculates risk scores based on side effects
4. **Symptom Triangle**: Maps symptom-disease-drug relationships
5. **Super Drug Score**: Ranks drugs by benefit/risk ratio
6. **Drug Conflicts**: Identifies drugs with overlapping side effects
7. **Network Visualization**: Generates graph data for disease-gene-drug networks
8. **Disease Symptom Diversity**: Analyzes symptom complexity across diseases

### Dashboard

- Interactive visualizations with Plotly
- Global search functionality
- Chart export (PNG/SVG)
- CSV data downloads
- Real-time filtering
- Network graph visualization
- Drug comparison tool

## Prerequisites

### System Requirements

- Python 3.8 or higher
- 4GB+ RAM recommended
- ~500MB disk space for data files

### Required Libraries

```python
pandas>=1.5.0
streamlit>=1.25.0
plotly>=5.15.0
networkx>=3.0
```

## Installation

### Clone or Download Project Files

```bash
# Create project directory
mkdir hetionet_analysis
cd hetionet_analysis
```

### (Optional) Docker Setup for Dashboard

You can run the Streamlit dashboard in a Docker container for easier deployment.

Dockerfile example (already present in project root)

```bash
FROM python:3.11-slim

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PIP_NO_CACHE_DIR=1

WORKDIR /app

COPY . /app

RUN pip install --upgrade pip \
    && pip install streamlit pandas plotly networkx neo4j

EXPOSE 8501

CMD ["bash", "-c", "python etl.py || true && exec streamlit run dashboard.py --server.port=8501 --server.address=0.0.0.0"]


```

Build Docker image

```bash
docker build -t etl-dashboard .
```

Run etl and dashboard in Docker

```bash
docker run -p 8501:8501 etl-dashboard
```

## Non Docker usage

### 1. Set Up Python Environment

```bash
# Create virtual environment
python -m venv etl_projekt

# Activate environment
# On macOS/Linux:
source etl_projekt/bin/activate
# On Windows:
# etl_projekt\Scripts\activate

# Install dependencies
pip install pandas streamlit plotly networkx
```

### 2. Run ETL Pipeline

Execute the ETL pipeline to process the Hetionet data:

```bash
python etl.py
```

**Expected Runtime**: ~ 1 minute

**Output**: Creates `neo4j_csv/` directory with CSV files

### 3. Launch Dashboard

Directly with Python

```bash
streamlit run dashboard.py
```

The dashboard will automatically open in your web browser at `http://localhost:8501`

## Explore Data

Navigate through the dashboard using the sidebar menu:

- **Overview**: Summary statistics and key metrics
- **Hotspot Genes**: Top genes by disease associations
- **Drug Repurposing**: Repurposing opportunities
- **Polypharmacy Risk**: Drugs ranked by side effects
- **Symptom Triangle**: Symptom-disease-drug connections
- **Super Drugs**: Best benefit/risk ratios
- **Drug Conflicts**: Overlapping side effects
- **Network Graph**: Interactive visualization
- **Compare Drugs**: Side-by-side drug comparison

## Project Structure

```editorconfig
hetionet_analysis/
├── hetionet-v1.0.json          # Input data (download separately)
├── hetionet_etl_final.py       # ETL pipeline
├── dashboard.py                # Streamlit dashboard
├── neo4j_csv/                  # Generated output directory
│   ├── nodes_*.csv             # Node files by type (11 files)
│   ├── edges_all.csv           # All relationships
|   |── edges_*.csv              # Splitted relationsship for neo4j import
│   ├── analysis_*.csv          # Analysis results (6 files)
│   ├── network_nodes.csv       # Network visualization nodes
│   └── network_edges.csv       # Network visualization edges
└── README.md                   # This file
```

## Data Analyses

### 1. Hotspot Genes

**Purpose**: Identify genes associated with multiple diseases for potential therapeutic targets.

**Method**: Counts disease associations (via `associates`, `regulates`, `upregulates`, `downregulates`, `binds` relationships) for each gene.

**Key Findings**:

- TNF: 48 disease associations
- TP53: 47 disease associations
- IL6: 41 disease associations

### 2. Drug Repurposing Opportunities

**Purpose**: Discover existing drugs that could treat new diseases based on shared genetic mechanisms.

**Method**:

1. Identify genes associated with each disease
2. Find other diseases sharing those genes
3. Identify drugs treating the related diseases
4. Exclude drugs already treating the target disease

**Output**: Disease-drug pairs with shared gene counts

### 3. Polypharmacy Risk Score

**Purpose**: Assess safety profiles of drugs based on documented side effects.

**Method**:

- Counts side effects per drug
- Calculates risk score: `side_effects / (diseases_treated + 1)`

**Key Metrics**:

- Higher score = higher risk per disease treated
- Enables comparison of drug safety profiles

### 4. Symptom-Disease-Drug Triangle

**Purpose**: Map relationships between symptoms, diseases, and treatments.

**Method**:

1. Count diseases presenting each symptom
2. Identify drugs treating those diseases
3. Calculate impact score: `diseases × treating_drugs`

**Applications**:

- Symptom-based drug discovery
- Understanding disease complexity

### 5. Super Drug Score

**Purpose**: Identify drugs with optimal benefit/risk ratios.

**Method**: `score = diseases_treated / (1 + side_effects)`

**Interpretation**:

- Higher score = better benefit/risk ratio
- Useful for first-line treatment selection

### 6. Drug Conflicts

**Purpose**: Identify drugs with overlapping side effects that may compound when combined.

**Method**:

1. Build drug-to-side-effects mapping
2. Compare all drug pairs
3. Calculate overlap percentage
4. Flag pairs with 10+ shared side effects

**Critical for**: Polypharmacy safety assessment

### 7. Network Visualization

**Purpose**: Provide graph-based view of disease-gene-drug relationships.

**Method**:

- Selects top 20 diseases by symptom count
- Includes connected genes (up to 150)
- Includes drugs treating those diseases (up to 50)

**Format**: NetworkX-compatible node/edge lists

### 8. Disease Symptom Diversity

**Purpose**: Quantify disease complexity by symptom count.

**Method**: Counts unique symptoms per disease via `presents` relationships.

**Insights**:

- Germ cell cancer: 116 symptoms
- Brain cancer: 88 symptoms
- Head and neck cancer: 79 symptoms

## Dashboard Features

### Global Search

Search across genes, diseases, or drugs by name. Results show top 5 matches with key metrics.

### Interactive Charts

All visualizations built with Plotly:

- Hover for detailed information
- Zoom and pan
- Export to PNG/SVG
- Responsive design

### Data Export

Download filtered data as CSV:

- Custom date ranges
- Filtered subsets
- Complete analysis results

### Statistics Boxes

Overview page displays key metrics:

- Average diseases per gene
- Average symptoms per disease
- Average side effects per drug

### Network Graph

Interactive force-directed graph showing:

- Red nodes: Diseases
- Blue nodes: Genes
- Green nodes: Drugs (Compounds)
- Edges: Relationships

### Drug Comparison

Side-by-side comparison of two drugs:

- Diseases treated
- Side effects
- Super score
- Recommendation based on benefit/risk ratio

## Output Files

### Node Files (11 files)

- `nodes_Gene.csv` - Gene entities with disease counts
- `nodes_Disease.csv` - Disease entities with symptom counts
- `nodes_Compound.csv` - Drug/compound entities
- `nodes_Symptom.csv` - Symptom entities
- `nodes_Side_Effect.csv` - Side effect entities
- Plus 6 additional node type files

### Edge File

- `edges_all.csv` - All 2.2M relationships with source, target, and type

### Analysis Files

- `analysis_drug_repurposing.csv` - Repurposing opportunities
- `analysis_polypharmacy_risk.csv` - Drug risk scores
- `analysis_symptom_triangle.csv` - Symptom connections
- `analysis_super_drugs.csv` - Drug rankings
- `analysis_drug_conflicts.csv` - Drug interaction warnings

### Network Files

- `network_nodes.csv` - Graph nodes for visualization
- `network_edges.csv` - Graph edges for visualization

## Technical Details

### Performance Optimizations

**Pre-filtering**: Edges filtered by type once, then reused across analyses

**Set Operations**: Uses Python sets for fast membership testing (O(1) vs O(n))

**Defaultdict**: Builds indices using defaultdict for efficient lookups

**Batch Processing**: Processes edges in batches for memory efficiency

### Data Type Handling

All IDs converted to strings for consistency:

```python
edges_df['source'] = edges_df['source'].astype(str)
edges_df['target'] = edges_df['target'].astype(str)
```

This prevents type mismatch errors when joining dataframes.

### Edge Direction Conventions

Hetionet uses directional relationships. Key conventions:

- `Disease -> Gene` for associations
- `Disease -> Symptom` for presentations  
- `Compound -> Disease` for treatments
- `Compound -> Side Effect` for adverse effects

### Scalability Considerations

The current implementation handles Hetionet v1.0 (47K nodes, 2.2M edges).
Loading the csv files with the extracted relationships even if separetely loaded into neo4j takes a siginificant amount of time. Unfortunately I have not been able to find a suitable solution in time and therefore I am currently only importing the relationships which are fewer in number.

### Data Quality

The analyses depend on Hetionet data quality. Known limitations:

- Not all drugs have documented side effects
- Gene-disease associations vary in evidence strength
- Network is not exhaustive of all biomedical knowledge

## Neo4j ETL & Analysis Pipeline

This repository includes a script for executing analysis queries on the dataset in a Neo4j database.

### Neo4j Prerequisites

Ensure that the following components are installed and ready to use:

- **Neo4j Desktop:** A local database instance must be created.
- **Python 3.x:** Installed on your system.
- **Python Driver:** Install the official Neo4j driver via pip in your virtual environment you created earlier:

    ```bash
    pip install neo4j
    ```

---

## Workflow Steps

Follow these steps exactly in the order provided:

### 1. Start Neo4j Database

  Open **Neo4j Desktop**.
  Select your project and click **Start** on the corresponding database.
  The database must be active before proceeding to the next steps.

### 2. Copy CSV Files

  After your ETL process has generated the CSV files, they must be moved to the Neo4j import directory.
  **Locating the path:** In Neo4j Desktop, click on `Open Folder` -> `Import`.
  Copy all CSV files from your ETL output into this folder.

### 3. Data Import via Cypher

  Navigate to the folder `neo4jqueries/loadingQueriesNeo4j`.
  Execute the Cypher scripts contained there within the **Neo4j Browser**.
  These scripts load the data from the import folder and create the nodes and relationships in the graph.

### 4. Execute Python Analysis

  Start the analysis script via your terminal:

```bash
python neo4j_etl.py
```

  **Input:** The script will ask you for your **database-username** (default: `neo4j`) and your **password**.

  **Processing:** The script automatically reads and executes cypher queries in the following directory `neo4jqueries/analysis_queries`.
  
  **Output:** Results of the analysis will be displayed on the terminal.

---

### Structure

| Directory / file                    | Functionality                                          |
| :---------------------------------- | :----------------------------------------------------- |
| `neo4j_etl.py`                      | Python-Script for execting analysis queries.           |
| `neo4jqueries/loadingQueriesNeo4j/` | Contains all Cypher files for the initial data import. |
| `neo4jqueries/analysis_queries/`    | Includes Cypher files for  analysis.                   |
|                                     |                                                        |

---

## Future Enhancements

Potential extensions to this project:

1. **Machine Learning**: Predictive models for drug efficacy
2. **Temporal Analysis**: Track knowledge graph changes over time
3. **API Development**: REST API for programmatic access
4. **Cloud Deployment**: AWS/GCP hosting for web access
5. **Additional Data Sources**: Integrate DrugBank, KEGG, etc.

## References

- Himmelstein, D. S. et al. (2017). Systematic integration of biomedical knowledge prioritizes drugs for repurposing. *eLife*, 6, e26726.
- Hetionet GitHub: [https://github.com/hetio/hetionet](https://github.com/hetio/hetionet)
- Neo4j Graph Database: [https://neo4j.com](https://neo4j.com)
- Streamlit Documentation: [https://docs.streamlit.io](https://docs.streamlit.io)
  
## License

This project processes publicly available Hetionet data. Refer to Hetionet licensing for data usage terms.

## Contact

For questions or issues, please refer to the project repository or documentation.

---

**Last Updated**: January 2026
**Version**: 1.0.0