CDPH Spark Analytics

A comprehensive big data processing solution for analyzing San Francisco's Complaint Data Portal (CDPH) dataset using Apache Spark. This project demonstrates modern data engineering techniques for handling, processing, and deriving insights from large-scale complaint resolution data.

Overview

CDPH Spark Analytics provides a robust framework for performing temporal, geographic, and operational analysis on complaint data. Built with Apache Spark, it leverages distributed computing to process large datasets efficiently and extract meaningful patterns from urban complaint resolution workflows.

Features

• Temporal Analysis: Track complaint trends and patterns across multiple years
• Geographic Analysis: Identify the top 10 most affected streets in San Francisco
• Resolution Metrics: Calculate average resolution times and delays by complaint type
• Robust Data Processing: Handle complex CSV data with proper parsing for quoted fields and multi-line records
• Batch Processing: Process large datasets in batch mode with optimized Spark DataFrames
• Streaming Capabilities: Support for real-time streaming analysis of complaint data
• Big Data Scale: Designed to handle massive datasets with efficient distributed processing
• HDFS Integration: Persist results in Hadoop Distributed File System for long-term storage

Project Structure

cdph-spark-analytics/
├── src/
│   ├── main/
│   │   ├── java/tn/spark/cdph/
│   │   │   ├── Main.java
│   │   │   ├── batch/
│   │   │   │   ├── ComplaintByResolution.java
│   │   │   │   ├── ComplaintByType.java
│   │   │   │   └── ComplaintByYear.java
│   │   │   ├── modern/
│   │   │   │   └── CDPHDataFrame.java
│   │   │   └── streaming/
│   │   │       └── CDPHStream.java
│   │   └── resources/
│   └── test/
│       └── java/
├── pom.xml
└── README.md

Technologies

• Apache Spark 3.5.0: Big Data processing framework
• Spark SQL: Structured data processing and analysis
• Spark Streaming: Real-time data processing
• Java 8: Core implementation language
• Maven: Build and dependency management
• HDFS: Distributed file storage system
• SLF4J: Logging framework

Building the Project

Ensure you have Maven and Java 8+ installed on your system.

cd cdph-spark-analytics
mvn clean package

This command will compile the project and create a JAR file in the target/ directory.

Running the Application

Prerequisites

• Apache Spark 3.5.0+ installed and configured
• CDPH dataset in CSV format
• HDFS cluster or local file system for output storage

Basic Execution

spark-submit --class tn.spark.cdph.modern.CDPHDataFrame \
  target/cdph-spark-1.jar \
  /path/to/input/data.csv \
  /path/to/output/directory

Using the Main Class

spark-submit --class tn.spark.cdph.Main \
  target/cdph-spark-1.jar \
  /path/to/input/data.csv \
  /path/to/output/directory

Analyses Performed

1. Dataset Schema Analysis

Displays the complete schema of the CDPH dataset and total record count for data validation.

2. Annual Complaint Trends

Tracks the evolution of complaints year by year, identifying seasonal patterns and long-term trends in complaint volumes.

3. Geographic Hotspots

Identifies the top 10 streets with the highest complaint frequency, enabling targeted urban planning and resource allocation.

4. Resolution Time Analysis

Calculates average resolution delays by complaint type, highlighting areas that require process improvements.

5. Data Persistence

Automatically exports aggregated results to HDFS in CSV format for downstream analysis and reporting.

Data Processing Capabilities

The application handles complex CSV data with:

• Multi-line record support
• Proper quote and escape character handling
• Null value filtering
• Date format parsing (MM/dd/yyyy)
• String manipulation and concatenation
• Aggregation and statistical calculations

Performance Optimization

• Distributed processing across cluster nodes
• Efficient filtering before aggregations
• Optimized date calculations
• Proper null value handling to prevent data loss

Output Format

Results are exported in CSV format with headers, containing:

• Annual complaint counts with year identifiers
• Top affected streets with frequency counts
• Average resolution delays by complaint type

Development Notes

• Spark SQL DataFrames provide type-safe data processing
• Functions like datediff(), to_date(), and avg() ensure accurate calculations
• Additional filtering mechanisms prevent null value propagation
• Output directory structure follows Spark conventions

License

This project is licensed under the Apache License 2.0. See LICENSE file for details.

Author

Meriem Melki

Contributing

Contributions are welcome. Please fork the repository, create a feature branch, and submit a pull request.

Support

For issues, questions, or suggestions, please open an issue on the GitHub repository.