Daily Text EPUB to JSON

Extract and process daily text content from Spanish EPUB files to JSON format. This tool processes Jehovah's Witnesses daily text publications, extracting scripture, explanations, and references into a structured JSON format. Available as a CLI tool, web interface, and REST API.

🚀 Features

🌐 Web Interface: Modern React-based UI with drag-and-drop support
🔧 REST API: Full-featured API server with real-time progress updates
💻 CLI Tool: Command-line interface with npx support
📱 Responsive Design: Works on desktop and mobile devices
📋 Copy to Clipboard: One-click JSON copying in the web interface
🔄 Real-time Progress: Server-Sent Events for live processing updates
📊 Year Auto-detection: Automatically detects year from EPUB content
🎯 Dual Parsing Strategy: HTML structure parsing with fallback
📅 Special Date Handling: Supports Memorial (Conmemoración) format
💾 Flexible Storage: Export to JSON file or MongoDB database
🏗️ Modern Architecture: ES6 modules with TypeScript support
✅ Comprehensive Testing: Jest test suite with high coverage
🚀 CI/CD Pipeline: Automated testing and deployment

📋 Prerequisites

Node.js 18.0.0 or higher
npm 8.0.0 or higher
MongoDB (optional, for database storage)

🛠️ Installation

Quick Start

# Clone the repository
git clone https://github.com/jjuanrivvera/daily-text-epub-to-json.git
cd daily-text-epub-to-json

# Install dependencies
npm install

# Start the web interface and API server
npm run dev:all

Open your browser at http://localhost:5174 to use the web interface.

Option 1: Use CLI with npx (no installation needed)

# Process an EPUB file directly
npx daily-text-epub-to-json es25_S.epub

Option 2: Install locally for all features

# Clone the repository
git clone https://github.com/jjuanrivvera/daily-text-epub-to-json.git
cd daily-text-epub-to-json

# Install dependencies
npm install

# For web interface development
cd web && npm install

⚙️ Configuration

Create a .env file in the project root (optional for CLI usage):

# Optional - Auto-detected from EPUB content if not specified
EPUB_FILE=es25_S.zip    # Name of the EPUB file in the Epubs folder
YEAR=2025                # Year of the daily texts (auto-detected if omitted)

# Optional - MongoDB configuration
MONGODB_URI=mongodb://localhost:27017/dailytexts

Directory Structure

daily-text-epub-to-json/
├── Epubs/              # Place your EPUB files here
├── Lab/                # Extracted XHTML files (auto-generated)
├── src/                # Source code (ES6 modules)
│   ├── index.js        # Main entry point
│   ├── parser/         # Parsing logic
│   ├── formatter/      # Date and reference formatting
│   ├── storage/        # JSON and MongoDB storage
│   └── utils/          # Utilities and constants
├── Models/             # Legacy MongoDB models (CommonJS)
├── output.json         # Generated JSON output
└── .env                # Configuration file

🎯 Usage

🌐 Web Interface

The web interface provides a modern, user-friendly way to process EPUB files:

Features

Drag & Drop Upload: Simply drag your EPUB file onto the upload area
Real-time Progress: Watch the extraction and processing progress live
Interactive JSON Viewer:
- Preview mode with expandable daily texts
- Raw JSON mode with syntax highlighting
- Copy to clipboard functionality
Year Detection Display: Shows detected year and text count
Download Results: Download processed JSON directly from the browser
Responsive Design: Works on desktop, tablet, and mobile devices

Starting the Web Interface

# Start both web interface and API server
npm run dev:all

# Or start separately:
npm run dev:server  # Start API server on port 3001
cd web && npm run dev  # Start web interface on port 5174

Access the web interface at: http://localhost:5174

Web Interface Usage

Open http://localhost:5174 in your browser
Drag and drop your EPUB file or click to select
Watch real-time processing progress
View results in Preview or Raw JSON mode
Copy JSON to clipboard or download as file

🔧 Server API

The REST API server provides programmatic access to the processing functionality:

API Endpoints

`POST /api/process`

Process an EPUB file and return JSON data.

curl -X POST http://localhost:3001/api/process \
  -F "epub=@es25_S.epub" \
  -H "X-Session-ID: unique-session-id"

Request:

Method: POST
Content-Type: multipart/form-data
Body: EPUB file as form data
Headers:
- X-Session-ID: Unique session identifier for SSE events

Response:

{
  "jobId": "job-1234567890",
  "status": "started",
  "message": "Processing started"
}

`GET /api/events/:sessionId`

Server-Sent Events endpoint for real-time progress updates.

const eventSource = new EventSource('/api/events/unique-session-id');
eventSource.onmessage = (event) => {
  const data = JSON.parse(event.data);
  console.log('Progress:', data);
};

Event Types:

connected: Connection established
progress: Processing progress update
complete: Processing completed
error: Processing error

`GET /api/download/:resultId`

Download processed JSON results.

curl http://localhost:3001/api/download/result-id-1234 -o output.json

`GET /api/health`

Health check endpoint.

curl http://localhost:3001/api/health

Response:

{
  "status": "ok",
  "timestamp": "2025-01-09T12:00:00.000Z",
  "uptime": 123.456
}

Starting the API Server

# Production mode
npm run start:server

# Development mode with auto-reload
npm run dev:server

# With custom port
PORT=8080 npm run dev:server

Default server URL: http://localhost:3001

💻 CLI Usage

# Basic usage with automatic year detection
npx daily-text-epub-to-json es25_S.epub

# With custom options
npx daily-text-epub-to-json es25_S.epub --year 2025
npx daily-text-epub-to-json es25_S.epub --output custom-output.json
npx daily-text-epub-to-json es25_S.epub --mongo --verbose

# Extract only (no processing)
npx daily-text-epub-to-json es25_S.epub --extract-only

# Process only (after extraction)
npx daily-text-epub-to-json es25_S.epub --process-only

CLI Options

Option	Description
`--year <year>`	Override automatic year detection
`--output <path>`	Custom output path for JSON file
`--mongo`	Enable MongoDB storage
`--verbose`	Enable verbose logging
`--extract-only`	Only extract EPUB, don't process
`--process-only`	Only process files, skip extraction
`-V, --version`	Display version information
`-h, --help`	Display help information

npm Scripts

# Complete build process (extract + process)
npm run build

# Using the CLI through npm
npm run cli es25_S.epub

# Step-by-step execution
npm run extract  # Extract EPUB to Lab folder
npm run process  # Process extracted files to JSON

# Legacy commands (CommonJS version)
npm run build:legacy

Output Format

The tool generates an output.json file with 365 daily texts:

[
  {
    "date": "2025-01-01",
    "text": "(Juan 3:16).",
    "textContent": "Porque tanto amó Dios al mundo que dio a su Hijo unigénito.",
    "explanation": "Detailed explanation of the scripture...",
    "reference": "w23.01 15 párr. 10"
  }
  // ... 364 more entries
]

Field Descriptions

date: ISO format date (YYYY-MM-DD)
text: Scripture reference in parentheses
textContent: The actual scripture text
explanation: Commentary and explanation
reference: Watchtower publication reference

🧪 Development

Git Hooks

This project includes pre-commit hooks to ensure code quality before commits:

Installation

# Install hooks automatically (runs on npm install)
npm install

# Or install manually
npm run install:hooks

What the Pre-commit Hook Checks

Prettier Formatting - Ensures consistent code style
ESLint - Checks for code quality issues
Tests - Runs the full test suite
Console.log Detection - Warns about console.log in production code

Bypassing Hooks (Emergency Only)

# Skip pre-commit checks (not recommended)
git commit --no-verify -m "Emergency fix"

Testing

# Run all tests
npm test

# Run tests with coverage
npm run test:coverage

# Run linting
npm run lint

# Auto-fix linting issues
npm run lint:fix

# Format code with Prettier
npm run format

Project Architecture

The project uses a modular ES6 architecture with three main components:

Core Processing Engine

Parser Module: Handles XHTML parsing with dual strategy
Formatter Module: Date and reference formatting
Storage Module: JSON and MongoDB persistence
Utils Module: Shared constants and logging

Web Interface (React + Vite)

React Components: Modular UI components
Custom Hooks: useEpubProcessor for state management
Real-time Updates: Server-Sent Events integration
Responsive Design: Mobile-first CSS approach
Features:
- Drag & drop file upload
- Progress tracking with animations
- JSON viewer with syntax highlighting
- Clipboard API integration

API Server (Express.js)

RESTful Endpoints: Clean API design
SSE Support: Real-time progress streaming
File Management: Multer for file uploads
Job Queue: In-memory job tracking
CORS Enabled: Cross-origin support
Error Handling: Comprehensive error responses

Key Technical Features

Automatic Year Detection
- Detects year from EPUB content metadata
- Multiple detection strategies (OPF, file patterns, title page)
- No dependency on filename conventions
Dual Parsing Strategy
- Primary: HTML structure parsing for accurate extraction
- Fallback: Sanitized text parsing for compatibility
Line Ending Compatibility
- Handles both \r\n (Windows) and \n (Unix) formats
- Dynamic detection and processing
Memorial Date Handling
- Special parsing for "CONMEMORACIÓN" format
- Extracts embedded date from special formatting
Error Recovery
- Graceful fallback mechanisms
- Comprehensive logging for debugging

🔄 CI/CD

The project includes GitHub Actions workflows for:

Continuous Integration: Runs on all pushes and PRs
Multi-version Testing: Tests on Node.js 18.x and 20.x
Code Quality: ESLint and Prettier checks
Security Scanning: npm audit for vulnerabilities
Automated Releases: Tag-based release creation
Dependency Updates: Automated Dependabot PRs

📦 Scripts Reference

Main Scripts

Script	Description
`npm run dev:all`	Start both web interface and API server
`npm run dev:server`	Start API server in development mode
`npm run dev:web`	Start web interface in development mode
`npm run start:server`	Start API server in production mode
`npm run build`	Complete extraction and processing
`npm run cli <file>`	Run CLI with an EPUB file

Processing Scripts

Script	Description
`npm run extract`	Extract EPUB to Lab folder
`npm run process`	Process Lab files to JSON
`npm run build:legacy`	Run legacy CommonJS version

Development Scripts

Script	Description
`npm test`	Run Jest test suite
`npm run test:coverage`	Run tests with coverage report
`npm run lint`	Check code with ESLint
`npm run lint:fix`	Auto-fix ESLint issues
`npm run format`	Format code with Prettier

🤝 Contributing

Fork the repository
Create your feature branch (git checkout -b feature/AmazingFeature)
Commit your changes (git commit -m 'Add some AmazingFeature')
Push to the branch (git push origin feature/AmazingFeature)
Open a Pull Request

📄 License

This project is licensed under the ISC License - see the LICENSE file for details.

🙏 Acknowledgments

Built with Node.js and modern JavaScript
Uses sanitize-html for safe HTML processing
Powered by Jest for testing
CI/CD with GitHub Actions

📞 Support

For issues or questions, please open an issue on GitHub.

Made with ❤️ by jjuanrivvera

Name		Name	Last commit message	Last commit date
.github		.github
Epubs		Epubs
Models		Models
scripts		scripts
server		server
src		src
web		web
.coderabbit.yaml		.coderabbit.yaml
.env.example		.env.example
.gitignore		.gitignore
.prettierrc		.prettierrc
README.md		README.md
eslint.config.js		eslint.config.js
extract-epub.cjs		extract-epub.cjs
jest.config.js		jest.config.js
load-texts-to-json.cjs		load-texts-to-json.cjs
package-lock.json		package-lock.json
package.json		package.json
unzip-node-stream-zip-version.cjs		unzip-node-stream-zip-version.cjs
unzip.cjs		unzip.cjs

Uh oh!

Latest commit

History

Folders and files

Repository files navigation