Version 2.0.55: Major refactoring and expanded format support

- Added comprehensive audio format support (MP3, FLAC, WAV, AAC, OGG, etc.)
- Expanded video format support (HEVC/H.265, ProRes, DNxHD, etc.)
- Enhanced image format support (HEIC/HEIF, RAW formats, etc.)
- Major code refactoring to eliminate redundancy
- Created utils.py with shared utilities (ProgressTracker, etc.)
- Improved performance with optimized file discovery and hash calculation
- Updated all documentation and version references

Author: TTLequals0

CHANGELOG.MD

@@ -7,6 +7,61 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [2.0.55] - 2025-01-20
+
+### Refactoring
+- **Eliminated Code Redundancy**:
+  - Created `utils.py` module with shared utilities:
+    - `ProgressTracker` class for unified progress calculation across all operations
+    - `log_operation_status()` for consistent operation logging
+    - `batch_process()` generator for consistent batch processing
+    - Common state manipulation functions (`mark_operation_complete`, `mark_operation_error`)
+  - Created `operation_handlers.py` with base classes for operations
+  - Refactored state models to use shared `create_state_dict()` function
+  - Replaced duplicate progress calculation code with centralized `ProgressTracker`
+  - Standardized error handling patterns with decorators
+  - Consolidated operation logging with `log_operation_status()`
+
+### Added
+- **Complete Audio Format Support**:
+  - Added support for all major audio formats (MP3, FLAC, WAV, AAC, OGG, Opus, WMA, etc.)
+  - Implemented audio-specific corruption detection using FFmpeg
+  - Added FLAC-specific validation for lossless integrity checking
+  - Deep scan mode for audio files to detect timestamp issues and packet errors
+  
+- **Expanded Video Format Support**:
+  - Added HEVC/H.265 support (.hevc, .h265 extensions)
+  - Added professional formats: ProRes (.prores), DNxHD/DNxHR (.dnxhd, .dnxhr), MXF (.mxf)
+  - Added broadcast formats: AVCHD (.mts, .m2ts, .avchd)
+  - Added legacy and additional formats: MPEG (.mpg, .mpeg, .vob), RealMedia (.rm, .rmvb), and more
+  
+- **Enhanced Image Format Support**:
+  - Added Apple HEIC/HEIF support (.heic, .heif)
+  - Added comprehensive RAW format support:
+    - Canon (CR2, CR3), Nikon (NEF, NRW), Sony (ARW, SRF, SR2)
+    - Adobe DNG, Olympus ORF, Panasonic RW2, Pentax PEF/PTX
+    - Fujifilm RAF, Sigma X3F, Kodak DCR/KDC, and more
+  - Added professional formats: PSD, OpenEXR, HDR, SVG
+  - Added scientific/specialized formats: FITS, Netpbm formats
+
+### Performance Optimizations
+- **File Discovery Performance**: 
+  - Replaced os.walk() with os.scandir() for 3-5x faster directory traversal
+  - Reduced file stat() calls by caching metadata during discovery
+  - Files are now discovered with creation time pre-fetched for efficient sorting
+  
+- **Database Operations**:
+  - Enhanced database write queue to process up to 100 items in a single transaction
+  - Improved batch processing to group similar operations together
+  - Added connection pooling with pool_size=10 and max_overflow=20 for better concurrency
+  - Optimized SQLite pragmas for better write performance
+  
+- **Hash Calculation**:
+  - Increased hash calculation chunk size from 4KB to 1MB (250x improvement in I/O operations)
+  - Use 4MB chunks for files larger than 1GB
+  - Added throughput metrics (MB/s) to hash progress logging
+  - Reduced system calls significantly for large file processing
+
 ## [2.0.53] - 2025-01-20
 
 ### Fixed

Dockerfile

@@ -22,7 +22,7 @@ ENV FLASK_APP=app.py
 ENV FLASK_ENV=production
 
 # Set version as build argument
-ARG APP_VERSION=2.0.53
+ARG APP_VERSION=2.0.55
 ENV APP_VERSION=$APP_VERSION
 
 CMD ["gunicorn", "--bind", "0.0.0.0:5000", "--workers", "4", "--timeout", "300", "app:app"]

README.md

@@ -149,19 +149,15 @@ Interactive exclusion management with modern UI:
 PixelProbe is available on Docker Hub as `ttlequals0/pixelprobe`. Check the [Docker Hub page](https://hub.docker.com/r/ttlequals0/pixelprobe/tags) for all available versions.
 
 **Current stable versions:**
-- **`ttlequals0/pixelprobe:latest`** - Latest stable release (v2.0.53)
+- **`ttlequals0/pixelprobe:latest`** - Latest stable release (v2.0.55)
+- **`ttlequals0/pixelprobe:2.0.55`** - Major refactoring for code quality, added comprehensive audio/video/image format support
 - **`ttlequals0/pixelprobe:2.0.53`** - Fixed file-changes progress tracking with async updates
-- **`ttlequals0/pixelprobe:2.0`** - Complete UI overhaul with modern responsive design
-- **`ttlequals0/pixelprobe:1.26`** - UI improvements and ImageMagick UTF-8 fixes
-- **`ttlequals0/pixelprobe:1.25`** - Database resilience for long-running scans
-- **`ttlequals0/pixelprobe:1.24`** - UI color visibility fixes
-- **`ttlequals0/pixelprobe:1.23`** - SQLite WAL mode and connection improvements
 
 You can specify a specific version in your `docker-compose.yml`:
 ```yaml
 services:
   pixelprobe:
-    image: ttlequals0/pixelprobe:2.0.53  # or :latest for newest
+    image: ttlequals0/pixelprobe:2.0.55  # or :latest for newest
 ```
 
 ### Development Setup
@@ -344,12 +340,24 @@ PixelProbe uses multiple methods to detect file corruption:
 - **Frame Validation**: Attempts to decode video frames to detect corruption
 - **Quick Scan**: Fast check of first 10 seconds for immediate feedback
 - **Stream Validation**: Verifies video and audio stream integrity
+- **HEVC/ProRes Support**: Specialized detection for modern codecs
 
 ### Image Files
 - **PIL Verification**: Uses Python Imaging Library for basic corruption detection
 - **ImageMagick**: Advanced image analysis and validation
 - **Dimension Checks**: Validates image dimensions and properties
 - **Format Validation**: Ensures files match their declared format
+- **RAW/HEIC Support**: Handles camera RAW files and Apple's HEIC format
+
+### Audio Files
+- **FFmpeg Audio Analysis**: Comprehensive audio stream validation
+- **Decode Testing**: Attempts to decode audio to detect corruption
+- **Header Validation**: Checks for missing or corrupted headers
+- **Format-Specific Tests**: 
+  - FLAC: CRC validation and built-in integrity checking
+  - MP3: Frame header validation
+  - Lossless formats: Bit-perfect verification
+- **Deep Scan Mode**: Full file analysis for timestamp and packet errors
 
 ### Detection Accuracy
 - **High Confidence**: 100% detection of files with broken headers, truncated files, and I/O errors
@@ -359,10 +367,33 @@ PixelProbe uses multiple methods to detect file corruption:
 ## Supported File Formats
 
 ### Video Formats
-- MP4, MKV, AVI, MOV, WMV, FLV, WebM, M4V
+- **Common**: MP4, MKV, AVI, MOV, WMV, FLV, WebM, M4V
+- **HEVC/H.265**: HEVC, H265 
+- **Professional**: ProRes, MXF, DNxHD, DNxHR
+- **Broadcast**: MTS, M2TS, AVCHD
+- **Legacy**: MPG, MPEG, VOB, RM, RMVB
+- **Other**: 3GP, 3G2, F4V, F4P, OGV, ASF, AMV, M2V, SVI
 
 ### Image Formats
-- JPEG, PNG, GIF, BMP, TIFF, WebP
+- **Common**: JPEG, PNG, GIF, BMP, TIFF, WebP
+- **Apple**: HEIC, HEIF
+- **RAW Formats**: 
+  - Canon: CR2, CR3
+  - Nikon: NEF, NRW
+  - Sony: ARW, SRF, SR2
+  - Adobe: DNG
+  - Others: ORF (Olympus), RW2 (Panasonic), PEF/PTX (Pentax), RAF (Fujifilm), X3F (Sigma), DCR/KDC (Kodak), MOS (Leaf)
+- **Professional**: PSD, EXR, HDR, SVG
+- **Other**: ICO, PBM, PGM, PPM, PNM, FITS
+
+### Audio Formats (NEW!)
+- **Lossy**: MP3, AAC, M4A, WMA, OGG, OGA, Opus, AMR
+- **Lossless**: FLAC, WAV, AIFF, APE, WV (WavPack), TTA, CAF
+- **Uncompressed**: WAV, AIFF, AU, SND, VOC
+- **High-Resolution**: DSF, DFF (DSD)
+- **Dolby/DTS**: AC3, DTS
+- **Container**: MKA (Matroska Audio), M4B (Audiobook)
+- **Legacy**: RA, RAM (RealAudio), GSM, MIDI
 
 ## Performance Considerations