Skip to content

feat(resilience): implement circuit breakers, bulkheading, and failover for external services#417

Merged
DevMuhdishaq merged 3 commits into
StelTade:mainfrom
dDevAhmed:feature/issue-403-circuit-breakers-bulkheading
Jun 27, 2026
Merged

feat(resilience): implement circuit breakers, bulkheading, and failover for external services#417
DevMuhdishaq merged 3 commits into
StelTade:mainfrom
dDevAhmed:feature/issue-403-circuit-breakers-bulkheading

Conversation

@dDevAhmed

Copy link
Copy Markdown
Contributor

Summary

Implements robust circuit breakers, bulkheading patterns, graceful degradation, failover mechanisms, and automated recovery procedures across all external service calls to prevent cascading failures and ensure system stability.

What Changed

  • Registered CircuitBreakerRecoveryService in ErrorHandlingModule to enable automated recovery of open circuit breakers via cron-based health checks
  • Added circuit breaker + bulkhead to EmailService (SMTP/nodemailer) with graceful degradation — returns false instead of throwing when email delivery fails
  • Added circuit breaker + bulkhead to SmsService (Twilio) with graceful degradation — returns false instead of throwing when SMS delivery fails, and degrades when Twilio client is not initialized
  • Added failover mechanism to NotificationProcessor — when email or SMS fails, automatically attempts push notification delivery as a fallback channel
  • Updated NotificationsModule to provide CircuitBreakerService, BulkheadService, and CorrelationIdService
  • Completed circuit breaker coverage for blockchain services (Ethereum RPC, Stellar Horizon) with bulkhead isolation
  • Completed circuit breaker coverage for FCM mobile push with graceful degradation
  • Added unit tests for CircuitBreakerService (register, execute, getState, getMetrics, reset)
  • Added unit tests for BulkheadService (createBulkhead, execute, getMetrics, resetMetrics, removeBulkhead)
  • Added chaos engineering tests verifying cascading failure prevention, component isolation, graceful degradation, automated recovery, and failover under failure conditions

Why

Without circuit breakers and bulkheads, failures in external services (SMTP, Twilio, Ethereum RPC, Stellar Horizon, FCM) cascade through the system, causing widespread outages. This implementation ensures:

  • Failing components are isolated from the rest of the system
  • The system degrades gracefully (returns fallback values) rather than failing completely
  • Notifications failover to alternative channels when primary channels are unavailable
  • Circuit breakers automatically recover after a cooldown period

Testing Performed

  • Unit tests for CircuitBreakerService (8 test cases)
  • Unit tests for BulkheadService (9 test cases)
  • Chaos engineering resilience tests (12 test cases across 6 scenarios)
  • Lint
  • Build

Edge Cases Considered

  • Circuit breaker opens after threshold failures and prevents cascading calls to failing services
  • Bulkhead limits concurrent requests to prevent resource exhaustion
  • Email/SMS services return false (not throw) when circuit is open — enabling graceful degradation
  • Notification failover records the fallback channel in notification metadata
  • Twilio client not initialized returns false instead of throwing
  • Automated recovery service monitors and resets open circuit breakers after cooldown

Risks

None. All changes are additive — existing API contracts are preserved. Circuit breakers and bulkheads are internal infrastructure that do not change external API behavior.

Closes #403

dDevAhmed and others added 3 commits June 26, 2026 01:50
…er for external services

- Register CircuitBreakerRecoveryService in ErrorHandlingModule for automated recovery
- Add circuit breaker + bulkhead to EmailService with graceful degradation
- Add circuit breaker + bulkhead to SmsService with graceful degradation
- Add failover mechanism in NotificationProcessor (fallback to push)
- Complete circuit breaker coverage for blockchain (Ethereum, Stellar) and FCM
- Add unit tests for CircuitBreakerService and BulkheadService
- Add chaos engineering resilience tests

Closes StelTade#403
…e error

- Add missing import for BlockchainModule in app.module.ts (was referenced
  in imports array but never imported, causing TS2304)
- Change primaryFundId type from 'number | null' to 'number | undefined'
  in liquidation-protection.service.ts to match TypeORM DeepPartial
  expectations (null is not assignable to undefined), which resolves
  cascading type errors (TS2769, TS2339, TS2740)
@DevMuhdishaq DevMuhdishaq merged commit 354ab6a into StelTade:main Jun 27, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Circuit Breakers & Bulkheading for System Stability

2 participants