Skip to main content
Version: MuddVision

Hangfire Background Job Processing POC

Author(s)​

  • Yeasa Mondal

Submission Date​

[2026/02/11]

Version History​

VersionDateChangesAuthor
1.02026-02-11Initial draftYeasa Mondal

Objective​

Implement Hangfire, an open-source background job processing framework for .NET, to manage and schedule recurring integration jobs with improved reliability, persistence, and monitoring capabilities.

Key Goals:

  • Run tasks in the background outside the main request/response flow
  • Provide reliability, persistence, and automatic retries
  • Replace complex manual scheduling logic with CRON expressions
  • Enable individual job control and monitoring through Hangfire Dashboard

Scope​

Current State​

We currently have:

  • 9 different sync jobs: Salesforce, Authenticom, DealerTrend, Campaign, CTM, GoogleAds, CDK, MetaAds, AdsCreatives
  • Single BackgroundService managing all syncs
  • Two scheduling modes: Daily (fixed times) or Interval (same interval for everything)
  • Manual scheduling logic that's becoming complex with multiple time calculations

Proposed Solution​

Migrate to Hangfire for background job processing with:

  • Individual recurring jobs for each integration
  • CRON expression-based scheduling
  • Dashboard for monitoring and manual control
  • PostgreSQL-backed persistence
  • Automatic retry logic with exponential backoff
  • Feature 1: Background job processing and scheduling system
  • Feature 2: Integration service job management
  • Feature 3: Monitoring and operational visibility for recurring jobs
  • Feature 4: Distributed job execution with proper locking

Technical Approach​

Why Hangfire?​

Pros:

  1. Built for this exact use case - Recurring jobs with different schedules

  2. CRON expressions - Powerful, flexible scheduling (e.g., 0 */2 * * * for every 2 hours)

    there are 5 fields:

    ┌──────── Minute (0 - 59)
    │ ┌────── Hour (0 - 23)
    │ │ ┌──── Day of month (1 - 31)
    │ │ │ ┌── Month (1 - 12)
    │ │ │ │ ┌─ Day of week (0 - 6) (Sunday = 0)
    │ │ │ │ │
    │ │ │ │ │
    * * * * *

  3. Dashboard - Monitor job execution, see failures, retry manually

  4. Persistence - Jobs survive application restarts (stores in PostgreSQL)

  5. Retry logic - Automatic retry with exponential backoff

  6. Distributed - Can scale across multiple instances with proper locking

  7. No complex timer logic - Hangfire handles all scheduling complexity

  8. Individual job control - Enable/disable/reschedule jobs independently

Cons:

  1. Additional dependency (but it's mature and well-maintained)
  2. Database overhead (but minimal for our scale)
  3. Learning curve (but straightforward)

Alternative Solutions (Less Optimal)​

SolutionProsCons
Quartz.NETVery powerful, cron supportMore complex than Hangfire, steeper learning curve
Multiple BackgroundServicesSimple, no dependenciesHard to manage configs, no dashboard, reinventing scheduling
Azure Functions/AWS LambdaServerless, scalableArchitectural shift, cost considerations, not on-premise friendly
NCrontab + BackgroundServiceLightweightStill need to build retry logic, monitoring, persistence

Implementation Plan​

1. NuGet Packages to Add​

  • Hangfire.Core (core functionality)
  • Hangfire.AspNetCore (ASP.NET Core integration)
  • Hangfire.PostgreSql (PostgreSQL storage)

2. Files to Create​

  • IntegrationService/Jobs/SalesforceRecurringJob.cs
  • IntegrationService/Jobs/AuthenticomRecurringJob.cs
  • IntegrationService/Jobs/DealerTrendRecurringJob.cs
  • IntegrationService/Jobs/CampaignRecurringJob.cs
  • IntegrationService/Jobs/CtmRecurringJob.cs
  • IntegrationService/Jobs/GoogleAdsRecurringJob.cs
  • IntegrationService/Jobs/CdkRecurringJob.cs
  • IntegrationService/Jobs/MetaAdsRecurringJob.cs
  • IntegrationService/Jobs/AdsCreativesRecurringJob.cs
  • IntegrationService/Configuration/SyncJobConfig.cs (strongly-typed config)
  • IntegrationService/Configuration/HangfireConfig.cs

3. Files to Modify​

  • Program.cs - Add Hangfire setup
  • appsettings.json - Add new config structure
  • appsettings.Development.json - Dev-specific schedules
  • appsettings.Staging.json - Staging schedules
  • appsettings.Production.json - Production schedules

4. Files to Delete/Deprecate​

  • ClientStatisticsSyncScheduler.cs (replace with Hangfire)
  • Old ClientStatisticsSync config section

5. Database Changes​

Hangfire will auto-create its schema tables:

  • hangfire.job
  • hangfire.state
  • hangfire.jobparameter
  • hangfire.jobqueue
  • etc. (about 10 tables)

Separate schemas per service:

  • IntegrationService → hangfire_integration
  • Future services → hangfire_[servicename]

Core Benefits of Hangfire Dashboard​

  1. Observability - You can SEE what's happening
  2. Manual Control - Operational superpower
  3. Failure Debugging - Huge in production
  4. Long-running Job Detection - Detect jobs taking too long
  5. Proof for Business/QA - Concrete evidence of job execution
  6. Queue & Throughput Insight - Monitor system performance

Development Tasks & Estimates​

NoTaskNameEstimate (Hours)DependenciesNotes
1Add Hangfire NuGet packages and setup2 hoursNo dependencyCore, AspNetCore, PostgreSql packages
2Create individual job classes (9 jobs)6 hoursTask 1One class per integration
3Create configuration classes2 hoursTask 1SyncJobConfig, HangfireConfig
4Modify Program.cs and configure Hangfire3 hoursTask 1, 3Add services, middleware, dashboard
5Update appsettings files (all environments)2 hoursTask 3Dev, Staging, Production configs
6Database schema setup and testing2 hoursTask 4Verify auto-creation of Hangfire tables
7Migrate existing job logic to new jobs8 hoursTask 2Port sync logic to new job classes
8Remove deprecated scheduler files1 hourTask 7Clean up old BackgroundService
9Testing and validation4 hoursAll tasksTest all jobs, dashboard, retries
10Documentation and team training2 hoursAll tasksDocument new system, training session
Total32 hours~4 developer days

Testing & Validation​

Test Scenarios​

  1. Job Scheduling Tests

    • Verify each job runs at correct CRON schedule
    • Test manual job triggering from dashboard
    • Validate job execution across different environments

  2. Persistence Tests

    • Stop and restart application
    • Verify jobs resume correctly
    • Check database state persistence

  3. Retry Logic Tests

    • Simulate job failures
    • Verify automatic retry with exponential backoff
    • Test manual retry from dashboard

  4. Dashboard Tests

    • Access dashboard UI
    • Monitor job execution in real-time
    • Test job management features (pause, resume, delete)

  5. Distributed Execution Tests

    • Run multiple application instances
    • Verify proper job locking
    • Ensure no duplicate executions

  6. Performance Tests

    • Monitor database overhead
    • Verify concurrent job execution
    • Check system resource usage

Success Criteria​

  • All 9 integration jobs running on individual schedules
  • Dashboard accessible and showing job history
  • Jobs surviving application restarts
  • Failed jobs automatically retrying
  • No manual timer logic required
  • Clean removal of old scheduler code

Risks & Mitigations​

RiskImpactProbabilityMitigation
Database overhead from Hangfire tablesMediumLowHangfire is optimized for minimal overhead; monitor performance
Learning curve for teamLowMediumProvide training session and documentation
Migration issues from old schedulerHighMediumThorough testing in dev/staging before production
Dashboard security concernsHighMediumImplement proper authentication and authorization
Job execution conflicts during migrationHighLowDeploy during maintenance window with proper rollback plan
Dependency on external libraryMediumLowHangfire is mature, well-maintained, and widely used
Configuration complexityLowMediumCreate clear documentation and strongly-typed config classes

Rollback Plan​

  1. Keep old scheduler code temporarily
  2. Deploy Hangfire alongside existing system initially
  3. Gradual migration: one job at a time
  4. Monitor for 48 hours before full cutover
  5. Quick rollback available if issues detected

Conclusion​

Hangfire is the optimal solution for managing our 9 recurring integration jobs. It provides:

  • Simplified scheduling with CRON expressions instead of complex manual timer logic
  • Built-in reliability with persistence and automatic retries
  • Operational visibility through the comprehensive dashboard
  • Scalability with distributed execution support
  • Reduced maintenance by eliminating custom scheduling code

The estimated 32 hours of implementation time is justified by:

  • Immediate operational improvements
  • Reduced complexity in codebase
  • Better monitoring and debugging capabilities
  • Future-proof architecture for additional background jobs

Recommendation: Proceed with Hangfire implementation as outlined in this POC.

Next Steps​

  1. Get approval for implementation
  2. Set up development environment with Hangfire
  3. Create first job as proof of concept
  4. Plan phased rollout strategy
  5. Schedule team training session