CDK Data Synchronization

Author(s)

• Reshmi Karan

Last Updated Date

2026-04-14

---

SRS References

---

Version History

Version	Date	Changes	Author
1.0	2026-04-14	Initial documentation for CDK sync	Reshmi Karan

---

Feature Overview

Objective: Establish automated synchronization of sales and inventory data from CDK (formerly Integration Management System) software via Own CDK API. This feature enables real-time data flow from Integration management systems into the MuddVision platform for analytics and reporting.

Scope:

• Data ingestion from CDK via Own CDK API
• Support for multiple dealerships with dealer-level isolation
• Handling of both historical (initial) and incremental (daily) data syncs
• Sales data processing and normalization
• Job status tracking and error handling

Dependencies:

• Fortellis API Service (IFortellisAPIService) - Vision API for data requests
• Integration Data Access Layer (IIntegrationDAL) - Data persistence
• CDK Contract Libraries - Data models and types
• Microsoft SQL Server - Sync log and status tracking
• Logging Framework (ILogger) - Monitoring and diagnostics

---

Requirements

Functional Requirements

1. Dealer Discovery: Retrieve all dealers configured for CDK integration from the database
2. Historical Data Sync: On first sync, fetch data from the past 1 year (configurable via constants)
3. Incremental Sync: On subsequent syncs, fetch only the past 4 days of data (configurable)
4. Subscription Validation: Validate dealer CDK subscriptions and department information
5. Job Tracking: Create and maintain sync job logs with status and request IDs
6. Duplicate Prevention: Skip dealers with ongoing sync jobs
7. Sales Data Processing: Extract and process sales records post-sync
8. Error Handling: Gracefully handle failures and continue processing other dealers
9. Status Reporting: Return success/error counts for each sync operation

Non-Functional Requirements

1. Performance: Process dealers serially to avoid API rate limits
2. Reliability: Retry pending jobs before creating new ones
3. Observability: Comprehensive logging at Info, Warning, and Error levels
4. Consistency: Maintain referential integrity with dealer and subscription data
5. Scalability: Support unlimited number of dealers

---

Design Specifications

Data Models

Vision Submit Request

public record VisionSubmitRequest
{
    public Guid JobId { get; init; }
    public Guid SubscriptionId { get; init; }
    public DateOnly StartDate { get; init; }
    public DateOnly EndDate { get; init; }
}

public record VisionSubmitResponse
{
    public Guid JobId { get; init; }
    public VisionJobStatus Status { get; init; }
    public Guid SubscriptionId { get; init; }
    public DateOnly StartDate { get; init; }
    public DateOnly EndDate { get; init; }
    public VisionCoverageSummary CoverageSummary { get; init; } = default!;
    public DateTime CreatedAt { get; init; }
}

public record VisionCoverageSummary
{
    public int TotalDays { get; init; }
    public int DaysAlreadyCached { get; init; }
    public int DaysToFetchFromCdk { get; init; }
}

public record VisionJobStatusResponse
{
    public Guid JobId { get; init; }
    public VisionJobStatus Status { get; init; }
    public DateTime CreatedAt { get; init; }
    public DateTime UpdatedAt { get; init; }
    public string? ErrorMessage { get; init; }
}

public record VisionPagination
{
    public int Limit { get; init; }
    public string? NextCursor { get; init; }
    public bool HasNext { get; init; }
}

public record VisionResultsResponse
{
    public Guid JobId { get; init; }
    public Guid SubscriptionId { get; init; }
    public string? ClientId { get; set; }
    public VisionPagination Pagination { get; init; } = default!;
    public List<VisionSalesResult> Data { get; init; } = [];
}

VisionJobStatus (Enum)

• Pending: Job submitted, awaiting processing in CDK
• Processing: CDK is retrieving data
• Completed: Data successfully retrieved
• Failed: Job failed in CDK system

Dealer Integration Model

public class Dealer
{
    public string ClientId { get; set; }              // Unique dealer identifier
    public string Name { get; set; }                  // Dealer name
    public string CdkId { get; set; }                 // CDK subscription ID (GUID format)
    public IntegrationType IntegrationType { get; set; }
}

CDK Result

public record VisionSalesResult
{
    // Customer
    public string? CustomerId { get; init; }
    public string? FullName { get; init; }
    public string? BusinessPhone { get; init; }
    public string? HomePhone { get; init; }
    public string? Email { get; init; }
    public string? Street { get; init; }
    public string? City { get; init; }
    public string? State { get; init; }
    public string? Zip { get; init; }
    public string? AddressLine1 { get; init; }
    public string? AddressLine2 { get; init; }
    public DateTime? BirthDate { get; init; }

    // Vehicle
    public string? Vin { get; init; }
    public int? Year { get; init; }
    public string? Make { get; init; }
    public string? Model { get; init; }
    public string? ModelName { get; init; }
    public string? ModelNo { get; init; }
    public int? Mileage { get; init; }
    public string? StockNo { get; init; }
    public string? Color { get; init; }

    // Deal
    public string? DealNo { get; init; }
    public string? DealType { get; init; }
    public DateOnly? ContractDate { get; init; }
    public DateTime? SalesDate { get; init; }
    public DateOnly? AccountingDate { get; init; }
    public DateOnly? CreationDate { get; set; }
    public DateOnly? DeliveryDate { get; init; }

    // Pricing / Gross
    public decimal? FrontGross { get; init; }
    public decimal? BackGross { get; init; }
    public decimal? GrossProfit { get; init; }
    public decimal? CashPrice { get; init; }
    public decimal? Msrp { get; init; }
    public decimal? CostPrice { get; init; }
    public decimal? FinanceAmt { get; init; }
    public decimal? FinanceCharge { get; init; }
    public decimal? Apr { get; init; }
    public decimal? PaymentAmt { get; init; }
    public int? Term { get; init; }
    public string? SalesPerson { get; init; }
    public string? FinanceType { get; init; }
    public decimal? BuyRateApr { get; init; }

    //cobuyer Customer
    public string? CobuyerCustomerId { get; init; }
    public string? CobuyerFullName { get; init; }
    public string? CobuyerBusinessPhone { get; init; }
    public string? CobuyerHomePhone { get; init; }
    public string? CobuyerEmail { get; init; }
    public string? CobuyerStreet { get; init; }
    public string? CobuyerCity { get; init; }
    public string? CobuyerState { get; init; }
    public string? CobuyerZip { get; init; }
    public string? CobuyerCountry { get; init; }
    public string? CobuyerAddressLine1 { get; init; }
    public string? CobuyerAddressLine2 { get; init; }
    public DateTime? CobuyerBirthDate { get; init; }
    //Trade
    public List<VehicleTradeDto>? Trades { get; init; }
}

Database Record

public record VisionDataRequest
{
    public Guid JobId { get; init; }
    public Guid SubscriptionId { get; init; }
    public DateOnly StartDate { get; init; }
    public DateOnly EndDate { get; init; }
    public VisionJobStatus Status { get; set; }
    public DateTime CreatedAt { get; init; }
    public DateTime UpdatedAt { get; set; }
    public string? ErrorMessage { get; set; }
}

---

API Interfaces

SyncCdkDataAsync()

Purpose: Initiates CDK data synchronization for all configured dealers

Method Signature:

public async Task<(int successCount, int errorCount)> SyncCdkDataAsync()

Return Value:

• successCount: Number of dealers with successfully submitted sync requests
• errorCount: Number of dealers that encountered errors

Process Flow:

1. Retrieve all dealers with CDK integration
2. Fetch active subscriptions from Fortellis API
3. Validate dealer CDK IDs and department information
4. For each dealer:
   • Check if historical data has been retrieved
   • Determine date range (1 year for first sync, 4 days for regular)
   • Check for existing in-progress jobs
   • Submit new Vision data request if needed
   • Update sync log with pending status

Error Handling:

• Logs detailed error messages but continues processing other dealers
• Returns partial success/error counts even if some dealers fail
• Returns (0, 1) if no dealers found or API subscription fetch fails

ProcessCdkSalesDataAsync()

Purpose: Process completed Vision jobs and retrieve sales data from CDK API

Method Signature:

public async Task<(int successCount, int errorCount)> ProcessCdkSalesDataAsync()

Returns via ProcessCdkSalesDataAsync:

Workflow:

public async Task ProcessCdkSalesDataAsync()

Return Value:

• successCount: Number of dealers with successfully processed sales records
• errorCount: Number of dealers that encountered errors

Process Flow:

1. Retrieve all dealers with CDK integration
2. For each valid dealer:
   • Fetch Vision job request(s) with status 'Processing' or 'Pending'
   • Call CDK API to get current job status
   • If job status is 'Completed':
     • Retrieve batchwise results from CDK API
     • Process each record (normalization, validation)
     • Store processed sales data in MuddVision database
     • Upsert purchase data using (DealNumber, ClientId, Customer) as the composite deduplication key; update the existing record if a match is found, otherwise insert a new record.
     • Update job status in sync log
   • If job status is 'Failed' or 'Pending':
     • Log status and skip batch retrieval
3. Return aggregated success/error counts

Error Handling:

• Logs errors but continues with next dealer
• Does not count skipped dealers (no unprocessed records) as errors
• Returns partial counts even if some dealers fail

---

API Interfaces:

Endpoint	Method	Parameters	Response	Status Codes
/api/v1/data-requests/	POST	VisionSubmitRequest	VisionSubmitResponse	200, 500,400,409
/api/v1/data-requests/{jobId}/status	GET	jobId (query)	VisionJobStatusResponse	200, 404, 500
api/v1/data-requests/{jobId}/results	GET	jobId, limit, cursor (query)	VisionResultsResponse	200, 500, 409, 404

---

Workflow

Daily Synchronization Flow

[Scheduled Trigger]
         ↓
[SyncCdkDataAsync]
         ↓
├─ Validate Dealers (CDK ID + Department)
│
├─ For Each Dealer:
│  ├─ Check History Status
│  ├─ Determine Date Range
│  │  ├─ First Sync: 1 year back
│  │  └─ Regular Sync: 4 days back
│  ├─ Check for Existing Jobs
│  ├─ Submit Vision Request → CDK API
│  └─ Update Sync Log (Status: Pending)
│
└─ Return Success/Error Counts

[CDK/Fortellis Processes Request]
         ↓
[Later: ProcessCdkSalesDataAsync via ProcessCompletedVisionJobAsync]
         ↓
├─ For Each Dealer:
│  ├─ Validate Dealer
│  ├─ Fetch Vision Job Requests (Status: Processing/Pending)
│  ├─ Check CDK API for Job Status
│  ├─ If Status = Completed:
│  │  ├─ Get Batchwise Results from CDK API
│  │  ├─ Process Each Record (Normalize, Validate)
│  │  ├─ Store in MuddVision Database
│  │  └─ Update Sync Log Status
│  └─ Skip if Status ≠ Completed
│
└─ Return Success/Error Counts

Data Flow Diagram

CDK Software (Dealership System)
         ↓
   [Fortellis Vision API]
         ↓
[SyncCdkDataAsync - Submit Request]
         ↓
[Sync Job Log - Status: Pending]
         ↓
[CDK Processes Request]
         ↓
[Job Status: Completed]
         ↓
[ProcessCompletedVisionJobAsync - Check Status & Retrieve]
         ↓
[Get Batchwise Results from CDK API]
         ↓
[Process & Validate Sales Records]
         ↓
[MuddVision Database]

---

Note: When showing data in the dashboard, deduplicate records by client_id, vin, and monthly purchase_date

Development Tasks & Estimates

No	Task Name	Status	Notes
1	CDK API Integration	Complete	Vision API implemented
2	Sync Job Log Management	Complete	DAL methods available
3	Dealer Validation & Filtering	Complete	CdkId and Department checks
4	Historical vs Incremental Logic	Complete	History flag tracked in DB
5	Error Handling & Logging	Complete	Comprehensive logging added
6	Sales Data Processing Pipeline	Complete	ProcessDealerCdkSalesAsync implemented
7	Status Tracking & Reporting	Complete	Return tuple standardized

---

Testing & Quality Assurance

Unit Tests

• Test dealer discovery and filtering
• Test date range determination (historical vs incremental)
• Test duplicate job prevention logic
• Test error handling and logging
• Test return value calculation (success/error counts)
• Test sales processing workflow

Integration Tests

• Mock Fortellis API responses
• Test end-to-end sync flow with multiple dealers
• Test handling of API failures
• Test concurrent access to sync logs
• Test transaction rollback on error

Acceptance Criteria

1. ✅ Dealers without CDK IDs are skipped with warning log
2. ✅ First sync retrieves data from 1 year back
3. ✅ Subsequent syncs retrieve only 4 days of data
4. ✅ In-progress jobs are not duplicated
5. ✅ All dealers are processed even if some fail
6. ✅ Sales records are correctly identified as processed/unprocessed
7. ✅ Comprehensive logging captures all major operations

Testing Tools

• NUnit / xUnit for unit testing
• Moq for mocking Fortellis API service
• SQL Server test database for DAL testing
• Application Insights for production monitoring

---

Deployment Considerations

Configuration Changes

-- Sync log table for CDK data requests
CREATE TABLE tblcdk_sync_log (
    jobid UUID PRIMARY KEY,
    subscriptionid UUID NOT NULL,
    status TEXT NOT NULL,
    startdate TIMESTAMP NOT NULL,
    enddate TIMESTAMP,
    startedat TIMESTAMP NOT NULL DEFAULT NOW(),
    completedat TIMESTAMP,
    errormessage TEXT
);

Migration Steps

1. Deploy service with new CDK sync functionality
2. Create initial sync logs table if not exists
3. Configure scheduled job to trigger SyncCdkDataAsync daily
4. Configure scheduled job to trigger ProcessCdkSalesDataAsync after CDK processing completes
5. Monitor initial syncs for dealers to verify data flow

Rollout Plan

1. Phase 1: Deploy to staging environment
2. Phase 2: Verify with subset of testing dealers
3. Phase 3: Monitor with canary dealer group
4. Phase 4: Full production rollout

---

Risks & Mitigations

Risk	Impact	Likelihood	Mitigation Strategy
Missing CDK ID for dealer	Medium	High	Clear validation and logging; manual review of dealers requiring CDK setup
Duplicate sync job submissions	High	Low	Check existing jobs before submission; implement idempotency
Data sync lag	Medium	Medium	Monitor job completion times; adjust thresholds if needed
Failed dealer processing blocks others	Medium	Low	Process dealers serially with individual error handling (current implementation)
Subscription ID validation failures	Medium	Medium	Validate CDK ID format; ensure Fortellis returns complete subscription list
Historical data overlap	Low	Low	Track history flag in database; prevent re-fetching historical data

---

Monitoring & Alerts

Key Metrics to Track

• Number of successful dealer syncs per day
• Number of failed dealer syncs per day
• Average time per dealer sync
• Total data volume synced (records/MB)
• Average time to process sales data after sync
• Job status distribution (Pending/Processing/Completed/Failed)

Alert Conditions

• Error count > 0 in any sync operation
• Sync takes longer than 30 minutes per dealer
• CDK API unavailable
• Database connection failures
• Missing dealer validation data

Log Levels

• Information: Sync start/completion, dealer processing start
• Warning: Dealers skipped due to missing CDK ID or department info
• Error: API failures, database errors, unexpected exceptions
• Debug: Detailed processing per dealer, job status checks

---

Review & Approval

• Reviewer: Ayon Das
• Approval Date: 16-04-2026

---

Notes

• The implementation processes dealers serially to respect API rate limits and avoid cascading failures
• Job IDs are reused for pending jobs to prevent duplicate submissions
• The DELTA_THRESHOLD_DAYS constant is critical and should match the CDK API processing window
• Sales data processing is intentionally decoupled from sync initiation via ProcessCompletedVisionJobAsync to allow CDK time to complete processing
• ProcessCdkSalesDataAsync orchestrates the overall sales data processing workflow, which internally calls ProcessCompletedVisionJobAsync to handle completed Vision jobs
• Batchwise retrieval from CDK API allows efficient handling of large result sets