Scraping Logs Management

Author(s)

• Faizal Khan

---

Last Updated Date

2026-03-11

---

SRS References

---

Version History

Version	Date	Author	Changes
1.0	2026-03-11	Faizal Khan	Initial scraping logs document

---

Feature Overview

The Scraping Logs Management feature provides admins and resellers the ability to view detailed, paginated data on scraping activities across the system.

Data Visibility & Authorization

• Admin View: When an Admin is viewing the scraping logs, they can see the scraping logs for each and every dealer under every reseller in the platform.
• Reseller View: When a particular Reseller is viewing the scraping logs, they can see the scraping logs for dealers only under them (dealers directly associated with reseller profile).

---

Request and Response Models

ScrapeLogsSortBy Enum

public enum ScrapeLogsSortBy
{
    SourceName = 1,
    DealerName = 2,
    CreatedAt = 3,
    UpdatedAt = 4
}

SortOrder Enum

public enum SortOrder
{
    Ascending = 1,
    Descending = 2
}

public enum ScrapingJobStatus
{
    Queued = 1,
    Fetching = 2,
    FetchedButNotEnriched = 3,
    Enriching = 4,
    Completed = 5,
    Failed = 6
}

ScrapeLog

public record ScrapeLog
{
    public string ScrapingJobId { get; init; } = string.Empty;
    public string DealerName { get; init; } = string.Empty;
    public string SourceName { get; init; } = string.Empty;
    public ScrapingJobStatus Status { get; init; }
    public DateTime? StartedAt { get; init; }
    public DateTime? CompletedAt { get; init; }
    public ScrapeDuration Duration { get; init; } = new();
    public int Total { get; init; }
    public int New { get; init; }
    public int Updated { get; init; }
}

ScrapeDuration

public record ScrapeDuration
{
    public int Hours { get; init; }
    public int Minutes { get; init; }
    public int Seconds { get; init; }
}

ScrapeMetrics

public record ScrapeMetrics
{
    public int Running { get; init; }
    public int Completed { get; init; }
    public int Failed { get; init; }
    public int TotalJobs => Running + Completed + Failed;
}

ServerPaginatedData<T>

public class ServerPaginatedData<T>
{
    public List<T> Data { get; set; } = [];
    public int TotalNumber { get; set; }
    public bool HasPreviousPage { get; set; }
    public bool HasNextPage { get; set; }
    public int TotalPages { get; set; }
    public int PageNumber { get; set; }
    public int RowsPerPage { get; set; }
}

ScrapeLogsFilter

public record ScrapeLogsFilter
{
    public DateOnly? StartDate { get; init; }
    public DateOnly? EndDate { get; init; }
    public ScrapingJobStatus? Status { get; init; }
    public string? SourceName { get; init; }
    public string? SearchKeyword { get; init; } // Searches against Dealer name
    public ScrapeLogsSortBy? SortBy { get; init; } // Defaults to CreatedAt
    public SortOrder? SortOrder { get; init; } // Defaults to Descending
    public int PageNumber { get; init; } = 1;
    public int RowsPerPage { get; init; } = 10;
}

---

API Interfaces

Endpoint	Method	Purpose	Parameters	Request Model	Response Model	Status Codes
/scraping/logs	GET	Retrieve paginated list of scraping logs, enforcing Admin/Reseller isolation restrictions	JWT in header, pagination & filter queries	ScrapeLogsFilter (as FromQuery)	ServerPaginatedData<ScrapeLog>	200, 400, 401, 403, 500
/scraping/metrics	GET	Retrieve aggregated scraping metrics across all logs under user's purview, enforcing Admin/Reseller isolation restrictions	JWT in header (No filter queries)	N/A	ScrapeMetrics	200, 401, 403, 500

---

API Examples

Get Scraping Logs Example

Request:

GET /scraping/logs?pageNumber=1&rowsPerPage=10&status=5&sortBy=3&sortOrder=2
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...

Success Response (200 OK):

{
  "data": [
    {
      "scrapingJobId": "0a9c2c4f-a1fe-4485-9fc2-c1658ae71a68",
      "dealerName": "Premium Auto Group",
      "sourceName": "Facebook Marketplace",
      "status": 5,
      "startedAt": "2026-03-10T10:13:01.571653Z",
      "completedAt": "2026-03-10T12:58:33.571653Z",
      "duration": {
        "hours": 2,
        "minutes": 45,
        "seconds": 32
      },
      "total": 1247,
      "new": 342,
      "updated": 905
    }
  ],
  "totalNumber": 30,
  "hasPreviousPage": false,
  "hasNextPage": true,
  "totalPages": 3,
  "pageNumber": 1,
  "rowsPerPage": 10
}

Error Response (401 Unauthorized):

{
  "status": 401,
  "message": "Invalid or expired token."
}

Error Response (403 Forbidden):

{
  "status": 403,
  "message": "You do not have permission to access these scraping logs."
}

Get Scraping Metrics Example

Request:

GET /metrics
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...

Success Response (200 OK):

{
  "running": 1,
  "completed": 5,
  "failed": 1,
  "totalJobs": 7
}

Error Response (401 Unauthorized):

{
  "status": 401,
  "message": "Invalid or expired token."
}

Error Response (403 Forbidden):

{
  "status": 403,
  "message": "You do not have permission to access scraping metrics."
}