File Management System

Author(s)

• Faizal Khan

---

Last Updated Date

[2026-04-14]

---

Version History

Version	Date	Changes	Author
1.0	2026-04-13	Initial Documentation for File Management	Faizal Khan

---

Feature Overview

Objective:
Provide a centralized, secure, and efficient file management system for the MarketPulse platform that supports multi-file operations, metadata tracking, and role-based access control. The system enables authenticated users to upload, retrieve, and delete files with automatic metadata persistence and public/private access management.

Scope:

• Multi-file upload with FileType-based folder organization (Images, Documents, General)
• Multi-file deletion with soft-delete capability (using fileId and fileType)
• File retrieval with FileType validation and access control (using fileKey)
• Comprehensive file metadata tracking and persistence
• Public and private file access management
• Authentication-based file access control
• Automatic folder creation based on FileType enum
• File size and type validation
• Audit trail for all file operations
• Dual identifier system: fileId (internal) and fileKey (public-facing retrieval)

Dependencies:

• PostgreSQL database for metadata persistence
• ASP.NET Core for HTTP file handling (IFormFileCollection)
• Logging framework for operation tracking
• JWT authentication for access control
• FileMetadata table for persistent tracking

---

Requirements

Functional:

1. Authenticated users can upload multiple files simultaneously.
2. Files are automatically renamed to unique GUID format with original extension preserved (e.g., 0e32463f-af34-4469-8ad4-21646ca4b649.jpg).
3. System creates appropriate folder structures automatically based on file type or provided folder name.
4. System persists comprehensive metadata for each uploaded file in the FileMetadata table.
5. File metadata includes: userId, fileId, fileKey, originalName, mimeType, fileSize, filePath, isDeleted, isPublic, createdAt.
6. Upload response returns both fileId (for deletion) and fileKey (for retrieval).
7. Authenticated users can delete multiple files simultaneously using fileId.
8. Soft-delete is implemented: files are marked as deleted but physically retained (data recovery).
9. Users can retrieve files using fileKey (unique public identifier).
10. Public files are accessible without authentication.
11. System returns file bytes, content type, and appropriate error messages on retrieval.
12. Multiple files can be processed in a single request with partial success/failure tracking.

Non-Functional:

1. Low latency (<500ms) for single file retrieval.
2. Support for batch operations: upload 100+ files, delete 50+ files, retrieve 20+ files simultaneously.
3. Secure storage with proper MIME type validation.
4. High reliability and data consistency.
5. Scalability for concurrent file operations.
6. Comprehensive error handling with detailed user feedback.
7. Support for files up to 50MB in size.

---

Design Specifications

Architecture Overview

The File Management System integrates with the ManagementService and utilizes a PostgreSQL database for metadata persistence:

1. Client/API submits file upload/delete/get requests with appropriate parameters.
2. FileManagementService processes operations and manages disk storage.
3. FileMetadataService persists and retrieves file metadata from PostgreSQL.
4. API Endpoints expose upload, delete, and get operations to authenticated clients.
5. Logging tracks all operations for audit purposes.

Data Models

FileMetadata Table

CREATE TABLE IF NOT EXISTS filemetadata (
    userid UUID NOT NULL,
    fileid VARCHAR(255) PRIMARY KEY,
    filekey VARCHAR(255) NOT NULL UNIQUE,
    originalname VARCHAR(255) NOT NULL,
    mimetype VARCHAR(100) NOT NULL,
    filesize BIGINT NOT NULL,
    filepath VARCHAR(500) NOT NULL,
    filetype VARCHAR(50) NOT NULL DEFAULT 'Images',
    isdeleted BOOLEAN DEFAULT FALSE,
    ispublic BOOLEAN NOT NULL,
    createdat TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updatedat TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
    deletedat TIMESTAMPTZ,
    CONSTRAINT fk_user FOREIGN KEY (userid) REFERENCES users(id) ON DELETE CASCADE
);

CREATE INDEX idx_filemetadata_userid ON filemetadata(userid);
CREATE INDEX idx_filemetadata_fileid ON filemetadata(fileid);

C# Models

// File Operation Models
public record FileUploadRequest
{
    public IFormFileCollection Files { get; init; }
    public string FolderName { get; init; } = "general"; // Default folder
    public bool IsPublic { get; init; } = false; // Default to private
    public string? Description { get; init; }
}

public record FileMetadata
{
    public string FileId { get; init; } // Unique identifier
    public string FileKey { get; init; } // Unique key for retrieval
    public Guid UserId { get; init; } // Uploader's user ID
    public string OriginalName { get; init; } // Original filename
    public string MimeType { get; init; } // e.g., image/jpeg
    public long FileSize { get; init; } // Size in bytes
    public string FilePath { get; init; } // Relative path on disk
    public FileType FileType { get; init; } // Images, Documents, General
    public bool IsDeleted { get; init; } = false;
    public bool IsPublic { get; init; } = false;
    public DateTime CreatedAt { get; init; }
    public DateTime? UpdatedAt { get; init; }
    public DateTime? DeletedAt { get; init; }
}

public enum FileType
{
    Images = 0,      // Image files
    Documents = 1,   // Document files
    Generl = 2
}

public record FileOperationResult
{
    public string FileId { get; init; }
    public string OriginalName { get; init; }
    public string UniqueFileName { get; init; } // GUID with extension
    public bool Success { get; init; }
    public string? ErrorMessage { get; init; }
    public string? FileKey { get; init; } // For retrieval
    public long FileSize { get; init; }
}

public record FileRetrievalResult
{
    public string FileId { get; init; }
    public byte[]? FileBytes { get; init; }
    public string? ContentType { get; init; }
    public bool Success { get; init; }
    public string? ErrorMessage { get; init; }
    public string OriginalName { get; init; }
}

public record FileDeletionResult
{
    public string FileId { get; init; }
    public string OriginalName { get; init; }
    public bool Success { get; init; }
    public string? ErrorMessage { get; init; }
}

public record CommonResponse
{
    public int Status { get; init; } // 200, 400, 401, 403, 404, 500
    public string? Message { get; init; }
    public object? Data { get; init; }
}

public record BatchOperationResponse<T>
{
    public List<T> Results { get; init; } = new();
    public int TotalItems { get; init; }
    public int SuccessCount { get; init; }
    public int FailureCount { get; init; }
    public DateTime ProcessedAt { get; init; } = DateTime.UtcNow;
}

File Naming Convention

Files are renamed to GUID format preserving the original extension:

• Original: product-details.pdf
• Renamed: 0e32463f-af34-4469-8ad4-21646ca4b649.pdf

FileKey Generation:

• FileKey is a unique identifier derived from folder type and fileId
• Example: images-0e32463f-af34-4469 (folder + first 19 chars of GUID)
• Used for public-facing retrieval API
• More user-friendly than full fileId

Benefits:

• Prevents filename collisions
• Eliminates special character issues
• Maintains file type information via extension
• FileKey provides clean URL-safe identifier for file retrieval
• FileId tracks internal file references and ownership

---

FileId vs FileKey: Clarification

Property	FileId	FileKey
Format	Full GUID (e.g., 0e32463f-af34-4469-8ad4-21646ca4b649)	Folder + first 19 chars (e.g., images-0e32463f-af34-4469)
Purpose	Internal identifier, database key, audit logging	Public-facing retrieval identifier
Use Case	Deletion, metadata queries, internal operations	File download API endpoint
API Endpoint	/file/delete (in request body as fileIds)	/file/{fileKey} (in URL path)
Unique	Yes (Primary key in database)	Yes (Unique constraint in database)
Human-Readable	No	Partially (includes folder name)

Key Points:

• Upload Response: Returns both fileId and fileKey
• File Retrieval: Use fileKey in GET request path
• File Deletion: Use fileId in DELETE request body
• Database: Both stored, both indexed for fast lookup
• Security: FileKey is URL-safe and indicates file category; FileId is more secure for internal tracking

Access Control Logic

File Retrieval Access Rules

STEP 1: DETERMINE AUTHENTICATION STATUS (Server-side by Controller)
    → Validate JWT token from Authorization header
    → Extract userId and userType from JWT claims
    → isAuthenticated = (valid JWT token found)

STEP 2: CHECK ACCESS
IF file.isPublic == TRUE
    → File accessible to anyone (no JWT required)
ELSE IF file.isPublic == FALSE
    IF isAuthenticated == TRUE
        IF userId == file.userId OR userType == "admin"
            → File accessible
        ELSE
            → Return 403 Forbidden (User not owner/admin)
    ELSE IF isAuthenticated == FALSE
        → Return 403 Forbidden (Authentication required)

File Modification Access Rules

Upload/Delete:
    → Require valid JWT token
    → Controller validates token and extracts userId, userType
    → Only authenticated users can upload
    → Only file owner or admin can delete
    → All operations logged with userId

---

API Specifications

Endpoints Overview

Endpoint	Method	Purpose	Auth Required	Parameters	Request Model	Response Model	Status Codes
/file/upload	POST	Upload single or multiple files	Yes	fileType (query param, required)	IFormFileCollection	BatchOperationResponse<FileOperationResult>	200, 400, 401, 413, 500
/file/delete	DELETE	Delete single or multiple files	Yes	fileIds and fileType (in request body)	DeleteFilesRequest	BatchOperationResponse<FileDeletionResult>	200, 400, 401, 404, 500
/file/{fileKey}	GET	Retrieve a single file	Conditional	fileKey (path param), fileType (query param)	–	File bytes (application/octet-stream)	200, 400, 403, 404, 500

---

API Interfaces

1. POST /file/upload - Upload Files API

Purpose:
Upload single or multiple files with automatic GUID renaming, metadata persistence, and access control configuration.

Authentication: JWT token in Authorization header (required)

Query Parameters:

• fileType (FileType enum, required): File type category (Images)
  • Images : Image files - stored in /images folder

Request Model:

public record UploadFilesRequest
{
    public IFormFileCollection Files { get; init; }
    public FileType FileType { get; init; } = FileType.Images;
}

Response Model: BatchOperationResponse<FileOperationResult>

Status Codes:

• 200 (OK) - Partial or complete success
• 400 (Bad Request) - Invalid parameters or empty file collection
• 401 (Unauthorized) - Missing or invalid JWT token
• 413 (Payload Too Large) - File size exceeds limit (50MB)
• 500 (Server Error) - Unexpected server error

Business Rules:

• Maximum file size per file: 50MB
• Maximum total batch size: 5GB
• FileType determines the folder where files are stored
• Supported MIME types: image/*, application/pdf, application/
• All files are renamed to GUID format with original extension
• Metadata is persisted immediately after successful upload
• Returns both successful uploads and failures in single response
• FileType is stored in metadata for validation on retrieval

Request Example:

POST /file/upload?fileType=Images
Authorization: Bearer {JWT_TOKEN}
Content-Type: multipart/form-data

files: [file1.jpg, file2.jpg]

Response (200 OK - Mixed Success/Failure):

{
  "results": [
    {
      "fileId": "0e32463f-af34-4469-8ad4-21646ca4b649",
      "originalName": "product-photo.jpg",
      "uniqueFileName": "0e32463f-af34-4469-8ad4-21646ca4b649.jpg",
      "success": true,
      "errorMessage": null,
      "fileKey": "prod-img-0e32463f-af34-4469",
      "fileSize": 2048576
    },
    {
      "fileId": "1f43574g-bf45-4570-9be5-32757db5c760",
      "originalName": "invalid-file.exe",
      "uniqueFileName": null,
      "success": false,
      "errorMessage": "File type 'application/x-msdownload' is not allowed",
      "fileKey": null,
      "fileSize": 0
    }
  ],
  "totalItems": 2,
  "successCount": 1,
  "failureCount": 1,
  "processedAt": "2026-04-13T10:30:45Z"
}

Error Response (400 Bad Request - No Files):

{
  "status": 400,
  "message": "No files provided in the request",
  "data": null
}

Error Response (401 Unauthorized):

{
  "status": 401,
  "message": "Missing or invalid JWT token"
}

---

2. DELETE /file/delete - Delete Files API

Purpose:
Soft-delete single or multiple files. Files are marked as deleted but retained on disk for recovery.

Authentication: JWT token in Authorization header (required)

Request Model: DeleteFilesRequest

public record DeleteFilesRequest
{
    public FileType FileType { get; init; } = FileType.Images; // File type category
    public List<string> FileIds { get; init; } // File IDs to delete
}

Response Model: BatchOperationResponse<FileDeletionResult>

Status Codes:

• 200 (OK) - Partial or complete success
• 400 (Bad Request) - Invalid request format
• 401 (Unauthorized) - Missing or invalid JWT token
• 404 (Not Found) - File(s) not found
• 500 (Server Error) - Unexpected server error

Business Rules:

• Only file owner (creator) or admin can delete files
• Deletion is soft-delete: mark isDeleted=true, set deletedat=now()
• Physical files remain on disk but metadata is updated
• Returns both successful deletions and failures in single response
• Prevents duplicate deletion attempts
• Audit log entry created for each deletion
• FileType is validated to ensure consistency

Request Example:

DELETE /file/delete
Authorization: Bearer {JWT_TOKEN}
Content-Type: application/json

{
  "fileType": "Images",
  "fileIds": [
    "0e32463f-af34-4469-8ad4-21646ca4b649",
    "1f43574g-bf45-4570-9be5-32757db5c760"
  ]
}

Response (200 OK):

{
  "results": [
    {
      "fileId": "0e32463f-af34-4469-8ad4-21646ca4b649",
      "originalName": "product-photo.jpg",
      "success": true,
      "errorMessage": null
    },
    {
      "fileId": "1f43574g-bf45-4570-9be5-32757db5c760",
      "originalName": "document.pdf",
      "success": true,
      "errorMessage": null
    }
  ],
  "totalItems": 2,
  "successCount": 2,
  "failureCount": 0,
  "processedAt": "2026-04-13T10:35:20Z"
}

Error Response (403 Forbidden - Not Owner):

{
  "status": 403,
  "message": "You do not have permission to delete this file"
}

Error Response (404 Not Found):

{
  "status": 404,
  "message": "One or more files not found",
  "data": {
    "notFoundFileIds": [
      "invalid-file-id-12345"
    ]
  }
}

---

3. GET /file/{fileKey} - Get File API

Purpose:
Retrieve a single file by its unique fileKey with access control and content delivery.

Authentication: JWT token in Authorization header (conditional)

• Required for private files (isPublic=false)
• Not required for public files (isPublic=true)

Path Parameters:

• fileKey (string): The unique file key (format: folder-<first-19-chars-of-guid>, e.g., images-0e32463f-af34-4469)

Query Parameters:

• fileType (FileType enum, required): File type category for validation (Images, Documents, General)

Authentication:

• The controller automatically determines authentication status by validating the JWT token in the Authorization header
• Controller checks: valid userId and userType from JWT claims

Response:

• On Success: File bytes with appropriate Content-Type header
• On Failure: JSON error response

Status Codes:

• 200 (OK) - File retrieved successfully
• 400 (Bad Request) - Invalid file key format or fileType mismatch
• 403 (Forbidden) - File is private and user not authenticated
• 404 (Not Found) - File not found or marked as deleted
• 500 (Server Error) - Unexpected server error

Business Rules:

• Public files (isPublic=true) accessible without authentication
• Private files (isPublic=false) require valid JWT token with userId and userType
• Controller determines authentication status by validating JWT token
• File ownership verified: private file userId must match authenticated userId
• FileType is validated against the file's stored metadata
• Deleted files (isDeleted=true) cannot be retrieved (404)
• Content-Type header matches file's MIME type
• Access attempts logged for audit

Request Example (Public Image File - No Authentication Required):

GET /file/images-0e32463f-af34-4469?fileType=Images

Request Example (Private Image File - Authenticated User):

GET /file/images-0e32463f-af34-4469?fileType=Images
Authorization: Bearer {JWT_TOKEN}

Response (200 OK):

HTTP/1.1 200 OK
Content-Type: image/jpeg
Content-Length: 2048576
Content-Disposition: attachment; filename="product-photo.jpg"

[Binary file content]

Error Response (403 Forbidden - Private File, Not Authenticated):

{
  "status": 403,
  "message": "You do not have permission to access this file. Authentication required."
}

Error Response (404 Not Found):

{
  "status": 404,
  "message": "File not found or has been deleted"
}

---

File Type Classification

Supported file types and their folders:

public static class FileTypeClassifier
{
    private static readonly Dictionary<string, string> ExtensionFolderMap = new()
    {
        // Images
        { ".jpg", "images" },
        { ".jpeg", "images" },
        { ".png", "images" },
        { ".gif", "images" },
        { ".webp", "images" },
        { ".svg", "images" },
    };
    
    public static string GetFolderFromExtension(string filename)
    {
        string extension = Path.GetExtension(filename).ToLower();
        return ExtensionFolderMap.ContainsKey(extension) 
            ? ExtensionFolderMap[extension] 
            : "general"; // Default folder for unknown types
    }
    
    public static string GetContentType(string filename)
    {
        var ext = Path.GetExtension(filename).ToLower();
        return ext switch
        {
            ".jpg" or ".jpeg" => "image/jpeg",
            ".png" => "image/png",
            ".gif" => "image/gif",
            ".pdf" => "application/pdf",
            ".zip" => "application/zip",
            ".docx" => "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
            _ => "application/octet-stream"
        };
    }
    
    public static bool IsAllowedExtension(string filename)
    {
        var ext = Path.GetExtension(filename).ToLower();
        var disallowed = new[] { ".exe", ".bat", ".cmd", ".sh", ".ps1" };
        return !disallowed.Contains(ext);
    }
}

---

Error Handling

All endpoints return standardized error responses:

public record ErrorResponse
{
    public int Status { get; init; }
    public string Message { get; init; }
    public Dictionary<string, string>? Details { get; init; }
    public DateTime Timestamp { get; init; } = DateTime.UtcNow;
}

---

Workflows

Workflow 1: User Uploads Files

1. User selects files from their system (e.g., 3 images for product listing).
2. Frontend submits POST /file/upload?folder=product-images&isPublic=true with JWT token.
3. Backend validates:
   • Authentication token
   • File types and sizes
   • Folder name format
4. For each file:
   • Generate unique GUID name: 0e32463f-af34-4469-8ad4-21646ca4b649.jpg
   • Create folder if not exists: files/upload/images/
   • Save file to disk
   • Persist metadata to database
   • Log operation to audit trail
5. Response returns:
   • Success: fileId, originalName, uniqueFileName, fileKey
   • Failure: errorMessage with reason
6. Frontend displays: Success count, failure count, and error details.

Workflow 2: User Retrieves a File

Scenario A - Public File:

1. Any user requests GET /file/images-0e32463f-af34-4469
2. Backend lookup: Query metadata by fileKey
3. Backend checks metadata: isPublic=true
4. Access granted: File bytes returned with image/jpeg content-type
5. User receives: File download in browser/app

Scenario B - Private File (Owner):

1. Authenticated user requests GET /file/images-0e32463f-af34-4469 with JWT Authorization header
2. Backend validates: JWT token and extracts userId, userType
3. Backend lookup: Query metadata by fileKey
4. Backend checks: isPublic=false, userId matches file owner, access authorized
5. Access granted: File bytes returned
6. User receives: File download

Scenario C - Private File (Unauthorized User):

1. Unauthenticated user requests GET /file/images-0e32463f-af34-4469 without JWT token
2. Backend validation: No valid JWT token in Authorization header
3. Backend lookup: Query metadata by fileKey
4. Backend checks: isPublic=false, authentication required but not provided
5. Access denied: 403 Forbidden response
6. User receives: Permission denied message

Workflow 3: User Deletes Files

1. User selects files to delete (e.g., 2 outdated product images).
2. Frontend submits DELETE /file/delete with fileIds array and JWT token.
3. Backend validates:
   • Authentication token
   • File ownership or admin role
   • File existence
4. For each file:
   • Check authorization
   • Mark isDeleted=true
   • Set deletedat=current_timestamp
   • Log operation with userId
5. Response returns:
   • Success/failure details for each file
   • Summary: totalItems, successCount, failureCount
6. Frontend displays: Deletion results to user.

---

Testing & Quality Assurance

Acceptance Criteria

• ✅ All files uploaded with GUID naming
• ✅ Metadata persisted in database for all files
• ✅ Public files accessible without authentication
• ✅ Private files require authentication
• ✅ Soft-delete removes file from retrieval but persists metadata
• ✅ Batch operations return detailed per-file results
• ✅ All operations logged with userId
• ✅ Proper error messages for all failure scenarios
• ✅ API endpoints documented with examples
• ✅ Code coverage >80%

---

References

• Database Schema: FileMetadata table definition
• API Error Handling Standards
• JWT Authentication Guide
• File Upload Security Best Practices

---