S3 File Management System

Author(s)

• Faizal Khan

Last Updated Date

2026-03-04

---

SRS References

• 3.4.1 S3 File Upload
• 3.4.3 Access Control & Visibility Rules

---

Version History

Version	Date	Changes	Author
1.0	2026-03-04	Initial draft for new S3-based File Management	Faizal Khan

---

Feature Overview

Objective:
Implement a centralized file management system for handling file uploads, metadata tracking, and secure S3 retrievals with type-based organization and simple authenticated-access enforcement.

Scope:

• Store physical files in AWS S3 and track metadata in the database.
• Granular access enforcement using a simple IsPublic boolean flag.
• Type-specific S3 folder routing: Main folder (Image/Document/Excel)/filetype/uniquefile.ext.
• Support for various filetypes (Profile, Inventory, Parts, ServiceRequest, Legal, Other).
• Auto-generate structured metadata for every upload.
• Record database interactions for tracking ownership, size, and mime types.

Dependencies:

• AWS S3 Storage Provider
• System Database for filemetadata

---

Requirements

Functional Requirements

1. File Upload

   • Accept file uploads and extract metadata (OriginalName, MimeType, FileSize).
   • Enforce an upload file size limit of 50 MB per file.
   • Generate unique identifiers (FileId, FileKey).
   • Determine Main folder dynamically based on extension.
   • Route to S3 following the structure: {Main Folder}/{FileType}/image.
   • Store exact file path (FilePath) in database.
   • Insert tracking metadata into the filemetadata table including UserId (Owner).

2. Access Control

   • IsPublic = true: Accessible by absolutely anyone.
   • IsPublic = false: Accessible only by authenticated users (a valid Token is required).

3. File Retrieval

   • Fetch valid URL/File streams from S3 via stored FileId or fetch dynamically from an absolute external URL.
   • If the request is an external full URL, the backend will download the payload via HTTP proxy and return the file to the client.
   • Enforce authentication logic: If the file is not public (IsPublic = false), the request MUST contain a valid Bearer token.
   • Exclude soft-deleted (IsDeleted = true) files.
   • Provide fallback mechanisms if a file is not found in the designated type folder. The image will be fetched from external server.

4. File Management

   • Soft-delete files (mark IsDeleted as true, potentially retain physical file in S3 or move to a cold bucket).
   • Validate FileType Enum values on upload.

Non-Functional Requirements

1. Security

   • Validate file types and content safety (MimeType mismatch protection).
   • Strictly evaluate authentication existence before dispensing non-public files.
   • Never expose raw S3 credentials or internal bucket namespaces directly if preventable.

2. Performance

   • Optimize metadata table calls with proper indexing on FileId and UserId.
   • Utilize S3 streaming to avoid excessive memory usage.

3. Scalability

   • S3-first architecture to accommodate virtually unlimited file scales.
   • Efficient handling of potentially large Excel or Document files.

4. Reliability

   • Audit trail tracking.
   • Transactional safety for database writes against S3 push confirms.

---

Design Specifications

UI/UX Design

(Portal / Admin Panel)

• File upload component with type selector dropdown (Inventory/Dealer)
• Drag-and-drop or click-to-browse interface
• Image preview after successful upload
• Inline error messages for validation failures

---

Workflow

Upload Process:

1. Client provides File(s), FileType. The backend will determine IsPublic automatically.
2. Backend determines file access based on FileType:
   • Profile, Inventory, Parts, Other -> IsPublic = true
   • ServiceRequest, Legal -> IsPublic = false
3. Backend determines target Main Folder:
   • Extensions with .jpg, .jpeg, png and .svg-> Main folder: Image
   • Extensions with .xlsx, .xls and.csv -> Main folder: Excel
   • Extension with .pdf -> Main folder: Document
4. Backend generates unique FileId (GUID string) and S3 FileKey (representing {UniqueFileName.ext}).
5. Constructed S3 Path = {Main Folder}/{FileType}/image.
6. Push binary to S3.
7. Write to database table filemetadata.
8. Return file metadata identifiers to client.

Retrieval Process:

1. Download API called with FileId.
2. Query filemetadata table.
3. Assess Access:
   • If IsPublic == true, proceed to fetch.
   • If IsPublic == false, check if request is authenticated (User.Identity.IsAuthenticated). Proceed if true.
4. If authorized, retrieve file from S3 using FileKey/FilePath.
5. Stream file to client.

---

Data Models

public class FileMetadata
{
    public Guid UserId { get; set; }
    public string FileId { get; set; }
    public string FileKey { get; set; }
    public string OriginalName { get; set; }
    public string MimeType { get; set; }
    public long FileSize { get; set; }
    public string FilePath { get; set; }
    public bool IsDeleted { get; set; }
    public FileType FileType { get; set; }
    public bool IsPublic { get; set; }
}

public enum FileType
{
    Profile,
    Inventory,
    Parts,
    ServiceRequest,
    Legal,
    Other
}

public record FileUploadRequest
{
    [Required]
    public List<IFormFile> Files { get; init; } 

    [Required]
    public FileType FileType { get; init; }
}

public record FileUploadResponse
{
    public List<UploadedFileInfo> UploadedFiles { get; init; }
    public int TotalUploaded { get; init; }
}

public record UploadedFileInfo
{
    public string FileId { get; init; }
    public string OriginalName { get; init; }
    public string MimeType { get; init; }
    public long FileSize { get; init; }
    public FileType FileType { get; init; }
    public bool IsPublic { get; init; }
    public string FilePath { get; init; }
}

Database Schema:

-- filemetadata table
CREATE TABLE filemetadata (
    userid UUID NOT NULL,
    fileid VARCHAR(255) PRIMARY KEY,
    filekey VARCHAR(255) NOT NULL UNIQUE,
    originalname VARCHAR(255) NOT NULL,
    mimetype VARCHAR(100) NOT NULL,
    filesize BIGINT NOT NULL,
    filepath VARCHAR(500) NOT NULL,
    isdeleted BOOLEAN DEFAULT FALSE,
    filetype VARCHAR(50) NOT NULL,
    ispublic BOOLEAN NOT NULL
);

CREATE INDEX idx_filemetadata_userid ON filemetadata(userid);
CREATE INDEX idx_filemetadata_filekey ON filemetadata(filekey);
CREATE INDEX idx_filemetadata_filetype ON filemetadata(filetype);

---

API Interfaces

Endpoint	Method	Payload	Response	Status Codes
/api/v1/files/upload	POST	files, fileType	FileUploadResponse	201, 400, 415, 500
/api/v1/files/delete	DELETE	fileIds (array in body)	CommonResponse	200, 403, 404, 500
/api/v1/files/download	GET	fileId or url	File (attachment)	200, 401, 404, 500

---

API Request/Response Examples

Upload File (Single or Multiple)

Request (Multiple Files):

POST /api/v1/files/upload
Content-Type: multipart/form-data
Authorization: Bearer <token>

files: <binary[]>
fileType: "Inventory"

Response (201):

{
  "uploadedFiles": [
    {
      "fileId": "3f8a1c2e-7b4d-4e9a-9c6f-2d1b8a5e7f90.jpg",
      "originalName": "car_front.jpg",
      "mimeType": "image/jpeg",
      "fileSize": 204800,
      "fileType": "Inventory",
      "isPublic": true,
      "filePath": "Image/Inventory/key_a3f4b2c15d6e4f7a.jpg"
    },
    {
      "fileId": "file_b4e5c3d26f7e5g8h",
      "originalName": "car_side.jpg",
      "mimeType": "image/jpeg",
      "fileSize": 156000,
      "fileType": "Inventory",
      "isPublic": true,
      "filePath": "Image/Inventory/key_b4e5c3d26f7e5g8h.jpg"
    }
  ],
  "totalUploaded": 2
}

---

Delete files from storage (Batch)

Request:

DELETE /api/v1/files/delete
Content-Type: application/json
Authorization: Bearer <token>

{
   "fileIds": ["file_a3f4b2c15d6e4f7a", "file_b4e5c3d26f7e5g8h"]
}

Response (200):

{
  "message": "Files successfully marked as deleted"
}

---

Download File FileId or URL

Request (Internal File):

GET /api/v1/files/download?fileId=file_a3f4b2c15d6e4f7a
Authorization: Bearer <token>

Request (External File):

GET /api/v1/files/download?url=https://example.com/some-external-image.jpg
Authorization: Bearer <token>

Response:

• Status 200: Returns the file binary sequence as an attachment.
• Status 401: If IsPublic is false and the requester is unauthenticated (Applies to internal FileIds).

---

Testing & Quality Assurance

Unit Tests

• Assert correct routing (Image/Inventory/, Document/Profile/, etc.) based on Mimetype.
• S3 Key generation correctness.
• IsPublic=true images fetch correctly for anonymous requests.
• IsPublic=false images reject requests from unauthenticated users with 401.

Integration Tests

1. Upload Flow

   • Send .xlsx -> verify /Excel/{FileType}/ classification.
   • Send .png -> verify /Image/{FileType}/ classification.
   • Post to DB works and returns matching structure.

2. Authorization Enforcement Flow

   • Request existing file without token -> 200 (if IsPublic=true) or 401 (if IsPublic=false).
   • Sign in as standard user -> Request IsPublic=false file -> Expect 200.

3. Error Handling

   • Invalid file sizes return 400.

Acceptance Criteria

• ✅ Upload API routes correctly into AWS S3 paths based on dynamic folder types.
• ✅ Table schemas rigorously track IDs, keys, and boolean access bounds.
• ✅ Enums correctly modeled and bounded.
• ✅ Authentication engine completely halts unauthenticated reads of private data.
• ✅ Mirrors reference architectural intent perfectly.

---

Deployment Considerations

Infrastructure

• Set up AWS IAM policies to let backend assume role mapping to mbdn-files (or new) bucket.
• Ensure S3 bucket is NOT public; all reads must funnel through backend endpoint to ensure Visibility enforcement.

Rollout Steps

1. Push database migration (filemetadata).
2. Deploy backend adjustments (FileController, S3Service, Visibility Handlers).
3. Validate S3 configurations safely store new items via staging environment.
4. Scale rollout into production.

---

Risks & Mitigations

Risk	Impact	Mitigation
Accidental S3 Bucket Public exposure	Critical	Block all public access via AWS console policies. Require proxy fetches.
Inadequate Authentication Logic	High	Ensure Authorization Guard unit tests exhaust all edge cases.
Latency on S3 Downloads	Medium	Stream bytes directly from AWS instead of loading fully into container memory.
Orphaned S3 Data	Medium	Soft-deletes should eventually trigger S3 lifecycle cleanups or batch hard-delete scripts.

---

Future Enhancements

• Implementation of background cron jobs parsing IsDeleted = true over 30 days old for permanent S3 removal.