📤 Export System Architecture
Background job-based data export pipeline
Executive Summary
The Export System provides asynchronous data export capabilities to prevent blocking API requests during large data retrieval. Exports are processed as background jobs with status polling and file download endpoints.
✅ Async Processing
Background jobs prevent timeouts
Background jobs prevent timeouts
✅ Multiple Formats
CSV, JSON implemented. Excel, PDF planned
CSV, JSON implemented. Excel, PDF planned
⚠️ CSV Only Now
Excel and PDF not implemented
Excel and PDF not implemented
🎯 Queue-Based
IHostedService for job processing
IHostedService for job processing
🔄 Export Job Lifecycle
Export Job Processing Flow
═══════════════════════════════════════════════════════════════════════════════
┌──────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Client │ │ ExportJobs │ │ ExportJob │ │ Blob │
│ (Angular)│ │ Controller │ │ Service │ │ Storage │
└────┬─────┘ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘
│ │ │ │
│ 1. Create Job │ │ │
│────────────────►│ │ │
│ │ │ │
│ │ 2. Queue Job │ │
│ │──────────────────►│ │
│ │ │ │
│ { jobId: 123 } │ │ │
│◄────────────────│ │ │
│ │ │ │
│ 3. Poll Status │ │ │
│────────────────►│ │ │
│ │ │ │
│ { status: │ │ │
│ "Processing" │ │ │
│◄───────────────│ │ │
│ ... │ │ │
│ │ │ │
│ 4. Background Processing │ │
│ │ │ │
│ │ │ 5. Query Data │
│ │ │──────────────────►│
│ │ │ (Tenant-scoped) │
│ │ │ │
│ │ │ 6. Build File │
│ │ │ (CSV/JSON) │
│ │ │ │
│ │ │ 7. Upload Blob │
│ │ │──────────────────►│
│ │ │ │
│ │ │ 8. Update Status │
│ │◄──────────────────│ │
│ │ │ │
│ 9. Complete │ │ │
│◄───────────────│ { status: "Completed", url: "..." } │
│ │ │ │
│ 10. Download │ │ │
│────────────────────────────────────────────────────────►│
│ │ │ │
│◄───────────────│ (file stream) │ │
│ │ │ │
Job Status States:
──────────────────
Pending → Processing → Completed
│ │ │
│ │ ▼
│ │ (Available for 24h)
│ │ │
└─────────┴──────► Failed
📊 API Inventory
| Controller | Route | Endpoints | Authorization |
|---|---|---|---|
| ExportJobsController | /api/export-jobs | 5 | SuperAdmin+Admin+Manager |
| DataExportController | /api/dataexport | 3 | SuperAdmin+Admin |
| AdminDashboardController | /api/admindashboard/export | 1 | SuperAdmin+Admin |
🔌 Export Endpoints
POST
/api/export-jobs
Create export job. Specify filters, date range, format, and data type.
SuperAdmin+Admin+Manager Existing
{
"dataType": "sessions|users|analytics",
"format": "csv|json|excel|pdf",
"dateRange": { "from": "2026-01-01", "to": "2026-01-31" },
"filters": { "userId": "...", "organizationId": "..." }
}
GET
/api/export-jobs/{id}
Get export job status and metadata. Poll this endpoint for progress updates.
Job Owner Existing
GET
/api/export-jobs
List export jobs for organization with filtering and pagination.
Tenant-scoped Existing
GET
/api/export-jobs/{id}/download
Download completed export file. Returns file stream with appropriate content type.
Job Owner Existing
📁 Export Format Support
| Format | Status | Implementation | Content-Type | File Extension |
|---|---|---|---|---|
| CSV | ✓ Implemented | Custom CSV builder (StringBuilder) | text/csv | .csv |
| JSON | ✓ Implemented | System.Text.Json | application/json | .json |
| Excel (XLSX) | ✗ Not Implemented | Planned: EPPlus or ClosedXML | application/vnd.openxml | .xlsx |
| ✗ Not Implemented | Planned: iTextSharp or QuestPDF | application/pdf |
⚙️ Export Job Queue Architecture
Background Job Processing Architecture
═══════════════════════════════════════════════════════════════════════════════
┌─────────────────────────────────────────────────────────────────────────────┐
│ Export Job Queue System │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ IExportJobService Interface │ │
│ │ │ │
│ │ Task CreateAsync(ExportRequest request, Guid orgId); │ │
│ │ Task GetByIdAsync(Guid jobId); │ │
│ │ Task ProcessQueueAsync(CancellationToken ct); │ │
│ │ Task GenerateDownloadUrlAsync(Guid jobId); │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ ExportJobBackgroundService │ │
│ │ (IHostedService) │ │
│ │ │ │
│ │ while (!cancellationToken.IsCancellationRequested) │ │
│ │ { │ │
│ │ // 1. Poll for pending jobs │ │
│ │ var pendingJobs = await _repo.GetPendingAsync(batchSize: 5); │ │
│ │ │ │
│ │ foreach (var job in pendingJobs) │ │
│ │ { │ │
│ │ // 2. Mark as processing │ │
│ │ await _repo.UpdateStatusAsync(job.Id, Processing); │ │
│ │ │ │
│ │ try │ │
│ │ { │ │
│ │ // 3. Execute export based on type │ │
│ │ var filePath = job.DataType switch │ │
│ │ { │ │
│ │ "sessions" => await ExportSessionsAsync(job), │ │
│ │ "users" => await ExportUsersAsync(job), │ │
│ │ "analytics" => await ExportAnalyticsAsync(job), │ │
│ │ _ => throw new NotSupportedException() │ │
│ │ }; │ │
│ │ │ │
│ │ // 4. Upload to blob storage │ │
│ │ var blobUrl = await _blobStorage.UploadAsync(filePath);│ │
│ │ │ │
│ │ // 5. Mark completed │ │
│ │ await _repo.CompleteAsync(job.Id, blobUrl); │ │
│ │ } │ │
│ │ catch (Exception ex) │ │
│ │ { │ │
│ │ // 6. Mark failed with error │ │
│ │ await _repo.FailAsync(job.Id, ex.Message); │ │
│ │ } │ │
│ │ } │ │
│ │ │ │
│ │ // 7. Wait before next poll │ │
│ │ await Task.Delay(TimeSpan.FromSeconds(30), ct); │ │
│ │ } │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ⚠ Current Implementation Notes: │
│ • In-memory queue (not distributed) │
│ • Single worker instance (no horizontal scaling) │
│ • 30-second poll interval (configurable) │
│ • Files stored in Azure Blob Storage (24-hour SAS expiry) │
│ • No retry logic for failed jobs │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
⚠️ Scalability & Security Concerns
Single Worker
Current implementation runs on single instance. Cannot scale horizontally without distributed queue (Azure Service Bus, RabbitMQ).
No Dead Letter Queue
Failed jobs stay in Failed status. No automatic retry or alerting mechanism.
Memory Pressure
Large exports build entire file in memory before upload. Risk of OOM for multi-GB exports.
Tenant Isolation
Export jobs scoped to requesting organization. Users cannot access other tenants' export files.
✅ Recommended Enhancements
Distributed Queue
Replace in-memory queue with Azure Service Bus or RabbitMQ for horizontal scaling.
Recommended High
Streaming Export
Stream data directly to blob storage without loading entire dataset into memory.
Recommended High
Retry Policy
Implement exponential backoff retry for transient failures (DB timeouts, network issues).
Recommended Medium
Excel/PDF Implementation
Complete planned format implementations using EPPlus (Excel) and QuestPDF (PDF).
Recommended Medium