Back to solutions
data engineering
scraping
analytics
CMBS
AWS

Data Forge

Data engineering platform with distributed scraping, full Data Engineering pipelines, and SRE practices on AWS (SQS, SNS, Step Functions). Real-world enterprise quality case for KBRA in the CMBS domain.

Data Forge

Objectives

Collect and process market data at scale with distributed AWS functions
Provide integrated data visualization interface for CMBS market advisors
Build reliable data pipeline with data validation and quality

Challenges

Collect market data from multiple heterogeneous sources reliably and at scale
Ensure data quality and validation in a high-volume information pipeline
Present complex market data in an accessible and useful way for advisors

Implemented Solutions

Solutions implemented to ensure scalability, security and efficiency.

5-stage data pipeline: scraping, management, fetch, preprocessing, and validation
AWS infrastructure with Terraform (S3, EventBridge, RDS, SQS, SNS, Step Functions) and SRE practices for enterprise reliability
Forge UI: React 19 dashboard with React Router 7, TailwindCSS, and SSR for data visualization

Data Forge — Market Data Engineering

Data Forge is a data engineering platform developed by DWCorp as outsourcing for KBRA, focused on distributed market data scraping and visualization for CMBS domain advisors.

Data Pipeline (5 Stages)

  1. deal_access_scraper — Market data collection
  2. deal_manager — Data management and organization
  3. file_fetcher — File retrieval
  4. preprocessor — Data preparation and cleaning
  5. file_validator — Validation and quality control

Infrastructure (Terraform)

  • AWS S3 for data storage
  • AWS EventBridge for event orchestration
  • AWS RDS for database
  • AWS SSO for authentication
  • Docker for containerization

Forge UI — Visualization Dashboard

  • React 19 with React Router 7 and SSR
  • TailwindCSS 4 for styling
  • Vite 6 for fast builds
  • Docker + Nginx for production deployment
  • ESLint + Prettier + Husky for code quality

Context

Developed as an outsourcing project for KBRA, Data Forge demonstrates DWCorp's capability in enterprise data engineering with high complexity and volume.

Data Forge deal documents

Data Forge loan details