Modern Data Architecture Enables Near Real-Time Mortgage Insights
Challenge
The client’s mortgage data was housed in Encompass, a hosted cloud application. While the client was able to access some information through API calls and ingest it into on-premises SQL Server databases, the existing process created several business and technical challenges.
Many of the required columns were not available through the API, limiting the completeness of the data available for reporting and analysis. The client also regularly reached API usage limits, resulting in denied requests and disrupted data access. In addition, Power BI reports and dashboards built from the on-premises data were slow, limiting the user experience and reducing confidence in reporting performance.
The client also lacked the ability to view earlier versions of data, making time-travel analysis unavailable. As new fields were added, they had to be manually coded because the existing environment did not support schema drift. This created additional maintenance effort and slowed the organization’s ability to adapt to new reporting and analytics needs.
The client needed a modernized data solution that would make all mortgage data available, support near real-time updates, improve reporting performance, and reduce manual intervention.
Solution
The client engaged INSPYR Solutions to design a modern data architecture capable of making mortgage data more complete, accessible, and actionable. After evaluating options based on sizing, complexity, scalability, and the need for near real-time updates, INSPYR Solutions developed an event-driven solution using Azure and Microsoft Fabric.
Key elements of the solution included:
- Initial Full Data Load: INSPYR Solutions designed a process to fetch all available loan data using one request per loan, retrieving a single JSON object with complete loan information.
- Parallelized Ingestion: Azure Data Factory was used to ingest data and load loans in parallel, accelerating the initial load process and improving overall efficiency.
- Near Real-Time Event Streaming: An event-driven process was created to stream loan updates in near real time, typically in under one second.
- Bronze and Silver Data Layers: Data was mirrored to Microsoft Fabric as the Bronze layer, while a Silver layer was created with business-friendly names, calculations, and transformations applied.
- Schema Drift Support: New columns were automatically appended to the Bronze layer and made available for Silver, reducing manual coding and improving adaptability as the client’s data evolved.
- Automated Data Correction: INSPYR Solutions built detection and correction logic so that if a vendor update was not received within the expected timeframe, the affected loan would trigger a full reload.
- Historical Data Change Tracking: Streaming updates were stored in Microsoft Fabric to support a history of changes and provide improved visibility into how loan data changed over time.
Outcome
The new data architecture transformed the client’s mortgage data environment from a slower, API-limited reporting model into a scalable, near real-time data foundation capable of supporting improved reporting, analytics, and operational visibility.
Benefits realized through the implementation included:
- Accelerated access to loan updates by processing changes from the Encompass application into the Bronze and Silver databases in approximately 3 to 15 seconds, depending on the number of fields updated.
- Improved reporting foundation by automatically mirroring Bronze and Silver tables to the Microsoft Fabric Lakehouse for stronger analytics and historical data access.
- Increased data accuracy through automated reload logic that triggered a full refresh for any loan update not received within the specified timeframe.
- Expanded access to complete mortgage data by reducing limitations caused by unavailable API columns.
- Reduced manual development effort by enabling new columns to be automatically appended to the Bronze layer and made available for Silver.
- Enhanced data visibility by storing streaming updates in Microsoft Fabric to support a history of loan data changes over time.
- Strengthened scalability and reliability through a modern, event-driven architecture designed to support near real-time data movement and future reporting needs.
By establishing a modern Azure and Microsoft Fabric data architecture, the client gained a stronger foundation for accurate reporting, faster access to mortgage data, and improved confidence in the completeness and reliability of its analytics environment.
Client Profile
The client has been providing mortgage services to homebuyers for more than 30 years. With over 150 branches across the United States, the organization is focused on helping buyers make informed decisions with guidance and support from experienced, knowledgeable loan officers.
Technologies Supported
Azure Service Bus, Azure Data Factory, Azure Event Hub, Azure SQL Database, Azure Functions, Microsoft Fabric Lakehouse
View Similar Tags
Looking for cutting-edge technology solutions?
"*" indicates required fields