Case Study

From Fragmented Workflows to Unified Intelligence: How Trigent Streamlined Data Pipelines with Databricks

Information Technology (IT)
AI/ML

About the Client​

A leading financial services provider in the retail investment space, the client offers a mobile-first trading platform for individuals to access and trade in stocks, derivatives, mutual funds, and commodities. With a focus on democratizing financial markets, the company provides personalized investment recommendations, portfolio management tools, and advanced analytics powered by AI and machine learning, enabling retail investors to make more informed decisions with easily and confidently.

Business Challenge​

Despite its digital-first vision, the client faced operational inefficiencies due to fragmented workflows and legacy systems. Key challenges included:

  • Disjointed data pipelines that created bottlenecks and delayed data processing.
  • Multiple data sources, such as Redshift and S3, that lacked centralized control and versioning.
  • Dependency on legacy DAGs and Glue PySpark scripts, which slowed development and hampered scalability.
  • Limited automation and deployment flexibility, affecting agility in responding to market changes.

The client needed a modernized data architecture that could enable real-time processing, seamless integration, and better governance.

Trigent Solution

Trigent collaborated with the client to overhaul their data processing environment and migrate to a more scalable, AI-ready ecosystem using Databricks. Key solution components included:

    • Migration of legacy DAGs to Databricks, modernizing workflows and simplifying pipeline orchestration.
    • Re-implementation of workflows in Databricks notebooks using PySpark to improve performance and maintainability.
    • Ingestion of historical data from Amazon Redshift into the Databricks Landing (LO) layer.
    • Pipeline extension to move data from the LO layer to the Logical Integrated Unit (LIU) layer using pre-existing frameworks.
    • GitHub integration for source control and continuous deployment of Databricks workflows.
    • Conversion of existing Glue PySpark code into Databricks-native PySpark scripts to unify execution environments.
    • Creation of Databricks Jobs aligned with each DAG to automate task execution.
    • Establishment of external connections to data sources such as S3 and Redshift for seamless data integration.

Client Benefits

Trigent’s solution delivered tangible benefits across development velocity, data quality, and operational performance:

  • Streamlined workflows with end-to-end migration to Databricks, reducing pipeline complexity.
  • Improved code manageability and deployment automation via GitHub integration.
  • Faster data processing and lower latency, enabling near real-time analytics for trading insights.
  • Unified platform for data engineering and AI workflows, laying the foundation for ML-driven personalization.
  • Better governance and version control, ensuring auditability and compliance in a highly regulated environment.