Bengaluru, Karnataka, India
Summary
Manages and optimizes complex data pipelines across multiple financial portfolios using Spark and Scala to ensure data integrity and drive critical business insights for Target's finance team.
Highlights
Managed and optimized critical data pipelines for Target's finance team, encompassing Fixed Assets, Vendor Income, and Department Profit & Loss portfolios, ensuring data integrity and financial reporting accuracy.
Designed and developed scalable, high-performance data pipelines using Apache Spark and Hadoop, significantly enhancing processing capabilities for large datasets.
Optimized Spark job performance by tuning configurations (memory, shuffling, partitioning), leading to improved cluster resource utilization and efficiency.
Implemented robust Data Quality (DQ) checks and automated alert systems via Slack and email, enabling rapid discrepancy resolution.
Developed reusable Spark functions for common ETL operations, resulting in significant development time savings for future data projects.