Author(s): Sudheer Singamsetty

Abstract: Data engineering, as the foundation of modern data-driven systems, often grapples with repetitive workflows, data quality issues, and real-time scalability challenges. This paper proposes a novel self-learning artificial intelligence (AI) framework aimed at optimizing data engineering pipelines through dynamic adaptation and task automation. By integrating reinforcement learning and meta-learning into traditional data workflows, our model enables intelligent scheduling, anomaly detection, and schema evolution management without human intervention. Experimental evaluations on open-source benchmark datasets including TPC-H, MovieLens, and Kaggle IoT logs show up to 47% improvement in pipeline execution time, 35% reduction in data cleaning overhead, and 28% enhancement in anomaly correction accuracy compared to baseline systems. This work demonstrates the potential of self-learning AI to revolutionize data engineering by significantly increasing adaptability, robustness, and throughput in big data environments.

DOI: 10.33545/27076571.2025.v6.i1c.154

Pages: 195-199 | Views: 641 | Downloads: 290

Download Full Article: Click Here