Author(s): Srikanth Peddisetti

Abstract: In the era of increasingly complex, multi-source, and dynamic data systems, traditional data engineering pipelines often struggle with adaptability, explainability, and effective reasoning over data transformations. To overcome these challenges, we propose the Neuro-Symbolic Data Engineering (NSDE) framework, which synergistically combines symbolic reasoning with neural networks. This hybrid approach leverages rule-based knowledge graphs, ontologies, and logic programming alongside advanced deep learning architectures to enable intelligent, interpretable, and high-performance data processing workflows. NSDE supports automated schema matching, explainable data imputation, semantic enrichment, and robust data fusion. We validate the NSDE framework through two real-world case studies in healthcare and finance. In the healthcare domain, using MIMIC-III patient records integrated with radiology reports, NSDE demonstrated a 12% improvement in accuracy over baseline deep learning pipelines and achieved a 92% success rate in missing value recovery-substantially outperforming both DL-only (78%) and symbolic-only (65%) methods. Expert evaluators rated NSDE’s explanation quality highly (4.5 out of 5), confirming its interpretability and reliability. In the financial domain, NSDE was applied to anonymized European bank transaction data for schema alignment between legacy and modern systems. It achieved a schema matching accuracy of 94%, surpassing DL-based (83%) and symbolic-only (74%) approaches, with transformation logic deemed highly explainable and human-verifiable. Ablation studies highlighted the critical roles of both symbolic and neural components in balancing interpretability and adaptability, while scalability analysis confirmed linear growth with dataset size and efficient parallelization. Overall, NSDE presents a scalable, adaptive, and explainable framework for next-generation AI-driven data engineering pipelines.

DOI: https://www.doi.org/10.33545/27076571.2022.v3.i1a.158

Pages: 55-61 | Views: 1197 | Downloads: 790

Download Full Article: Click Here