Karthick Venkatesan

About

Lead Data Engineer with 10+ years of experience designing and implementing end-to-end data pipelines, scalable data platforms, and cloud-native solutions. Expertise in AWS cloud environments, healthcare data engineering/compliance, and AI Engineering, with strong skills in Python, SQL, PySpark, Apache Airflow, and CI/CD automation. Passionate about using data and generative/agentic AI to improve healthcare workflows and patient outcomes.

Experience

• Lead Data Engineer · Elevance Health05-01-2024

  • Led the development of Generative AI pipeline for L&D Benefits Document extraction using OpenAI and LangChain, integrating AI into core healthcare business processes • Led the development of Agentic AI solution for Automatic Error and Warning scrapper from Webserver logs and provide AI generated solution for the Error • Designed and implemented end-to-end data pipelines using AWS Glue, PySpark, and Apache Hudi, processing healthcare provider data • Migrated legacy Informatica workflows to modern PySpark and Hudi-based jobs, improving processing efficiency by 25% and reducing operational costs • Built and managed data orchestration workflows using Apache Airflow, automating complex data pipelines and reducing manual intervention

• Senior Data Engineer · Threesides02-01-2021 – 07-31-2023

  • Designed and built robust data ingestion pipeline using AWS Redshift and AWS Glue • Led database migration from PostgreSQL to DynamoDB using AWS DMS, achieving 20% cost reduction and significant read latency improvement • Optimized ETL jobs in AWS Glue, achieving 35% reduction in processing time and decreasing cloud compute costs • Implemented CI/CD pipelines using GitHub Actions for automated testing and deployment of data engineering workflows • Collaborated with cross-functional stakeholders to translate business requirements into robust technical data solutions

• Senior Software Engineer · Sigma Red AI Labs08-01-2020 – 02-28-2021

  • Developed real-time data pipeline in Python to migrate AI model outputs to Dashboard Application, enabling immediate business intelligence • Implemented cost-effective AWS Serverless Architecture (VPC, API Gateway, AppSync, Amplify) with improved CI/CD deployment pipeline • Accelerated feature deployment by 40% through DevOps workflow optimization and automation

• Data Engineer · Accenture04-01-2015 – 06-30-2018

  • Created high-volume data ingestion jobs using Talend to migrate data from legacy systems to Hadoop Data Lake, supporting CDC and incremental loads • Automated job scheduling processes using Selenium, reducing manual operational efforts by 30% • Maintained data infrastructure performance and proactively troubleshooted issues to ensure reliability and scalability

Education

• KLN College of EngineeringB.Tech, Information Technology

Similar talent on Pangea

Meghana G.Senior Data Engineer | Analytics Engineering & ELT Pipelines (dbt, Snowflake, AWS)Boston, MAAAkash R.Data Engineer | Data Analytics & Infrastructure | Healthcare, SaaS, and Life Sciences | Data Pipeline & Dashboard SpecialistMcKinney, TXSaad A.AI & Medicine Professional | Data Science & Analytics for Healthcare InnovationDallas, TXDiwakar V.Data Engineer (Azure + Databricks) | 4+ Years Building Scalable Data PlatformsGurugram, Haryana