Jay Palaniappan
Sr. Solutions Architect at Databricks
Jay has 25 years of IT expertise, with a decade focused on designing and managing Big Data solutions in the Cloud. As a Senior Specialist Solutions Architect at AWS from August 2020 to April 2022, he guided leading entertainment companies in optimizing their big data ecosystems. Since May 2022, Jay has been a Senior Solutions Architect at Databricks, where he supports Digital Native businesses in running Data Engineering, Machine Learning, and AI workloads. In addition to his professional work, Jay is a technical blogger and speaker, sharing insights and innovations in Data Engineering to help others excel in the field.
Sessions
Evolution of Apache Spark Structured Streaming - From Then to Now
Evolution of Apache Spark Structured Streaming, from its early micro-batching model to its modern, unified API for batch and streaming workloads. Key advancements include event-time processing, fault tolerance, and performance optimizations, enabling scalable, real-time data analytics with improved latency and throughput.