● Lead an onshore and offshore data engineering team, specializing in providing tailored support for Microsoft Azure Databricks clients

● Resolve client issues in Azure Databricks, ensuring timely solutions that improve platform reliability and performance

● Collaborate with Microsoft stakeholders to gather and refine requirements, translating business needs into actionable data engineering objectives

● Collaborating with the Microsoft Data Governance team to translate business requirements into actionable features for the data infrastructure

● Leading the architectural design and development of a Delta Lake solution within Apache Spark Pools ensuring geo-replication across Azure Regions to meet GDPR - Data Residency policies

● Setting up Dev, QA and Production environments with Azure DevOps to implement release pipelines

● Developing ML pipelines in Azure ML Studio to create predictive analysis for the telemetry logs 

Azure and AWS data cloud services for batch, streaming architectures and machine learning

Data Engineer Consultant for Retail Companies

Specializing in comprehensive data and analytics solutions, including:

• Data Engineering: Designing and developing Kappa Architecture for real-time data processing and analytics

• Data Testing: Creating and executing test cases for data quality using Spark and MPP (Databricks, Synapse) tools

• Data-Driven Architecture: Architecting scalable solutions with Apache Kafka, serverless functions, and Event Hubs for robust, event-driven data systems

● Development in HDFS - Hadoop ecosystem to build a data architecture and development on data integration scripts in python (SSH, Web Scraping, REST APIs, ODBC connections to unify data sources

● Data migration plan into AWS - S3, lambda functions, EC2 and Redshift for data warehousing, and also move existing Azure infra into AWS

● ML models to predict trends in suppliers and materials used in ericsson to build towers, sentiment analysis to categorize sellers, and build a conversational bot to query data about financial reports using RASA Framework and NLP techniques such as LSTM

● The development of ML was done on pandas dataframes and the ELT in Pyspark

● Backend development with Django to create CRUD endpoints for SQL Server DB

● Data Integration with Python Scripts to feed a Kimball Data Mart architecture

● Development of ML notebooks to clean, categorize medical records using LSTM and Language Models

● Forecast evolution of medical conditions in the scope of medical insurance policy, cost of diseases related medical problems at company level to provide insights to the sales personal and provide tailored solutions in the insurance broker representatives in new coverage plans

● Involved with medical personal to create catalogs for diseases and treatments following the international Classification of Diseases - ICD10

● The ML models were performed on pandas dataframes or Numpy arrays depending of the statistic to provide in the Dashboards

● Data Migration into a new OLTP systems in MySQL

● Web Development in frontend (Javascript- Angular, VueJs) and backend (PHP-Laravel)

● Maintenance old web app built in Java in the Tomcat server and optimization of the infrastructure using linux- RedHat distro,