Senior Databricks Engineer

Azure Databricks, Python, PySpark, SQL

This individual will collaborate with developers, data owners, governance leads, and business analysts across IT and aligned application stakeholders.

• Critical Requirements
• Financial services or similarly regulated environment experience strongly preferred

• Design, develop, test, and support cloud data solutions focusing on data ingestion, data quality, tuning, and performance of upstream cybersecurity data sources using Databricks
• Provide indirect leadership and mentorship to junior team members; lead data feed development efforts and design initiatives
• Participate in development meetings to align priorities and objectives, assign tasks, and share experiences across applications under development
• Consult with technology and development teams to coordinate integration of applications within the broader software ecosystem
• Capture and document metadata for Key Data Elements (KDEs) to ensure accuracy and completeness for Data Quality (DQ) rules and daily dataset processing
• Work with the data architecture team to align KDEs to logical data models, develop physical data structures, and document physical data names, definitions, and types
• Partner with data owners and stakeholders to create technical requirements and DQ rules around data elements needed in the data warehouse
• Partner with Business Management and Data Owners to identify critical metrics and data fields required for metric dashboards
• Establish views to encapsulate data for downstream consumption and ensure alignment with DQ rules established on metadata
• Utilize IBM TWS production job scheduling and adhere to standards around daily scheduling and batch monitoring of production jobs
• Identify and resolve data quality issues including inaccuracies and incomplete information; implement improved procedures and processes

Qualifications & Skills

Azure & Cloud Platform

Azure Databricks & Unity Catalog

Azure Data Factory

Azure Data Lake Storage

Azure Functions

Azure Event Grid

Azure Log Analytics & Monitor

Development & Data Engineering

Python / Java

SQL Server, Oracle / PL-SQL, T-SQL

ETL / ELT

REST API & Microservices

CI/CD - Git, Jenkins, Azure DevOps

Collibra DQIM/DQE, IBM Infosphere DA

• Strong knowledge of enterprise Information Security data domains: Phishing, Identity Management, Privileged Access, Cloud Security, Incident Response, Vulnerability Management, and Threat Detection
• Data knowledge of PaaS/SaaS products including ServiceNow, CrowdStrike, MS Purview, Proofpoint, WIZ.IO, JIRA, SharePoint, and Azure Active Directory
• Understanding of information security frameworks (NIST, CIS, CRI Profile) and regulatory compliance (NYSDFS, GDPR, CCPA)
• Nice to Have
• Knowledge of Microsoft Sentinel for SIEM
• Knowledge of IBM Tivoli Workload Scheduler
• Exposure to Power BI for data visualization and reporting

#LI-EW1