Custom Software Systems, Inc.
Azure Databricks Engineer
3 days ago
Leesburg
Job Description Custom Software Systems, Inc. (CSS) is seeking an Azure Databricks Engineer to join our team. The Azure Databricks Engineer will serve as a critical technical resource within the Financial Regulatory Agency's Data Modernization Section, supporting the Cloud Data Management and Analytics Platform. This platform is the enterprise foundation for the agency's data management, advanced analytics, and AI capabilities. Working within the Chief Data Officer Staff organization, this role is essential to the agency’s mission of maintaining stability and confidence in the nation's financial system through data-driven modernization. The engineer will lead the vision, design, development, and transformation of the agency's data strategy, helping to transition from siloed data management to one where enterprise data becomes a strategic resource that is securely shared and used for critical regulatory functions. Responsibilities 1. Enterprise Data Platform Development & Operations • Design and build scalable ETL/ELT pipelines using Azure Databricks, Delta Lake, and Apache Spark to process over 30 terabytes of structured and unstructured financial examination data, supporting over twenty-five active business initiatives., • Architect and continuously mature the lakehouse solution, implementing advanced data modeling, efficient partitioning, and Delta Lake optimizations (e.g., designing medallion architectures with bronze, silver, and gold layers)., • Configure, manage, and optimize Databricks workspaces, clusters, and compute resources, while implementing comprehensive role-based access controls, security policies, and cost optimization strategies., • Develop reusable architecture patterns and cloud technology reference models to accelerate adoption and standardization across the enterprise. 2. Legacy System Modernization and NLP Analytics • Lead the technical migration and modernization of a legacy on-premises Natural Language Processing (NLP) solution to Azure-native tools within the cloud platform., • Develop and optimize pipelines that extract intelligence from unstructured and semi-structured examination data, leveraging Azure AI Services and Azure Cognitive Services for advanced NLP capabilities (e.g., entity identification, relationship mapping)., • Design solution patterns that can be replicated for similar ML/NLP document extraction needs across the agency. 3. Real-Time and Batch Processing Architecture • Develop comprehensive real-time data processing solutions using Structured Streaming to support time-sensitive financial monitoring and regulatory examination workflows, ensuring low-latency data availability and exactly-once processing semantics., • Implement high-performance batch processing jobs for data transformation, aggregation, and integration, designing and maintaining hybrid processing architectures (lambda or kappa) that seamlessly handle both streaming and batch workloads. 4. Machine Learning Operations and Data Science Support • Collaborate with data scientists to deploy ML models into production environments, implementing and managing MLflow for comprehensive model lifecycle management (versioning, experiment tracking, model registry)., • Create automated ML pipelines for model training, validation, and inference that integrate with cloud platform services, supporting operational enterprise AI capabilities. 5. Infrastructure as Code and DevOps Practices • Develop and maintain Terraform Infrastructure as Code (IaC) scripts for cloud platform operational builds, ensuring reproducibility and consistency., • Implement continuous integration and deployment (CI/CD) practices using the agency's Enterprise GitHub, and follow the agency's Change Control Board release process. 6. Advanced Data Security and Governance Implementation • Implement advanced data security capabilities, including enterprise data labeling, fine-grained access control (sub-document/chunk level), and data tokenization and masking techniques. · Must be clearable. Citizenship · US Citizenship Required Qualifications • 5+ years of progressive experience as a Data Engineer., • Expert-level proficiency in Apache Spark and Delta Lake, with deep understanding of distributed data processing architectures, performance optimization techniques (e.g., Spark internals, partitioning, broadcast joins), and Delta Lake's ACID transaction capabilities., • Advanced capabilities in Python, SQL, and Scala for developing robust data engineering solutions., • Proven experience architecting and optimizing large-scale data pipelines handling terabytes of data., • Comprehensive, hands-on experience with the integrated Cloud Data Management and Analytics Platform suite:, • Azure Databricks (Primary data engineering platform), • Azure Data Factory (Orchestration of complex workflows), • Azure Data Lake Storage (Enterprise data repository), • Azure Synapse Analytics (Integrated analytics capabilities), • Proficiency with supporting services: Azure Machine Learning, Azure AI Services (including Azure OpenAI), Cosmos DB, App Services, Functions, and API Management., • Proficiency with Terraform for Infrastructure as Code (IaC)., • Exceptional analytical and problem-solving abilities with a proven track record of resolving complex data engineering challenges., • Excellent written and verbal communication skills, with the ability to explain complex technical concepts to diverse audiences (business users, executives, technical teams)., • Deep experience with Agile and Scrum methodologies and demonstrated flexibility to support multiple simultaneous initiatives., • Commitment to maintaining current knowledge in rapidly evolving Azure and Databricks technologies., • Microsoft Certified Azure Data Engineer Associate (Preferred)., • BA/BS/MS in Computer Science, Engineering, Data Science, or equivalent professional experience. Why Join Us? This role offers the opportunity to be instrumental in modernizing the agency's data capabilities, directly enabling advanced analytics and AI solutions that support critical financial examination and regulatory functions. You will be shaping enterprise solution patterns that create lasting impact and will be involved in cutting-edge initiatives, including enterprise AI model management and multi-cloud data fabric implementations. Your work will directly contribute to the agency’s mission to maintain stability and confidence in the nation's financial system. Compensation & Benefits[1][2] · Wage Range: Negotiable · General Benefits: Custom Software Systems, Inc. offers our employees a competitive benefits package that may include: ü Health insurance plans ü Health Savings Account (HSA) ü Dental ü Vision ü Long-term disability ü Short-term disability ü Basic term life insurance ü Supplemental term life insurance for employees, spouses, and dependents ü Simple IRA ü Parking/Commuting expense reimbursement ü Training/Education [1] Compensation range must be coordinated with and approved by the CSS Chief Operating Officer (COO). [2] Compensation & Benefits information is required for all Maryland Employers effective October 1, 2024. Company DescriptionCompany Background: Headquartered in Leesburg, Virginia, Custom Software Systems, Inc. (CSS) is a Woman-Owned (WOSB) and HUBZone certified small business. Built on a foundation of trusted client partnerships, CSS has fostered a stakeholder-centric yet disciplined approach to IT solutions development. This ensures our ability to consistently meet or exceed our customers' expectations. Benefits: CSS is a very employee oriented company knowing that well trained, professional associates are what make our company great. We offer a competitive benefits package that includes: paid holidays and paid time off; medical insurance that includes vision; dental insurance; company paid long and short-term disability and life insurance; a Simple IRA plan (similar to 401k); parking and commuter reimbursement. We also work with our employees on training and professional certification plans that benefit the employee, the client and CSS - a win-win-win strategy. Equal Opportunity Employer: CSS provides equal employment opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, gender, sexual orientation, gender identity or expression, national origin, age, disability, genetic information, marital status, amnesty, or status as a covered veteran in accordance with applicable federal, state and local laws. CSS complies with applicable state and local laws governing non-discrimination in employment in every location in which the company has facilities. This policy applies to all terms and conditions of employment, including, but not limited to, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation, and training. CSS expressly prohibits any form of unlawful employee harassment based on race, color, religion, gender, sexual orientation, gender identity or expression, national origin, age, genetic information, disability, or veteran status. Improper interference with the ability of CSS employees to perform their expected job duties is absolutely not tolerated.Company Background:\r\nHeadquartered in Leesburg, Virginia, Custom Software Systems, Inc. (CSS) is a Woman-Owned (WOSB) and HUBZone certified small business. Built on a foundation of trusted client partnerships, CSS has fostered a stakeholder-centric yet disciplined approach to IT solutions development. This ensures our ability to consistently meet or exceed our customers' expectations. \r\n\r\nBenefits:\r\nCSS is a very employee oriented company knowing that well trained, professional associates are what make our company great. We offer a competitive benefits package that includes: paid holidays and paid time off; medical insurance that includes vision; dental insurance; company paid long and short-term disability and life insurance; a Simple IRA plan (similar to 401k); parking and commuter reimbursement. We also work with our employees on training and professional certification plans that benefit the employee, the client and CSS - a win-win-win strategy.\r\n\r\nEqual Opportunity Employer:\r\nCSS provides equal employment opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, gender, sexual orientation, gender identity or expression, national origin, age, disability, genetic information, marital status, amnesty, or status as a covered veteran in accordance with applicable federal, state and local laws. CSS complies with applicable state and local laws governing non-discrimination in employment in every location in which the company has facilities. This policy applies to all terms and conditions of employment, including, but not limited to, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation, and training.\r\n\r\nCSS expressly prohibits any form of unlawful employee harassment based on race, color, religion, gender, sexual orientation, gender identity or expression, national origin, age, genetic information, disability, or veteran status. Improper interference with the ability of CSS employees to perform their expected job duties is absolutely not tolerated.