Databricks Architect / Admin (W2 only C2H)
18 hours ago
Hartford
Position Title: Databricks Architect / Admin (W2 only C2H) Employment Type: Location: Hartford, CT (Hybrid – 3 Days Onsite / 2 Days Remote) Schedule: Monday–Friday, 8:00 AM – 5:00 PM Duration: 6-month Contract-to-Hire Team Size: 10–12 members (Onshore + Offshore) Work Environment: Agile, daily stand-up at 9:00 AM, ServiceNow workflow management Interview Process: 1–2 rounds (possible third round), panel with hiring manager and team members; technical + behavioral focus, no formal assessment. Travel: Minimal (commute only) A senior Databricks architect with deep platform administration experience who can function as a strategic advisor rather than a day-to-day operator. The ideal candidate combines strong Databricks architecture knowledge with AWS infrastructure expertise, cost optimization skills, Linux administration, and enterprise-scale data platform experience. Position Overview The Databricks Architect/Admin is a senior individual contributor responsible for architecting, governing, optimizing, and supporting the enterprise Databricks platform. This role is primarily consultative with selective hands-on responsibilities and serves as the subject matter expert for platform architecture, administration, governance, cost optimization, and enterprise integrations. The environment is already established; the selected candidate will provide strategic guidance, optimization recommendations, and ongoing support rather than building from scratch. This position is a backfill for a retiring resource and requires deep expertise to guide engineering teams and platform direction. The role requires a strong combination of: • Databricks architecture expertise, • Advanced administration knowledge, • Cloud platform experience (AWS), • Unix/Linux administration, • Cost management and optimization skills, • Data engineering and integration experience, • Strategic advisory capability Key Responsibilities Platform Architecture & Design • Architect and govern enterprise Databricks environments including:, • Workspace topology, • Unity Catalog structures, • Access control frameworks, • Define and enforce:, • Cluster standards, • Runtime versions, • Instance pool strategies, • Auto-scaling policies, • Design scalable and performant pipelines using:, • Delta Live Tables (DLT), • Databricks Workflows, • Structured Streaming, • Define architectural standards for:, • Delta Lake formats, • Partitioning, • Z-ordering, • OPTIMIZE/VACUUM strategies, • Lead integration architecture with upstream platforms:, • Fivetran, • Ab Initio Databricks Consulting & Advisory • Serve as the primary Databricks consultant across teams, • Provide guidance on platform best practices, • Support technical decision-making, • Act as the escalation point for Databricks platform issues, • Mentor teams on platform usage and optimization, • Recommend new Databricks capabilities and roadmap enhancements Cost Management & Optimization A major focus area during the first 3–6 months. Responsibilities include: • Own DBU consumption reporting and tracking, • Identify optimization opportunities across:, • Interactive clusters, • Jobs, • SQL Warehouses, • Detect inefficiencies such as:, • Excessive DBU consumption, • Looping jobs, • Misconfigured workloads, • Build cost attribution/chargeback models, • Support:, • Capacity planning, • Contract forecasting, • Vendor discussions, • Collaborate on monthly cost analyses Unix/Linux Infrastructure & Operations • Administer and troubleshoot Linux environments supporting Databricks, • Manage:, • Init scripts, • Cluster lifecycle processes, • Shell scripting, • Build Bash and Python automation solutions, • Support:, • Monitoring, • Log aggregation, • Maintenance workflows, • Manage:, • File systems, • Permissions, • Data movement operations, • Perform EC2/VM diagnostics with infrastructure teams Governance, Security & Compliance • Design governance frameworks in Unity Catalog, • Implement:, • Data lineage, • Tagging, • Access auditing, • Partner with cybersecurity teams, • Support:, • Secrets management, • Network isolation, • Encryption, • Maintain audit documentation and compliance artifacts Automation, AI & Machine Learning • Implement automation for:, • Cluster lifecycle management, • Scheduling, • Alerting, • Self-healing workflows, • Leverage:, • AutoML, • MLflow, • Model Serving, • Integrate:, • Databricks Assistant, • GitHub Copilot, • Support scalable feature engineering and ML deployment pipelines, • Evaluate emerging:, • AI capabilities, • Generative AI tools, • LLM-integrated workflows Required Qualifications • 7+ years in data engineering or platform engineering, • Minimum 4+ years of hands-on Databricks implementation experience, • Deep expertise in:, • Unity Catalog, • Delta Lake, • Databricks Workflows, • Delta Live Tables, • SQL Warehouses, • Strong Unix/Linux skills:, • Shell scripting, • Process management, • File systems, • Cron, • Python and PySpark expertise, • AWS or cloud platform experience:, • EC2, • Storage, • Networking, • IAM/security, • SQL expertise for reporting and analysis, • Experience designing scalable and cost-efficient enterprise platforms, • Experience with automation frameworks, • Familiarity with AI/ML ecosystem, • Oracle database experience, • Git and CI/CD knowledge, • ServiceNow and Jira exposure, • Strong communication skills Preferred Qualifications • MLflow deployment and model registry experience, • Generative AI tools:, • LangChain, • LlamaIndex, • RAG workflows, • Workflow orchestration:, • Apache Airflow, • Databricks Workflows, • Fivetran integration expertise, • Ab Initio administration or development, • Data governance platforms:, • Collibra, • Alation, • Experience in regulated industries:, • Insurance, • Financial services, • Terraform/Ansible experience, • Disaster recovery and resiliency planning Core Technical Skills Databricks Ecosystem • Databricks, • Unity Catalog, • Delta Lake, • Delta Live Tables, • Databricks Workflows, • SQL Warehouses, • MLflow, • AutoML Cloud & Infrastructure • AWS, • EC2, • S3, • Networking, • Private Links, • Firewalls Programming & Automation • Python, • PySpark, • Bash/Shell scripting, • SQL, • Git Integration Technologies • Fivetran, • Ab Initio, • Oracle Platforms & Operations • ServiceNow, • Jira, • Linux/Unix First 3–6 Month Priorities / Challenges Primary focus areas: • Cost optimization and Databricks usage tracking, • Stabilization of Databricks onboarding application (currently MVP), • Identification of billing discrepancies and inefficiencies, • Strategic capacity planning and vendor support, • Providing platform guidance and consultation to teams Hiring Notes • Hartford candidates strongly preferred, • Exceptional candidates from alternate locations may be considered, • Budget is below current market expectations, • Hiring manager noted limited flexibility but open to strong profiles Applicant Notices & Disclaimers • For information on benefits, equal opportunity employment, and location-specific applicant notices, click _ At SPECTRAFORCE, we are committed to maintaining a workplace that ensures fair compensation and wage transparency in adherence with all applicable state and local laws. This position's starting pay is: $45.00/daily.