Security Observability Engineer

Job Overview

Key Responsibilities

• End-to-End SIEM Pipeline Management & Optimization
• Lead the migration of log sources from Splunk ingestion to Cribl Stream/Edge
• pipelines, ensuring load-balanced, fault-tolerant delivery and processing of security and IT logs.
• Architect and manage scalable, resilient pipelines—including design,
• implementation, and operation of load balancing solutions (e.g., Cribl worker groups, external load balancers, or syslog load distribution) for high ingest volumes.
• Analyze, tune and securely onboard all log sources (firewalls, EDR, cloud,
• authentication, proxies, etc.)—covering parsing, filtering, and data reduction techniques to minimize Splunk ingest/storage costs without sacrificing security coverage.
• Develop and maintain Cribl and Splunk configurations, including advanced
• transformations, field normalization, masking, and enrichment for security analytics.
• Ensure optimal distribution of logging workload across Cribl worker nodes and
• Splunk indexers to prevent bottlenecks, data loss, or single points of failure. • Security Event Visibility and SOC Enablement
• Collaborate with SOC, IR, and threat detection teams to ensure all security logs
• reach the SIEM eRiciently and reliably, and logs are tuned for actionable detection. • Actively monitor, test, and remediate pipeline balancing and ingestion health to
• maximize uptime and forensic visibility.
• Respond to and resolve SIEM and pipeline issues that impact security, detection, or
• compliance visibility.
• Governance, Compliance & Documentation
• Apply and document security policies for log routing, load balancing, event
• retention, and integrity—ensuring pipeline architecture meets audit, legal, and privacy requirements.
• Track and report ingest reduction, Splunk cost savings, pipeline health, and event
• loss/drop rates across the load-balanced infrastructure.
• Maintain clear, up-to-date documentation of log flows, pipeline topology, load
• balancing strategies, and operational procedures.
• Required Skills & Quali fi cations
• Extensive hands-on experience with Splunk SIEM engineering (indexers, search
• heads, clustering, forwarders, CIM, performance/load optimization).
• 2+ years with Cribl Stream/Edge, including deployment and tuning of distributed,
• load-balanced pipelines.
• Deep understanding of machine data transport (syslog, HEC, TCP, UDP), log
• balancing strategies (syslog balancers, DNS round-robin, Cribl worker groups, etc.), and high-availability logging environments.
• Proven expertise onboarding, parsing, and tuning security log sources (firewalls,
• cloud, EDR/XDR, IAM, authentication, and networking) for best possible coverage and SOC/IR support.
• Advanced Splunk SPL, data model, event parsing, and alert tuning skills; practical
• SIEM optimization experience.
• Scripting/automation ability (Python, shell/CLI, or similar) for pipeline management
• and validation.
• Strong troubleshooting, monitoring, and operational dashboard skills for both
• pipeline and SIEM health.
• Preferred Quali fi cations
• Hands-on experience designing and operating clustered/HA Cribl and Splunk
• deployments (worker groups, clustered indexers, resilient data forwarders, etc.). • Splunk ES or Cribl certifications.
• Key Success Metrics
• No loss of security data or alert coverage during/after pipeline migration and
• optimization.
• Documented, measurable reductions in Splunk ingest volume and
• operational/storage cost.
• Consistently balanced log throughput and minimal risk of bottlenecks or
• overloads—pipeline health and reliability metrics maintained above SLA. • Well-documented, adaptable pipeline and load balancing architecture.
• The estimated salary range for this position is 90k-120k
• Starr is an equal opportunity employer, which means we'll consider all suitably qualified applicants regardless of gender identity or expression, ethnic origin, nationality, religion or beliefs, age, sexual orientation, disability status or any other protected characteristic. We recruit and develop our people based on merit and we're committed to creating an inclusive environment for all employees. We offer first class training and development opportunities to all employees. Our aim is to grow our own talent and bring out the best in people.
• Splunk indexers to prevent bottlenecks, data loss, or single points of failure.
• Security Event Visibility and SOC Enablement
• reach the SIEM eRiciently and reliably, and logs are tuned for actionable detection.
• Actively monitor, test, and remediate pipeline balancing and ingestion health to
• deployments (worker groups, clustered indexers, resilient data forwarders, etc.).
• Splunk ES or Cribl certifications.
• overloads—pipeline health and reliability metrics maintained above SLA.
• Well-documented, adaptable pipeline and load balancing architecture.