Kafka Architect / Streaming Data Architect

Company

Location

About Us

Architecture & Design

• Design
• enterprise-scale Kafka architecture
• aligned with Bank Lakehouse (Medallion - Bronze/Silver/Gold).
• Define
• event-driven architecture patterns
• for real-time ingestion and processing.
• Establish
• topic strategy, partitioning, replication, retention policies
• .
• Architect
• multi-cluster Kafka setup
• (dev/test/prod, DR strategy).
• Ensure
• exact point-in-time recovery and fault tolerance
• .

Platform Implementation

• Lead setup and configuration of:
• Kafka brokers, Zookeeper / Kraft mode
• Kafka Connect framework
• Schema Registry
• Stream processing (Kafka Streams / Spark Streaming / Flink)
• Integrate Kafka with:

• Data Warehouse (Teradata)

• Denodo (for virtualization layer)

Real-Time Data Engineering

• Design pipelines for:
• **Event streaming ingestion**
• **Near real-time batch ingestion**
• Build CDC (Change Data Capture) pipelines using Kafka Connect (Debezium).
• Enable ingestion from
• source systems
• .
• Support
• incremental and historical loads integration
• .

Performance & Scalability

• Ensure:
• **High throughput and low latency**
• Support for
• concurrent consumers without degradation
• Optimize:
• Partition strategy
• Consumer group scaling
• Producer configurations (acks, batching, compression)

Resilience & Reliability

• Implement:
• Failover strategies
• Data replication and durability (\<99.9% uptime requirement)
• Handle:
• Hardware failures
• Data corruption scenarios
• Design
• retry, dead-letter queues (DLQ), error handling frameworks

Data Governance & Security

• Enforce:

• Role-Based Access Control (RBAC)

• Encryption (in transit & at rest)
• Data masking/tokenization support (aligned with classification)
• Integrate with:
• Metadata and lineage tools
• Governance frameworks (NDMO/NDI compliance)

Monitoring & Operations

• Implement monitoring using:
• Cloudera Manager
• Kafka metrics (JMX, Prometheus, Grafana)
• Define:
• Alerting thresholds
• SLA monitoring for ingestion pipelines
• Ensure observability for:
• Lag monitoring
• Throughput and error rates

Phase 1 Remediation Support

• Assess existing Kafka pipelines and components.
• Identify:
• Defective pipelines
• Missing ingestion flows
• Redesign/rebuild where required.
• Ensure alignment with
• target architecture and data models (FSLDM/FSAS)
• .

Reporting Enablement Support

• Ensure:
• Real-time data availability for downstream reporting
• Support:
• Query performance optimization
• Data consistency across Bronze/Silver/Gold
• Resolve issues impacting reporting (data mismatches, delays).

Documentation & Governance

• Deliver:
• High-level architecture design
• Low-level design (LLD) for Kafka pipelines
• Topic catalog and schema definitions
• Maintain:
• Data lineage documentation
• Source-to-target mappings
• Participate in:
• Architecture governance boards
• Change management processes

Ecosystem Tools

• Cloudera (CDP)

• Apache Flink (preferred)

• Informatica (integration awareness)
• Teradata
• Denodo

Data & Integration

• CDC tools (Debezium preferred)
• REST APIs, Microservices integration
• ETL/ELT Design patterns

DevOps & Infra

• Kubernetes / Docker (optional but preferred)
• CI/CD for pipeline deployment
• Linux environments

Banking Domain Experience (Mandatory)

• Experience in
• BFSI domain
• Understanding of:
• Regulatory reporting
• Data reconciliation (≥99% accuracy)
• AML, Payments, Trade finance systems

Soft Skills

• Strong stakeholder management (business + IT)
• Ability to work in
• multi-vendor environment
• Experience in
• large-scale transformation programs
• Strong problem-solving and RCA skills
• Thank you for your interest in applying for this position with TCS.
• We will review your application and will get back to you if we are considering your interest in this opportunity.

Privacy Note