DTCC / Financial Services / SRE Interview Notes

1. DTCC Overview

DTCC (Depository Trust & Clearing Corporation) is a critical financial market infrastructure organization responsible for clearing, settlement and transaction processing for financial institutions.

Key Requirements

2. Payment Processing Architecture

Client Application ? API Gateway ? Payment Orchestrator ? Bank / Payment Provider ? Webhook / Callback ? Transaction Reconciliation

Example Scenario

Customer clicks "Pay ?10,000". Payment provider receives request but response times out. Customer refreshes and retries. Without idempotency two payments may be processed. With idempotency only one transaction succeeds.

Best Practices

Pattern Purpose Real Example
Idempotency Avoid duplicate transactions UPI Payment Retry
Retry Recover transient failures Bank API timeout
Circuit Breaker Prevent cascading failures External payment gateway down
Timeout Avoid hanging requests Connection timeout after 5 sec
TLS Security Encrypt communication HTTPS/TLS 1.3
Observability Logs, Metrics, Traces Splunk + Prometheus + Grafana

3. Resilience Patterns

Pattern Purpose Example
Circuit Breaker Stop repeated failures Bank API unavailable
Retry Recover temporary failures HTTP 503 retry 3 times
Bulkhead Isolate failures Separate thread pools
Failover Switch to backup system Primary DB ? Secondary DB
Rate Limiting Protect backend systems 100 requests/minute
Caching Reduce load Redis Cache
Graceful Degradation Partial service instead of outage Show cached balance
Health Checks Monitor service health Kubernetes probes

4. Real Production Incident Example

Scenario

Payment service started failing during Black Friday peak traffic.

Root Cause

Immediate Actions

Permanent Fix

5. SLA, SLO, SLI (Frequently Asked)

Term Definition Example
SLI What is measured Availability = 99.95%
SLO Target objective Maintain >99.9%
SLA Customer commitment 99.5% guaranteed

Interview Answer

SLI is the actual metric measured. SLO is the target value for that metric. SLA is the business agreement with customers.

6. Kubernetes Interview Examples

Concept Example
Liveness Probe Restart pod if application hangs
Readiness Probe Receive traffic only when healthy
Horizontal Pod Autoscaler Scale pods based on CPU
Rolling Update Deploy without downtime
Namespace Environment isolation

7. Disaster Recovery Strategy

DR Component Example
Database Replication Primary ? Secondary Region
Backup Daily snapshots
Failover Automatic region switch
RTO 15 minutes
RPO Near Zero

Interview Example

Our application was deployed across two AWS regions. In case of primary region failure, Route53 redirected traffic to the secondary region and database replicas were promoted automatically.

8. Saga Pattern Example

E-Commerce Order Flow

Order Service ? Payment Service ? Inventory Service ? Shipping Service

Failure Scenario

Payment succeeds but Inventory fails. Saga executes compensating transaction and refunds payment.

Pattern Usage
Choreography Event-based coordination
Orchestration Central coordinator controls flow
Compensation Rollback completed steps

9. Infrastructure Concepts

Concept Interview Answer
High Availability System remains operational during failures.
Fault Tolerance System continues despite component failures.
Resilience Quick recovery from disruptions.
Scalability Handle increased workload efficiently.
Observability Logs + Metrics + Traces.
Auto Scaling Adjust resources automatically.
Load Balancer Distribute traffic across servers.

10. GenAI for SRE

Application Logs ? Kafka ? AI Processing Layer ? LLM Analysis ? Dashboard / Alerts

Enterprise Use Cases

11. Most Expected Interview Questions

  1. Tell me about yourself.
  2. Describe a Sev-1 incident.
  3. How do you perform RCA?
  4. Explain SLA, SLO, SLI.
  5. What is Error Budget?
  6. How does Circuit Breaker work?
  7. How do you troubleshoot Kubernetes pods?
  8. How do you handle payment failures?
  9. Explain Blue-Green deployment.
  10. Explain Canary Release.
  11. How do you improve reliability?

12. Strong Keywords For Interviews

High Availability, Fault Tolerance, Reliability Engineering, Observability, Resilience, Incident Management, Root Cause Analysis, Disaster Recovery, Capacity Planning, Performance Optimization, Automation, Production Stability

SRE Interview Questions & Answers

1. Describe a Sev-1 Incident

A Sev-1 (Severity-1) incident is the most critical production issue that causes a complete outage, major business impact, or affects a large number of users. It requires immediate attention from multiple teams.

Sample Interview Answer

In one of our payment processing applications, the payment service started failing during peak traffic hours. Customers were unable to complete transactions, causing significant business impact.

Result: Service availability restored quickly with minimal revenue loss and preventive measures were implemented.

2. How Do You Perform RCA (Root Cause Analysis)?

RCA is a structured process used to identify the actual cause of a production issue and implement preventive actions.

RCA Process

  1. Collect incident timeline.
  2. Analyze application and infrastructure logs.
  3. Review monitoring metrics.
  4. Identify triggering event.
  5. Determine root cause.
  6. Implement permanent fix.
  7. Create preventive action items.
Example: Memory leak caused pod restarts. Heap dump analysis identified an unclosed cache object consuming memory.

3. Explain SLA, SLO, and SLI

Term Description Example
SLI Service Level Indicator Availability = 99.95%
SLO Service Level Objective Target Availability = 99.9%
SLA Service Level Agreement Customer Contract = 99.5%

Simple Explanation

Payment System Example:
SLI = API Availability 99.95%
SLO = Maintain above 99.90%
SLA = Customer compensation below 99.50%

4. What Is an Error Budget?

An Error Budget represents the acceptable amount of failure allowed while still meeting the SLO.

Formula

Error Budget = 100% - SLO
Example:
SLO = 99.9% Availability
Error Budget = 0.1% Downtime

Monthly Duration = 43,200 Minutes
Allowed Downtime = 43 Minutes

If the error budget is exhausted, feature releases may be paused and reliability improvements become the priority.

5. How Does Circuit Breaker Work?

Circuit Breaker prevents repeated calls to failing services and protects the overall system from cascading failures.

States

State Description
Closed Normal operation, requests allowed.
Open Failures detected, requests blocked.
Half-Open Limited requests allowed for testing recovery.
Example:
Payment gateway becomes unavailable.
Circuit Breaker opens after 5 failures and prevents further calls until recovery.

6. How Do You Troubleshoot Kubernetes Pods?

Step-by-Step Approach

  1. Check pod status.
  2. View pod logs.
  3. Describe pod events.
  4. Check resource utilization.
  5. Verify readiness and liveness probes.
  6. Check service and ingress connectivity.
  7. Validate deployment configuration.

Useful Commands

kubectl get pods kubectl describe pod pod-name kubectl logs pod-name kubectl top pod kubectl get events
Example:
Pod repeatedly restarted due to OutOfMemoryError. Increased memory limits and optimized application memory usage.

7. How Do You Handle Payment Failures?

Best Practices

Example:
Customer submits payment.
Payment provider times out.
Retry mechanism executes.
If uncertain state exists, reconciliation process validates transaction status.

8. Explain Blue-Green Deployment

Blue-Green deployment uses two identical production environments.

Blue Environment (Current Production) ? Switch Traffic ? Green Environment (New Version)

Advantages

Example:
Version 1 runs in Blue environment.
Version 2 deployed in Green environment.
Traffic switched after validation.

9. Explain Canary Release

Canary deployment gradually releases a new version to a small percentage of users before full rollout.

Version 1 ? 95% Traffic Version 2 ? 5% Traffic Monitor ? Increase Gradually

Advantages

Example:
Deploy Version 2 to 5% users.
Monitor errors and latency.
Increase traffic gradually to 25%, 50%, and finally 100%.

10. How Do You Improve Reliability?

Reliability is improved through proactive monitoring, automation, resilience engineering, and continuous improvement.

Key Strategies

Interview Answer:
I improve reliability by implementing proactive monitoring, resilience patterns such as Circuit Breakers and Retries, conducting RCA for production incidents, automating operational tasks, tracking SLA/SLO metrics, and continuously optimizing system performance and scalability.

SRE Interview Closing Statement

My experience in production support, incident management, Kubernetes troubleshooting, payment systems, resiliency engineering, monitoring, and cross-functional collaboration enables me to build highly available, scalable, and reliable systems while ensuring business continuity and operational excellence.