DTCC / Financial Services / SRE Interview Notes
1. DTCC Overview
DTCC (Depository Trust & Clearing Corporation) is a critical financial market
infrastructure organization responsible for clearing, settlement and transaction
processing for financial institutions.
Key Requirements
- High Availability (99.99%+)
- Disaster Recovery
- Zero Data Loss
- Low Latency Processing
- Secure Transactions
- Regulatory Compliance
- Auditability
- Scalability
2. Payment Processing Architecture
Client Application
?
API Gateway
?
Payment Orchestrator
?
Bank / Payment Provider
?
Webhook / Callback
?
Transaction Reconciliation
Example Scenario
Customer clicks "Pay ?10,000".
Payment provider receives request but response times out.
Customer refreshes and retries.
Without idempotency two payments may be processed.
With idempotency only one transaction succeeds.
Best Practices
| Pattern |
Purpose |
Real Example |
| Idempotency |
Avoid duplicate transactions |
UPI Payment Retry |
| Retry |
Recover transient failures |
Bank API timeout |
| Circuit Breaker |
Prevent cascading failures |
External payment gateway down |
| Timeout |
Avoid hanging requests |
Connection timeout after 5 sec |
| TLS Security |
Encrypt communication |
HTTPS/TLS 1.3 |
| Observability |
Logs, Metrics, Traces |
Splunk + Prometheus + Grafana |
3. Resilience Patterns
| Pattern |
Purpose |
Example |
| Circuit Breaker |
Stop repeated failures |
Bank API unavailable |
| Retry |
Recover temporary failures |
HTTP 503 retry 3 times |
| Bulkhead |
Isolate failures |
Separate thread pools |
| Failover |
Switch to backup system |
Primary DB ? Secondary DB |
| Rate Limiting |
Protect backend systems |
100 requests/minute |
| Caching |
Reduce load |
Redis Cache |
| Graceful Degradation |
Partial service instead of outage |
Show cached balance |
| Health Checks |
Monitor service health |
Kubernetes probes |
4. Real Production Incident Example
Scenario
Payment service started failing during Black Friday peak traffic.
Root Cause
- Connection Pool Exhaustion
- Database connections not released properly
- Pod memory spikes
Immediate Actions
- Checked Grafana dashboards
- Analyzed Splunk logs
- Scaled pods from 10 ? 20
- Increased DB connection pool
- Restarted affected pods
Permanent Fix
- Connection leak detection
- Prometheus alerts
- Pool monitoring dashboards
- Load testing with JMeter
5. SLA, SLO, SLI (Frequently Asked)
| Term |
Definition |
Example |
| SLI |
What is measured |
Availability = 99.95% |
| SLO |
Target objective |
Maintain >99.9% |
| SLA |
Customer commitment |
99.5% guaranteed |
Interview Answer
SLI is the actual metric measured. SLO is the target value for that metric.
SLA is the business agreement with customers.
6. Kubernetes Interview Examples
| Concept |
Example |
| Liveness Probe |
Restart pod if application hangs |
| Readiness Probe |
Receive traffic only when healthy |
| Horizontal Pod Autoscaler |
Scale pods based on CPU |
| Rolling Update |
Deploy without downtime |
| Namespace |
Environment isolation |
7. Disaster Recovery Strategy
| DR Component |
Example |
| Database Replication |
Primary ? Secondary Region |
| Backup |
Daily snapshots |
| Failover |
Automatic region switch |
| RTO |
15 minutes |
| RPO |
Near Zero |
Interview Example
Our application was deployed across two AWS regions.
In case of primary region failure, Route53 redirected traffic to the
secondary region and database replicas were promoted automatically.
8. Saga Pattern Example
E-Commerce Order Flow
Order Service
?
Payment Service
?
Inventory Service
?
Shipping Service
Failure Scenario
Payment succeeds but Inventory fails.
Saga executes compensating transaction and refunds payment.
| Pattern |
Usage |
| Choreography |
Event-based coordination |
| Orchestration |
Central coordinator controls flow |
| Compensation |
Rollback completed steps |
9. Infrastructure Concepts
| Concept |
Interview Answer |
| High Availability |
System remains operational during failures. |
| Fault Tolerance |
System continues despite component failures. |
| Resilience |
Quick recovery from disruptions. |
| Scalability |
Handle increased workload efficiently. |
| Observability |
Logs + Metrics + Traces. |
| Auto Scaling |
Adjust resources automatically. |
| Load Balancer |
Distribute traffic across servers. |
10. GenAI for SRE
Application Logs
?
Kafka
?
AI Processing Layer
?
LLM Analysis
?
Dashboard / Alerts
Enterprise Use Cases
- Incident Summarization
- RCA Generation
- Alert Correlation
- Runbook Recommendations
- Knowledge Assistant
- Log Analysis
11. Most Expected Interview Questions
- Tell me about yourself.
- Describe a Sev-1 incident.
- How do you perform RCA?
- Explain SLA, SLO, SLI.
- What is Error Budget?
- How does Circuit Breaker work?
- How do you troubleshoot Kubernetes pods?
- How do you handle payment failures?
- Explain Blue-Green deployment.
- Explain Canary Release.
- How do you improve reliability?
12. Strong Keywords For Interviews
High Availability,
Fault Tolerance,
Reliability Engineering,
Observability,
Resilience,
Incident Management,
Root Cause Analysis,
Disaster Recovery,
Capacity Planning,
Performance Optimization,
Automation,
Production Stability
SRE Interview Questions & Answers
1. Describe a Sev-1 Incident
A Sev-1 (Severity-1) incident is the most critical production issue that causes a complete outage, major business impact, or affects a large number of users. It requires immediate attention from multiple teams.
Sample Interview Answer
In one of our payment processing applications, the payment service started failing during peak traffic hours. Customers were unable to complete transactions, causing significant business impact.
- Declared Sev-1 incident and initiated war-room call.
- Analyzed application logs and monitoring dashboards.
- Identified database connection pool exhaustion.
- Scaled Kubernetes pods and increased connection pool size.
- Restored service within 20 minutes.
- Performed RCA and implemented permanent monitoring.
Result: Service availability restored quickly with minimal revenue loss and preventive measures were implemented.
2. How Do You Perform RCA (Root Cause Analysis)?
RCA is a structured process used to identify the actual cause of a production issue and implement preventive actions.
RCA Process
- Collect incident timeline.
- Analyze application and infrastructure logs.
- Review monitoring metrics.
- Identify triggering event.
- Determine root cause.
- Implement permanent fix.
- Create preventive action items.
Example:
Memory leak caused pod restarts. Heap dump analysis identified an unclosed cache object consuming memory.
3. Explain SLA, SLO, and SLI
| Term |
Description |
Example |
| SLI |
Service Level Indicator |
Availability = 99.95% |
| SLO |
Service Level Objective |
Target Availability = 99.9% |
| SLA |
Service Level Agreement |
Customer Contract = 99.5% |
Simple Explanation
- SLI ? What is measured.
- SLO ? Desired target.
- SLA ? Customer commitment.
Payment System Example:
SLI = API Availability 99.95%
SLO = Maintain above 99.90%
SLA = Customer compensation below 99.50%
4. What Is an Error Budget?
An Error Budget represents the acceptable amount of failure allowed while still meeting the SLO.
Formula
Error Budget = 100% - SLO
Example:
SLO = 99.9% Availability
Error Budget = 0.1% Downtime
Monthly Duration = 43,200 Minutes
Allowed Downtime = 43 Minutes
If the error budget is exhausted, feature releases may be paused and reliability improvements become the priority.
5. How Does Circuit Breaker Work?
Circuit Breaker prevents repeated calls to failing services and protects the overall system from cascading failures.
States
| State |
Description |
| Closed |
Normal operation, requests allowed. |
| Open |
Failures detected, requests blocked. |
| Half-Open |
Limited requests allowed for testing recovery. |
Example:
Payment gateway becomes unavailable.
Circuit Breaker opens after 5 failures and prevents further calls until recovery.
6. How Do You Troubleshoot Kubernetes Pods?
Step-by-Step Approach
- Check pod status.
- View pod logs.
- Describe pod events.
- Check resource utilization.
- Verify readiness and liveness probes.
- Check service and ingress connectivity.
- Validate deployment configuration.
Useful Commands
kubectl get pods
kubectl describe pod pod-name
kubectl logs pod-name
kubectl top pod
kubectl get events
Example:
Pod repeatedly restarted due to OutOfMemoryError. Increased memory limits and optimized application memory usage.
7. How Do You Handle Payment Failures?
Best Practices
- Idempotency Keys
- Retry Mechanism
- Circuit Breaker
- Timeout Management
- Transaction Reconciliation
- Monitoring & Alerts
Example:
Customer submits payment.
Payment provider times out.
Retry mechanism executes.
If uncertain state exists, reconciliation process validates transaction status.
8. Explain Blue-Green Deployment
Blue-Green deployment uses two identical production environments.
Blue Environment (Current Production)
?
Switch Traffic
?
Green Environment (New Version)
Advantages
- Zero downtime deployment.
- Instant rollback.
- Reduced deployment risk.
Example:
Version 1 runs in Blue environment.
Version 2 deployed in Green environment.
Traffic switched after validation.
9. Explain Canary Release
Canary deployment gradually releases a new version to a small percentage of users before full rollout.
Version 1 ? 95% Traffic
Version 2 ? 5% Traffic
Monitor ? Increase Gradually
Advantages
- Lower deployment risk.
- Early issue detection.
- Controlled rollout.
Example:
Deploy Version 2 to 5% users.
Monitor errors and latency.
Increase traffic gradually to 25%, 50%, and finally 100%.
10. How Do You Improve Reliability?
Reliability is improved through proactive monitoring, automation, resilience engineering, and continuous improvement.
Key Strategies
- Monitoring & Alerting
- Observability
- Auto Scaling
- Disaster Recovery Planning
- Load Testing
- Chaos Engineering
- Incident Management
- RCA & Preventive Actions
- Automation of Repetitive Tasks
- SLA/SLO Monitoring
Interview Answer:
I improve reliability by implementing proactive monitoring, resilience patterns such as Circuit Breakers and Retries, conducting RCA for production incidents, automating operational tasks, tracking SLA/SLO metrics, and continuously optimizing system performance and scalability.
SRE Interview Closing Statement
My experience in production support, incident management, Kubernetes troubleshooting, payment systems, resiliency engineering, monitoring, and cross-functional collaboration enables me to build highly available, scalable, and reliable systems while ensuring business continuity and operational excellence.