Legacy Banking Infrastructure Support Skill

=== LEGACY BANKING INCIDENT RUNBOOK ===

Runbook title
Overnight Core Banking Batch Job Failure Runbook

System/capability
Core banking batch processing

Purpose
Provide structured triage for recurring overnight batch failures that may affect account updates, downstream reconciliation, reporting, and start-of-day operations.

Scope

Included

• batch failure classification
• predecessor and successor job review
• input/output file checks
• log review checklist
• escalation path
• rerun decision support
• downstream impact checks
• post-incident summary

Out of scope

• unauthorized reruns
• direct database edits
• ledger corrections
• production changes without approval
• final decision to rerun or rollback

Severity model
SEV-1:
Batch failure affects core ledger, account balances, settlement, or regulatory deadline.

SEV-2:
Batch failure threatens start-of-day operations or downstream reconciliation deadline.

SEV-3:
Batch failure is isolated and can be corrected within SLA.

Incident triggers

• job fails with non-zero return code
• job exceeds runtime threshold
• predecessor job missing
• input file missing
• output control totals mismatch
• downstream reconciliation job blocked
• repeated failure across multiple nights

Initial triage

1. Confirm job name, job ID, and environment.
2. Confirm first failure timestamp.
3. Check predecessor jobs.
4. Check input file arrival.
5. Check control totals if applicable.
6. Check whether downstream settlement/reconciliation/reporting deadlines are at risk.
7. Check recent changes to batch schedule, job parameters, database, files, or vendor feed.
8. Preserve logs before retry or rerun.

Dependency map

Primary job
[Batch job name]

Upstream dependencies

• source transaction files
• prior posting jobs
• vendor file transfer
• scheduler calendar

Downstream dependencies

• reconciliation job
• general ledger interface
• regulatory reporting feed
• customer statement generation
• start-of-day availability

Evidence to collect

• job ID
• job start/end time
• return code
• scheduler log
• application log
• file arrival timestamp
• input file size/control total
• rejected records summary with sensitive data masked
• downstream blocked jobs
• recent change record
• prior incident reference

Log review checklist

Application/batch logs

• first error
• repeated error pattern
• rejected record count
• file format errors
• timeout errors
• dependency unavailable

Scheduler logs

• predecessor completion
• job calendar
• retry attempts
• resource constraints

Database metrics

• locks
• connection pool
• storage
• deadlocks
• long-running queries

Decision tree

IF input file is missing
Check file transfer status.
Check vendor delivery confirmation.
Escalate to file transfer/vendor owner if not delivered.

IF input file arrived but job rejected records:
Review rejection summary.
Mask sensitive data.
Identify common rejection pattern.
Escalate to application owner or source-system owner.

IF predecessor job failed
Follow predecessor job runbook.
Do not rerun current job until dependency is resolved.

IF downstream deadline is at risk
Escalate to major incident process and business owner.

IF rerun is considered
Confirm approval, data integrity impact, duplicate-processing risk, and downstream communication.

Escalation path

Incident commander
[Major incident lead if SEV-1/SEV-2]

Batch operations
[Team/contact]

Application owner
[Team/contact]

Database owner
[Team/contact]

File transfer/vendor owner
[Team/contact]

Business owner
[Operations/reconciliation owner]

Compliance/regulatory contact
[If deadline or reporting risk exists]

Rollback/rerun decision guidance
Rollback or rerun must be approved by authorized operations and application owners. Before rerun, confirm duplicate-processing risk, input file integrity, downstream status, and reconciliation impact.

Workaround assessment
Manual processing may require dual control, audit record, business approval, and reconciliation plan.

Data integrity checks

• duplicate transactions?
• missing transactions?
• partial processing?
• control totals match?
• ledger affected?
• downstream reconciliation affected?

Communication templates

Technical update
Batch job [name] failed at [time] with [return code]. Initial impact assessment: [impact]. Evidence collected: [evidence]. Current owner: [owner]. Next update: [time].

Business update
Overnight batch processing for [capability] is delayed. Current known impact: [impact]. Reconciliation/settlement deadline risk: [risk]. Next update: [time].

Post-resolution checks

• job completed successfully
• downstream jobs resumed
• control totals verified
• reconciliation checks passed
• business owner notified
• incident record updated

Post-incident summary template

Incident ID

Job

Timeline

Root cause

Impact

Resolution

Approvals

Data integrity checks

Action items

Runbook updates

Owner and review cadence
Batch operations owner. Review quarterly or after any SEV-1/SEV-2 batch incident.