Building a PGP-Encrypted Flat File Exchange with Dual MFT Events

Introduction

Most integration write-ups focus on APIs — REST, webhooks, message queues. But a huge share of real enterprise data movement still happens the old-fashioned way: a flat file, encrypted, dropped on an SFTP server, picked up on a schedule. This post walks through a pattern I've built more than once on IBM webMethods — a PGP-encrypted flat file exchange, driven by two independent Managed File Transfer (MFT) events with an Integration Server flow doing the actual transformation in between.

It's not a glamorous architecture. It's a reliable one, and the details that make it reliable are exactly the ones that don't show up in a five-minute demo.

Business Problem

A source system exports a flat file — a CSV with multiple columns — on a schedule. Because the file may contain sensitive data, it's PGP-encrypted before it's placed on an SFTP server. On the receiving side, that file needs to be picked up, decrypted, transformed by business logic, and delivered — re-encrypted — to a different SFTP location for a downstream partner or system to consume.

The requirement isn't just "move the file." It's: never let sensitive data sit around unencrypted longer than necessary, keep an audit trail of exactly what happened at each stage, and make failures in one direction (inbound) clearly distinguishable from failures in the other (outbound).

High-Level Architecture

The flow is three hops:

Source SFTP (PGP-encrypted)
        │
        ▼
MFT Event #1 (scheduled)
  → fetch file to internal work directory
  → decrypt with ESB private key
  → invoke Integration Server
  → [after IS returns] delete decrypted CSV, archive original PGP file, log
        │
        ▼
Integration Server
  → parse against flat file schema, row by row
  → apply business logic per row
  → batch-write target file (50–100 records/batch, configurable)
  → write to a second work directory
        │
        ▼
MFT Event #2 (scheduled)
  → find target file in work directory #2
  → encrypt with partner's public key
  → copy to target SFTP
  → move local copy to Done directory, log
        │
        ▼
Target SFTP (partner)

Two MFT events, not one. That's a deliberate choice, and it's worth explaining why.

Why Two Independent MFT Events, Not One

It would be simpler, on paper, to have a single MFT event handle the whole thing end to end — fetch, decrypt, invoke, wait, encrypt, deliver. In practice, that couples two very different failure modes together.

An inbound failure (the source file never arrives, or fails to decrypt) is a completely different operational problem than an outbound failure (the target SFTP is unreachable, or the partner's public key has rotated). Splitting them into two independent MFT events means:

• Each hop has its own schedule, its own retry policy, and its own success/failure log — so when something breaks, you know immediately which side of the flow it's on, without digging through a single merged log.
• The two hops can be tuned independently. If the source system's exports are bursty, Event #1's schedule doesn't need to match Event #2's.
• Integration Server sits cleanly in the middle, decoupled from the transport concerns on either side.

PGP Decryption on the Way In

The source system encrypts the file with the ESB's PGP public key before it's ever transmitted. Only the ESB's matching private key — held in a secure keystore, never exported — can decrypt it. This happens inside the MFT event itself, before Integration Server is ever invoked, and the decrypted file stays inside an internal work directory the whole time. At no point does a plaintext version of the file touch anything externally reachable.

Flat File Processing — Row by Row, Not All at Once

Once Integration Server is invoked, the file is parsed against a flat file schema defined against the known column layout. The important design decision here is the same one that matters in any large-file integration: records are read and processed one row at a time, not loaded into memory as a whole. Business logic — validation, enrichment, whatever transformation rules apply — runs per row.

Batching the Output, Not the Input

The output side works differently from the input side, deliberately. Rather than writing one row at a time to the target file (expensive) or holding the entire transformed dataset in memory before a single write (risky at scale), records are written in batches of 50–100, with the batch size exposed as a configuration value rather than hardcoded.

That number isn't arbitrary — it's a tuning knob. A larger batch size trades higher per-write throughput against a bigger unit of work if something fails mid-batch; a smaller one is more conservative. Making it configurable means that trade-off can be revisited without a code change as data volumes grow.

Cleanup Isn't Optional — It's Part of the Design

Two cleanup steps happen after the inbound hop, and neither is an afterthought:

1. The decrypted CSV is deleted from the work directory once Integration Server has consumed it. If the source data includes anything sensitive, a plaintext file sitting on disk indefinitely is exactly the kind of thing that turns into a real problem during a security review.
2. The original PGP-encrypted file is archived, not deleted — moved to an archive location so it's available for audit or reprocessing without needing to go back to the source system.

The same pattern repeats on the outbound side: once the encrypted file is confirmed delivered to Target SFTP, the local copy moves to a "done" directory rather than being deleted outright.

PGP Encryption on the Way Out

Before delivery, the target file is encrypted again — this time with the receiving partner's public key, so only their private key can open it. It's the mirror image of the inbound step, and the same rule applies: the unencrypted target file never leaves the ESB's internal work directory.

Failure Handling

Each hop fails independently and logs independently:

• If MFT Event #1 can't fetch or decrypt the source file, that's logged as a Hop 1 failure — Integration Server is never invoked, so there's no risk of processing a partial or corrupted file.
• If MFT Event #2 can't reach the target SFTP, that's a Hop 3 failure, logged separately from anything that happened upstream.

This separation is what makes production support tractable. A one-line log entry saying "outbound delivery failed" is immediately actionable in a way that a single ambiguous "integration failed somewhere" alert is not.

Security Best Practices

• Data is PGP-encrypted end-to-end outside the ESB's own boundary — encrypted at rest on both source and target SFTP, and in transit.
• The plaintext file exists only transiently, inside an internal work directory, for the duration of a single processing run.
• Private keys never leave their respective keystores; only the matching public key is ever shared externally.

Conclusion

There's nothing exotic in this pattern — SFTP, PGP, a flat file schema, two scheduled events. What makes it hold up in production is the discipline around the boring parts: deleting sensitive plaintext promptly, archiving instead of deleting the audit trail, keeping inbound and outbound failure domains separate, and making the one genuine performance lever (batch size) a config value instead of a constant buried in code. Enterprise integration reliability is usually built from decisions exactly this unglamorous.