Modernizing Legacy Systems Into Scalable Cloud-Native Architectures

IA FORUM MEMBER INSIGHTS: ARTICLE

By Shailesh Kadam, Artificial Intelligence & Cloud Architect, SAKS GLOBAL

For many enterprises, the journey from legacy, on-premises infrastructure to a modern, cloud-native ecosystem is not merely a technology upgrade - it’s a fundamental transformation of how the business operates, scales, and competes. Organizations that delay this transition accumulate technical debt that compounds year over year: aging Informatica ETL pipelines, monolithic Oracle EBS deployments, Unix-based shell scripts, and custom file transfer mechanisms all become increasingly expensive to maintain, impossible to scale elastically, and opaque to modern observability tooling.

This article will trace the end-to-end journey of one such transformation - from a complex web of legacy systems, through a pragmatic hybrid state, and ultimately into a fully cloud-native architecture on Amazon Web Services (AWS). Along the way, it outlines the strategic pillars, architectural patterns, and operational practices that enable teams to modernize without disrupting the business operations that depend on them.

The Legacy Landscape: Understanding What We Are Migrating From

Before any modernization effort can begin in earnest, teams must develop a complete and honest inventory of the systems they are working with. In the case study examined here, the legacy landscape was rich and complex. Source systems included Retail Management System (RMS) and ATG commerce platform feeding into an Informatica-powered Operational Data Store (ODS) layer. The ODS pipeline proceeded through multiple stages: ODS staging, dimensional modeling (Dim & Facts), aggregated staging, and a final feed layer.

Beyond the core data pipeline, the ecosystem included an extensive list of vendor integrations and internal applications: platforms such as BloomReach, Bluecore, Borderfree, CheetahMail, Curalate, Granify, iProspect, Pinterest, Rich Relevance, ShopRunner, ShopStyle, Experian, UGAM, Compliance Networks, SAS, Omniture, Everage, and Veraction - alongside internal tools including WMS, EDW/ERA, CXP, iSell, Oracle EBS, Kronos, and advanced analytics platforms.

Data moved through Informatica-managed SFTP jobs, with nightly full-batch loads complemented by hourly delta syncs. Final delivery was in XML and flat-file format - formats that were never designed for real-time, API-driven consumption. ETL control was managed through a POBOX construct, and vendor delivery happened through external SFTP sites.

Figure 1 - Legacy Architecture: Informatica-powered ODS pipeline with nightly batch delivery to vendor SFTP sites

This architecture, while functional, presented serious limitations: batch-only delivery windows meant that downstream vendors received stale data; Informatica licensing and operational overhead were substantial; the pipeline had no native cloud integration; and horizontal scaling required expensive hardware procurement cycles rather than on-demand elasticity.

Assessment & Strategic Planning: The Modernization Framework

A successful cloud modernization program begins with structured assessment and strategic planning - not with technology selection.

The framework applied in this case centered on five interconnected pillars:

• Assessment & Strategic Evaluation: The first step was a systematic evaluation of every legacy workload: its dependencies, data volumes, SLA requirements, and organizational pain points. The goal was not just to understand what exists, but to classify systems by migration complexity, business criticality, and readiness for re-platforming. This assessment drives the sequencing of the migration program and surfaces dependencies that could otherwise derail later phases.

• Adopting Cloud-Native Design Principles: Legacy monolithic architectures were targeted for decomposition into microservices with clear API boundaries, event-driven communication patterns, and containerized deployments managed via Docker and Kubernetes. This shift is architectural and cultural - it requires teams to abandon the comfort of tightly coupled, procedural batch flows in favor of loosely coupled, asynchronous services that can be deployed, scaled, and replaced independently.

• Incremental Migration with the Strangler Pattern: Rather than a high-risk "big bang" migration, the program adopted the strangler fig pattern: new cloud-native capabilities are built alongside the legacy system, progressively absorbing traffic and data flows until the legacy component can be decommissioned. This approach dramatically reduces migration risk and allows teams to validate the new architecture under real production load before committing fully.

• Leveraging Cloud-Native Services & Automation: On AWS, the architecture leaned into managed services - Amazon S3 for object storage, AWS Lambda for serverless compute, AWS Glue for managed ETL, Amazon Aurora for relational workloads, Amazon DynamoDB for high-throughput NoSQL, Amazon SQS for message queuing with dead-letter queue (DLQ) support, Kinesis Data Streams for real-time event streaming, and Amazon API Gateway for exposing data as developer-friendly APIs. CI/CD pipelines automated the deployment and testing of all pipeline components, reducing the manual toil that had been endemic in the legacy environment.

• Security, Governance & Operational Excellence: Zero-trust security principles were embedded from the outset, with identity-based access controls, encryption in transit and at rest, and cloud governance frameworks enforcing policy guardrails. Site Reliability Engineering (SRE) practices - including Service Level Objectives (SLOs), error budgets, and structured on-call rotations - were adopted to manage the operational complexity of a distributed, multi-account AWS environment.

The Hybrid Architecture: A Pragmatic Bridge State

Cloud migrations of this scale rarely move directly from legacy to fully cloud-native. The hybrid architecture represents the critical bridge phase: legacy components continue to serve their existing consumers while new cloud-native capabilities are stood up in parallel and progressively take over.

Figure 2: Hybrid Architecture — AWS and legacy ODS layers operating in parallel, enabling phased migration without business disruption

In the hybrid phase, the new ICC (Integrated Commerce Cloud) AWS layer was introduced above the existing ICC ODS layer. The AWS layer included S3 buckets for data ingestion from the DT (Digital Transformation) AWS environment, AWS Lambda functions for data transformation, AWS Glue for schema management and ETL, Amazon Aurora for structured relational storage, Amazon DynamoDB for the Product Availability Layer (PAL), and a Glue Scheduler to orchestrate workload timing.

Crucially, the legacy ODS pipeline continued to run in parallel - feeding Informatica-managed SFTP jobs, producing XML and flat-file feeds, and delivering data to vendor SFTP sites via nightly batch and hourly delta processes. The new AWS layer, meanwhile, introduced modern delivery mechanisms: data streaming via webhooks and WebSockets for near-real-time vendor consumption, and JSON file feeds through managed file transfer as a steppingstone away from XML.

This parallel operation is expensive in the short term - teams must maintain two systems - but it is far less costly than a failed migration that disrupts vendor integrations or leaves downstream consumers without data. The hybrid phase also creates a natural validation environment: the cloud-native outputs can be compared against legacy outputs to detect discrepancies before the legacy system is retired.

The Cloud-Native Architecture: The Target State

The fully cloud-native target architecture eliminates the Informatica ODS layer entirely. All data ingestion, transformation, storage, and delivery are handled within the AWS ecosystem. The ICC AWS layer - anchored by S3, Lambda, Glue, Aurora, and DynamoDB - becomes the single source of truth for all vendor data delivery.

Figure 3 - Cloud-Native Architecture - Fully AWS-hosted pipeline with Developer Portal APIs, data streaming, and managed file transfer replacing the legacy ODS

In this target state, vendors consume data through three delivery mechanisms: a Developer Portal for API-based consumption, data streaming via webhooks and WebSockets for event-driven near-real-time delivery, and JSON file feeds with managed file transfer for vendors requiring batch delivery. The ETL Control POBOX and Informatica SFTP infrastructure are retired. Nightly full-batch and hourly delta schedules are retained only where legacy vendor contracts require it - everything else moves to event-driven delivery.

The shift from batch to event-driven delivery is perhaps the most transformative outcome of the migration. Where the legacy architecture required vendors to wait for nightly jobs before receiving updated product, inventory, or pricing data, the cloud-native architecture can propagate changes in near-real-time - enabling downstream systems to reflect the current state of the business rather than yesterday's state.

Detailed Architecture: Source Feeds Replication & Inventory Management

Source Feeds Replication from DT to ICC

The detailed data replication architecture illustrates how individual product data domains - Products, SKUs, Catalog, Cart, CartItem, CartSKUItem, NonRules, Rules, and Inventory - are replicated from the DT NMO (Neiman Marcus Online) environment into the ICC AWS environment.

Figure 4 - Detailed Architecture Part 1: Kinesis Data Streams and Lambda-based feed replication with full-load S3 export and DynamoDB updates in ICC

For streaming updates, Kinesis Data Streams provide an independent data stream for each feed domain. Lambda functions consume from these streams, performing data transformation and forwarding messages to SQS. A second Lambda function places each record into the ICC account's DynamoDB tables. A Dead Letter Queue (DLQ) captures messages that cannot be processed, ensuring no data is silently dropped.

For full-load scenarios, DT exports data to S3 buckets - one per feed domain. In the ICC environment, API Gateway receives a trigger to invoke a Lambda-driven Batch Job that fetches the full load data from the DT S3 buckets and populates the ICC DynamoDB tables. This two-path architecture - streaming for real-time updates, S3-based batch for full loads - provides both the freshness of event-driven delivery and the reliability of periodic full refreshes.

NM Inventory Final Feed Architecture

The inventory feed architecture is particularly complex, given the criticality of accurate, timely inventory data for downstream commerce systems. From DT NMO, both full-load exports and Kinesis Data Streams carry nm_store_inventory_by_sku and SKUs data into the ICC environment.

Figure 5 - Detailed Architecture Part 2: NM Inventory Final Feed with DynamoDB streams, Aurora RDS, PAL Load, and Oracle OnPrem DB integration

In ICC, an S3 Full Load Bucket triggers SQS → Lambda → Batch processing pipelines. DynamoDB upserts and streams feed downstream Lambda functions that populate RDS Stage Tables in Aurora RDS (PostgreSQL) with RDS Proxy for connection pooling. PAL Queries and other DynamoDB queries populate Final Feed tables, which are persistent in both DynamoDB and RDS Final Tables. DynamoDB Streams trigger additional Lambda functions for further downstream processing.

The PAL Load component manages the PAL (Product Availability Layer) database through a combination of full-load batch processing via SQS and event-driven Lambda processing of SQS messages. Notably, the architecture also maintains a connection to an Oracle OnPrem DB - an intentional accommodation for systems that are not yet cloud-ready, consistent with the strangler pattern philosophy of incremental modernization.

Key Lessons & Best Practices

Organizations embarking on similar cloud modernization journeys will find the following principles invaluable based on the architecture and strategy detailed above:

• Inventory before architecture. You cannot modernize what you do not fully understand. A complete inventory of systems, dependencies, data flows, and consumer SLAs is the indispensable first step - before any cloud technology is selected or any migration sequence is defined.

• Embrace the hybrid phase. Resistance to a prolonged hybrid state often leads to rushed migrations and production incidents. The hybrid phase is a feature, not a failure - it’s the mechanism through which the organization safely transfers risk from the legacy system to the cloud-native replacement.

• Design for the target state from day one. Even during the hybrid phase, build the cloud-native components as though the legacy system will be gone tomorrow. This discipline prevents the accumulation of hybrid-specific technical debt that makes the final decommissioning step more painful than it needs to be.

• Instrument everything. The loss of observability is one of the greatest risks in cloud migration. Invest early in CloudWatch metrics, distributed tracing, structured logging, and SLO-based alerting. These capabilities are what transform a cloud deployment from a black box into a manageable production system.

• Manage organizational change alongside technical change. The shift from batch-oriented, Informatica-centric data engineering to event-driven, AWS-native engineering requires significant skills development, new tooling, and new ways of working. Technical architecture decisions must be paired with an organizational change management program that brings engineering teams, data teams, and vendor partners along the journey.

Conclusion

Modernizing legacy systems into scalable, cloud-native architectures is one of the most consequential engineering programs an organization can undertake. It touches every layer of the technology stack, every team that owns or consumes data, and every vendor that depends on reliable data delivery. Done well, it eliminates years of accumulated technical debt, unlocks elasticity and resilience that legacy infrastructure cannot provide, and positions the organization to move faster in response to market changes.

The journey from Informatica and Oracle on-premises to a fully AWS-native architecture - through structured assessment, cloud-native design principles, the strangler pattern migration approach, managed service adoption, and embedded security and governance - demonstrates that this transformation is achievable without disrupting the business operations that depend on it. The key is discipline: discipline in planning, discipline in execution, and discipline in resisting the temptation to shortcut the hybrid phase in favor of speed.

Author Disclaimer: The views and opinions expressed herein are those of the Author alone and are shared in a personal capacity, in accordance with the Chatham House Rule. They do not reflect the official views or positions of the Author’s employer, organization,