The process of testing and validating the proper functioning of large systems that integrate multiple third-party components is not always a straightforward task. With modern solutions generally incorporating multiple interconnected processes and systems that work in concert, corner cases are often found that can have unforeseen consequences if not previously identified and addressed using comprehensive testing strategies.
My team was tasked by a large e-retailer to remake its entire OMS (Order Management System) and build a solution that integrates multiple third-party APIs (Shopify, Accertify, Avalara, etc.) used by the company. The project was part of a larger retail strategy that involved moving the customer from an old-school monolithic legacy systems approach to a more modern, scalable, and cost-efficient cloud approach leveraging state-of-the-art AWS tools and services.
This article describes the various challenges and scenarios faced by my team and provides insight into the approaches explored in order to test and validate the proper functioning of the OMS.
E2E Testing: What It Is & Why It’s Important​
E2E (End-to-End Testing) testing refers to a software testing methodology that tests an application or a system’s workflow from the beginning to the end. The E2E testing approach aims to replicate real user scenarios so that the system can be validated for integration and data integrity.
E2E testing determines whether the various dependencies of an application are working accurately and also serves as an effective means to validate that information is being correctly communicated between the multiple components of the system.
How the E2E Tests Are Executed​
We leveraged AWS CodeBuild, AWS’s fully-managed build service, to conduct the E2E tests. The team has an AWS CodeCommit repository where it pushes the code and every time a PR (Pull Request) is created, a code pipeline is triggered. AWS CodeBuild deploys the entire OMS system (which includes the deployment of a system of Serverless APIs, AWS Lambda functions, AWS Step Functions, and Amazon SQS queues) in a unique environment on the AWS Cloud that simulates the production environment.
Each build returns a message to the team to notify them whether the build failed or not. We have a special Slack channel for those notifications. All the resources deployed to run the tests are automatically deleted at the end of each build to avoid unnecessary costs. This approach enables the team to execute multiple E2E tests simultaneously, yet cost-efficiently.
Scenario & Challenges we faced​
Approach #1: Fake Payloads​
Our initial approach was to build E2E tests with fake payloads (data sent by Shopify to the OMS). We created payloads, pushed them to the system, and proceeded to check if the SQS messages were being sent (an indication that the workflow was functioning without errors).
Drawbacks of the Approach
This approach proved to be overly time-consuming and inefficient because multiple bugs would not be detected and we ran into the same problems over and over again. These regressions slowed down the entire process.
Approach #2: Framework with Payloads (“Golden Testing”)​
Having acknowledged the inefficiency of the first approach, I came with a second approach called “Golden Testing”.
How “Golden Testing” Works & Its Benefits
There are 50 “golden orders”, or scenarios, that represent the edge cases of the application. Instead of writing the logic for each of these orders, this second approach only requires the payloads (data sent by Shopify and other services) and a description of the orders (all the steps the order data needs to go through and all the information it needs to push the order out).
The approach requires the UAT (User-Acceptance Testing) team to put together a spreadsheet that contains all the bugs. To find bugs, the UAT team executes manual tests. They place orders on Shopify, the order management system receives these orders, and based on the responses that are provided, the UAT team fills a spreadsheet containing details of all the bugs/errors found in the system as a result of these manual tests. We used the information provided in the spreadsheet to identify and isolate errors within the OMS.
To execute the E2E tests, the TrackIt team leverages JavaScript and uses Jest, a JavaScript testing framework designed to ensure the correctness of any JavaScript codebase. The test script (as seen below) checks and validates whether the correct outputs (“Requirements”) are being sent out by each of the “StepFunctions”. When a test fails, the team proceeds to make the necessary corrections to the OMS. Despite still being time-consuming, this second approach provided the team with a deterministic approach to prevent regressions.
const E2eClass = require('../E2eClass');
const {
StepFunctions: {
ORDER_ENTRY,
FRAUD_ACCEPT,
SECONDARY_TAX_RESULT,
PAYMENT_CAPTURE_RESULT,
REFUND_ENTRY,
REFUND_TAX_REVERSAL,
REFUND_RECORD_ORIGINAL_PAYMENT,
},
Requirements: {
SQS_ACCERTIFY_FRAUD_REQUEST,
SQS_AVALARA_CREATE_TRANSACTION_REQUEST,
SQS_PAYMENT_CAPTURE_REQUEST,
SQS_AVALARA_COMMIT_TRANSACTION_REQUEST,
SQS_SHOPIFY_SALES_ORDER,
SQS_AVALARA_CREATE_TRANSACTION_REVERSAL,
SQS_NO_SHIP_CANCELLATION,
SQS_AVALARA_COMMIT_TRANSACTION_REQUEST_REFUND,
TRANSACTION_100,
TRANSACTION_105,
},
} = require('../constants');
describe(
'Test Case 4', () => {
const runnerClass = new E2eClass('Test Case 4',
{
orderNumber: 1772,
orderId: 2850372747397,
scenario: [
{
stepFunction: ORDER_ENTRY,
requirements: [SQS_ACCERTIFY_FRAUD_REQUEST],
},
{
stepFunction: FRAUD_ACCEPT,
requirements: [SQS_AVALARA_CREATE_TRANSACTION_REQUEST],
},
{
stepFunction: SECONDARY_TAX_RESULT,
requirements: [SQS_PAYMENT_CAPTURE_REQUEST],
},
{
stepFunction: PAYMENT_CAPTURE_RESULT,
requirements: [
SQS_AVALARA_COMMIT_TRANSACTION_REQUEST,
SQS_SHOPIFY_SALES_ORDER,
TRANSACTION_100,
],
},
{
stepFunction: REFUND_ENTRY,
requirements: [SQS_AVALARA_CREATE_TRANSACTION_REVERSAL],
},
{
stepFunction: REFUND_TAX_REVERSAL,
requirements: [SQS_NO_SHIP_CANCELLATION],
},
{
stepFunction: REFUND_RECORD_ORIGINAL_PAYMENT,
requirements: [
SQS_AVALARA_COMMIT_TRANSACTION_REQUEST_REFUND,
TRANSACTION_105,
],
},
],
});
runnerClass.run();
},
);
E2E Testing vs. Unit Testing​
Working on the OMS gave us an opportunity to assess the importance of considering different options for testing to identify the most comprehensive and efficient approach. We recognized that despite being easy to write (after building the framework), E2E tests also introduce a fair degree of complexity to the process of testing intricate systems containing multiple integrated third-party components.
- Benefit: Easier to Execute
E2E tests are easy to write when an entire system needs to be tested as a whole. Using Unit tests to comprehensively test a system would be a much lengthier and less efficient process.
- Drawback: Slow
One of the fundamental drawbacks of running E2E tests is that they take a long time to be completed. Using E2E tests for bugs that could have been resolved with Unit tests can slow things down. This proves to be an issue when one compares the time required to run the 50 E2E tests needed for this project (which takes 5–10 mins in this example with the 2nd “Golden Testing” approach) to the time required to execute all the Unit tests (10 seconds).
- Drawback: Difficulty of Identifying Bugs
Contrary to Unit testing where each component of the system can be tested in an isolated manner, E2E testing does not provide testers with the ability to immediately identify where a specific error has occurred.
A Combination of E2E Testing and Unit Testing​
Having acknowledged the limitations in efficiency arising from choosing to solely execute E2E tests, Our approach was to execute more Unit tests while continuing to leverage “Golden Testing”. This addition of Unit tests provided us the necessary flexibility to isolate and test individual components almost instantly, whereas continuing to use “Golden Testing” enabled the prevention of regressions.