한국어Email Jake

Delivery API gateway for an e-commerce platform

Jake Tae-moon Jo · Engineering lead · https://jake-jo-portfolio.vercel.app/en/work/commerce-delivery-gateway/

Put four local couriers' different authentication, address codes and status codes behind one API, so the client's service only ever calls the gateway.

local couriers integrated
4
first commit to first staging
77 days
first commit to production record
262 days
common delivery statuses
9
Email JakeDownload summary PDF

Problem and context

The client's delivery management service is a product used on an e-commerce platform. It had to create, cancel and track shipments with several local couriers. Every courier had its own authentication, request format and status codes, and authentication alone came in four kinds.

1 more paragraph

Addresses did not match either. A shop order carries the address as free text with administrative area names, while each courier's API wanted its own area codes. The service needed a server in the middle to absorb all of these differences.

Project facts
Period
Oct 2020 – Dec 2021
Role
Sole developer (design, build, tests, deployment, docs)
Team
Just me (all 217 commits on main)
Platforms
REST API server (staging and production)
Stack
Node.js · Express · Sequelize · MariaDB · Passport · JWT · axios · winston · PM2 · Mocha · Chai
Context
Integration between a client's delivery management service, used on an e-commerce platform, and several local couriers in Southeast Asia

System overview

The delivery management service calls a single gateway API, and the adapters, address code lookup and status mapping absorb every courier-specific difference.

Click a node, or Tab to it and press Enter, to highlight its note. Dashed boxes are external systems.

REST · JWTencryptedname → codeHTTPSstatus webhookDelivery serviceClient productMariaDBSequelize · field encryptionGateway APIExpress · JWTWebhook intakeCourier A statusAddress lookupIn memory · 5 tablesCourier adapters4 × create/cancel/trackCouriers A–DLocal courier APIsStatus mapping9 common statuses
  • Delivery service → Gateway API: REST · JWT
  • Gateway API → MariaDB: encrypted
  • Gateway API → Courier adapters
  • Courier adapters → Address lookup: name → code
  • Courier adapters → Couriers A–D: HTTPS
  • Courier adapters → Status mapping
  • Couriers A–D → Webhook intake: status webhook
Node notes
The client's product, used on an e-commerce platform. After sign-up and login it registers each shop's courier credentials, then asks for waybills to be issued, tracked or cancelled. The API contract was agreed with the client in six sequence diagrams.
MariaDB with Sequelize: 5 models (user, courier, credentials, shipment, item), 20 migrations and 4 seeds. 35 fields of personal data and courier credentials are AES-encrypted just before writing and decrypted on read.
An Express REST API: /api/health plus 13 routes under /api/v1, 14 endpoints in all. It authenticates with JWT (passport-jwt), and login has brute-force protection (backed by the database) and password strength rules. Responses carry a request ID header, and logs record it. It runs as a PM2 cluster.
Receives status webhooks from courier A, converts them to a common status and updates the stored shipment status right away. Separately, when a shipment is looked up and its last update is more than an hour old (configurable), the gateway asks the courier again.
At startup the server loads 5 per-courier address code tables into memory. Each request turns area names into that courier's codes. It looks for an exact match first, then falls back to similarity matching; below the thresholds (middle administrative level 0.89, lowest level 0.95) it returns 400.
Each courier has its own request builder and client. All four expose create, cancel and track in the same shape. There is one set of credentials per shop and courier, in one of 4 schemes: fixed token, ID and password, ID and password plus customer code, or customer ID.
APIs of four local couriers. Courier A also sends status webhooks, B uses token login, and C signs requests with a key. A fifth courier only got credential registration and its address code table.
Maps each courier's own status codes to 9 common statuses: ready, picked up, cancelled, in transit, delivery failed, returning, returned, delivered and exception. Anything that doesn't match becomes unknown.

What I built

  • Agreed the API contract first: six sequence diagrams, worked out with the client, covering sign-up and auth, registering and reading credentials, courier registration, waybill issue, the status-change hook and cancellation.
  • Built 14 REST API endpoints. Authentication uses JWT, and login has brute-force protection and password strength rules. 35 fields of personal data and courier credentials are AES-encrypted just before saving.
1 more
  • Gave four courier adapters one shape. Each courier has its own request builder and client, and create, cancel and track all come out the same. Credentials are stored per shop and courier, in four schemes.
2 more
  • Mapped every courier's status codes to 9 common statuses. The stored status is refreshed two ways: a lookup re-asks the courier only if the last update is more than an hour old, and courier A's status webhooks are applied as soon as they arrive.
  • Added what operations needed: request IDs in response headers and logs, daily rotating logs, Swagger UI turned off in production, a PM2 cluster, and deploy scripts for staging and production. Tests were written alongside the code from three days in; there are 79 active test cases.

Key decisions

An address below the threshold is rejected, not sentDecision I wrote two separate tools and pulled them in: an extractor that fetches the code tables and a lookup library that finds the codes.Result The thresholds have not changed since November 2020, and the production patch of June 2021 refreshed every courier's address tables.
Problem
A shop order carries the address as free text with administrative area names, while each courier's API wanted its own area codes. Even getting those code tables differed by courier: three offered an API, two only spreadsheets.
Decision
I wrote two separate tools and pulled them in: an extractor that fetches the code tables and a lookup library that finds the codes. The extractor strips spelling differences such as administrative unit words, then writes each table to CSV. It looks for an exact match first, then falls back to similarity matching. The thresholds are 0.89 for the middle administrative level and 0.95 for the lowest, and the top level must match exactly. Below the threshold nothing goes to the courier; the gateway returns 400 with per-field errors.
Result
The thresholds have not changed since November 2020, and the production patch of June 2021 refreshed every courier's address tables. Unit test cases in the library describe both cases: slightly misspelt middle- and lowest-level names are still found, and a wrong address is not.
Cost & remaining risk
Only the single highest score counts; the gap between first and second place is not checked. The chosen lowest administrative level is not returned in the response, so the caller cannot see which one an address was matched to. The gateway has no step where a person confirms the match, and why these thresholds were chosen is not recorded.

How one waybill gets issued

Before the courier call the gateway rewrites the address and request in that courier's format, and after it turns the reply back into a common shape, so every courier takes the same request and gives the same response.

  1. Issue requestExternal

    The delivery service sends a JWT plus order, address and items (POST /api/v1/shippings)

  2. Validate inputSystem

    Required fields, phone number format

  3. Find credentialsSystem

    The ones this shop registered for this courier; 404 if none

  4. Convert to courier formatSystem

    Missing buyer name filled with the recipient's, area names turned into that courier's codes (400 if not found), phone number normalised

  5. Call courier APIExternal

    Through that courier's adapter

  6. Normalise replySystem

    Waybill number and fee in a common shape

  7. SaveSystem

    Shipment and items stored, personal data fields encrypted

  8. RespondSystem

    The stored shipment returned as JSON

Results

The first staging deploy went up with courier A already done, and the four couriers were added one by one over four months.

  • Shipped the first staging deploy after finishing create, track and cancel for courier A.
  • Added the four couriers one at a time, so the delivery service could handle any of them with the same request and the same response.
1 more result
  • The first record of running in production (a production patch note) came on day 262.
1 more result
  • Kept going after that: an item model and updated courier requests let one shipment carry several items.

This page covers only what the code and commit history confirm. Production scale and how long it ran in production are left out.

From first commit to the first production record

First staging deploy on day 77; couriers were added one at a time, and the first production patch note came on day 262.

to the first staging deploy
77 days
to the first production patch note
262 days
2021AprJulOct123456789To first staging · 77 daysFour couriers · 127 days
  1. Development starts
  2. Courier A

    Create, track, cancel; done Dec 11

  3. First staging deploy

    Day 77

6 more
  1. Courier B
  2. Courier C

    Acceptance check Mar 24

  3. Courier D

    Through Apr 15

  4. Production readiness

    Brute-force protection, phone validation, production deploy script, InnoDB (through May 20)

  5. Production patch note

    First record of running in production; day 262

  6. Multiple items per shipment

    Last main commit Dec 29