Logistics forecasting with flow simulators for port operations

Context and challenge

A general cargo and container port operation on the Pacific coast managed its operational planning with manual methods and spreadsheets. Berth, crane, and yard assignment planning was done per shift with a maximum horizon of 8 hours, leaving the operational team unable to anticipate congestion and react with sufficient lead time. Vessel waiting times in the bay averaged 6.4 hours, peaking at 18 hours during high-demand periods, generating contractual penalties and damaging relationships with shipping lines.

The root of the problem was structural: maritime and land cargo flow involves at least six interdependent variables (vessel ETAs, cargo type and volume, berth availability, crane assignment, yard capacity, and truck pickup scheduling) that a human team cannot simultaneously optimize with a 72-hour horizon. Each locally suboptimal decision generated cascading effects that only became visible 4 to 8 hours later, by which time it was too late to react.

Data landscape & sources

The project required integrating data sources with radically different origins and formats: internal port transactional systems, international maritime feeds, and land transport data. The historical volume analyzed covered 3 years of operations with more than 12,000 vessel calls and 2.8 million container moves recorded in the TOS.

• TOS (Terminal Operating System, Navis N4): container moves, yard slot allocation, crane scheduling, and stowage plans with real-time updates
• Maritime AIS (MarineTraffic API + local AIS receiver): position, speed, and ETA of 800+ active vessels in the region updated every 2 minutes
• Cargo manifests and customs documentation: type, weight, and volume of declared cargo per vessel with 24-96 hours advance notice depending on service type
• Truck scheduling systems: pickup windows, license plates, and expected land transport volumes with hourly granularity
• Maritime meteorology: sea state, wind, and visibility to model effective berth availability and ship crane performance
• Operational performance history: crane cycle times, productivity per berth, equipment breakdown rates, and 36-month operational incident records

Methodology & analysis

The analysis began with reconstructing the 500+ historical port congestion scenarios identified in the 36-month records. Each scenario was decomposed into its root causes — combinations of simultaneous vessel arrivals, equipment failures, land transport delays, and planning errors — to build the dependency model feeding the flow simulator. Causal analysis work with shift supervisors was essential: many congestion events had tacit explanations not documented in any system.

• Analysis of 500+ historical congestion scenarios with causal decomposition and classification by type and severity
• Vessel ETA modeling with Prophet (trend component + weekly and annual seasonality) combined with a custom LSTM to capture delay patterns dependent on route type and shipping line; MAE=1.8 hours at 72-hour horizon
• Berth and crane assignment optimization with Mixed Integer Linear Programming (MILP, CBC solver via PuLP) over 4-hour windows with re-optimization every 30 minutes
• Yard congestion prediction with XGBoost on features of current occupancy, expected arrivals, and pickup scheduling; precision=83%, recall=78% for congestion alerts > 85% occupancy
• Contingency scenario simulation with Monte Carlo over empirical distributions of operating times, crane breakdowns, and land transport delays
• Berth interdependency analysis with dependency graphs to model cascade effects of delays between adjacent berthing positions

Model architecture / technical design

• Ingestion and synchronization: native connector to Navis N4 via REST API; AIS consumer with local buffer for connectivity fault tolerance; operational state updated every 90 seconds; on-premise deployment with no data leaving the port perimeter
• Predictive engine: Prophet for ETA time series with confidence intervals; LSTM (2 layers, 128 units, dropout=0.3) trained per route and shipping line to capture specific patterns; weighted ensemble with RMSE as weighting criterion
• Assignment optimizer: MILP with 200-400 decision variables per 4-hour window; CBC solver with 45-second time limit guaranteeing feasible solution; automatic re-optimization when ETA changes exceed 30 minutes
• Contingency simulator: 500+ scenarios pre-computed nightly with Monte Carlo; response time < 3 seconds for existing scenario query; ad-hoc new scenario generation in < 2 minutes
• Operations panel: real-time web dashboard with berth, yard, and crane occupancy map; configurable threshold alerts; 72-hour visualization horizon updated every 5 minutes
• Infrastructure: client on-premise servers (port security requirement); Linux/Docker stack; TimescaleDB for time series; automatic backup and failover to secondary server with RTO < 15 minutes

Implementation details

The biggest technical challenge was integrating with Navis N4 TOS in real-time read mode without impacting the port's critical operational system. The N4 REST API has strict rate limits and does not support webhooks, requiring an intelligent priority-based poller: berths with active vessel operations are queried every 30 seconds, while yard slots with no expected movement in the next 2 hours are updated every 5 minutes. The AIS connector presented a different challenge: international feeds have variable latencies of up to 8 minutes, so a local VHF AIS receiver was implemented to complement the external feed with high-frequency coverage in the approach area.

The on-premise deployment mandated by port security requirements demanded a high-availability design without public cloud dependency. A primary/secondary server scheme was implemented with synchronous TimescaleDB replication and automatic failover via Keepalived. Machine learning models are retrained weekly on the secondary server during low-load windows and promoted to primary via blue-green deployment with automatic metric validation before promotion.

Validation & testing

The system operated in shadow mode for 8 weeks before go-live, generating alternative assignment plans in parallel with shift planners' decisions. 312 significant divergences between the system plan and the manual plan were documented, of which 71% favored the system's solution when evaluated retrospectively. The vessel ETA model was validated against a 180-day test set with MAE=1.8 hours at the 72-hour horizon and MAE=0.9 hours at the 24-hour horizon, outperforming the expert planner benchmark (MAE=3.1h and 1.4h respectively) in both cases. The MILP optimizer was backtested against the 500+ historical congestion scenarios, resolving 94% of cases with assignments that would have prevented the documented congestion.

A 3-week user acceptance test (UAT) was conducted with shift supervisors across all three port shifts. The main UAT objections centered on extreme edge cases not covered by the model (general power failures, strikes) for which a manual override mode was added with documented justification logging. The failover system stress test simulated a primary server failure during operations, verifying automatic recovery in 8 minutes with zero data loss thanks to synchronous replication.

Results & business impact

• ✓-28% average vessel waiting time in the bay (from 6.4h to 4.6h in the first 4 months)
• ✓+34% berth infrastructure utilization through MILP assignment optimization with re-optimization every 30 minutes
• ✓100% elimination of contractual delay penalties in the first full quarter of system operation
• ✓Vessel ETA prediction MAE: 1.8h at 72h horizon vs 3.1h manual benchmark (42% improvement)
• ✓500+ contingency scenarios available in < 3 seconds, enabling informed operational response to any deviation
• ✓ROI calculated at 6.2x in 12 months accounting for penalty elimination, throughput increase, and planning overtime reduction

Key lessons learned

• ✓A 72-hour predictive horizon is only useful if the ETA model outperforms the human benchmark across the entire horizon — validating this before go-live is non-negotiable
• ✓TOS integration in read mode without operational impact requires an intelligent priority-based polling design; a naive poller generates unnecessary load and can degrade the critical system
• ✓On-premise deployment imposes high-availability constraints that must be designed from the start — adding failover after the fact is far more costly and risky than including it in the original design
• ✓Expert planners hold critical tacit knowledge about edge cases that no data system captures: the UAT phase is the only opportunity to extract that knowledge and convert it into documented override rules
• ✓The MILP optimizer must have a strict time limit with guaranteed feasible solution — in port operations, a suboptimal solution in 45 seconds is infinitely better than the optimal solution in 10 minutes that arrives too late
• ✓International AIS feeds have variable and unpredictable latencies; a local AIS receiver is a small investment that eliminates dependency on a critical external single point of failure for the ETA model