Time Discrepant Shipments in Manifest Data

Abstract

Manifest data is a log of container shipments from foreign lading ports to U.S. unlading ports. We provide several time varying network-based representations of this data in order to extract its most “discrepant” port pairs and contents patterns. We treat this time varying network representation as a combinatorial set system and use its discrepancy and firing rate (Abello et al. (2010) Detecting Novel Discrepancies in Communications Networks, International Conference on Data Mining, ICDM 2010: 8–17, Sydney, Australia and Chazelle (2000) The Discrepancy Method: Randomness and Complexity, Cambridge University Press) as the main statistics to track the most “salient” network elements. The output of the entire process is a “fossil” sub-network that encodes those port pairs and contents that exhibit unusual time varying patterns. It is expected that substantial deviations from these patterns will be useful triggers for further content inspections. The applicability of the proposed techniques is not limited to manifest data.

Appendices

Appendix 1: Sample Records of Original Data

5.1.1 Human Readable version

5.1.2 Corresponding Original Text Data

Appendix 2: Sample Fossil Information as Seen in Figs. 5.6 and 5.7

Recall that an edge is selected to be in the i-fossil at time t if its weight is at least i-standard deviations away from the overall data discrepancy at that time. The following are samples of edges in the i-fossil, where i is the maximum number of standard deviations at time t. Our fossil frequency calculations ignore timestamp 1 (denoted as FOS_FREQ in the tables below).

5.1.1 Ascend Descend Coloring Fossil

5.1.2 Random Coloring Fossil

Appendix 3: Salient Edges Selected by Both Coloring Schemes

The following are examples of edges that are included in the fossil with respect to both random coloring and ascend descend coloring.