Combining Machine Learning and Statistical Disclosure Control to Promote Open Data

  • Nasca PengEmail author
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 996)


We proposed machine learning-oriented statistical disclosure control as a novel solution for New Zealand Transport Agency to release more person-related variables in its open crash data for privacy-preserving data mining. Instead of making arbitrary decisions in variable aggregation and using perturbation to guard against reidentification attacks at the cost of data distortion, we creatively drew upon feature engineering and dimensionality reduction techniques such as correspondence analysis to make evidence-based data manipulation without distortion. In addition, we built random forest classifiers along the way to directly monitor the impact of our data manipulation on users’ modelling, rather than relying on traditional utility metrics such as information loss and difference of eigenvalues that are less interpretable for users. The dataset produced using our method satisfied 10-anonymity based on 11 quasi-identifiers, with less than 3% suppression, compared with only 3-anonymity based on no more than 8 quasi-identifiers with far more than 3% suppression commonly reported in literature. Furthermore, our method enabled random forest classifier to achieve 0.996 for AUC and 0.895 for F-score in predicting crash severity.


Statistical disclosure control Privacy-preserving data mining Confidentiality Dimensionality reduction 


  1. 1.
    Skinner, C.: Statistical disclosure control for survey data. In: Handbook of Statistics, pp. 381–396. Elsevier, Amsterdam (2009)Google Scholar
  2. 2.
    Sweeney, L., Samarati, P.: Protecting privacy when disclosing information: k-anonymity and its enforcement through generalization and suppression, Computer Science Laboratory, SRI International (1998)Google Scholar
  3. 3.
    Elliot, M.J., Manning, A., Mayes, K., Gurd, J., Bane, M.: SUDA: a program for detecting special uniques. In: Joint UNECE/Eurostat Work Session on Statistical Data Confidentiality, Geneva (2005)Google Scholar
  4. 4.
    Mateo-Sanz, J.M., Domingo-Ferrer, J., Sebé, F.: Probabilistic information loss measures in confidentiality protection of continuous microdata. Data Min. Knowl. Discov. 11(2) 181–193 (2005)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Templ, M., Kowarik, A., Meindl, B.: Statistical disclosure control for micro-data using the R package sdcMicro. J. Stat. Softw. 67(1), 1–36 (2015)Google Scholar
  6. 6.
  7. 7.
  8. 8.
    Ho, T.K.: Random decision forests. In: Proceedings of 3rd International Conference on Document Analysis and Recognition (1995)Google Scholar
  9. 9.
    Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321–357(2002)CrossRefGoogle Scholar
  10. 10.
    Kassambara, A.: Practical Guide to Cluster Analysis in R: Unsupervised Machine Learning, STHDA (2017)Google Scholar
  11. 11.
    Bendixen, M.: A practical guide to the use of correspondence analysis in marketing research. Mark. Bull. 14, 16–38 (2003)Google Scholar
  12. 12.
    Jaro, M.A.: Advances in record linkage methodology as applied to the 1985 census of Tampa Florida. Stat. Med. 14, 491–498 (1989)Google Scholar
  13. 13.
    Winkler, W.E.: String comparator metrics and enhanced decision rules in the fellegi-sunter model of record linkage. In: Proceedings of the Section on Survey Research Methods (1990)Google Scholar
  14. 14.
    Blei, D.M., Ng, A.Y., Jordan, M.: Latent dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022(2003)Google Scholar
  15. 15.
    Machanavajjhala, A., Gehrke, J., Kifer, D., Venkitasubramaniam, M.: L-diversity: privacy beyond k-anonymity. In: 22nd International Conference on Data Engineering (ICDE 2006) (2006)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  1. 1.Statistics New ZealandWellingtonNew Zealand

Personalised recommendations