Acquisition and Preparation of Data for OSINT Investigations

Gibson, Helen

doi:10.1007/978-3-319-47671-1_6

Helen Gibson¹³

Part of the book series: Advanced Sciences and Technologies for Security Applications ((ASTSA))

4083 Accesses
6 Citations
3 Altmetric

Abstract

Underpinning all open-source intelligence investigations is data. Without data there is nothing to build upon, to combine, to analyse or draw conclusions from. This chapter outlines some of the processes an investigator can undertake to obtain data from open sources as well as methods for the preparation of this data into usable formats for further analysis. First, it discusses the reasons for needing to collect data from open sources. Secondly, it introduces different types of data that may be encountered including unstructured and structured data sources and where to obtain such data. Thirdly, it reviews methods for information extraction—the first step in preparing data for further analysis. Finally, it covers some of the privacy, legal and ethical good practices that should be adhered to when accessing, interrogating and using open source data.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Chapter: USD 29.95; Price excludes VAT (USA)

eBook: USD 149.00; Price excludes VAT (USA)

Softcover Book: USD 199.99; Price excludes VAT (USA)

Hardcover Book: USD 199.99; Price excludes VAT (USA)

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

1.
https://www.bellingcat.com/.
2.
https://dev.twitter.com/cards/markup.
3.
https://developers.facebook.com/docs/reference/opengraph/.
4.
https://www.w3.org/TR/html5/.
5.
https://www.w3.org/WAI/intro/aria.
6.
http://datamarket.azure.com/dataset/bing/search.
7.
https://beta.companieshouse.gov.uk.
8.
http://download.geonames.org/export.
9.
https://www.ordnancesurvey.co.uk/business-and-government/products/opendata-products-grid.html.
10.
https://gnip.com/.
11.
http://datasift.com/.
12.
www.snapchat.com.
13.
https://dev.twitter.com/rest/public.
14.
https://dev.twitter.com/streaming/overview.
15.
https://dev.twitter.com/rest/public/search.
16.
https://chrome.google.com/webstore/detail/twitter-archiver/pkanpfekacaojdncfgbjadedbggbbphi?hl=en.
17.
http://nodexl.codeplex.com/.
18.
www.facebook.com.
19.
www.linkedin.com.
20.
https://transparencytoolkit.org/project/icwatch/.
21.
https://developer.bbc.co.uk/content/bbc-platform-api/.
22.
http://open-platform.theguardian.com/.
23.
https://developer.ap.org/ap-content-api.
24.
http://developer.nytimes.com/docs/times_newswire_api/.
25.
http://emm.newsbrief.eu/overview.html.
26.
http://gdeltproject.org/.
27.
http://www.bbc.co.uk/monitoring.
28.
https://feedly.com.
29.
https://digg.com/reader.
30.
https://risk.thomsonreuters.com/products/world-check.
31.
https://www.tracesmart.co.uk/datasets.
32.
http://www.gbgplc.com/uk/products.
33.
https://www.experianplc.com/.
34.
www.torproject.org.
35.
https://tails.boum.org.
36.
http://opencatalog.darpa.mil/MEMEX.html.
37.
http://www.alchemyapi.com/.
38.
https://flipboard.com/.
39.
https://evernote.com/webclipper/.
40.
https://github.com/GravityLabs/goose.
41.
https://pypi.python.org/pypi/goose-extractor/.
42.
http://www.alchemyapi.com/products/alchemylanguage/text-extraction.
43.
http://aylien.com/article-extraction/.
44.
http://www.rosette.com/.
45.
http://nlp.stanford.edu/software/.
46.
http://www.opener-project.eu/.
47.
http://www.sas.com/en_us/software/analytics.html#text-analytics.
48.
http://www-03.ibm.com/software/products/en/i2-analyze.
49.
https://www.paterva.com/web6/.
50.
https://www.palantir.com/.
51.
http://www.textronsystems.com/products/advanced-information/axis-pro.
52.
http://osirtbrowser.com/.
53.
http://sotera.github.io/Datawake/.
54.
2008. Data Protection Directive 95/46/EC—EUR-Lex—Europa.eu. http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=CELEX:31995L0046:en:HTML.
55.
2011. Protection of personal data—European Commission. http://ec.europa.eu/justice/data-protection/.

References

Bayerl PS, Akhgar B (2015) Surveillance and falsification implications for open source intelligence investigations. Commun ACM 58(8):62–69
Article Google Scholar
Bazzell M (2016) Open source intelligence techniques: resources for searching and analyzing online information. CCI Publishing
Google Scholar
Bird S (2006) NLTK: the natural language toolkit. In: Proceedings of the COLING/ACL on interactive presentation sessions. Association for Computational Linguistics, July 2006, pp 69–72
Google Scholar
Bradbury D (2011) In plain view: open source intelligence. Comput Fraud Secur 2011(4):5–9
Article Google Scholar
Cavoukian A (2011) 7 Foundational principles of privacy by design. https://www.ipc.on.ca/images/Resources/7foundationalprinciples.pdf
Chen H (2011) Dark Web: exploring and mining the dark side of the web. In: 2011 European intelligence and security informatics conference (EISIC). IEEE, Sept 2011, pp 1–2
Google Scholar
College of Policing (2013) Investigation process. In: Authorised professional practice. https://www.app.college.police.uk/app-content/investigations/investigation-process/#material
College of Policing (2015) Intelligence cycle. In: Authorised professional practice. https://www.app.college.police.uk/app-content/intelligence-management/intelligence-cycle/
Cunningham H, Tablan V, Roberts A, Bontcheva K (2013) Getting more out of biomedical documents with GATE’s full lifecycle open source text analytics. PLoS Comput Biol 9(2):e1002854
Article Google Scholar
DARPA (2014) Memex aims to create a new paradigm for domain-specific search. In: Defense Advanced Research Projects Agency. http://www.darpa.mil/news-events/2014-02-09
Defense Technical Information Center (DTIC), Department of Defense (2007) Joint intelligence. http://www.dtic.mil/doctrine/new_pubs/jp2_0.pdf
FBI Intelligence Cycle (n.d.) In: Federal Bureau of Investigation. https://www.fbi.gov/about-us/intelligence/intelligence-cycle
Finkel JR, Grenager T, Manning C (2005) Incorporating non-local information into information extraction systems by Gibbs sampling. In: Proceedings of the 43rd annual meeting on Association for Computational Linguistics. Association for Computational Linguistics, June 2005, pp 363–370
Google Scholar
Fu T, Abbasi A, Chen H (2010) A focused crawler for Dark Web forums. J Am Soc Inform Sci Technol 61(6):1213–1231
Google Scholar
Gibson S (2004) Open source intelligence. RUSI J 149:16–22
Article Google Scholar
Greenwald G, MacAskill E, Poitras L (2013) Edward Snowden: the whistleblower behind the NSA surveillance revelations. In: The guardian. http://www.theguardian.com/world/2013/jun/09/edward-snowden-nsa-whistleblower-surveillance
Hansen D, Shneiderman B, Smith MA (2010) Analyzing social media networks with NodeXL: insights from a connected world. Morgan Kaufmann, Los Altos
Google Scholar
HMIC (Her Majesty’s Inspectorate of Constabulary) (2011) The rules of engagement: a review of the August 2011 riots. https://www.justiceinspectorates.gov.uk/hmic/media/a-review-of-the-august-2011-disorders-20111220.pdf
Hoepman JH (2014) Privacy design strategies. In: IFIP international information security conference. Springer, Berlin, June 2014, pp 446–459
Google Scholar
Imran M, Elbassuoni S, Castillo C, Diaz F, Meier P (2013) Practical extraction of disaster-relevant information from social media. In: Proceedings of the 22nd international conference on World Wide Web. ACM, May 2013, pp 1021–1024
Google Scholar
Lohr S (2014) For big-data scientists, “Janitor Work” is key hurdle to insights. In: The New York Times. http://mobile.nytimes.com/2014/08/18/technology/for-big-data-scientists-hurdle-to-insights-is-janitor-work.html?_r=2
Madhavan J, Ko D, Kot Ł, Ganapathy V, Rasmussen A, Halevy A (2008) Google’s deep web crawl. Proc VLDB Endowment 1(2):1241–1252
Article Google Scholar
Manning CD, Surdeanu M, Bauer J, Finkel JR, Bethard S, McClosky D (2014) The Stanford CoreNLP Natural Language Processing Toolkit. In ACL (System Demonstrations), June 2014, pp 55–60
Google Scholar
Mercado SC (2009) Sailing the sea of OSINT in the information age. Secret Intell Reader 78
Google Scholar
NATO (2001) NATO open source intelligence handbook
Google Scholar
Omand D, Bartlett J, Miller C (2012) Introducing social media intelligence (SOCMINT). Intell Natl Secur 27(6):801–823
Article Google Scholar
Pallaris C (2008) Open source intelligence: a strategic enabler of national security. CSS Analyses Secur Policy 3(32):1–3
Google Scholar
Rogers C, Lewis R (eds) (2013) Introduction to police work. Routledge, London
Google Scholar
Shein E (2013) Ephemeral data. Commun ACM 56:20
Article Google Scholar
Warden P (2010) How I got sued by Facebook. In: Pete Warden’s blog. https://petewarden.com/2010/04/05/how-i-got-sued-by-facebook/

Download references

Author information

Authors and Affiliations

CENTRIC/Sheffield Hallam University, Sheffield, UK
Helen Gibson

Authors

Helen Gibson
View author publications
You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Helen Gibson .

Editor information

Editors and Affiliations

School of Computing and Management Sciences, Sheffield Hallam University, Sheffield, United Kingdom
Babak Akhgar
Rotterdam School of Management, Erasmus University, Rotterdam, The Netherlands
P. Saskia Bayerl
Office of the Police and Crime Commissioner for West Yorkshire, Wakefield, United Kingdom
Fraser Sampson

Rights and permissions

Reprints and permissions

Copyright information

About this chapter

Cite this chapter

Gibson, H. (2016). Acquisition and Preparation of Data for OSINT Investigations. In: Akhgar, B., Bayerl, P., Sampson, F. (eds) Open Source Intelligence Investigation. Advanced Sciences and Technologies for Security Applications. Springer, Cham. https://doi.org/10.1007/978-3-319-47671-1_6

Download citation

DOI: https://doi.org/10.1007/978-3-319-47671-1_6
Published: 03 January 2017
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-47670-4
Online ISBN: 978-3-319-47671-1
eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics