Skip to main content
SpringerLink
Log in

Climbing the Data Mountain: Processing of SFX Data

  • Chapter
  • First Online:
X-ray Free Electron Lasers

Abstract

Serial femtosecond crystallography experiments produce mountains of data that require FEL facilities to provide many petabytes of storage space and large compute clusters for timely processing of user data. The route to reach the summit of the data mountain requires peak finding, indexing, integration, refinement, and phasing. Those who reach the summit get a crystal clear view of the “radiation damage-free” structure of a protein that is most consistent with the observed measurements. Data processing plays a critical role in the ability to measure accurate structure factor intensities from individual diffraction snapshots and combine them in three-dimensional space. Current developments in SFX aim to take into account the huge complexity of SFX experiments, modeling variations in the beam and crystals, uncertainties in geometry, partiality, mosaicity, and figures of merit that are unique to SFX.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 119.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 159.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Allahgholi, A., Becker, J., Bianco, L., Delfs, A., Dinapoli, R., Goettlicher, P., et al. (2015). AGIPD, a high dynamic range fast detector for the European XFEL. Journal of Instrumentation, 10, C01023.

    Article  Google Scholar 

  2. Bajt, S., Chapman, H. N., Spiller, E. A., Alameda, J. B., Woods, B. W., Frank, M., et al. (2008). Camera for coherent diffractive imaging and holography with a soft-x-ray free-electron laser. Applied Optics, 47, 1673–1683.

    Article  Google Scholar 

  3. Barends, T. R. M., Foucar, L., Ardevol, A., Nass, K., Aquila, A., Botha, S., et al. (2015). Direct observation of ultrafast collective motions in CO myoglobin upon ligand dissociation. Science, 350, 445–450.

    Article  CAS  Google Scholar 

  4. Barends, T. R. M., Foucar. L., Botha, S., Doak, R. B., Shoeman, R. L., Nass, K., (2014). De novo protein crystal structure determination from X-ray free-electron laser data. Nature, 505, 244–247.

    Article  CAS  Google Scholar 

  5. Barends, T. R. M., Foucar, L., Shoeman, R. L., Bari, S., Epp, S. W., Hartmann, R., et al. (2013). Anomalous signal from S atoms in protein crystallographic data from an X-ray free-electron laser. Acta Crystallographica D, 69, 838–842.

    Article  CAS  Google Scholar 

  6. Barty, A., Boutet, S., Bogan, M. J., Hau-Riege. S., Marchesini, S., Sokolowski-Tinten, K., et al. (2008). Ultrafast single-shot diffraction imaging of nanoscale dynamics. Nature Photonics, 2, 415–419. http://dx.doi.org/10.1038/nphoton.2008.128

    Article  CAS  Google Scholar 

  7. Barty, A., Kirian, R. A., Maia, F. R. N. C., Hantke, M., Yoon, C. H., White, T. A., et al. (2014). Cheetah: Software for high-throughput reduction and analysis of serial femtosecond X-ray diffraction data. Journal of Applied Crystallography, 47(3), 1118–1131.

    Article  CAS  Google Scholar 

  8. Batyuk, A., Galli, L., Ishchenko, A., Han, G. W., Gati, C., Popov, P. A., et al. (2016). Native phasing of x-ray free-electron laser data for a G protein-coupled receptor. Science Advances, 2, e1600292.

    Article  Google Scholar 

  9. Beyerlein, K., White, T. A., Yefanov, O., Gati, C., Kazantsev, I. G., Fog-Gade, N., et al. (2017). FELIX: An algorithm for indexing multiple crystallites in X-ray free-electron laser snapshot diffraction images. Journal of Applied Crystallography, 50, 1075–1083.

    Article  CAS  Google Scholar 

  10. Blaj, G., Caragiulo, P., Carini, G., Carron, S., Dragone, A., Freytag, D., et al. (2015). X-ray detectors at the Linac Coherent Light Source. Journal of Synchrotron Radiation, 22(3), 577–583. http://dx.doi.org/10.1107/S1600577515005317

    Article  CAS  Google Scholar 

  11. Boutet, S., Foucar, L., Barends, T. R. M., Botha, S., Doak, R. B., Koglin, J. E., et al. (2015). Characterization and use of the spent beam for serial operation of LCLS. Journal of Synchrotron Radiation, 22, 634–643. https://doi.org/10.1107/S1600577515004002

    Article  CAS  Google Scholar 

  12. Brehm, W., & Diederichs, K. (2014). Breaking the indexing ambiguity in serial crystallography. Acta Crystallographica Section D, 70, 101–109.

    Article  CAS  Google Scholar 

  13. Carini, G. A., Boutet, S., Chollet, M., Dragone, A., Haller, G., Hart, P. A., et al. (2014). Experience with the CSPAD during dedicated detector runs at LCLS. Journal of Physics Conference Series, 493, 012011.

    Article  Google Scholar 

  14. Casanas, A., Warshamanage, R., Finke, A. D., Panepucci, E., Olieric, V., Nöll, A., et al. (2016). EIGER detector: Application in macromolecular crystallography. Acta Crystallographica. Section D, Structural Biology, 72(9), 1036–1048. http://doi.org/10.1107/S2059798316012304

    Article  CAS  Google Scholar 

  15. Chapman, H. N., Barty, A., Bogan, M. J., Boutet, S., Frank, M., Hau-Riege, S. P., et al. (2006). Femtosecond diffractive imaging with a soft-X-ray free-electron laser. Nature Physics, 2, 839. http://dx.doi.org/10.1038/nphys461

    Article  CAS  Google Scholar 

  16. Chapman, H. N., Fromme, P., Barty, A., White, T. A., Kirian, R. A., Aquila, A., et al. (2011). Femtosecond x-ray protein nanocrystallography. Nature, 470, 73–77.

    Article  CAS  Google Scholar 

  17. Conrad, C. E., Basu, S., James, D., Wang, D., Schaffer, A., Roy-Chowdhury, S., et al. (2015). A novel inert crystal delivery medium for serial femtosecond crystallography. IUCrJ, 2, 421–430.

    Article  CAS  Google Scholar 

  18. Damiani, D., Dubrovin, M., Gaponenko, I., Kroeger, W., Lane, T. J., Mitra, A., et al. (2016). Linac Coherent Light Source data analysis using psana. Journal of Applied Crystallography, 49, 672–679.

    Article  CAS  Google Scholar 

  19. Dauter, Z. (2006). Estimation of anomalous signal in diffraction data. Acta Crystallographica Section D, 62, 867–876.

    Article  Google Scholar 

  20. Duisenberg, A. J. M. (1992). Indexing in single-crystal diffractometry with an obstinate list of reflections. Journal of Applied Crystallography, 25, 92–96.

    Article  CAS  Google Scholar 

  21. Foucar, L. (2016). CFEL-ASG Software Suite (CASS): usage for free-electron laser experiments with biological focus. Journal of Applied Crystallography, 49(4), 1336–1346.

    Article  CAS  Google Scholar 

  22. Galli, L., Son, S. K., Barends, T. R. M., White, T. A., Barty, A., Botha, S., et al. (2015). Towards phasing using high X-ray intensity. IUCrJ, 2, 627–634.

    Article  CAS  Google Scholar 

  23. Galli, L., Son, S. K., Klinge, M., Bajt, S., Barty, A., Bean, R., et al. (2015). Electronic damage in S atoms in a native protein crystal induced by an intense X-ray free-electron laser pulse. Structural Dynamics, 2, 041703.

    Article  CAS  Google Scholar 

  24. Gildea, R. J., Waterman, D. G., Parkhurst, J. M., Axford, D., Sutton, G., Stuart, D. I., et al. (2014). New methods for indexing multi-lattice diffraction data. Acta Crystallographica Section D, 70, 2652–2666.

    Article  CAS  Google Scholar 

  25. Ginn, H. M., Brewster, A. S., Hattne, J., Evans, G., Wagner, A., Grimes, J. M., et al. (2015). A revised partiality model and post-refinement algorithm for X-ray free-electron laser data. Acta Crystallographica Section D, 71, 1400–1410.

    Article  CAS  Google Scholar 

  26. Ginn, H. M., Evans, G., Sauter, N. K., & Stuart, D. I. (2016). On the release of cppxfel for processing X-ray free-electron laser images. Journal of Applied Crystallography, 49, 1065–1072.

    Article  CAS  Google Scholar 

  27. Ginn, H. M., Messerschmidt, M., Ji, X., Zhang, H., Axford, D., Gildea, R. J., (2015) Structure of CPV17 polyhedrin determined by the improved analysis of serial femtosecond crystallographic data. Nature Communications 6, 6435.

    Article  CAS  Google Scholar 

  28. Ginn, H. M., Roedig, P., Kuo, A., Evans, G., Sauter, N. K., Ernst, O., et al. (2016). TakeTwo: An indexing algorithm suited to still images with known crystal parameters. Acta Crystallographica Section D, 72, 956–965.

    Article  CAS  Google Scholar 

  29. Hattne, J., Echols, N., Tran, R., Kern, J., Gildea, R. J., Brewster, A. S., et al. (2014). Accurate macromolecular structures using minimal measurements from X-ray free-electron lasers. Nature Methods, 11, 545–548.

    Article  CAS  Google Scholar 

  30. Heisen, B. C., Boukhelef, D., Esenov, S., Hauf, S., Kozlova, I., Maia, L., et al. (2013). Karabo: An integrated software framework combining control, data management, and scientific computing tasks. In Proceedings of ICALEPCS, San Francisco.

    Google Scholar 

  31. Hunter, M. S., Segelke, B., Messerschmidt, M., Williams, G. J., Zatsepin, N. A., Barty, A., et al. (2014). Fixed-target protein serial microcrystallography with an x-ray free electron laser. Scientific Reports, 4, 6026. http://dx.doi.org/10.1038/srep06026

    Article  CAS  Google Scholar 

  32. Hunter, M. S., Yoon, C. H., DeMirci, H., Sierra, R. G., Dao, E. H., Ahmadi, R., et al. (2016). Selenium single-wavelength anomalous diffraction de novo phasing using an X-ray-free electron laser. Nature Communications, 7, 13388.

    Article  CAS  Google Scholar 

  33. Hutchison, C. D. M., Cordon-Preciado, V., Morgan, R. M. L., Nakane, T., Ferreira, J., Dorlhiac, G., et al. (2017). X-ray free electron laser determination of crystal structures of dark and light states of a reversibly photoswitching fluorescent protein at room temperature. International Journal of Molecular Sciences, 18 (1918). https://doi.org/10.3390/ijms18091918

    Article  Google Scholar 

  34. Kabsch, W. (1988). Evaluation of single-crystal x-ray diffraction data from a position-sensitive detector. Journal of Applied Crystallography, 21, 916–924.

    Article  CAS  Google Scholar 

  35. Kabsch, W. (2014). Processing of X-ray snapshots from crystals in random orientations. Acta Crystallographica Section D, 70, 2204–2216.

    Article  CAS  Google Scholar 

  36. Karplus, P. A., & Diederichs, K. (2012). Linking crystallographic model and data quality. Science, 336, 1030–1033.

    Article  CAS  Google Scholar 

  37. Kirian, R. A., Wang, X., Weierstall, U., Schmidt, K. E., Spence, J. C. H., et al. (2010). Femtosecond x-ray protein nanocrystallography — data analysis methods. Optics Express, 18, 5713–5723.

    Article  Google Scholar 

  38. Könnecke, M., Akeroyd, F. A., Bernstein, H. J., Brewster, A. S., Campbell, S. I., Clausen, B., et al. (2015). The NeXus data format. Journal of Applied Crystallography 48, 301–305. http://dx.doi.org/10.1107/S1600576714027575

    Article  Google Scholar 

  39. Kraft, P., Bergamaschi, A., Broennimann, C., Dinapoli, R., Eikenberry, E. F., Henrich, B., et al. (2009). Performance of single-photon-counting PILATUS detector modules. Journal of Synchrotron Radiation 16(3), 368–375. http://doi.org/10.1107/S0909049509009911

    Article  CAS  Google Scholar 

  40. Kroon-Batenburg, L. M. J., Schreurs, A. M. M., Ravelli, R. B. G., & Gros, P. (2015). Accounting for partiality in serial crystallography using ray-tracing principles. Acta Crystallographica Section D, 71, 1799–1811.

    Article  CAS  Google Scholar 

  41. Lyubimov, A. Y., Uervirojnangkoorn, M., Zeldin, O. B., Brewster, A. S., Murray, T. D., Sauter, N. K., et al. (2016). IOTA: Integration optimization, triage and analysis tool for the processing of XFEL diffraction images. Journal of Applied Crystallography, 49, 1057–1064.

    Article  CAS  Google Scholar 

  42. Maia, F. R. N. C. (2012). The coherent x-ray imaging data bank. Nature Methods, 9(9), 854–855. http://dx.doi.org/10.1038/nmeth.2110

    Article  CAS  Google Scholar 

  43. Mancuso, A. P., Aquila, A., Borchers, G., Giewekemeyer, K., & Reimers, N. (2013). Technical design report: scientific instrument single particles, clusters, and biomolecules (SPB). https://doi.org/10.3204/XFEL.EU/TR-2013-004

    Google Scholar 

  44. Mariani, V., Morgan, A., Yoon, C. H., Lane, T. J., White, T. A., O’Grady, C. P., et al. (2016) OnDA: Online data analysis and feedback for serial X-ray imaging. Journal of Applied Crystallography, 49(3), 1073–1080.

    Article  CAS  Google Scholar 

  45. Mozzanica, A., Bergamaschi, A., Cartier, S., Dinapoli, R., Greiffenberg, D., Johnson, I., et al. (2014) Prototype characterization of the JUNGFRAU pixel detector for SwissFEL. Journal of Instrumentation, 9, C05010.

    Article  Google Scholar 

  46. Nakane, T., Joti, Y., Tono, K., Yabashi, M., Nango, E., Iwata, S., et al. (2016). Data processing pipeline for serial femtosecond crystallography at SACLA. Journal of Applied Crystallography, 49, 1035–1041.

    Article  CAS  Google Scholar 

  47. Nakane, T., Song, C., Suzuki, M., Nango, E., Kobayashi, J., Masuda, T., et al. (2015). Native sulfur/chlorine SAD phasing for serial femtosecond crystallography. Acta Crystallographica Section D, 71, 2519–2525.

    Article  CAS  Google Scholar 

  48. Nass, K., Meinhart, A., Barends, T. R. M., Fourcar, L., Gorel, A., Aquila, A., et al. (2016). Protein structure determination by single-wavelength anomalous diffraction phasing of X-ray free-electron laser data. IUCrJ, 3, 180–191.

    Article  CAS  Google Scholar 

  49. Pande, K., Hutchison, C. D. M., Groenhof, G., Aquila, A., Robinson, J. S., Tenboer, J., et al. (2016). Femtosecond structural dynamics drives the trans/cis isomerization in photoactive yellow protein. Science, 352, 725–729.

    Article  CAS  Google Scholar 

  50. Pixel array detectors. http://bigbro.biophys.cornell.edu/research/pad. Accessed 20.11.2017.

  51. Powell, H. R. (1999). The Rossmann Fourier autoindexing algorithm in MOSFLM. Acta Crystallographica Section D, 55(10), 1690–1695. https://doi.org/10.1107/S0907444999009506

    Article  CAS  Google Scholar 

  52. Rossmann, M. G. (1979). Processing oscillation diffraction data for very large unit cells with an automatic convolution technique and profile fitting. Journal of Applied Crystallography, 12, 225–238.

    Article  CAS  Google Scholar 

  53. Rossmann, M. G., Leslie, A. G. W., Abdel-Meguid, S. S., & Tsukihara, T. (1979). Processing and post-refinement of oscillation camera data. Journal of Applied Crystallography, 12, 570–581.

    Article  CAS  Google Scholar 

  54. Sauter, N. K. (2015). XFEL diffraction: Developing processing methods to optimize data quality. Journal of Synchrotron Radiation, 22, 239–248.

    Article  CAS  Google Scholar 

  55. Sauter, N. K., Hattne, J., Brewster, A. S., Echols, N., Zwart, P. H., & Adams, P. D. (2014). Improved crystal orientation and physical properties from single-shot XFEL stills. Acta Crystallographica Section D, 70, 3299–3309.

    Article  CAS  Google Scholar 

  56. Schreurs, A. M. M., Xian, X., & Kroon-Batenburg, L. M. J. (2010). EVAL15: A diffraction data integration method based on ab initio predicted profiles. Journal of Applied Crystallography, 43, 70–82.

    Article  CAS  Google Scholar 

  57. Thayer, J., Damiani, D., Ford, C., Dubrovin, M., Gaponenko, I., O’Grady, C. P., et al. (2017). Data systems for the Linac coherent light source. Advances in Structural Chemical Imaging, 3(1), 3. http://dx.doi.org/10.1186/s40679-016-0037-7

    Article  CAS  Google Scholar 

  58. Uervirojnangkoorn, M., Zeldin, O. B., Lyubimov, A. Y., Hattne, J., Brewster, A. S., Sauter, N. K., et al. (2015). Enabling X-ray free electron laser crystallography for challenging biological systems from a limited number of crystals. eLife, 4, e05421.

    Article  Google Scholar 

  59. White, T. A. (2014). Post-refinement method for snapshot serial crystallography. Philosophical Transactions of the Royal Society B 369, 20130330.

    Article  Google Scholar 

  60. White, T. A., Barty, A., Stellato, F., Holton, J. M., Kirian, R. A., Zatsepin, N. A., et al. (2013). Crystallographic data processing for free-electron laser sources. Acta Crystallographica D, 69, 1231–1240.

    Article  CAS  Google Scholar 

  61. White, T. A., Mariani, V., Brehm, W., Yefanov, O., Barty, A., Beyerlein, K. R., et al. (2016). Recent developments in CrystFEL. Journal of Applied Crystallography, 49, 680–689.

    Article  CAS  Google Scholar 

  62. Yamashita, K., Kuwabara, N., Nakane, T., Murai, T., Mizohata, E., Sugahara, M., et al. (2017). Experimental phase determination with selenome-thionine or mercury-derivatization in serial femtosecond crystallography. IUCrJ, 4, 639–647.

    Article  CAS  Google Scholar 

  63. Yamashita, K., Pan, D., Okuda, T., Sugahara, M., Kodan, A., Yamaguchi, T., et al. (2015). An isomorphous replacement method for efficient de novo phasing for serial femtosecond crystallography. Scientific Reports, 5, 14017.

    Article  Google Scholar 

  64. Yefanov, O., Mariani, V., Gati, C., White, T. A., Chapman, H. N., Barty, A. (2015). Accurate determination of segmented X-ray detector geometry. Optics Express, 23, 28459.

    Article  Google Scholar 

  65. Yoon, C. H., DeMirci, H., Sierra, R. G., Dao, H. E., Ahmadi, R., Aksit, F., et al. (2017). Se-SAD serial femtosecond crystallography datasets from selenobiotinyl-streptavidin. Scientific Data, 4, 170055. http://dx.doi.org/10.1038/sdata.2017.55

    Article  CAS  Google Scholar 

  66. Zeldin, O. B., Brewster, A. S., Hattne, J., Uervirojnangkoorn, M., Lyubimov, A. Y., Zhou, Q., et al. (2015). Data exploration toolkit for serial diffraction experiments. Acta Crystallographica Section D, 71, 352–356.

    Article  CAS  Google Scholar 

  67. Zhu, D., Feng, Y., Stoupin, S., Terentyev, S. A., Lemke, H. T., Fritz, D. M., et al. (2014). Performance of a beam-multiplexing diamond crystal monochromator at the Linac Coherent Light Source. Review of Scientific Instruments 85(6), 063106.

    Article  Google Scholar 

Download references

Acknowledgements

TAW acknowledges the Helmholtz Association via Programme-Oriented Funds. Portions of this research were carried out at the Linac Coherent Light Source (LCLS) at the SLAC National Accelerator Laboratory. LCLS is an Office of Science User Facility operated for the US Department of Energy Office of Science by Stanford University. Use of the Linac Coherent Light Source (LCLS), SLAC National Accelerator Laboratory, is supported by the US Department of Energy, Office of Science, Office of Basic Energy Sciences under Contract No. DE-AC02-76SF00515.

Author information

Authors and Affiliations

  1. Linac Coherent Light Source, SLAC National Accelerator Laboratory, Menlo Park, CA, USA

    Chun Hong Yoon

  2. Center for Free-Electron Laser Science, German Electron Synchrotron DESY, Hamburg, Germany

    Thomas A. White

Authors
  1. Chun Hong Yoon
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thomas A. White
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Thomas A. White .

Editor information

Editors and Affiliations

  1. SLAC National Accelerator Laboratory, Menlo Park, CA, USA

    Sébastien Boutet

  2. School of Molecular Sciences, Arizona State University, Tempe, AZ, USA

    Petra Fromme

  3. SLAC National Accelerator Laboratory, Menlo Park, CA, USA

    Mark S. Hunter

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Yoon, C.H., White, T.A. (2018). Climbing the Data Mountain: Processing of SFX Data. In: Boutet, S., Fromme, P., Hunter, M. (eds) X-ray Free Electron Lasers. Springer, Cham. https://doi.org/10.1007/978-3-030-00551-1_7

Download citation

Publish with us

Policies and ethics

Search

Navigation