Five pitfalls of empirical scheduling research

  • J. Christopher Beck
  • Andrew J. Davenport
  • Mark S. Fox
Session 6
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1330)


A number of pitfalls of empirical scheduling research are illustrated using real experimental data. These pitfalls, in general, serve to slow the progress of scheduling research by obsfucating results, blurring comparisons among scheduling algorithms and algorithm components, and complicating validation of work in the literature. In particular, we look at difficulties brought about by viewing algorithms in a monolithic fashion, by concentrating on CPU time as the only evaluation criteria, by failing to prepare for gathering of a variety of search statistics at the time of experimental design, by concentrating on benchmarks to the exclusion of other sources of experimental problems, and, more broadly, by a preoccupation with optimization of makespan as the sole goal of scheduling algorithms.


Schedule Problem Schedule Algorithm Constraint Satisfaction Search Statistic Constraint Graph 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. AAAI Empirical Workshop (1994). Proceedings of the AAAI-94 Workshop on Experimental Evaluation of Reasoning and Search Methods.Google Scholar
  2. Allen, J. F. (1983). Maintaining knowledge about temporal intervals. Communications of the ACM, 26(11):832–843.Google Scholar
  3. Beasley, J. E. (1990). OR-library: distributing test problems by electronic mail. Journal of the Operational Research Society, 41(11):1069–1072. Also available by ftp from ftp:// Scholar
  4. Beck, J. C. (1994). A schema for constraint relaxation with instantiations for partial constraint satisfaction and schedule optimization. Master's thesis, Department of Computer Science, University of Toronto.Google Scholar
  5. Beck, J. C. (1997). A generic framework for constraint-directed search and scheduling. Technical report, Department of Industrial Engineering, University of Toronto, 4 Taddle Creek Road, Toronto, Ontario M5S 3G9, Canada.Google Scholar
  6. Beck, J. C., Davenport, A. J., Sitarski, E. M., and Fox, M. S. (1997a). Beyond contention: extending texture-based scheduling heuristics. In Proceedings of AAAI-97. AAAI Press, Menlo Park, California.Google Scholar
  7. Beck, J. C., Davenport, A. J., Sitarski, E. M., and Fox, M. S. (1997b). Texture-based heuristics for scheduling revisited. In Proceedings of AAAI-97. AAAI Press, Menlo Park, California.Google Scholar
  8. Beck, J. C. and Jackson, K. (1997). Constrainedness and the phase transition in job shop scheduling. Technical report, School of Computing Science, Simon Fraser University.Google Scholar
  9. Brucker, P. and Thiele, O. (1996). A branch & bound method for the general-shop problems with sequence dependent set-up times. OR Spektrum, 18:145–161.Google Scholar
  10. Carlier, J. and Pinson, E. (1989). An algorithm for solving the job-shop problem. Management Science, 35(2):164–176.Google Scholar
  11. Caseau, Y. and Laburthe, F. (1996). Cumulative scheduling with task intervals. In Proceedings of the Joint International Conference and Symposium on Logic Programming. MIT Press.Google Scholar
  12. Cheeseman, P., Kanefsky, B., and Taylor, W. (1991). Where the really hard problems are. In yProceedings of IJCAI-91, volume 1, pages 331–337.Google Scholar
  13. Cheng, C. C. and Smith, S. F. (1996). Applying constraint satisfaction techniques to job shop scheduling. Annals of Operations Research, Special Volume on Scheduling: Theory and Practice, 1. Forthcoming.Google Scholar
  14. Cohen, P. R. (1995). Empirical Methods for Artificial Intelligence. The MIT Press, Cambridge, Mass.Google Scholar
  15. Davenport, A. J., Beck, J. C., and Fox, M. S. (1997). Propagation over the meets temporal constraint. Technical report, Department of Industrial Engineering, University of Toronto.Google Scholar
  16. Drummond, M., Bresina, J., and Swanson, K. (1994). Just-in-case scheduling. In Proceedings of AAAI-94, pages 1098–1104, Menlo Park, CA. AAAI Press/MIT Press.Google Scholar
  17. ECAI Empirical Workshop (1996). Proceedings of the ECAI-96 Workshop on Empirical Artificial Intelligence.Google Scholar
  18. Erschler, J., Roubellat, F., and Vernhes, J. P. (1976). Finding some essential characteristics of the feasible solutions for a scheduling problem. Operations Research, 24:772–782.Google Scholar
  19. Erschler, J., Roubellat, F., and Vernhes, J. P. (1980). Characterising the set of feasible sequences for n jobs to be carried out on a single machine. European Journal of Operational Research, 4:189–194.Google Scholar
  20. Fox, M. S. (1983). Constraint-Directed Search: A Case Study of Job-Shop Scheduling. PhD thesis, Carnegie Mellon University, Intelligent Systems Laboratory, The Robotics Institute, Pittsburgh, PA. CMU-RI-TR-85-7.Google Scholar
  21. Fox, M. S. (1990). Constraint-guided scheduling-a short history of research at CMU. Computers in Industry, 14:79–88.Google Scholar
  22. Gent, I. P., Grant, S. A., MacIntyre, E., Prosser, P., Shaw, P., Smith, B. M., and Walsh, T. (1997). How not to do it. Technical Report 97.27, School of Computer Studies, University of Leeds.Google Scholar
  23. Gent, I. P., MacIntyre, E., Prosser, P., and Walsh, T. (1996). The constrainedness of search. In Proceedings ofAAAI-96, volume 1, pages 246–252.Google Scholar
  24. Harvey, W. D. (1995). Nonsystematic backtracking search. PhD thesis, Department of Computer Science, Stanford University.Google Scholar
  25. Harvey, W. D. and Ginsberg, M. L. (1995). Limited discrepancy search. In Proceedings of IJCAI-95, pages 607–613.Google Scholar
  26. Hildum, D. W. (1994). Flexibility in a knowledge-based system for solving dynamic resourceconstrained scheduling problems. PhD thesis, Department of Computer Science, University of Massachusetts, Amherst, MA. 01003-4610. UMass CMPSCI TR 94-77.Google Scholar
  27. Hooker, J. N. (1994). Needed: An empirical science of algorithms. Operations Research, 42:201–212.Google Scholar
  28. Hooker, J. N. (1996). Testing heuristics: We have it all wrong. Journal ofHeuristics, 1:33–42.Google Scholar
  29. Le Pape, C. (1994a). Implementation of resource constraints in ILOG Schedule: A library for the development of constraint-based scheduling systems. Intelligent Systems Engineering, 3(2):55–66.Google Scholar
  30. Le Pape, C. (1994b). Using a constraint-based scheduling library to solve a specific scheduling problem. In Proceedings of the AAAI-SIGMAN Workshop on Artificial Intelligence Approaches to Modelling and Scheduling Manufacturing Processes.Google Scholar
  31. Lhomme, O. (1993). Consistency techniques for numeric CSPs. In Proceedings of IJCAI-93, volume 1, pages 232–238.Google Scholar
  32. Muscettola, N. (1992). Scheduling by iterative partition of bottleneck conflicts. Technical Report CMU-RI-TR-92-05, The Robotics Institute, Carnegie Mellon University.Google Scholar
  33. Nuijten, W. and Aarts, E. (1997). A computational study of constraint satisfaction for multiple capacitated job shop scheduling. European Journal of Operational Research. To appear.Google Scholar
  34. Nuijten, W. P. M. (1994). Time and resource constrained scheduling: a constraint satisfaction approach. PhD thesis, Department of Mathematics and Computing Science, Eindhoven University of Technology.Google Scholar
  35. Ow, P. S., Smith, S. F., and Thiriez, A. (1988). Reactive plan revision. In Proceedings of AAAI-88, pages 77–82. AAAI.Google Scholar
  36. Sadeh, N. (1991). Lookahead techniques for micro-opportunistic job-shop scheduling. PhD thesis, Carnegie-Mellon University. CMU-CS-91-102.Google Scholar
  37. Saks, V. (1992). Distribution planner overview. Technical report, Carnegie Group, Inc., Pittsburgh, PA, 1522.Google Scholar
  38. Smith, S. F. and Cheng, C. C. (1993). Slack-based heuristics for constraint satisfaction scheduling. In Proceedings AAAI-93, pages 139–144.Google Scholar
  39. Taillard, E. (1993). Benchmarks for basic scheduling problems. European Journal of Operational Research, 64:278–285.Google Scholar
  40. Xiong, Y., Sadeh, N., and Sycara, K. (1992). Intelligent backtracking techniques for job-shop scheduling. In Proceedings of the Third International Conference on Principles of Knowledge Representation and Reasoning, Cambridge, MA. Google Scholar
  41. Zweben, M., Davis, E., Daun, B., and Deale, M. (1993). Informedness vs. computational cost of heuristics in iterative repair scheduling. In Proceedings of IJCAI-93, pages 1416–1422.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1997

Authors and Affiliations

  • J. Christopher Beck
    • 1
  • Andrew J. Davenport
    • 2
  • Mark S. Fox
    • 1
    • 2
  1. 1.Department of Computer ScienceUniversity of TorontoTorontoCanada
  2. 2.Department of Industrial EngineeringUniversity of TorontoTorontoCanada

Personalised recommendations