The Needle in a Haystack Problem

Speaker: Finding a needle in a haystack is quote for a search task that is effectively impossible. The reason it’s impossible is that there’s a lot of hay in a haystack and it’s all in a random heap. To find a needle in a haystack, you would have to take the stack apart, examining every stock of hay as you did, searching for the glint of the almost invisible tiny needle. Any thought of finding the needle quickly would be certain to be disappointed. Clearly, if your data is important to you, you do not want to store it in a haystack. The data equivalent of a haystack is an unordered collection of data items. To find the item you want, you must scan the entire collection until you find it. The item you want might be one of the first items you scan. It might be one of the last, or it might be anywhere in between. On average, you’ll have to individually look at about half of your data before you find the item you want. Worst case, you’ll have to scan it all. For large databases, that’s impractical.

There is a better way. Organize your data in a structure that makes it easy to find any given record. To find records quickly, create an index that serves as a pointer to any record that you might want to retrieve later. In this lesson, we’ve talked about where SQL came from and what it is used for. We talked about the importance of organizing data to make information that we want to pull out of it easily retrievable. In lesson two, we’ll examine the relational model, which is the most common way of structuring database data today. If you structure your data according to the relational model, you’ll be able to retrieve the data you want and do so very quickly.