Advertisement

Behavior Research Methods

, Volume 50, Issue 5, pp 1971–1980 | Cite as

Computationally constructing a repository of compound remote associates test items in American English with comRAT-G

  • Ana-Maria Olteţeanu
  • Holger Schultheis
  • Jonathan B. Dyer
Article

Abstract

The Remote Associates Test (RAT) has been used to measure creativity, however few repositories or standardizations of test items exist, like the normative data on 144 items provided by Bowden and Jung-Beeman. comRAT is a computational solver which has been used to solve the compound RAT in linguistic and visual forms, showing correlation to human performance over the normative data provided by Bowden and Jung-Beeman. This paper describes using a variant of comRAT, comRAT-G, to generate and construct a repository of compound RAT items for use in the cognitive psychology and cognitive modeling community. Around 17 million compound Remote Associates Test items are created from nouns alone, aiming to provide control over (i) frequency of occurrence of query items, (ii) answer items, (iii) the probability of coming up with an answer, (iv) keeping one or more query items constant and (v) keeping the answer constant. Queries produced by comRAT-G are evaluated in a study in comparison with queries from the normative dataset of Bowden and Jung-Beeman, showing that comRAT-G queries are similar to the established query set.

Keywords

Remote associates test Creative cognition Computational creativity Cognitive modeling 

Introduction

Imagine you set out to test or manipulate the creative ability of some of your participants using the Remote Associates Test (Mednick & Mednick 1971). This test consists of giving participants three words—like water, mine, and shaker—and asking them what word might be related to all three such words.1 You would have to first find and select a set of RAT queries, and then control as many variables as you can about them.

Various creativity tests are available: the Alternative Uses Test (Guilford 1967), the Remote Associates Test (Mednick and Mednick 1971), the Torrance Tests of Creative Thinking (Kim 2006), the Wallach–Kogan tests (Wallach and Kogan 1965), riddles (used by Whitt & Prentice, 1977; Qiu et al., 2008), empirical insight tests from Duncker (1945), Maier (1931), and Saugstad and Raaheim (1957) and others. However, some of these tests are not easily available (TTCT), some are non-standardized or do not provide any norms, and others provide small sets of stimuli. Thus, various creativity tests could highly benefit from modernization by being normed, having more factors controlled for and by developing ampler sets of stimuli, in order to provide more varied testing conditions.

The Remote Associates Test (Mednick & Mednick 1971), has been used to measure creativity and adapted to various other languages—for example in Japanese (Baba 1982) and Dutch (Chermahini, Hickendorff, & Hommel, 2012). The test has been rated as the second most used creativity test in a meta-analysis surveying 45 neuroimaging studies (Arden, Chavez, Grazioplene, & Jung, 2010). The RAT is assumed to measure creative convergent thinking, unlike other creativity tests, which are better suited to measure divergent thinking—e.g., the Alternative Uses Test. The RAT is very useful in measuring insight theory effects, as performance in the RAT has been shown to correlate with performance in insight problems (Schooler and Melcher 1995).

Various types of investigations in and beyond creative cognition use the Remote Associates Test. Amongst others, these include the study of the effects of incubation with GO players (Sio & Rudowicz 2007) and baseball players (Wiley 1998), the relation between REM sleep and creativity (Cai, Mednick, Harrison, Kanady, & Mednick, 2009), synesthesia and creativity (Sitton & Pierce, 2004; Ward, Thompson-Lake, Ely, & Kaminski, 2008), the role of affect in problem-solving (Fodor 1999), memory (Storm, Angello, & Bjork, 2011), peripheral attention (Ansburg and Hill 2003) etc. Because of its wide use, the scientific community would thus benefit from an ampler set of standardized stimuli for the RAT.

Worthen and Clark (1971) made the case that different stimuli categories can be distinguished within the original stimuli of Mednick and Mednick (1971)—specifically they differentiated between functional items and structural items (sometimes also appearing in the literature under the name of compound items).

The CreaCogs framework (Olteţeanu, 2014, 2016) for creative problem-solving uses knowledge organization to support a unified set of core creative problem solving processes, like association, associative convergence, re-representation, restructuring, search, and substitution. The validation of the framework and processes is done in a comparative manner to human performance (Olteţeanu, Falomir, & Freksa, in press) through implementing systems which can show creative problem-solving abilities and solve creativity tests for humans. Among such systems, the comRAT-C system (Olteţeanu & Falomir 2015) has explicitly addressed the computational solving of compound RAT queries. When solving the RAT, comRAT-C calculated the probability of finding a solution based on the frequency of query and answer words, as will be shown in “Generating new remote associates test items with comRAT-G”. A highly significant correlation has been observed between the results of comRAT-C and the difficulty of RAT queries for humans, as expressed in percentage of solvers and response times in the human normative data (Bowden & Jung-Beeman 2003). This correlation ranged between 0.3 and 0.52 for different solving times. This correlation showed that the frequency of query items plays an important role into how difficult a query is for human participants, and a standardized set of RAT stimuli should take this into account. Furthermore, (Olteţeanu & Schultheis in press) modified frequency and probability factors independently, keeping them at low and high levels. This showed that frequency and probability both are factors which influence accuracy and response times when solving the RAT.

For the Remote Associates Test, normative data from human participants does exist from Bowden and Jung-Beeman (2003), which provides data on mean time-to-solution and percentage of participants solving for 144 compound RAT problems, with four different time limits. Though very useful, this work does not provide standardized queries based on frequency of occurrence of word or answer stimuli, the importance of which has been shown by Olteţeanu & Falomir (2015). This paper aims to enrich the existing pool of compound Remote Associates Test items and provide a standardized treatment that allows control over the frequency of occurrence and probability of finding an answer. Seventeen million new compound RAT items are constructed, using the entire space of frequent noun expressions in American English - thus providing the largest standardized treatment of the compound RAT test to date. These items are computationally generated by adapting the previously implemented computational solver of the Remote Associates Test, comRAT-C (Olteţeanu & Falomir 2015), to a generative variant - comRAT-G. The frequency of items from the COCA corpus2 and comRAT-C’s probability of finding an answer are indexed in the provided repository, and can be used to generate a set of controlled queries. Subsets of queries in which one word or the answer are kept constant can also be extracted.

The rest of this paper is structured as follows. A brief overview of the comRAT solver and its transformation to comRAT-G is provided in “From comRAT to comRAT-G”. The methodology of generating new Remote Associates Test items with comRAT-G is explained, together with examples, in “Generating new remote associates test items with comRAT-G”. The evaluation of 100 queries with human participants is shown in “Evaluation”. The type of data generated in the repository is described in “The Repository”, and various possible uses are showcased. Finally, future work is discussed. The 100 items used for evaluation are presented in the Annex.

From comRAT to comRAT-G

The comRAT-C (Olteţeanu & Falomir 2015) system has been built to solve the compound variant of the Remote Associates Test, in the tradition of Psychometric AI (Bringsjord 2011) and as an exploration of the processes of the CreaCogs creative problem-solving framework (Olteţeanu 2014; Olteţeanu & Falomir 2016).

As data, comRAT-C takes the most frequent 2-grams from the COCA corpus. Knowledge organization plays an important role in CreaCogs and in comRAT’s knowledge base in the following way: whenever a 2-gram is given to comRAT-C, it is stored as an object of the Expression class, which is constructed from two objects of the Concept class and a Link between them. If one of the Concepts is already present, the other Concept and the new Link are added. If both Concepts are present, only the Link is added. For example, if the 2-gram “cake flour” is read, comRAT-C will check to see if it knows the Concepts “cake” and “flour”. If it doesn’t know one of them, the item will be added as a Concept3; if it doesn’t know either, both items will be added. Then it will add a link between those Concepts, with a numeric tag attached—the frequency of the 2-gram as taken from the corpus.

Over time, each Concept will end up with a set of Links to all the other Concepts it has been in an Expression with, as shown in Fig. 1. This associative structure constitutes the knowledge organization of comRAT-C. When three words are given to comRAT-C, as they would in the context of a RAT query, each of these words activates the set of Concepts they are linked to. An overlapping activation starting from two or three of the initial words can sometimes be observed. In Fig. 1, the initial words are depicted in green - cottage, Swiss, and cheese. A two-item convergence of activation is observed for the word chocolate, and a three-item convergence for the word cheese. The three-item convergences are possible RAT query responses. Multiple two and three-item convergences are of course possible for the same query.
Fig. 1

A graphical depiction of the link structure obtained in comRAT-C. Only a few links are shown for visual readability

The comRAT-C system performs well in answering compound RAT queries even without the links between Concepts being weighted using the frequency of 2-gram metrics; however, adding these metrics improves comRAT-C’s performance and helps break ties between multiple possible three-item convergences.

An analysis of comRAT-C’s probability to find an answer based on the two-gram frequency data revealed a correlation to human performance normative data. This correlation shows that the frequency of 2-grams, on which the probability of finding the answer is based, might have an influence on the process of solving compound RAT queries. In order to keep compound queries controlled for the frequency variable, to check for other influences (like order), and to understand these influences in more detail, frequency-based probability data needs to be gathered on a large set of queries. A large enough set of queries can also be used to keep part of the query words or the answer word (different queries, same answer) constant. In order to gather such data, and construct a large set of queries, we proposed reverse-engineering our computational approach in order to generate new compound RAT items.

Thus, instead of using the organization structure of comRAT to provide answers, this will be used to provide queries. From this vantage point, each word w a n s that has at least three links, let’s say to words w a , w b , w c , is a potential answer to a RAT query.

Generating new remote associates test items with comRAT-G

The process of generating new Remote Associates Test items unfolds as follows. All the noun–noun high frequency 2-grams in COCA are organized in Concepts and Links in the knowledge base of comRAT-G. The selection of noun–noun 2-grams is done using the UCREL CLAWS7 Tagset4 (tags as per this tagset are provided with the 2-grams dataset). comRAT-G uses nouns alone in this current version, unlike comRAT-C, which used more parts of speech, as described in Olteţeanu & Falomir (2015). Thus the set of most frequent 2 million 2-grams is reduced to a set of 43,908 expressions.

First, comRAT-G iterates through the words and provides preliminary results, which henceforth we shall call type 1 results, a sample of which is shown in Table 1. Type 1 results consider each word as a potential answer word. Thus in Table 1, w a n s stands for the answer word and w q for a potential query word which can be used to get the answer word. Terms w q can be further integrated in a RAT query in positions w a , w b or w c . The third column represents the frequency of association between the query word w q and the answer word w a n s . The fourth column represents the frequency of association between the query word and any word. The fifth column represents the probability of answer w a n s given specific query word w q .5 This is calculated as shown in Eq. 1. Query words w q are only generated for the w a n s , which have at least three w q . Applying the process to get to type 1 results yields a total of ∼ 81500 unique w a n s , w q combinations, based on 9601 unique answer words.
$$ P[w_{ans} \mid w_{q}]=\frac{fr(w_{q} w_{ans})}{\sum\limits_{k=1}^{n} (w_{q}, w_{k})} $$
(1)
Table 1

An example of preliminary results, focusing on words as answers to possible queries

w a n s

w q

f r(w a n s , w q )

\({\sum }_{k=1}^{m} (w_{q}, w_{k})\)

P(w a n s |w q )

Health

animal

95

6558

0.0145

 

bone

159

4048

0.0393

 

brain

25

7139

0.0035

 

breast

46

10302

0.0045

 

child

149

16550

0.009

 

city

49

16666

0.0029

In this case the answer word is “health”, and only a few of its 256 links are shown

After type 1 results have been produced and stored, the new RAT queries are generated, using a combinatoric approach. For each answer word, the set of query words are retrieved and three-word combinations are generated. This applies the well-known combinatorics formula shown in Eq. 2, with n being the number of query items connected to a specific answer word, and k being 3.
$$ \begin{array}{c}n \\ k \end{array} = \frac{n!}{k!(n-k)!} $$
(2)
For each possible answer with n < 100 (and of course n > 2), all unique combination triples are produced. We capped n at 100 because of computational costs (\({100 \choose 3}\) is 161,700 possible combinations with the terms connected to the same answer word) and diminishing returns—an answer word w a n s connected to over 100 items might be a very common word, or form much weaker bonds with either of the words; thus RAT items constructed from its terms might not be too interesting or intuitive to solve (lower associative power of triggering result).
In order to construct all such combinations in a computationally feasible manner, comRAT-G uses Alan Tucker’s combinatorics algorithm (Tucker 2006). For 9601 query answer words and capping n at 100 (which translates into only using about 9200 answers), we obtain about 17 million possible RAT triples. The probability of answering the query is calculated based on the conditional probability of the answer to be triggered by each of the three query items. The probability thus currently considers an equal weighting of the three items, as shown in Eq. 3 (as in comRAT-C (Olteţeanu & Falomir 2015)). However, different weighting schemes can be considered for modeling purposes—which is why we also provide the conditional probability of each item, as shown in the Appendix. The various types of data items captured by this ample list of possible RAT queries and the roles in which such data can be used in empirical research are presented in “The Repository”.
$$ P(w_{ans}) = \frac{ P(w_{ans} \mid w_{a})+P(w_{ans} \mid w_{b})+P(w_{ans} \mid w_{c})}{3} $$
(3)

Evaluation

In order to check whether the queries created by comRAT-G are suitable, valid and reliable RAT queries, which can be solved by human participants and are coherent with existing RAT datasets, we have set up a study in which the human performance on comRAT-G queries is compared to that on normative data from Bowden and Jung-Beeman (2003).

Method

Two sets of query items, one comprising randomly selected comRAT-G queries and the second comprising randomly selected queries from the normative data of Bowden and Jung-Beeman (2003), were presented mixed in random order to native speakers in an online study. Accuracy and response times for solving the items were recorded. The purpose of the study was to check whether: (i) correlations between performance indicators hold between comRAT-G and Bowden & Jung-Beeman items, thus showing validity of comRAT-G items and (ii) whether comRAT-G queries are a reliable tool, as measured by Cronbach’s alpha, compared to Bowden & Jung-Beeman items.

Participants

A total of 113 native English speakers, 72 female and 41 male, were recruited at University of Pittsburgh and on Crowdflower and volunteered to answer our study, which was set up online. Participants had a wide range of ages, education levels and self-rated their creativity on a wide set of levels, as shown in Table 2.
Table 2

Descriptive data on the age, education, and self-rated creativity level of the participants, n = 113

Indicator

Level

No. of participants

Age

Under 20

2

 

20–30

33

 

30–40

36

 

40–50

17

 

50–60

20

 

60–70

4

 

Over 70

1

Education

Secondary school

7

level

High school diploma

27

 

Enrolled in undergraduate courses

20

 

Completed undergraduate courses

34

 

Enrolled in postgraduate courses

6

 

Completed postgraduate courses

19

Creativity

Low

5

self-rated

Average

39

 

Above average

41

 

High

22

 

Very high

6

Materials

Fifty compound RAT queries were randomly selected from the items produced by comRAT-G. Another 50 queries were randomly selected from the query set of Bowden & Jung-Beeman. The comRAT-G queries can be found in the Appendix. From Bowden and Jung-Beeman (2003) we used queries 5, 6, 11, 15, 17, 20, 21, 22, 24, 26, 28, 29, 30, 37, 38, 40, 45, 46, 50, 51, 53, 58, 62, 65, 68, 71, 72, 74, 76, 79, 82, 84, 87, 90, 95, 96, 99, 106, 110, 111, 114, 116, 122, 124, 130, 131, 133, 136, 139 and 144.

Procedure

The task was explained with two query examples. Then, five training queries were presented. These queries were taken from Bowden & Jung-Beeman items and did not overlap with our random selection of 50 items. After the participants attempted to solve the training queries, feedback including the correct answer was presented. Then, the 100 queries (50 from Bowden & Jung-Beeman, 50 from comRAT-G) were presented in random order.

Results

The dependent variables were (i) accuracy, measured as the number of correct responses for each participant in comRAT-G and Bowden & Jung-Beeman queries, and (ii) response times, measured as the number of seconds each participant spent on answering each comRAT-G and Bowden & Jung-Beeman query.

As Table 3 shows, the mean accuracy was 26.20, standard deviation (SD) = 7.03) problems correctly solved (52.4 %) for comRAT-G items and 26.41 (S D = 11.24) problems correctly solved (52.82%) for Bowden & Jung-Beeman items. The mean response times for correct solutions (n = 1126) was 14.52 s (S D = 9.89) for comRAT-G items and 16.56 s (S D = 12.84) for Bowden & Jung-Beeman items, as shown in Table 4.
Table 3

Descriptive statistics on accuracy in number of queries solved, n = 113

 

Mean accuracy

Std.

95 % Conf.

interval

 

no. solved (S D)

Error

L.B.

U.B.

comRAT-G

26.20(7.03)

0.66

24.89

27.51

Bowden & Jung-Beeman

26.41(11.24)

1.06

24.31

28.50

Both

52.64(16.16)

1.52

49.62

55.65

Table 4

Descriptive statistics on response times (RT) in seconds for queries solved, n = 112

 

Mean RT in

Std.

95 % Conf.

interval

 

seconds (S D)

Error

L.B.

U.B

comRAT-G

14.52 (9.89)

0.93

12.67

16.38

Bowden & Jung-Beeman

16.56 (12.84)

1.21

14.15

18.97

As shown in Table 5, the mean number of participants solving each comRAT-G query was 59.28, and the mean number of participants solving each Bowden & Jung-Beeman query was 59.92. The mean time spent per comRAT-G query, independent of whether it was solved or not, was 21.9 s, while the mean time spent per each Bowden & Jung-Beeman query was 23.12.
Table 5

Descriptive statistics on number of participants solving per query (of 113), and mean time spent per query (whether a correct answer was given or not)

 

Mean no. of

Std.

95 % Conf.

interval

 

participants solving (S D)

Error

L.B.

U.B

Solved comRAT-G

59.28 (33.63)

4.76

49.72

68.84

Solved Bowden & Jung-Beeman

59.92 (21.18)

3.0

53.9

65.94

Solved both

59.6 (27.96)

2.8

54.05

65.15

 

Mean time

Std.

95 % Conf.

interval

 

spent solving

Error

L.B.

U.B

Time comRAT-G

21.9 (10.3)

1.46

18.97

24.83

Time Bowden & Jung-Beeman

23.12 (7.1)

1.00

21.11

25.14

Time both

22.51 (8.82)

0.88

20.76

24.26

Accuracy and response times per query for the comRAT-G dataset are shown in the Appendix.

Accuracy showed an average significant correlation between the comRAT-G and the Bowden & Jung-Beeman datasets of r = 0.54, p < 0.0001 (Fig. 2a). Response times showed a highly significant large correlation between the two datasets of r = 0.75, p < 0.0001 (Fig. 2b). Note that response times were calculated only for correct answers.
Fig. 2

Correlations on a accuracy and b response times

We then measured the scale reliability of the two datasets (comRAT-G items and Bowden & Jung-Beeman items) using Cronbach’s alpha as an internal consistency measure. As Table 6 shows, Cronbach’s alpha on accuracy was 0.851 for comRAT-G items, 0.932 for Bowden & Jung-Beeman items and 0.936 for both sets of queries taken together. Cronbach’s alpha on response times (on both correct and incorrect answers) was 0.991 for comRAT-G items, 0.99 for Bowden & Jung-Beeman items and 0.995 for both set of queries taken together.
Table 6

Cronbach’s alpha internal consistency measures

Set of queries

Performance

n

Cronbach’s

 

measure

 

Alpha

comRAT-G

Accuracy

113

0.851

Bowden & Jung-Beeman

Accuracy

113

0.932

Both

Accuracy

113

0.936

comRAT-G

Response times

112

0.991

Bowden & Jung-Beeman

Response times

112

0.99

Both

Response times

112

0.995

As a final point, we checked to see whether the accuracy and response times data we obtained on the Bowden & Jung-Beeman dataset with our participants correlated with that obtained by Bowden & Jung-Beeman. As shown in Table 7, all accuracy measures and all but one response times measures correlated significantly.
Table 7

Correlation of performance on the Bowden & Jung-Beeman queries, between our participants and Bowden & Jung-Beeman’s participants

Measure type

Bowden & Jung-Beeman measure

Our measure

Correlation

Accuracy

Accuracy 2s

Accuracy

r = 0.66, p < 0.0001

 

Accuracy 7s

Accuracy

r = 0.76, p < 0.0001

 

Accuracy 15s

Accuracy

r = 0.78, p < 0.0001

 

Accuracy 30s

Accuracy

r = 0.81, p < 0.0001

Response

RT 7s

RT

r = 0.45, p < 0.0001

times

RT 15s

RT

r = 0.15,−

 

RT 30s

RT

r = 0.72, p < 0.0001

Discussion

The descriptive data is similar between the comRAT-G and Bowden & Jung-Beeman’s set of queries, on both mean accuracy and mean response times. The average and high correlations obtained between the performance of the participants on the comRAT-G and Bowden & Jung-Beeman sets of items on both accuracy and response times show the validity of the comRAT-G dataset, pointing to the fact that we are measuring the same skill with comRAT-G as with Bowden & Jung-Beeman sets. The high Cronbach’s alpha internal consistency scores, which remain the same or increase when putting the two item sets together, show that both sets are highly reliable, and consistent with each other. Thus the comRAT-G data are in all crucial respects similar to the established query set.

The Repository

In the following, the RAT queries generated and the data which accompanies them will be explained as a function of data items (columns), ability to search for and order items and some examples of possible empirical research using this data. Table 8 shows a sample of the generated queries data and their form.
Table 8

An example of generated queries, organized by w a n s which is ability

w a

w b

w c

f r(w a n s ,

f r(w a n s ,

f r(w a n s ,

f r

f r

f r

P(w a n s

P(w a n s

P(w a n s

P(w a n s )

w a n s

   

w a )

w b )

w c )

(w a )

(w b )

(w c )

|w a )

|w b )

|w c )

  

coping

language

leadership

88

58

76

1237

6405

4556

0.0711

0.0091

0.0167

0.0323

ability

coping

language

problem-solving

88

58

50

1237

6405

606

0.0711

0.0091

0.0825

0.0542

ability

coping

language

reading

88

58

65

1237

6405

4292

0.0711

0.0091

0.0151

0.0318

ability

coping

language

reasoning

88

58

37

1237

6405

121

0.0711

0.0091

0.3058

0.1287

ability

coping

language

student

88

58

35

1237

6405

26153

0.0711

0.0091

0.0013

0.0272

ability

coping

language

thinking

88

58

34

1237

6405

1239

0.0711

0.0091

0.0274

0.0359

ability

coping

language

writing

88

58

29

1237

6405

2515

0.0711

0.0091

0.0115

0.0306

ability

The generated compound RAT queries can thus be ordered in the following ways:
  1. (1)

    Alphabetically by the first, second and third word (on w a , w b and w c ). This ability to search for alphabetically ordered RAT queries allows empirical research keeping the first letter or the entire query word (or more than one word) constant. This can be used in various forms, at its extension allowing for keeping the entire query constant and checking for different possible answers.

     
  2. (2)

    By the answer (w a ). This can allow for comparisons of query difficulty in which the query terms differ, and the answer is kept constant.7 Thus for the queries a) health, child and center and b) insurance, hair and child, the answer is the same - care, and so is one of the given terms, child. However, the likelihood of reaching this answer is not the same. Keeping the answer the same can help check upon the influence the different terms and their frequency have on the performance.

     
  3. (3)

    By frequency of the favorable cases (f r(w a , w a n s ), f r(w b , w a n s ), f r(w c , w a n s )).

     
  4. (4)

    By the (sum) frequency of the given words (f r(w a ), f r(w b ), f r(w c )). This allows for the study of frequency based influence separately from probability. Empirical exploration of whether keeping the frequency constant throughout the words (query or answer) among different queries has an impact on answer performance, and the function and interinfluences between frequency and answer performance should thus be possible.

     
  5. (5)

    By the probability of the answer to be found. Multiple queries with similar probabilities can thus be analyzed, and queries from different probability classes can be analyzed (low probability, medium probability, high probability) together with their influence on human performance.

     

The frequency of the various query words with the answer, and the frequency of the query words occurring in other combinations has been provided here separately, as the probability of finding the answer has been calculated here taking the influence from the three words to be equal. It might be the case that the first two words have a higher influence (see Olteţeanu, 2014), and showing frequency explicitly for each of the query words allows the study of order effects.

An interface permitting access to the queries constructed by comRAT-G can be found here: http://creacogcomp.com/comRAT-G.html.

Conclusions and future work

An ample set of about 17 million queries was generated using a variant of a computational RAT solver - comRAT-G. This set of queries aims to fill a gap in the area of providing normative frequency-based data and an ampler set of stimuli for cognitive and computational creativity research. Frequency and frequency-based probability of finding the answer have been computed for all the generated queries and are provided with this data. The contributed repository allows further control over variables when testing for the influence of frequency, keeping words constant and word order in Remote Associates queries.

The entire list of queries or a subset thereof can be obtained by contacting the authors. As future work, we will aim to make the following contributions:
  1. (i)

    Improve the online interface with more search and selection features, for easy access.

     
  2. (ii)

    Generate an updated version of this repository by also parsing compound nouns from the corpus automatically, and offer queries based on compound nouns as another controllable variable. The motivation for this is that items of the form (w a n s , w q ) which are parsed from compound nouns might be associated more tightly than items which have 2-grams as a point of origin.

     
  3. (iii)

    Generate a version of the repository which includes queries made of other parts of speech than nouns alone.

     
  4. (iv)

    Offer the ability to collapse plural and singular forms to modelers.

     
  5. (v)

    Add free association norms data to the query-answer pairs, if available.

     
  6. (vi)

    Enable control of query and answer word length.

     
  7. (vii)

    Enable control over semantic domain of words.

     
  8. (viii)

    Rate a part of these queries for interestingness and hardness, in order to further refine the generating algorithm.

     

Footnotes

  1. 1.

    The answer to this RAT query is salt.

  2. 2.

    Corpus of Contemporary American English (COCA): http://corpus.byu.edu/coca/

  3. 3.

    The reason we call these items Concepts is that we entertain the possibility that they can be more than linguistic items, and another system - comRAT-V (Olteţeanu, Gautam, & Falomir, 2015)- aims to solve a visual variant of the RAT using visual associations.

  4. 4.
  5. 5.

    Further restrictions would have been possible, for example, by only considering words as valid query words, if their conditional probability is above a certain threshold. While this may have restricted possible query words to those that have a more valid/reasonable appeal, we thought it better to start with an all-inclusive set, on which then modelers can put their own thresholds.

  6. 6.

    We removed one outlier that spent 177 min on the task, as this indicated solving the RAT was not their main focus.

  7. 7.

    This can also allow for keeping just the initial part of the answer constant, and checking for phonetico-syntactical rather than semantical influences in answer difference.

  8. 8.
  9. 9.

    Response times and SD for queries 29, 32, 34, 43, 46 and 48 have also been shown with corrections for outliers. All the data analyses have been done without these corrections.

Notes

Acknowledgements

Ana-Maria Olteţeanu gratefully acknowledges the support of the Deutsche Forschungsgemeinschaft (DFG) for the Creative Cognitive Systems (CreaCogs) project.8 The support offered by the RISE DAAD program is acknowledged by Ana-Maria Olteţeanu and Jonathan B. Dyer. We thank Thansuda Kraisangka from Mahidol University for helping us create the interface to the generated test items.

References

  1. Ansburg, P. I., & Hill, K. (2003). Creative and analytic thinkers differ in their use of attentional resources. Personality and Individual Differences, 34(7), 1141–1152.CrossRefGoogle Scholar
  2. Arden, R., Chavez, R. S., Grazioplene, R., & Jung, R. E. (2010). Neuroimaging creativity: a psychometric view. Behavioural Brain Research, 214(2), 143–156.CrossRefGoogle Scholar
  3. Baba, Y. (1982). An analysis of creativity by means of the remote associates test for adult revised in Japanese (JARAT FORM a). The Japanese Journal of Psychology, 52(6), 330–336.Google Scholar
  4. Bowden, E. M., & Jung-Beeman, M. (2003). Normative data for 144 compound remote associate problems. Behavior Research Methods, Instruments, and Computers, 35(4), 634–639.CrossRefGoogle Scholar
  5. Bringsjord, S. (2011). Psychometric artificial intelligence. Journal of Experimental & Theoretical Artificial Intelligence, 23(3), 271–277.CrossRefGoogle Scholar
  6. Cai, D. J., Mednick, S. A., Harrison, E. M., Kanady, J. C., & Mednick, S. C. (2009). REM, not incubation, improves creativity by priming associative networks. Proceedings of the National Academy of Sciences, 106(25), 10130–10134.CrossRefGoogle Scholar
  7. Chermahini, S. A., Hickendorff, M., & Hommel, B. (2012). Development and validity of a Dutch version of the remote associates task: An item-response theory approach. Thinking Skills and Creativity, 7(3), 177–186.CrossRefGoogle Scholar
  8. Duncker, K. (1945). On problem solving. Psychological Monographs, 58(5), i–113.CrossRefGoogle Scholar
  9. Fodor, E. (1999). Subclinical inclination toward manic-depression and creative performance on the remote associates test. Personality and Individual Differences, 27(6), 1273–1283.CrossRefGoogle Scholar
  10. Guilford, J. P. (1967). The nature of human intelligence. McGraw-Hill.Google Scholar
  11. Kim, K. H. (2006). Can we trust creativity tests? A review of the Torrance tests of creative thinking (TCT). Creativity Research Journal, 18(1), 3–14.CrossRefGoogle Scholar
  12. Maier, N. R. (1931). Reasoning in humans. ii. The solution of a problem and its appearance in consciousness. Journal of Comparative Psychology, 12(2), 181.CrossRefGoogle Scholar
  13. Mednick, S. A., & Mednick, M. (1971). Remote associates test: Examiner’s manual. Houghton Mifflin.Google Scholar
  14. Olteţeanu, A.-M. (2014). Two general classes in creative problem-solving? An account based on the cognitive processes involved in the problem structure - representation structure relationship. In: Besold, T., Kühnberger, K. U., Schorlemmer, M., & Smaill, A. (Eds.) Proceedings of the international conference on computational creativity, Vol. 01-2014: Osnabrück.Google Scholar
  15. Olteţeanu, A.-M. (2016). From simple machines to eureka in four not-so-easy steps. Towards creative visuospatial intelligence. In: Müller, V. (Ed.) Fundamental issues of artificial intelligence.  https://doi.org/10.1007/978-3-319-26485-1_11, (Vol. 376. pp. 159–180): Springer.Google Scholar
  16. Olteţeanu, A.-M., & Falomir, Z. (2015). ComRAT-c - a computational compound remote associates test solver based on language data and its comparison to human performance. Pattern Recognition Letters, 67, 81–90.  https://doi.org/10.1016/j.patrec.2015.05.015.CrossRefGoogle Scholar
  17. Olteţeanu, A.-M., & Falomir, Z. (2016). Object replacement and object composition in a creative cognitive system. Towards a computational solver of the alternative uses test. Cognitive Systems Research, 39, 15–32.  https://doi.org/10.1016/j.cogsys.2015.12.011.CrossRefGoogle Scholar
  18. Olteţeanu, A.-M., & Schultheis, H. (in press). What determines creative association? revealing two factors which separately influence the creative process when solving the remote associates test. The Journal of Creative Behaviour.  https://doi.org/10.1002/jocb.177
  19. Olteţeanu, A.-M., Gautam, B., & Falomir, Z. (2015). Towards a visual remote associates test and its computational solver. In: Proceedings of the third international workshop on artificial intelligence and cognition 2015, (Vol. 1510. pp. 19–28): CEUR-Ws.Google Scholar
  20. Olteţeanu, A.-M., Falomir, Z., & Freksa, C. (in press). Artificial cognitive systems that can answer human creativity tests: an approach and two case studies. IEEE Transactions on Cognitive and Developmental Systems.  https://doi.org/10.1109/TCDS.2016.2629622 Google Scholar
  21. Qiu, J., Li, H., Yang, D., Luo, Y., Li, Y., Wu, Z., & Zhang, Q. (2008). The neural basis of insight problem solving: an event-related potential study. Brain and Cognition, 68(1), 100–106.CrossRefGoogle Scholar
  22. Saugstad, P., & Raaheim, K. (1957). Problem-solving and availability of functions. Acta Psychologica, 13, 263–278.CrossRefGoogle Scholar
  23. Schooler, J. W., & Melcher, J. (1995). The ineffability of insight. In: Ward, T., & Finke, R. (Eds.) The creative cognition approach. (pp. 249–268). Cambridge, MA: The MIT Press.Google Scholar
  24. Sio, U. N., & Rudowicz, E. (2007). The role of an incubation period in creative problem solving. Creativity Research Journal, 19(2-3), 307–318.CrossRefGoogle Scholar
  25. Sitton, S. C., & Pierce, E. R. (2004). Synesthesia, creativity and puns 1. Psychological Reports, 95(2), 577–580.CrossRefGoogle Scholar
  26. Storm, B. C., Angello, G., & Bjork, E. L. (2011). Thinking can cause forgetting: memory dynamics in creative problem solving. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37(5), 1287.PubMedGoogle Scholar
  27. Tucker, A. (2006). Applied combinatorics. Wiley.Google Scholar
  28. Wallach, M. A., & Kogan, N. (1965). Modes of thinking in young children: A study of the creativity-intelligence distinction. Holt, Rinehart & Winston.Google Scholar
  29. Ward, J., Thompson-Lake, D., Ely, R., & Kaminski, F. (2008). Synaesthesia, creativity and art: What is the link? British Journal of Psychology, 99(1), 127–141.CrossRefGoogle Scholar
  30. Whitt, J. K., & Prentice, N. M. (1977). Cognitive processes in the development of children’s enjoyment and comprehension of joking riddles. Developmental Psychology, 13(2), 129.CrossRefGoogle Scholar
  31. Wiley, J. (1998). Expertise as mental set: The effects of domain knowledge in creative problem solving. Memory & Cognition, 26(4), 716–730.CrossRefGoogle Scholar
  32. Worthen, B. R., & Clark, P. M. (1971). Toward an improved measure of remote associational ability. Journal of Educational Measurement, 8(2), 113–123.CrossRefGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2017

Authors and Affiliations

  1. 1.Cognitive Systems, Bremen Spatial Cognition CenterUniversity of BremenBremenGermany
  2. 2.Department of Computer ScienceUniversity of PittsburghPittsburghUSA

Personalised recommendations