Research cycles
or
how to evaluate theories and models

The meanings of the terms "theory" and "model" vary somewhat with research domains, and indeed, the border between theories and models is fuzzy. In general, however, a theory rather is a verbal account of conceptual ideas, whereas a model rather is an applicable elaboration thereof. This implies that both theories and models may have predictive power, but that falsifiability is an issue which applies to models rather than to theories. Hence, the conceptual plausibility of ideas formulated in theories can be assessed as such, while to these ideas, models are vehicles that can be tested more directly on things such as predictive power and practical feasibility.

The foregoing indicates three aspects by which theories and models can be evaluated, namely, conceptual plausibility, predictive power, and practical feasibility. This yields three cycles of research to enhance, revise, or reject, theories and models (see also the figure below):

• the theoretical cycle --- which aims to assess the conceptual plausibility of ideas and assumptions.
• the empirical cycle --- which aims to test ideas and assumptions by way of controled experiments.
• the tractability cycle --- which aims to assess if ideas and assumptions allow for feasible implementations.

In cognitive (neuro)science, the dominant cycle is the empirical cycle. Experimental data, however, are not only relevant but often also multi-interpretable. Moreover, cognitive (neuro)science is typically suited for a multidisciplinary approach, so that, to evaluate a theory or model, evidence from the theoretical and tractability cycles may be just as relevant to arrive at convincing scientific conclusions.

To be more specific:

• The idea of the theoretical cycle is to formalize conceptual ideas and assumptions, to see if they can be underpinned by a derivation from first principles. This method is characteristic of mathematics, in which a theorem usually starts as a conjecture that calls for a proof. The search for a proof may be successful, but it may also lead to the conclusion that the conjecture is false or that it has to be adapted in order to be provable. A successful proof means that the correctness of the conjecture can be derived logically from facts proven earlier, and thereby, from first principles (i.e., virtually undeniable assumptions, or axioms, like the ones Euclid used around 300 BC as the starting point of geometry). Such derivations may provide theoretical evidence for or against the ideas and assumptions in a theory or model.

• The idea of the well-known empirical cycle of research is to conduct controled experiments -- not only to explore unchartered terrain, but also to test concrete predictions that can be inferred from proposed theories or models. This method, which has roots in physics, was promoted in the behavioral sciences by de Groot (1961/1969), and may provide empirical evidence for or against ideas and assumptions in a theory or model.

• The idea of the tractability cycle is to assess if models allow for feasible process implementations, which is a criterion that holds equally for computers and brains (cf. van Rooij, 2008). This method has roots in computer science and artifical intelligence research, but it is also a fruitful method in cognitive (neuro)science. For instance, a model may yield plausible predictions but such a prediction may, according to the model, be the result of a selection of one outcome out of a highly combinatorial number of candidate outcomes. If this implies that each and every one of these candidate outcomes is to be judged separately, the model is not realistic because, then, the selection may easily require more time than is available in this universe. Such a selection process is therefore said to be intractable, no matter whether it is to be performed by computers or by brains. The tractability cycle may reveal such things and may lead to adjustments of the model such that it becomes tractable and, thereby, realistic.

In my research, this distinction between research cycles is a guiding methodological principle (see also Marr's levels and Metaphors of cognition). For instance:

• For my passes through the theoretical cycle, see Transparant holographic regularity and Simplicity versus likelihood.

• For my passes through the empirical cycle, see The holographic approach to goodness and Information measurement.

• For my passes through the tractability cycle, see Transparallel processing by hyperstrings and Human cognitive architecture.

There are no clear-cut borders between the research cycles, and the topics I addressed so far can be positioned by interlocking the three research cycles as follows: