Critical Rationalism vs Inductivism: A Response to Kieren

The following is a response to Kieren’s claim that Deutsch’s Hard to Vary (HTV) criteria is really just Induction in disguise. (his original post) I made a short response to him pointing out that in reality we do not look for good theories via only constraint of past observations but instead by how well it’s constrained against other theories. Kieren responded back to me making the following arguments (Twitter link):

1. “…these other theories are themselves constrained by observations.”
2. “Any theory that says something new about the world… [makes claims that must either be] explained by other theories… [or were] accepted as knowledge via induction.” Where he specifically defines “induction” as “constraint by past observation.”
3. “…you might rely on a theory of human behaviour relating to trade, but this theory itself is constrained by our observations of how humans behave in such situations.”
4. “I was just taking an additional step than you to see where our theory was being constrained by observation.”

Given these arguments, Kieren maintains that HTV is still just crypto-Induction .

After reading Kieren’s feedback, I’m no longer certain I understand what he means by “induction.” The concept of “induction” is usually understood as something more than “constraint by past observation.” (i.e. his point #4.)

In this article, I’m going to consider my own understanding of Induction (particularly as used by my own field of Machine Learning) and then consider how that seems differs from the way Kieren seems to be using it. Finally, I’ll explain why I believe Kieren is actually failing to address address the Critical Rationalist position at all and how I believe it’s likely he’s actually making a circular argument in the first place. I’ll also make specific suggestions of how Kieren can improve his argument by addressing the substance of the difference between Inductivism and Critical Rationalism (CR).

What is Induction?

Let’s start with the original “Baconian” understanding of “Inductivism” as Wikipedia explains it:

[Inductivism is where] one observes nature, proposes a… law to generalize an observed pattern, [then] confirms it by many observations

Now Inductivism (as defined above) is obviously problematic in many ways as a description of science. For example, it’s not possible to confirm via observations — you can really only refute via observations. And science does not start with observations but instead starts with a problem with a current explanation that seems worth addressing. That explanation might not even be a scientific theory; it might just be a myth or even an innate theory built into the software of the mind. So if by “Inductivism” we mean how Wikipedia defines it, then clearly CR (including now Deutsch’s HTV criteria) and Inductivism are not the same thing and are fully incompatible as descriptions of science.

Is Induction Even a Real Thing?

Is Inductivism a real thing at all? Popper wasn’t convinced. [1] I’ve seen Deutsch claim it’s not a real thing either. However, in both cases, it might be they simply meant that there is no scientific Inductive method, which I agree is true.

In any case, certainly Inductivism as originally understood by Bacon as the basis for science is not a real thing. For example, Bacon was quite convinced that the way Induction worked was you cleared your mind of prejudice (he invited the term!) and then observe the world without any preconceived notions. Then, based on this unprejudiced observations, you somehow arrived at a generalization of the observations. [2] Popper pointed out that this is a false view because all observations are already theory-impregnated. [3]

Even Kieren isn’t arguing that Bacon’s view is strictly true. For example, his response to me makes it clear that he has no problem at all with that HTV criteria including constraint by past theories (as Popper insists upon) whereas Bacon had no such concept. So I don’t think there is necessarily a difference of opinion here between Critical Rationalists and Kieren on the fact that original Baconian Inductivism — as the basis for science — is not a real thing. Presumably Kieren accepts what he feels is a more up-to-date version of Bacon’s theory.

The Real Theory of Induction

But we have a modern theory of Induction (that presumably was inspired by Bacon’s ideas) that is quite real and has proven quite effective in my own field of study: Machine Learning. Indeed, one might even say that the entire field of statistics is the science of how Induction actually works. So “Induction” is real, it just doesn’t happen to be how science works, despite its misleading use of some similar vocabulary as science.

Machine Learning and Induction

In machine learning ‘induction’ is the process where you generalize a rule from observations (in a training set) using statistical learning, or something related to statistical learning, such as neural nets (which have no specific theory behind them yet, but historically came out of statistical learning.)

The idea is that you start with a number of observations in the “Training Set” and you pick a learning method (say, linear regression) by which to search through a “hypothesis space” via some sort of “inductive bias.”

In the case of linear regression, the bias is “hypotheses” that fit to a line. This means the search through the hypothesis space is limited to trying to find a line that minimizes sum of squared error. The end result is a ‘hypothesis’ that will with some (hopefully) high probability properly predict (via new observations) what the right outcome will be. We then test the final “hypothesis” via a “Test Set” to “confirm” it’s really working as a generalization of the observations and not just “overfitting” the data.

Now it seems to me that “Induction,” as used by Machine Learning, does fit well to how Baconian Induction is defined by Wikipedia. So I’d argue that Induction is real, it’s really used, and it works somewhat similarly to what Bacon thought.

What Bacon Got Wrong About Induction

Bacon did get some things wrong. For example, an “inductive bias” is actually a specific “prejudice” or, in other words, theory about the data. Contrary to Bacon’s ideas, it’s been shown mathematically that for Induction to work you must have an inductive bias. [4] Or put another way, real Induction requires theory to even get off the ground.

Further, you have to use different inductive biases for different problems due to the No Free Lunch Theorem. That is to say, there is no single method of Inductive learning superior to all others; all succeed or fail equally well given the right observations (data).

Induction is also severely limited in other ways. For example, it gives only probabilistic results and the results will be wrong (with high probability) if the probability distribution you used for your training set (the original observations) does not match the probability distribution of the real world. Since the world is constantly changing, this poses a problem for Machine Learning, namely that you have to constantly update your observations and retrain so that the model (hypothesis) you’re using matches what the world looks like today. [5]

Einstein and General Relativity

So Induction is real, but it has nothing to do with, say, how Einstein came up with General Relativity.

When Einstein came up with General Relativity (GR), he first started with a problem: light seems to be a maximum speed. How was that even possible?

He wanted a good explanation for how to understand reality such that light would always be the maximum speed. He also knew whatever conjectured hypothesis he came up with needed to explain why Newtonian physics has seemed to work so well for centuries with no one noticing that it wasn’t quite correct. That is to say, the new theory had to encompass the old one, including all observations that worked for that old theory but also producing new critical tests where the old theory made the wrong prediction and the new one made the right prediction.

Undoubtedly Einstein also knew his proposed hypothesis must not violate other known physical theories, say maybe the laws of thermodynamics. (Though I’m not aware of anything that suggest he specifically used thermodynamics to inspire his theory.)

So Einstein came up with his theory not from some large training set of observations that he then generalized, but from an attempt to solve a problem and constraints against past and existing theories.

He then subjected his theory to a series of critical tests that the old theory couldn’t predict correctly and the new theory could. He put his theory out for criticism of other physicists. (Who initially rejected it!)

His final theory was not probabilistic on whether or not it makes good predictions. And while undoubtedly we’ll eventually have to discard GR for a better theory (to cohere with quantum theory, for example) that updated theory will have nothing to do with the world changing its probability distribution. In fact, the process Einstein used has nothing at all to do with probability distributions of anything, like real induction uses.

So it’s easy to see that Einstein’s scientific process was completely different than Induction as used in Statistics and Machine Learning and that that process of Induction really has nothing to teach us about science.

Kieren’s Inductivism Isn’t About Statistical Induction

Now I suspect Kieren will object here that when he speaks of Induction he’s not talking about Induction as used today in Machine Learning and Statistics. I would anticipate (based on his response to me above) that he’d make two main points to me: 1) that light was a maximum speed is still an observation, and 2) all the theories constraining GR themselves had observations — including all those observations that did match Newtonian physics for centuries.

Therefore, Kieren would probably still conclude that the process Einstein used to come up with GR is still “Inductivism” even if it has basically nothing to do with Induction in Statistics and Machine Learning.

But I think the key point here is that Bacon’s original Inductivism has no relationship to how Einstein came up with GR (other than, as Kieren points out, the fact that observations were somehow involved). However, Einstein’s approach to discovering GR is a nearly perfect match to Critical Rationalism — including now Deustch’s HTV criteria.

Is Kieren’s “Inductivism” Really Just Crypo-Critical Rationalism?

Kieren has only so far insisted on understanding “Inductivism” as nothing more than “constraint by past observation.” I’ll openly admit that a problem is generally also an “observation” in the regular English sense of the term, though not the way I would normally think of it when thinking of actual Induction. I also admit that all theories we use today at one point had their own problems that inspired those theories, which were therefore also observations. So given Kieren’s definition of “Inductivism” (which clearly is not the same as either Baconian Inductivism nor actual Induction as used in ML and statistics) that CR is a form of “Inductivism” if this is the way we’re going to define it.

So I think my first response to Kieren’s objections here would be to quote Popper, who had no problem at all calling the Critical Rational method he was proposing “Inductivism” if that is what we want to call it:

So I must say now that I do not believe there is such a thing as an inductive method or an inductive procedure — unless indeed you decide to use the name ‘induction’ for that method of critical discussion and of attempted refutation which I have described here. I never quarrel about words, and I have of course no serious objection if you wish to call the method of critical discussino ‘induction.’ But if you do, then you should be aware of the fact that it is very different from anything that has ever been called ‘induction’ in the past. [6]

What Does Kieren Actual Mean by Inductivism?

My guess would be that Kieren will not be satisfied with this, because it would mean that what he is calling “Inductivism” is really just crypto-Critical Rationalism (and not the other way around as all the insight comes from CR, not Inductivism). If this was all he meant, it would be a fairly misleading statement at best and certainly has no insight to offer us.

So I think Kieren is actually meaning something else. But what does he possibly mean?

Do Observations Always Come Before Theory?

Honestly, even after reviewing Kieren’s original post and his responses to me again, I’m still not clear what he’s getting at entirely.

Kieren stated: “I was just taking an additional step than you to see where our theory was being constrained by observation.”

But why stop there? Why not then notice that all observations are theory-impregnated and therefore note that we can go back another step back to a theory? Would this prove something? If not, why does his ability to go back a step to an observation prove something?

One possibility here is that Kieren does not accept the idea that all observations are theory-impregnated. If so, I do agree — I feel even the likes of Kuhn overwhelmingly demonstrated that observation is always theory-impregnated. But if this is not something Kieren accepts, then I would see this as the actual point of disagreement between him and Critical Rationalists and this would be why his article actually fails to show HTV criteria implies Inductivism. To prove that, he’d have to actually address why observations are not theory-impregnated, which he doesn’t do.

Another possibility that comes to mind is that he feels it’s intuitively obvious that observations play a primal role compare to theory, i.e. that the chicken and egg nature of theory and observation must start with observation, not theory. However, if this was his point, he never actually makes it. Further, if it’s his point, I don’t think he’s correct.

Which Comes First, The Theory or the Observation?

Popper’s answer to the actual chicken and egg problem is to reference biological evolution — each chicken and each egg that came before was a earlier kind of ‘egg’ or ‘chicken’ until it’s no longer a chicken (or maybe even an egg) anymore. [7]

That is also Popper’s answer to the Theory and Observation paradox. If every theory we know of comes from observations (at a minimum, a problem which is a type of observation) and every observation is theory-impregnated, we can go back in time until we find that we’re no longer dealing with ‘theories’ of science but merely myth. Go back further, and eventually we’re not even dealing with myth, we’re dealing with biological theories built into our genome before humans even existed.

Interestingly, this means that Kieren is wrong that “A HTV theory is one in which each of its concepts is constrained by some past observed phenomena” unless he wants to stretch the word “observed” to include the observerless biological natural selection process.

Now perhaps Kieren doesn’t agree with that formulation, but again, his write-up never addresses this. Given this is a substantive part of how a Critical Rationalist would see the world, this is something he really needs to have addressed in his argument.

Conclusions

It seems to me, especially give Kieren’s point #2, that Kieren is merely making a circular argument. He believes observation via induction produces explanations/theories. Therefore, so long as he can show that Deutsch’s Hard to Vary criteria is constrained by past observations, he figures that’s sufficient to prove his point because it’s obvious (to him) that observations vs induction produce explanations/theories. In other words his conclusion and his starting premise are the same.

By comparison, a Critical Rationalist does not deny that theories come from observations — particularly if that observation is in fact a problem to be solved such as with Einstein and the speed of light — but rather they believe observations and theory are a chicken and egg problem where problems (which are observations), which give rise to theory, which give rise to observations, etc. This explains why a Critical Rationalist is not impressed with Kieren’s argument so far.

I’d encourage Kieren to address the actual differences between Inductivism and Critical Rationalism by explaining why he does not accept that observations are theory-impregnated or why he believes theory and observation are not just a chicken and egg problem that, in fact, stops at theory, not observation.

Notes

[1] “So I must say now that I do not believe there is such a thing as an inductive method or an inductive procedure…” I quote this elsewhere in the article. See The Myth of the Framework, p. 104.

[2] “The idea that we can purge our minds of prejudices at will and so get rid of all preconceived ideas or theories… is naïve and mistaken. The rule “Purge yourself of prejudice!” can therefore have only the dangerous result that, after having made an attempt or two, you may think that you have succeeded — with the result, of course, that you will stick more tenaciously to your prejudices and dogmas, especially to those of which you are unconscious.” (Myth, p. 86)

[3] “…even observations, and reports of observations, are under the sway of theories…. Indeed, there is no such thing as an uninterpreted observation, an observation which is not theory-impregnated.” (Myth, p. 58)

[4] See Machine Learning by Tom Mitchell, p. 42: “The Futility of Bias-Free Learning.”

[5] I would also note that when Kieren originally used the term “observation” I had the Inductive sense of the term in mind. I.e. some sort of set of observations that I then am supposed to generalize. I did not initially have in mind that constraint by any other theory counted also the observations (problems) that led to that theory. This doesn’t seem much like induction to me at all, as I’ll explain in the rest of this article.

[6] See Myth of the Framework, p. 104. Popper goes on to say: “For induction was always supposed to establish a theory , or generalization, while the method of critical discussion does not establish anything. Its verdict is always and invariably ‘not proven.’” This is another place where I’d like to see Kieren address better what he believes “Induction” to be. I do not see him insisting that observations can verify or establish a theory a proven true. Therefore, his version of “Induction” may just be something else entirely different than how the term has been used in the past.

[7] Example taken from Bryan Magee’s Karl Popper, p. 26–27.