Against the Burden of Knowledge

If we look for example at the career of Kepler, we see that he had to accumulate expert knowledge of both systems and that he arrived at his later so-called laws not by a sudden insight, but through tireless work, trying again and again over several years to make sense out of series of measurements that did not really make sense in either the Ptolemaic or Copernican system.

It was not the epicycles that worried the late medieval astronomers, but the fact that according to their geocentric theory the earth was not really exactly in the center of the deferent. Correspondingly, however, the sun was not really at the center of the Copernican system either. Both systems lacked elegance, and both systems were more or less equaly in accordance with the observational data. To solve this debate and to make progress, the strategy of the most famous astronomer of his time, Tycho Brahe, was to collect better measurements. And it were these accumulated measurements that enabled his pupil Kepler to develop a solution. In a sort, he was lucky. Only the data for the planet Mars, which he considered first, was exact enough to match an ellipsis. His data for Jupiter or Saturn would not have allowed him to come up with one.

Accumulated knowledge was not an obstacle, but the basis for Keplers insights. That we might get a different impression is a result of simplifications that happen after a scientific community reaches a conclusion. At that stage such a theory becomes textbook knowledge: a student needs no longer acquire the accumulate knowledge of the previous period, but only the successful doctrine. These doctrines appear and are all too often presented as ingenious insights from geniuses, instead of cumulation points of a collective work of generations of scientists and scholars.

There has never been such a ripe time to find such ample reason to propose and discuss new and radical ideas. And there has never been such a hostile time to looking into it.

In physics we’ve lost 50-70 years to string theory and related nonsense and the people still working on effective quantum field theories have had to re-brand as working on “quantum information / quantum computing” to stay in rent and lab desks. There aren’t many serious QFT people who doubt Everettian epistemology but it’s still nothing undergraduates hear about. Most serious non-string gravity work comes out of Perimeter and Marletto/Deutsch.

In economics and finance we still put our hand into the sink shredder of the Chicago-style strong-form EMH, as resoundingly and empirically disproved by Simons (may he rest). And we continue to embrace Friedman-style supply side economic policy, without peer the nastiest wealth transfer from those with surplus away from those facing scarcity at the barrel of a gun in absolute terms ever. It never trickles down.

In the fundamental epistemology of our day we are surrendering the reigns of influence to people who market chat bots trained to lie convincingly at scale.

Knowledge, and the new ideas that approximate at the limit the derivative of knowledge are under attack. They’re under attack by some rich guy’s choice for the job he did well and decided to keep in the family.

If the rent in a city is too high you are not going to get the MOST interesting restaurants, bars, and clubs. You are going to get only the businesses that will DEFINITELY convince an investor to write a check to dump on such high rents; regardless of whether that is a good idea or not.

The PhD cohorts for R1 Universities hasn't really gotten any bigger than 50 years ago. The number of academic jobs hasn't really gotten any bigger either. The only people having success in the system are the types of people that seem low-risk to the system.

So of course we should expect a decline in innovative ideas as time rolls on. The only way to reverse this is to literally create more tenured jobs (or perhaps temporary tenure ex: 10 years you're guaranteed employment) and increase the size of PhD student cohorts so that they are large enough that iconoclasts can fit in again.

Given that, is is surprising that progress does not scale proportionally? Maybe, maybe not. For sure my comment is not backed by anything but anecdotes.

I think another cause could be bureaucratic. From what I understand, several hundred years ago, research was funded by rich patrons who wanted some credit in relation to discoveries. Nowadays there is a giant, lurking, centrally planned grant machine that distributes money to researchers. And as we know from economics, central planning becomes increasingly untenable as the system becomes larger. Results get worse as the complexity skyrockets.

Additionally, if we are going to blame the “burden of knowledge” in any capacity, we have to acknowledge the abysmal education system (in the US). An anecdote which I will never forget is, in 6th grade we had a student who had recently immigrated from India. By his own telling, he was an average student there. But compared to native students, he was light years ahead in math. We learned multiplication in third grade, division in fourth grade, long division and fractions in fifth grade, or something like that. He had learned multiplication in first grade, and all of division in second grade. While our smartest students in sixth grade were grappling with pre algebra, he was bored in classes with the eighth grade algebra students. Our education system has, since experiencing that, seemed to me deeply flawed; there really is no reason an efficient and effective public education system should take 12+ years to be able to have a child ready for college and then another 4+ in college to have them ready to contribute to an isolated field.

disagree here. it does not make it easier. to show why a new model is right or superior means having to understand the old models well enough to show it's wrong or suboptimal. hence knowledge.

The hard part is what to do about it.

“ Ptolemy and his astronomical ancestors explained these “retrograde” motions with the extra loops you see in the map above called “epicycles.” By the 15th century astronomers had accumulated centuries of meticulous measurements and incorporated them into complex orbital paths, matching their observations. Learning these models and taking enough measurements to improve one of them took an entire lifetime of monastic devotion to studying the stars. The burden of knowledge was immense.

But then, Copernicus came along with the heliocentric model which, in its simplest initial form, made worse prediction than the tuned-up Ptolemaic model. But the burden of knowledge was dissolved in an instant. Improving the Copernican model meant shifting orbital paths from perfect circles to ellipses. It had nothing to do with the epicycles and perihelions of the Ptolemaic model and none of that burdensome knowledge was necessary to expand the frontier anew.”

First, it agrees with the notion of a burden of knowledge, by agreeing that improving the Ptolemaic model took huge amounts of time to learn its intricacies.

Secondly, the Copernican alternative worked not because it was simpler, but because the Ptolemaic model was wrong. There is never a guarantee that a model we currently is is wrong and can be replaced by a substantially simpler model. Maybe someone brilliant will come along and find a simpler model than quantum electrodynamics, but maybe they won't: it's absolutely possible that, say, QED is the right model, and we can at best make it even more complex to explain more details of phenomena.

Finally, tools only get you so far. Tools don't necessarily encode some of the scientific knowledge you need to advance fields. They are a separate track from it.

The problem is that most new ideas are, in fact, cumulative, and hence increases the burden at a faster rate than disruptive ideas reduce it

The idea of "Burden of knowledge" only requires that most knowledge accumulates. The fact that there are exceptions that do not accumulate does not disprove it

PS: Since these are articles penned by academic economists rather than pro scientists, it'd be nice to hear criticism from an economic (or finance) angle..

It means that to push boundaries we need more money to buy/construct new tests. In other words, you need exponentially more money. Which leads to the kind of blockbusterization of science. You don't get hundred blockbusters each year, instead you get two, by team with track record. Which sounds a lot like institutional decay.

We are in a period of widespread outright institutional collapse. Institutional decay should be the null hypothesis when considering the causal factors behind any widespread problem