Writes Large Correct Programs (2008)

I’ve worked on and finished some extremely large programs over my years in non-tech enterprise. I’m also an external examiner for CS students, and I’ve regularly talked about how I think the curriculum is outdated. In Denmark where I’m from it’s rare to get a programming application from someone without a related degree. It wasn’t so rare 20 years ago, but I can’t remember when I saw one last. I agree that degrees, and especially CS, doesn’t guarantee that people can code. We use them mostly as a “safety net” in our hiring processes here in my region of the world. Basically you can view hiring the wrong person as the most expensive mistake you can make as a manager, and educations are a sort of risk management. You might think that the “can this person actually program” risk is worst for people who are fresh out of their education but there are a lot of factors which can play into it. Older devs may be set in their ways, maybe even religious about some sort of programming philosophy. On the flip side they will start producing value right away.

Anyway, by far I think the biggest hurdle in our industry right now is pseudo-jobbers like project managers, business process owners, scrum masters, various architects and what not. Not everyone is a waste of time, some of them do excellent work and function as exponential productivity catalysts. The vast majority of them, however, spend so much time engineering the process, architecture, whatever that their teams never ship on time or within budget. In this sense I think “correct programs” is hard to value. Because often the “incorrect large program” that doesn’t scale, will be much more valuable for a business than a “correct program” which never even gets the chance because it took to long to get out there.

This reminds me strongly of reaching the final year industry projects in my software engineering degree, and seeing a significant portion of my colleagues unable to develop software in any meaningful way.

There was a curriculum correction in the years afterwards I think, but so many students had zero concept of version control, of how to start working on a piece of software (sans an assignment specification or scaffold), or how to learn and apply libraries or frameworks in general. It was unreal.

When I was a prof (many years ago), I was working on a database for a political campaign. The code was a mess. I asked a couple of my colleagues (successful comp. sci. profs) to help out. It became clear very quickly that there are two kinds of comp. sci. profs: those who can program and those who cannot.

Can absolutely relate and understand.

I taught myself C++ by writing games with SDL2.

The first game -- snake took about a couple of hundred lines and I put everything in one .CPP file. And I felt pretty good.

The second game, well, I forgot what it is, not Tetris nor Breakout, but it was complex enough that I realized that I need to put code into header files and source files.

The last game of that project was a Ultima-spinoff. I even used the same sprite sheet. The complexity completely drowned me. I was constantly asking myself how should I arrange things properly so I don't need a lot of global variables that every file can see, because naturally the Game class needs to see and know all other classes, and the Renderer class needs to see and know many other classes too, etc.

Eventually I dropped the project. A few years ago I picked it up again and figured out something that is close to Entity - System (not ECS just ES). I didn't complete the project but then firmly believe that it was the right architecture, and I was simply too burnt out to complete it.

This year I learned about ECS from pikuma. I think it's over complicated for small-medium games. Anyway I'm trying to say that I agree that writing a 10,000 line project is way more complicated than 10 1,000 line projects.

My non-technical co-founder decided to teach himself coding in the past couple of years.

He can write quite complex logic but he tries to make everything as generic and flexible as it can possibly be and so his code is very hard to read as it invents a lot of concepts. He had to refactor it many times due to bugs or unforeseen technical limitations. Took him months to write. On the other hand, I wrote a script which does a similar thing as his but for a different environment (over a network with added latency, bandwidth limits, network instability) and only covers the essential cases but it only took me about a day to write and has been working without significant flaws since the beginning. Only had like 1 bug for a minor edge case. Also, my code is very short relative to his.

This experience reinforces a rule for coding that I've had since the past 5 years or so. It's basically:

"If you can't explain what your code does, from start to finish, in simple language to a non-technical person (who has the required business domain knowledge), and in a way which answers all questions they might have, then your code is not optimal."

Abstractions should always move you closer to the business domain, not towards some technical domain which you invented and only exists in your head.

The part about "start to finish" is important but doesn't mean "every line". You should have abstractions which may be beyond the non-technical person's understanding but you should be able to explain what each abstraction does at a high level without jumping into each function. You should be able to walk through any major feature by walking through your code, without having to jump around many files and without having to pause constantly to explain abstractions as

It seems to me that in other areas of tech, companies generally hire electrical engineers, mechanical engineers, civil engineers, etc. On the other hand, software companies feel that they don't need to hire computer scientists.

Then periodically there is a discussion on Hacker News that boils down to "all of the other engineering disciplines can make reliable predictions and deadlines; why can't software?" or "why is this company's code so shoddy?" or "why are we drowning in technical debt?".

Perhaps the these are all related?

Computer scientists advance in their careers by writing papers, software developers do by writing programs. Some CS grad students and profs are genius programmers, but mostly CS researchers write a program that at best lacks the polish of a real product and at worst almost works.

When I was in physics grad school I had a job writing Java applets for education and did a successful demo of two applications at an CS conference in Syracuse and was congratulated for by bravery. I was not so brave, I expected these programs to work every time for people who came to our web site. (Geoff Fox, organizer of the conference, did an unsuccessful demo where two Eastern European twins tried to make an SGI supercomputer show some graphics and said “never buy a gigabyte of cheap RAM!”)

I'd prefer maintainable programs of any size.

Other than non-trivial academic samples, the odds of a program needing to change over its lifetime or large, and it's current apparent correctness has little to do with someone else adapting it to the ever changing environment.

The number of times I've heard "it seems to work and we don't dare change it" is far too many

One thing that’s missing: programs are mutable. A good programmer writes programs that are easy to maintain and extend.

When choosing my degree back in the mid-1990s, I chose a BEng in Software Engineering, and not a BSc in Computer Science because I wanted to enter a career writing software, not a deep study of the theory of a branch of applied mathematics.

I was fortunate enough to have figured this out for myself, and whenever I met a CS grad in my early career it was obvious that the production of actual software terrified them.

Meanwhile I'd learned how to build (and how not to build), working programs in C including a simple OS on an M68k chip on a VME bus. I struggled with my final year project because it became too theoretical and CS-ish (trying to write a Prolog to SQL interpreter), so my grade took a hit, but I am really glad I entered industry with useful, practical skills that employers valued.

There's always going to be a place for pure CS, I'm glad it exists as a discipline, but more kids should understand the difference, and more colleges should be offering degrees that teach people how to software is built (and how to build software yourself), not just write papers.

I graduated in CS at a renowned university over two decades ago. The faculty was very theory-oriented, basically every lecture was just theorem, proof, example - even the programming lectures.

As a matter of fact, there was only a single lecture I took (and which I didn't need to take) where we needed to use computers for the weekly exercises.

I think also, being able to write a minimal amount of code to implement any given feature is important.

However, your code should anticipate a range of possible future requirements changes... But then it shouldn't try to be a silver bullet either. Experience helps a lot as it allows you to see hurdles and limitations ahead of time so you know exactly how much silver you can put in your bullet for any given part of your code.

When I was in the college, I'd made some consultancies to final students from CS degree which don't know how to program and even bad, the project was a simple copy and paste some classes, change attributes and manually test it. So, the main problem I'd seen was they don't have the ability to figures out the system and it's a basic expectation!

Computer Science is to Software Engineering as Physics is to Civil Engineering.

Software Engineering is to programming as Civil Engineering is to construction.

Related:

Writes large correct programs - https://news.ycombinator.com/item?id=2556270 - May 2011 (32 comments)

I can definitely write a 1KLOC program to solve a 10-line problem.

The key observation about the Dunning-Kruger study is that humility, hard graft, and honest perseverance towards improving oneself are the three most important factors in achieving proficience in crafting large software data flow systems. How one designs and implements such systems is the art of any form of creative endeavor.

Designing and implementing large and correct systems is a matter of growing them, from small, trusted pieces into larger interconnected systems of systems, with ever greater care, knowing that the entire thing can collapse at any time if the wrong decisions are made or have been made.

There are many axes of complexity. Routine line of business systems, say an inventory management system for a car dealer, with a proper architecture costs should be additive instead of multiplicative, a certain cost to develop features such as “autocomplete control that draws values from a database row” and a certain cost to deploy that control. Double the number of screens using the same features and you double the cost but it feels like sub linear scaling because for the second tranche of forms you don’t have to redevelop the components and it can go much faster.

That ideal can be attained and you can be in control in application development but often we are not. When you are in control the conventional ideas anout project management apply.

As you get very big you start having new categories of problems, for instance a growing social system will have problem behaviors and you’d wish it was out of scope to control it but no, it is not out of scope.

Then there are projects which have a research component whether it is market research (iterate on ideas quickly) or research to develop a machine learning system or develop the framework for that big application above or radically improved tools.

A compiler book makes some of those problems look more regular like application programs but he project management model for research problems involves a run-break-fix trial of trying one thing and another thing which you will be doing even if you are planning to do something else.

Livingston, in Have fun at work says go along with the practices in the ocean you swim in (play planning poker if you must) but understand you will RBF are two knobs on run-break-fix: (a) how fast you can cycle and (b) the probability distribution of how many cycles it will take. Be gentle in schooling your manager and someday you might run your own team that speaks the language of RBF.

Unit tests put a ratchet in RBF and will be your compass in the darkest days. They enable the transition to routine operation (RBF in operations is the devil’s antipattern!)

They are not a religion. You do not write them because a book told you so, you write them for the same reason a mountain climber wears a rope and if you don’t feel that your tests are muda, waste as they say in Japan.

> how to organize software so that the complexity remains manageable as the size increases

So John is missing the role of software architect here. Science, art, and development - 3 roles. Not all visits to the stratosphere are misadventures.