Zen and the art of (Simulink) software maintenance.

call for help.. (and a long post)

As Rob Pirsig had said, the mind is a bundle of ghosts, ideas that exist on their own accord, evolve over time and help or hurt us. Newton's laws, common sense, and the golden rule are some of the best. They are reinforced by the affirmation from repeated experiments or experiences and by the social rewards, cooperation they enable. But there are others.. the wretched ones.. which cause the mind to go out of balance and lose its connection with reality, to lose its calm and dive into spirals of anger, hatred, doubt, cynicism, conspiracy theory, prejudices, inferiority or supremacy complexes.. the list goes on. In some sense.. the mind can become infinitely more complex than the reality it deals with.. and therein lies the rub. To be effective, the mind needs a system to reign in or weed out undue complexity and biases and have a clear north star, or values in Pirsig's words that guides its quest to make sense and forge meaning out of living. 

Well..  you may be asking by now.. what does this rant have to do with the software? As a control engineer, who writes software for large and small real-time systems, I often find myself struck by Pirsig's analogy. Software, after all, is the ghost in the mind of the machine, and as I tell my son, we control engineers teach machines  how to think. The value of good software only becomes clear when you have suffered the pain of a bad one. Over a career, one often sees as much variety in software quality as the patients walking in a psychologist's office.. 

My first brush with the code complexity was a 6000+ line code in Matlab that simulated a horse's gait, that I inherited from a graduate student during engineering. It was a pure spaghetti code, without comments, and full of go-to blocks.. I had to take a double-sided small type print-out, fill all margins with notes and try and give up 6 times before actually making sense of it. Nowadays, a typical car has 100 million lines of code running through its mind. Although it has become harder to write spaghetti code, to my surprise, it is rather too common to see wretched ghosts roaming in minds of softwares that run our world. It often sticks out like a big tumor of technical debt that is risky to operate on, should the patient bleed to death.

Specifically I want to highlight a common problem for Simulink based real-time control software which I have seen repeat often. Simulink revolutionized control software development by its bullet-proof auto-code generation with safety guarantee. But it is a poorly understood tool. Its visual appeal lies in being able to 'see' the complexity. But in wrong hands, it can cause the complexity to persist and grow, because it is hard to see diffs in the code, it is harder to modularize, bus objects need to broken up into component signals before passing to deepest blocks, it is hard to implement object oriented design and it is not clear where we should use Simulink vs. text blocks. Organizations often are stuck in limbo between developers that want to do everything in Simulink versus those that want a mix.

Typical clinical picture looks like this: team of more than dozen developers, design history that spans more than a decade, use of Simulink auto-coding to generate production code. At some point in the past, the code was in C++, which somebody replicated line by line in Simulink, or put lots of hairy Simulink wrappers for legacy c blocks. Multi-layered bus architecture to route signals between pairs of block or global bus. End result is the mind-numbing code complexity.

Inability of a single person to understand the whole, keeps the zombi code around because nobody realizes they are dead. Dependencies spiral out of control and referred pain passes between unrelated blocks.  Interaction matrices, battery of tests, requirements, reviews, design rules and red tapes aim to incise the tumor or starve it, but fall short of curing the patient. This complex software works to everyone's surprise and relief until it doesn't, and it takes heroic efforts to debug and fix the symptoms, but the root cause, the complexity, marches on. It feeds on itself as developers work outside and through the spaghetti, adding to it. Clean slate design often tempts developers, but the sheer burden of enumerating requirements that legacy software satisfies, and the risk of breaking something, discourages the leap of faith and the march down the valley, to the bigger hill.

Software engineers in other text based languages, through trial by fire, have figured out how to tame such complexity by refactoring, simplification, modularity and pervasive and automated unit and system testing. But I am yet to see design patterns, best practices emerge for Simulink driven real time control development that pay off the technical debt, maximize "simplicity" and cure the patient of this schizophrenia of unneeded complexity. Please let me know if you know any..