When building a new piece of software or a new piece of architecture, a common practice is what I would describe as wasteful by design. Often, it is no longer a question of how to use technology to solve a specific problem but how to shoehorn technology into a problem without a genuine regard for specific requirements or actual needs.
When business requirements conflict with engineering reality
A case may be made for when projected scalability requirements are over-optimistic but wasteful by design is not about this business reality; it is about designing a system for a set of requirements or numbers without actually thinking about what is required to build.
It was racks of shiny appliances; then too many virtual machines, it was “elastic compute” then containers, then Kubernetes, and now serverless. Every time one of these trends appears, pragmatism disappears in a swarm of tech debt.
It was SQL databases, then NoSQL, then serverless databases. It is often unclear what is talked about, a query language, a database engine, a replication technology, a storage paradigm, or a marketing term?
This confusion makes for hopeless choices, which, more often than not, does not fit the domain problem well. That is how we see MongoDB being used to develop banking software, Kubernetes to deploy static websites, and Hadoop to process excel-scale data.
Finding Excuses
Often, when telling someone how absurd some technology choice ultimately is, a multitude of excuses are brought up. The goal is to make the original domain problem overreach to absurd lengths to appear for someone pragmatic. Nevertheless, that is not about being practical but appearing as someone with deep domain expertise and not protruding on the initial choice by portraying criticism as a lack of expertise or understanding of the problem at hand.
Showing Real Pragmatism
When an engineer shows pragmatism, he is now a pessimist or someone lacking proficiency for not choosing the “right” solution. Even worse, this train of thought leads to fashion-based designing, where the last trend is the right engineering choice.
Many examples exist in the real world, be it in programming languages, where the latest construct is better than the old one, and frameworks where the newest design pattern must replace the previous architecture, in deployment solutions where freshest rhymes with best.
Simple is bland; complicated is superior
When something becomes too well known or too prominent, even when previously considered state-of-the-art, it is now a vulgarity that one should avoid.
Because people may implement simple concepts earlier on, they become well known not only due to time but also by the nature of being easy to teach; it is then natural for novel concepts to be more convoluted.
Some people then get attached to complicated concepts just because they are complicated. Be-it through not wanting to lose the time they invested learning about some notion but also because it may make them feel intellectually superior for using convoluted technology.
Laziness and Lack of Expertise
It is easier to over-engineer than to engineer a solution just right. Significantly over-engineering a solution allows for a complete lack of proper planning and understanding by wasting resources.
When technology choices become fashion-driven, it is even easier to promote inefficient or just inappropriate solutions as an expert knows or uses the latest trend.