It's been around 5 years in the software space as a software engineer. In this journey, I started with the backend, jumped around between different full-stack technologies and currently settling down as the DevOps Engineer, occasionally handling difficult and complex parts of the system development.

Throughout my career, I've worked with numerous frontend frameworks, backend frameworks and operational strategies and technologies. This brings back a lot of painful and happy memories. In this blog, I've written about my views from my experience which may or may not be correct.

One problem I've always faced in the majority of the projects is as the projects grow it gets increasingly difficult to change things, observe things and gain a better understanding of the system. When we aren't able to have a good sense of the system, we can't properly architect the new change.

One recent example; I was leading in moving the auth-module of a system from self-backend implementation to Google provider. When I started the project it was really simple, I had implemented the auth module as a separate testable entity that could be plugged into other modules and easily modified. As the project grew with other developers, they started plugging in middlewares, modifying attributes of the auth module and injecting unneeded things to the auth state just because it was properly designed to hold the state in the request-response lifecycle.

Now, I thought to change the internal implementation of the auth module, and sweet hell Mary, everything was a mess. Our middleware broke, and authorization didn't work. Most of the modules that depended on the auth module broke.

Another example; I was leading the development of a data labeling system. We envisioned this system would be complex, so we chose to separate systems into 3 different components. This time, I had bootstrapped the projects and the code was in proper shape in the initial commit. But, soon after from Day 2, there were blockers. Separate components had separate developers working on them. The interaction between components was over HTTP, and communication issues between these components ate up a lot of the development time. By a lot, I mean the development time was extended significantly. Now, when I think about it, it boils down to "premature optimization is the root of all evil".

Another example; This was a streaming site built with a claimed "Super Fast" NodeJS framework at the time. I thought, okay streaming is a tough job. Let's write this service with NodeJS and a custom ffmpeg wrapper in NodeJS. The super-fast framework was so slow to develop, that it had no community plugins. I wrote a lot of things myself which I could have got for free. If I'd sacrificed 1% speed and used a slower framework, my productivity would be 100x.

What did I learn from it?

I learned two key things from it:

1. Developers often fail to understand the intention behind the code.

2. You don't need to think about scaling to 10 million when you're writing your first line of code.

3. There is no such thing as "It worked for X corp, it should surely work for us."

4. Don't go with "Yeah, I've worked with X framework between these choices, and I think it's a good fit."

5. Performance isn't always the top priority.

How do I do it now?

Now, I've seen enough failures to identify the correct one. I'm confident enough to tell that whatever you're building, it should follow these characteristics:

• should have as little magic as it can.

• dx is not a thing to ignore.

• Most of the things should be automated.

• Interfaces and boundaries should be thought of at first.

• The system should be observable.

• There should be a proper document in place to understand "why" for all ADRs (architecture decision records)

In the starting, I'll start with some BaaS (Firebase, Supabase, Strapi maybe) because it's not worth investing a lot of code and developer time when your (potential) clients and software don't even understand each other.

Most of the BaaS makes things observable, you can know which services are getting most requests, which queries are taking the most time, what time you are getting a lot of traffic, which is the bottleneck in your system etc. You also get sweet other things out of the box such as authentication/authorization, and data persistence layers and the best of them all is dx with it.

After a certain period, you'll get a perfect understanding of which way your software is heading. You can slowly start separating the bottlenecks of the BaaS platform into separate services. You write your code this time, as serverless functions if possible for your service. And you plug it into your BaaS.

Now, you need to start finding out how that BaaS does certain things and how'd you on about doing that on your own, research on the database performance, language, toolings etc.

After a certain level of maturity, you'll realize your business logic is getting too complicated to tune your BaaS and you need to do it on your own. This is the moment you start slowly rewriting your backend and shifting your API into your backend, one service at a time.

At last, this is my personal experience. Let me know if you have any conflicting views or I misunderstood something, I'm learning every day and would be very happy to correct myself.