Fatigue Driven Development (FDD)

Hello, my name is 👨🏻 David Shekunts 👴🏿 and I'm tired of debugging at night, I'm tired of unmaintainable code, I'm tired of technologies that break. Here I collect best practices that help me rest more. I hope they will help you too.

📚 Table of Contents

📖 Dictionary

• "Data source" – database, cache, third-party API, basically anywhere we get data from or send data to.

• Application Interface (API) – a way to interact with a running application (stdin, stdout, TCP, UDP, )

• "Endpoint" – one of the API methods (e.g., HTTP API URL, gRPC command description, mutation/query/subscription in GQL, etc.)

• Message Broker (MB) – ability to publish messages (usually by "topics") and subscribe to their appearance

• Message Queue (MQ) – same as MB, but guarantees delivery sequence

• Event Driven Architecture (EDA) – we throw out "Events" (id, name in past perfect tense, data) and consume them with any number of services.

• CI / CD – automated processes required for building and deploying services

• "Infrastructure" – databases, CI/CD, servers, orchestration, network, and everything else

• Orphan data – data that only made sense when the data it was tied to existed (essentially, any data we want to make ON DELETE CASCADE)

• "Projection" – readonly data calculated based on other data. For example, we save all messages from a device to one table, and then calculate projections of its current and historical temperature readings / battery status / location / etc.

• System degradation – when new functionality breaks old functionality, considered one of the worst types of errors.

• Feature flag – boolean value (true | false) that you pass into code (via env) and depending on whether it's on or off, you enable or disable functionality

• Real-time – systems that involve processing and responding in milliseconds (maximum 1-3 seconds)

• Application (app) – code running as a process

• Instance – a unit of a running application

• Transport – any way two processes interact, involving sending/receiving data (TCP, HTTP, MQ, 2IP, stdin, etc.)

• Internal communication – calling business logic within one application by calling a function from within code

• External communication – calling business logic from another application using some Transport

🍔 App structure

i. Monorepo

1. Atomic deployments – both backend, frontend, and infrastructure code are deployed to production at once

2. Everything in one place – even with 5 repositories, problems start with finding the right things

3. Shared code – ability to use local code between services without publishing

1. Development and staging environments, so that development is done in the first one, and changes that will then go to production go to the second one

2. If you need to publish libraries, be sure to add a CI/CD stage with manual control for their building and publishing (main releases the main version, development releases beta, other branches release alpha)

ii. Monolith

1. One large codebase

2. Runs in approximately 1 instance

1. Avoids distributed problems (cross-communication and state synchronization)

2. Fast deployment

3. Less hassle with Infrastructure

4. Easier to debug

1. All the above pros work up to a certain size, after which they completely disappear

2. Difficulty of horizontal scaling

3. Single point of failure

4. Without code writing rules, it turns into spaghetti

iii. Microservices

1. Codebase separated from each other

2. Ideally, their own resources (different servers, databases, caches, etc.)

1. Can isolate codebase in a separate place (security, management)

2. Theoretically maximum horizontal scaling

1. Microservice boundary errors are much worse than any monolith, no matter how big

2. Complexity of communication and state synchronization at maximum

3. Difficult to deploy

4. Difficult to monitor infrastructure

iv. Modular monolith

1. Reuse the same codebase but run in different instances

2. Use the same data sources

1. All the advantages of monoliths, without the point of irreversible bloat

2. All the advantages of microservices, minus the complexity of cross-communication and state synchronization

1. Sharing resources (e.g., database queries) can cause them to overload

2. More difficult to deploy and maintain than a monolith

3. Without code writing rules, it turns into spaghetti

v. What to choose?

1. Start with a modular monolith, trying to avoid cross-service communication as much as possible

2. Extract something into microservices out of NECESSITY, meaning you'll simply see that there's really no other way, then extract something into microservices (this point may never even arrive)

vi. How to divide microservices

a. Separate team

b. Security

c. Geo-dependence

d. Stateful

e. Separate feature set

f. Dedicated resources

g. Interferes with others

h. Deployment independence

i. Business logic independence

j. Mini-product

vii. FALSe - Best Project Structure

1. Features - code with business logic of the application, divided by domains.

2. Apps - code that launches the application in different configurations.

3. Libs - code that could become private or public libraries.

4. Scripts - code that we occasionally want to run locally or from CI/CD.

/features
	/auth
		login.ts
		register.ts
	/user-management
		get-user.ts
		delete-user.ts

/apps
	/main-http
		/http-api
			schema.ts
		config.ts
		index.ts
	/cron
		index.ts

/libs
	/@my-company
		/specific-lib
			index.ts
	/logger
		index.ts

/scripts
	/inactivate-stale-user
		index.ts

viii. Graceful-shutdown

1. Start a timer that will kill the application even if it hasn't finished

2. Close APIs (HTTP, MQ, etc.)

3. Turn off all cron jobs and intervals

4. Wait for current business logic processes to complete

5. Close external connections (DB, MQ, etc.)

ix. Separate your Cron

🎛️ API

i. RPC

// # Request
{
	id: string
	name: "GetUser",
	params: {
		userId: string
	},
	meta: {
		ts: Date
		requesterId: UUID
		traceId: UUID
	}
}

// # Success Response
{
	id: string // same as in request
	result: {
		case: "success",
		success: {
			email: string
			avatar: string
		}
	}
}

// # Failure Response
{
	id: string // same as in request
	result: {
		case: "failure",
		failure: {
			code: number
			message: string
		}
	}
}

{
	id: string
	name: "UserCreated",
	payload: {
		userId: string
	},
	meta: {
		ts: Date
		requesterId: UUID
		traceId: UUID
	}
}

ii. CQRS

1. If you're returning data, you have no right to change the system state (write anything to DB, cache, change variable values)

2. If you're changing the system state, you can only respond with "OK" or "Error"

iii. (a)Sync Communication

iv. Schema

1. OpenAPI or GQL for HTTP / MQ / MB / WS

2. Protobuf for gRPC / MQ / MB / WS

v. Push | Pull

vi. Create IDs on the client

1. This allows the client to independently request the necessary data in case of success

2. Easier to implement streaming / real-time communication

3. Allows the use of Eventual Consistency

4. The unique identifier will also act as an idempotency key

5. On the backend, you can create entities, link them with this UUID, and only then write to the database (instead of making a record, getting a SERIAL, and only then creating the next related entity)

👴🏿 Type it

i. Branded & Nominal Types and validation on type

Функционально Ориентированное Программирование (ФОП)

Функционально Ориентированное Программирование (ФОП) – функциональная альтернатива ООП.

https://fop.davidshekunts.ru/#d04ae9b9e8e24f3ab46144f1e8816fc2

ii. Algebraic Data Types (ADT) and Invariants

Функционально Ориентированное Программирование (ФОП)

Функционально Ориентированное Программирование (ФОП) – функциональная альтернатива ООП.

https://fop.davidshekunts.ru/#d04ae9b9e8e24f3ab46144f1e8816fc2

iii. implements – incredible evil

// business-logic/some-fn.ts

interface User {
	id: string
	email: string
}

interface UserDataSource {
	getUserById(id: string): User
}

function someLogic(uds: UserDataSource) {
	// ...
}

// databases/pgsql.ts

interface UserTable = {
	id: UUID
	email: string
}

const UserTableService = (conn: PgConnection) => {
	return {
		getUserById(id: string): UserTable => {
			// ...
		}
	}
}

// app/main.ts

const pg = PgConnection()
const userTable = UserTableService(pg)
someLogic(userTable)

I prefix – evil

1. It's a semantic error – you don't name all classes with a prefix C or all numbers with a postfix Int

2. With this naming convention, you tie the interface to the implementation, but the essence of an interface is precisely to decouple one from the other

interface IPaymentService {
	// ...
}

function extendSubscription(ps: IPaymentService) {
	// ...
}

class User {
	constructor(
	public id: string,
	public email: string
	) {}
}

// When you use the User class in typing, you're actually using
// its interface
const someFn = (user: User) => {
	// ...
}

// That is, in fact, the User class consists of 2 parts:

// 1. The class type
interface User {
	id: string
	email: string
}

// 2. The class runtime (conditionally)
const User = {
	new(id: string, email: string): User => {
		return {id, email }
	}
} 

interface UserRepository = {
	insert(user: User) => void
}

// PSQL implementation
class UserRepositoryPostgreSQL {
	insert(user: User): void {}
}

// Mongo implementation
class UserRepositoryMongoDB {
	insert(user: User): void {}
}

🐞 Error Handling

i. Classified Errors

// . First the base error

type BaseError = {
	_isBaseError: true // for simple checking
	statusCode: number // always use HTTP Status Code
	type: string // unique keys of the error dictionary (more on this below)
	message: string // internal error description
	data: JSON // additional data
}

// . The first layer is usually about whether they are available internally or externally

type InternalError = BaseError & {
		_isInternalError: true
}

type PublicError = BaseError & {
		_isPublicError: true
		publicMessage: string // this error message will be shown to users
}

// . Then standard types

type ForbiddenError = PublicError & {
	_isForbiddenError: true
	statusCode: 403
}

type NotFoundError = PublicError & {
	_isNotFoundError: true
	statusCode: 404
}

// and so on

// . Then your specific ones

type SubscriptionEndedError = ForbiddenError & {
	_isSubscriptionEndedError: true
	publicMessage: "Your susbscription ended"
}

type PostsNotFoundError = NotFoundError & {
	_isPostsNotFoundError: true
	publicMessage: "No posts"
}

const errorHandler = (request: Request, error: Error) => {
	if (error._isBaseError) {
		if (error._isPublicError) {
			return request.status(error.statusCode).message(error.publicMessage)
		} else {
			return request.status(500).message("Internal error")
		}
	}
}

ii. Error Dictionary

// errors.json
{
	// key is unique, and the value is any more or less understandable string
	"user_email_validation_error": "Email is incorrect",
	"value_to_long": "Value too long: ",
	"you_are_not_owner": "You are not owner"
}

iii. Return, not throw

1. Program clarity increases tenfold

2. You can type the returning errors

3. Debugging is simplified

4. Code for error handling looks cleaner

5. It's harder to forget to handle all the necessary cases

🧠 Architecture

i. Functionally Oriented Programming (FOP)

ФОП - Функционально Ориентированное Программирование

Функциональная альтернатива ООП ФОП - парадигма программирования, основанная на Процедурном программировании с использованием техник Функционального программирования, разработанная для мультипарадигмальных языков. Для работы с ФОП в вашем языке должно быть всего 2 сущности: Описание структуры данных - описание структур с которыми будут работать наши функции.

https://fop.davidshekunts.ru/101ffcdcef71463c81c9ecf20130d74b

ii. Data Oriented Architecture

Data Oriented Architecture

Функционально Ориентированное Программирование (ФОП) – функциональная альтернатива ООП.

https://fop.davidshekunts.ru/data-oriented-architecture

iii. Dirty vs Clean Architecture

1. Any abstractions of business logic increase Artificial complexity (link), and in CA and DDD, this is maximized.

2. Code is the servant of data (DB), not vice versa, so the fewer abstractions over data in your code, the easier it will be to work with it (link)

3. If you create a program so that you can change, for example, PG to MongoDB, then your program will work poorly with both databases

4. Abstraction over the database immediately assumes that (1) you won't be able to use important features of a specific database, (2) you won't be able to optimize for a specific database, (3) you will work unoptimized and very quickly reach the ceiling of resources you need

5. The more reusable code (for example, the Entities layer), the more danger there is of breaking something that works when adding a new feature / changing an old one (over time, this probability reaches 100%)

iv. Vertical Slices

1. This gives us true independence of code bases, and the chance of affecting something else when changing a feature approaches zero

2. Logic related to a specific feature doesn't pollute the common codebase but stays where it's needed, which incredibly helps in understanding the codebase

3. Testing such code is much easier

4. We can easily enable/disable/move features anywhere

5. Each feature can use its own technologies (you can even change the Query builder, database, library, etc.)

1. Keeping 1000 features in one folder is inconvenient, so feature folders should be distributed across domain area folders (auth, shop, reports, blog-system, etc.)

2. Within a domain area, you can have reusable code for that domain area

3. And common reusable code (like a DB schema) can always be organized as a "library" / SDK that individual features will use

v. Event Driven Architecture (EDA)

1. An Event consists of
1. id: uuid
2. name: string – in past perfect form (UserRegistered, PostCreated, etc.)
3. payload: JSON – content
4. timestamp: UnixTimestamp – creation time
5. traceId: uuid – identifier passing from the very first call through all calls and events to build a complete picture

2. Events have sizes
1. S – name + entity id
2. M – name + useful data
3. L – name + all data
4. XL – name + new and old data

3. Each case requires its own "size," experiment

4. If some data affecting the event or handlers might change over time, put it in the event

5. If it's possible to reproduce some logic using events (i.e., when you don't need to wait for a response), it's better to do so

6. Use queue systems with delivery guarantees and replication (like Kafka)

7. It's better if events are persisted (like in Kafka) so they can be re-read

8. Learn the Transaction Outbox pattern for very complex and dangerous operations

9. For state synchronization, use Saga or Event-based Orchestrator

10. Never expect an event to execute in a constant time

vi. Say "NO" to master-master

1. Too much data?
1. Try separating cold and hot data, where cold data is archived and set aside for the future
2. If data is only needed for projection, move it to horizontally scalable OLAP databases (Clickhouse) and remove it from the main database

2. INSERT not fast enough? Use batching layers, like Kafka, where you dump all records that will later be batched and put into final storage

3. UPDATEs too slow? Often you can change UPDATE to INSERT or use Event Sourcing / EDA – save events about changes, then aggregate and calculate projections from the other end

4. DELETEs too slow? Mark data for deletion (deletedAt) and clean them once via cron

vii. Say "yes" to master-slave

viii. Horizontal scaling

1. You'll need external storage for state synchronization (Redis-like)

2. Absolutely all processes become concurrent, which means you need to either know how to distribute (for example, round-robin on queues) or know how to lock (Redis-like / etcd-like)

3. Problems can occur on only some instances, so in resource monitoring, you need to separate each individual instance

💾 Databases

i. ORM or not

ii. Migration first

1. Write migrations

2. Apply them to the database

3. Perform introspection – export the table schema into your language's type system and constants (table names, column names, etc.)

iii. Optimistic & Pessimistic Concurrency Control

iv. Transactions

1. Read uncommitted – we read data that is not yet committed (maximum speed, minimum reliability, suitable when the data we read cannot fail to be written, for example, because they have no Constraints)

2. Read committed – we read only committed data (reliable, simple, works)

3. Repeatable read – within a transaction when reading, we will always get the same data as if reading from a snapshot (more complex, but a common case, such as when using a sub-query)

4. Serializable – we put all transactions in a queue (minimum speed, maximum reliability, prepare for deadlocks)

1. Design your architecture to minimize or avoid using transactions (and if you do use them, not higher than Repeatable read). And yes, architectural solutions can allow working without transactions.

2. Never delay transactions – slowing down 1 transaction can lead to cascading growth and errors throughout the system.

3. If you had to use Serializable, then most likely, you were just lazy / don't have time to do it differently.

v. Distributed Transactions

vi. Drop Relations

1. ON DELETE CASCADE is an incredibly dangerous construct that (1) can delete necessary data, (2) significantly slows down the database, (3) is very difficult to control and debug. It can only be used with one-to-one relationships; in other cases, it's better to delete orphaned data with a cron job during periods of low load.

2. Even more often, you'll notice that when deleting data, you don't actually need to delete related data—in fact, it's harmful—making ON DELETE CASCADE completely pointless.

3. ON DELETE / UPDATE ... transfers business logic to the database, which leads to tons of debugging and not understanding "why it doesn't work."

4. FOREIGN KEY for checking the existence of an entity is often meaningless—if you received some id, then most often you either have to verify its existence in advance or it definitely exists. And even if not, the cron job from the first point will eventually delete this data.

5. Orphaned data most often won't interfere with your queries because if we've deleted the linking data, they simply won't appear in standard queries (possible collisions only in OLAP queries, but there you need to monitor many aspects anyway).

6. Sooner or later, you'll need to store some data in one database and some in another, and at that point, you'll lose relations anyway. If you reject them from the beginning, you'll be able to use as many different databases as you want and horizontally scale your storage 100 times easier.

vii. Drop Constraints

viii. How to Choose a Database

1. Will it handle the required QPS

2. Look at the required transaction levels and atomicity

3. Check the number of available connections

4. Check replication availability

5. Check if it's suitable for many small inserts or only large ones

6. Is there Update functionality

7. Is there Upsert functionality

8. Are there bulk loading methods (e.g., COPY in PG)

9. How MVCC and GC are structured (to understand the complexity of Insert/Update/Delete, as well as the causes of pauses)

10. What mechanisms provide Optimistic/Pessimistic Concurrency Control

11. Check types of numbers, dates, arrays, and the presence of JSON/unstructured types, as well as different methods of working with them

12. Check for normal libraries: adapter, query builder, migrator, and introspection

13. Row-based or column-based

14. OLAP or OLTP

15. Is there CDC

16. Eventual or Strong consistency

17. If master-master, what consensus algorithm

18. Is something additional required for clustering (e.g., zookeeper, etcd)

ix. Use UUID

1. You can prepare multiple entities in code that are related to each other by id and insert them into the database at once (in the case of serial, you would have to insert, get the id, and only then insert the next entity)

2. Allows the client to send the entity id and after a successful operation request it through separate endpoints/mechanisms (for example, receive it from WS)

3. Allows for delayed insertion (for example, if you want to batch entities and insert them later, but related entities can already appear in the database)

4. If you sum up the 3 points above: allows building Eventual Consistency systems

5. In emergency situations, allows you to go through almost all tables to find which entity this id belongs to

x. INSERT-s / UPDATE-s / DELETE-s must be batched

1. If you don't persist messages in a third-party system (e.g., Kafka), you can lose them

2. Batched messages won't be available at the moment they actually appear

xi. "Storages like Onions"

1. Connection establishment – establishing a connection with the client

2. Processor allocation – allocating a unit that will process the request (process in PG, thread in MySQL, goroutine in CockroachDB, thread per core in RedPanda)

3. Query processing – parsing the received message into logical steps (for example, parsing SQL)

4. Schema validation – checking that the sent data corresponds to the schema

5. Execution planning – determining if, in what sequence, and from where to retrieve data

6. Indexes
1. Verification – ensuring we don't violate existing indexes
2. Creation – creating new ones (especially worth noting the atomicity of UNIQUE indexes)

7. Concurrent access – for example, MVCC (creating new record versions or rollback journal)

8. Transactions – transaction management

9. Commit – confirming that the operation will succeed regardless of circumstances (for example, WAL record or RAFT commit)

10. Respond with metadata – sometimes it's necessary to respond to the client with metadata in advance, for example, about the type of returned data

11. Persistence
1. Communication with the storage layer – sometimes this is part of the processing instance (PG), and sometimes it exists separately (YDB, TiDB)
2. Compression – compressing data
3. Storage optimization – enables efficient storage
4. Batching – aggregating data in memory for subsequent flush
5. Flush – unloading from memory to disk

12. Cleanup – most storage systems will have one mechanism or another for cleanup, for example, VACUUM in PG or sector deletion in Kafka

13. Clustering
1. Membership – discovering and joining a cluster
2. Leadership – determining leaders
3. Health-checks – checking the availability of cluster components
4. Anti split-brain – preventing Split-brain
5. Recovery – recovering after separation from the cluster
6. Rehydration – restoring data to the required level
7. Configuration synchronization – synchronizing the final state of cluster configuration and individual nodes
8. Index synchronization – synchronizing indexes
9. Data synchronization – synchronizing the data itself
10. Master-Slave replication – replicating data for further reading

14. Partitioning
1. Local – dividing large master tables into smaller ones by key within 1 instance to optimize IO operations
2. Sharding – dividing and storing master tables on different instances

15. Backups – some storages are capable of automatic backup of hot/cold data, for example, to S3 (TimescaleDB, Redpanda)

xii. Event bus

1. Almost always, first of all, you will need a persistent storage like Kafka:
1. This gives you the ability to process messages in batches
2. Re-read messages when invalidation is necessary
3. Write in average constant time
4. Have horizontal scaling

2. If you are willing to sacrifice delivery reliability and persistence for the sake of speed, then use a Message Broker (NATS.io / EMQX)

xiii. SQLite

• Since SQLite's performance depends on language speed + processor power + IO throughput, theoretically it's one of the fastest databases, at minimum because it completely lacks all the network complexity of standard databases

• While other embedded storages are just key-value (rocksdb, leveldb, badger), or NoSQL with unique SDKs (couchdb-like), SQLite is a full-fledged SQL database, comparable in SQL capabilities to PostgreSQL: schema, indexes, transactions, locks, joins, constraints, everything is there

• Consequently, experience with any other database will be relevant, making SQLite very attractive to developers

• This also means you can switch from SQLite to PostgreSQL / MySQL almost painlessly when/if the time comes

• Due to its simplicity, SQLite is either already integrated (browsers, mobile devices, native apps) or easily added (I've seen hardware with minimal Linux that uses SQLite)

• You can work with one instance from multiple processes simultaneously

• To backup the database, you just need to upload files to S3

• Super simple integration testing because you can have a separate SQLite instance for each test (e.g., in-memory)

• If you create a SaaS without multitenancy and distribute it as a boxed solution, you can easily open an instance per client

• As free as possible

• Classic network file systems don't allow multiple servers/containers to work with the same SQLite database (I only found information about VFS, but I don't yet understand how viable this option is)

• If something happens to the file system during data writing, there's a chance it will break beyond recovery

• Local database for applications with frontend (web, mobile, desktop)

• Local database for remote agents (applications that collect and send data to the cloud from devices, from a server in a warehouse)

• Cache for a single instance (for example, for actors)

• Database for startups/projects requiring work with large volumes of data, but without wanting to pay a lot for PostgreSQL/MySQL instances

• Sync Read replicas, so you can switch when the master fails

• WAL Streaming Backup for reliable backups

• Pocketbase (https://pocketbase.io/) – admin panel, Firebase-like HTTP API, email, auth, file storage, logs and much more out of the box

• Turso (https://turso.tech/) – distributed SQLite in an Edge environment

• Electric (https://electric-sql.com/) – SQLite on the client side that synchronizes with PostgreSQL using CRDT, turning SQLite into a multi-master edge database

• LiteFS (https://github.com/superfly/litefs) – SQLite database replication at the file system level

• libSQL (https://github.com/tursodatabase/libsql/) – SQLite fork on which Turso is built, with the ability to deploy servers, replicas, auto-sync WAL to S3, and so on

• rqlite (https://github.com/rqlite/rqlite) – turning SQLite into a full-fledged database with read replicas, written in Go

• dqlite (https://github.com/canonical/dqlite) – roughly the same as rqlite, but in C

• First, it's clear that SQLite is evolving into a database for Edge environments because it can operate with minimal resources

• Also, it's definitely destined to become a p2p database (in the style of couchbase) because it's already integrated/easily integrates with any client

• Adding built-in Sync Read replicas will make it more likely to be used in production applications

• If I had to develop a standard web application for business now, I wouldn't overthink it and would use PostgreSQL

• For my own projects, I would happily use it

• If LiteFS or libSQL works well, I would more seriously consider using it in production applications

🔎 Testing

i. General

1. Business logic is tested absolutely always, everything else is optional

2. If a function doesn't use infrastructure, write unit tests

3. In all other cases, always write integration tests

4. Tests are simple (you just need to learn how to set them up)

5. Tests are an extra hour of coding that saves you 10 hours of sleep

6. How many tests should you write for a function? One, then more if desired

7. When you find a bug, first write a test to reproduce it, then fix the bug

ii. Unit, Integration, E2E tests

1. Unit tests – testing without environment (DB, caches, API, etc.), meaning all integrations are replaced with mocks, tests are part of the code.

2. Integration tests – we expose integrations (DB, caches, API, etc.) for tests, tests are part of the code.

3. E2E – we expose the full application with all integrations and call its API, tests are a separate codebase.

iii. Integration tests

1. The launch script can be written in Makefile

2. Before running tests, set up a local environment (preferably through docker-compose)

3. Wait for startup

4. Apply migrations and fixtures

5. Take a DB snapshot before each test

6. Create a separate connection for each test

7. After each test, delete the snapshot and close the connection

8. At the end of tests, DO NOT close the environment

🌎 Logging, metrics, tracing

i. Metrics are your spider sense, Traces are your map, Logs are your eyes

1. At large volume, the only way to see if everything is (not) ok is through metrics

2. The secondary tool is tracing

3. And only as a last resort do we use logging to clarify details

ii. Meta-info of logs

1. Add commit hash

2. Level

3. Service name

4. Unique id

5. Service start time

6. Function call stack

7. Request ID

8. Trace ID

iii. Technical

1. Use zero-allocation loggers

2. Write to stdout

3. Verify that the logger writes asynchronously

👨‍👧‍👦 Leading

i. Triptych – the ideal structure of a technical team

• Team lead (ears and mouth) – a single entry point for business, responsible for delivery timelines, technical backlog, and team condition; creates working conditions for Senior and Middle

• Senior (brain) – responsible for the quality and functionality of the entire system, and therefore: makes technical decisions, has veto power, puts out fires, creates technical conditions for Middle work

• Middle (hands) – responsible for the functionality of the code they write: aligns decisions with Senior, builds features and ensures they work

• I called them "Team lead", "Senior" and "Middle" because there are no more appropriate words (except those in parentheses); in reality, a person at the "Senior" level can be in the "Middle" role

• These are ROLES, which means they can be combined by the same person

• There should be a maximum of one Team Lead and one Senior, but there can be many Middles, ideally within the limit of 10

• Team Lead may not have technical skills (I call these PMs, but this doesn't remove the responsibilities described above)

• I deliberately didn't include QA, DevOps, CTO, Architect, etc. because they either can reuse what's written above, or are more hyperbolic versions (for example, CTO is a Team Lead who is also responsible for payroll)

• Decisions become (in)correct only after you've applied them, so someone simply needs to take responsibility for choosing a path and waiting for the result. If several people try to approve such a decision, they will have big problems, therefore, the captain (Senior) should be only one.

• Communication with business is painful. There can be an incredible amount of it. Both incoming to the team and outgoing. At the same time, business often doesn't know how to communicate with developers and vice versa, so letting them interact on a regular basis is definitely not worth it. They can be introduced and left for some time (feature creation), but all communication and responsibility should be in one person's hands (Team Lead)

• Seniors complement Middles, and Middles complement Seniors: Middles can get all the knowledge they need, Seniors can realize themselves as "senseis" and at the same time learn by structuring their knowledge while transferring it to Middle developers. Such an Ouroboros allows both to grow and enjoy the process.Two Senior developers can (initially, have conflicts, but I wrote about this above, so in this situation) ask each other too few questions (for example, due to embarrassment) and build some incredibly absurd thing simply because they didn't dare to discuss it in advance (this is generally about maturity, but that's a separate topic)

• Openly and explicitly discuss who takes on which role, ensuring that Seniors clearly transfer decision rights on business matters to the Team Lead, while Middles explicitly transfer rights for technical decisions to Seniors

• Establish good, open, and constant communication between all these links

• Give Seniors the ability to make technical decisions and hire people in accordance with them

• If the Team Lead has poorly developed soft skills and lacks the "steel balls" to say "no" to both developers and business, everyone will burn out, business will suffer, and turnover will begin

• If you choose a bad Senior, they will lead the entire team into the abyss, but if you don't give Seniors the right to make risky decisions, good Seniors won't be able to accomplish anything either

• If Seniors aren't ready to listen to Middles, or Middles aren't willing to agree with Seniors' final decisions, the scheme won't work, which is why it should be the Senior who builds the team

💻 Programming

i. Everything is concurrent

ii. Avoid Mutexes

iii. Program as if everything is already broken

1. If you have a state machine, don't forget to write a cron job that will move hung state machines to some final state.

2. If there's a set of data that must be written together or not written at all, then use: (1) prepare all operations on entities in advance and write them in one atomic operation at the end, (2) combine this data into one table/message in a queue (and if you need to separate them later, do it as a separate process), (3) create and run a state machine (sooner or later it will do what's needed), (4) use transactions.

3. If you need data at a specific moment in time, write it as soon as you can gather it together, BUT make sure that the write operation is idempotent (meaning that if the same input data enters the same code logic again, it won't create a duplicate record).

4. You might be lucky and the application will crash with SIGINT or SIGTERM, so take care to implement graceful shutdown.

iv. And much, much more

(λ) Функционально Ориентированное Программирование (ФОП)

Функционально Ориентированное Программирование (ФОП) – функциональная альтернатива ООП.

https://fop.davidshekunts.ru/#dd250ed82406451f9dc45525d34b045a

👨🏻 About the Author