Historical materialism as a method for analysing technology


One of the core traits of a good technologist is the ability to understand technology not just in terms of code, but also in terms of history: how it came to be, the people that created it, and what influenced them (the companies they worked for, the dominant languages, etc.). To borrow terminology from another domain:

Historical materialism is a method for understanding the development of human society and history through the lens of material circumstances as opposed to individuals and ideas. That is to say, we can define society and how it has progressed in terms of the material realities of its productive forces and the relation of people to these productive forces.

Consider a technology such as Delta Lake. To fully appreciate it, we must consider a few elements:

The core argument to understand here is thus: Technology does not develop by accident or in a vacuum, there are myriad forces that influence it and cause it to take shape. The reason for understanding this is then that we can use that to inform our choices, building a nuanced perspective on a technology that we might not be otherwise able to understand.

For example, we can shortcut a lot of the Rust vs Go arguments that came out in the early days of these languages to the following:

The "which language to choose" argument becomes easy when looking through this lens: If you're looking to build low-latency user-facing software with a strong focus on security and robustness, preference Rust. If you're looking to pick a standardised language for a medium-to-large company that is productive to use and deploy, preference Go. Now, that's not to say you can't use either of these languages for either of these things (in fact, Rust is becoming increasingly promising for networked services), but that they were designed in a fairly specific way to address specific concerns.

Both of these languages also show how development continues as more varied use-cases and users get involved, for example, Go transitioning to modules instead of GOPATH as users outside of monorepos need to take advantage of it, and Rust's development of async/await and Tokio as people want to use it to build fast and correct networked systems software.

Going forward, critically examine the technology you use not just from a purely technical lens, but also from a historical and cultural one. What company created it, why did they create it, why did people outside of that company adopt it (or not). Doing this will start to reveal patterns in the technological world that gives you more insight into decisions, and provide you with better heuristics for evaluation.