Historical materialism as a method for analysing technology

One of the core traits of a good technologist is the ability to understand technology not just in terms of code, but also in terms of history: how it came to be, the people that created it, and what influenced them (the companies they worked for, the dominant languages, etc.). To borrow terminology from another domain:

Historical materialism is a method for understanding the development of human society and history through the lens of material circumstances as opposed to individuals and ideas. That is to say, we can define society and how it has progressed in terms of the material realities of its productive forces and the relation of people to these productive forces.

Consider a technology such as Delta Lake. To fully appreciate it, we must consider a few elements:

• It's lineage and the forces of the previous generations that shaped it - In this case, created by Databricks, which was founded on Spark, which can be traced to Hadoop/MapReduce, papers written by Google, and so-on.
  • Consider also Parquet as an influence on highly-performant file formats, and Orc as an influence on file formats supporting transactional operations
  • Consider Hive and Iceberg as examples of metadata storage at scale
  • Consider Arrow for data transfer
• The productive forces that created it and influenced it and its generations - Written in Scala similar to most of the Spark ecosystem, which is heavily influenced by Java in the Hadoop ecosystem.
• The relation of the technology to the companies (capital) owning it - Delta Lake is a component of Databricks built to address a tangible issue they see customers facing.
• The superstructure or culture of the developers and the company - Databricks has a strong open source foundation in Spark, which is why Delta Lake is open-source.

The core argument to understand here is thus: Technology does not develop by accident or in a vacuum, there are myriad forces that influence it and cause it to take shape. The reason for understanding this is then that we can use that to inform our choices, building a nuanced perspective on a technology that we might not be otherwise able to understand.

For example, we can shortcut a lot of the Rust vs Go arguments that came out in the early days of these languages to the following:

• Go came out of Google which predominantly writes server-side code in C++, Java, and Python. We can see how Go is an attempt to mesh elements from these (such as the static compilation of C++, and the GC of Java) to build something purposeful, an attempt to avoid mistakes from some of these (fast compilation as opposed to C++, easy single binary deployment as opposed to Python), and an attempt to subvert some second-order effects (ease of use for a large employee base compared to C++).
• Rust came out of a predominantly open source culture which primarily worked with C/C++ and ML. While it went through many iterations, the Rust we have today is an attempt to bring rigour and correctness to the systems world, trying to blend C++'s templating (low/zero-cost abstractions), ML's logical correctness (error monads and modules), and C++'s performance. We can also see how the focus on crafting a systems language (directed by efforts such as being able to build a browser) gave us a very specific kind of language.

The "which language to choose" argument becomes easy when looking through this lens: If you're looking to build low-latency user-facing software with a strong focus on security and robustness, preference Rust. If you're looking to pick a standardised language for a medium-to-large company that is productive to use and deploy, preference Go. Now, that's not to say you can't use either of these languages for either of these things (in fact, Rust is becoming increasingly promising for networked services), but that they were designed in a fairly specific way to address specific concerns.

Both of these languages also show how development continues as more varied use-cases and users get involved, for example, Go transitioning to modules instead of GOPATH as users outside of monorepos need to take advantage of it, and Rust's development of async/await and Tokio as people want to use it to build fast and correct networked systems software.

Going forward, critically examine the technology you use not just from a purely technical lens, but also from a historical and cultural one. What company created it, why did they create it, why did people outside of that company adopt it (or not). Doing this will start to reveal patterns in the technological world that gives you more insight into decisions, and provide you with better heuristics for evaluation.