Technological materialism

One of the core traits of a good technologist is the ability to understand technology not just in terms of code, but also in terms of history: how it came to be, the people that created it, and what influenced them (the companies they worked for, the dominant languages, etc.). To steal borrow terminology from another domain:

Historical materialism is—incredibly roughly—a method for understanding the development of human society and history through the lens of material circumstances as opposed to individuals and ideas. That is to say, we can examine society and how it has progressed in terms of the material realities of its productive forces and the relation of people to these productive forces.

Consider a technology such as Delta Lake. To fully understand it, we must consider a few elements:

• It's lineage and the forces of the previous generations that shaped it: created by Databricks, which was founded on Spark, which can be traced to Hadoop/MapReduce at Google, which favours horizontally scaling on commodity hardware, and so-on. Consider also:
  • Parquet as an influence on highly-performant file formats, and Orc as an influence on file formats supporting transactional operations;
  • Hive and Iceberg as examples of metadata storage at scale; and
  • Arrow as a data interoperabiltiy format.
• The productive forces that created it and influenced it: heavily utilises Scala similar to most of the Spark ecosystem, which is heavily influenced by Java in the Hadoop ecosystem.
• The relation of the technology to the companies (capital) owning it: Delta Lake is a component of Databricks built to fit their product (and potentially others).
• The superstructure or culture of the developers and the company: we can ask why Delta Lake is open-source, and what incentives does Databricks have for it.

The core argument to understand here is that technology does not develop by accident or in a vacuum, there are many forces that influence it and cause it to take shape. With this we can inform our choices, building a nuanced perspective on a technology that might not be easy understand otherwise.

For example, we can shortcut a lot of the Rust vs Go arguments that came out in the early days of these languages to the following:

• Go came out of Google which predominantly writes server-side code in C++, Java, and Python. We can see how Go is an attempt to:
  • Mesh elements from these languages, such as the static compilation of C++, and the garbage collector of Java;
  • Avoid mistakes such as slow compilation (C++), and complex dependency management (Python); and
  • Subvert some second-order effects such as ease of use for a large employee base compared to C++.
• Rust (of a certain time) came out of a predominantly open source culture which primarily worked with C++ and ML. While it went through many iterations, the Rust we have today is an attempt to bring rigour and correctness to the systems world, trying to blend C++'s low(er)-level, structural elements from ML (error monads and modules), low-cost abstractions from languages like Haskell to match C's performance. We can also see how the focus on crafting a systems language—directed by efforts such as Servo—gave us a very specific kind of language.

The "which language to choose" argument becomes easy when looking through this lens: If you're looking to build low-latency user-facing software with a strong focus on security and robustness, preference Rust. If you're looking to pick a standardised language for a medium-to-large company that is productive to use and deploy, preference Go. That's not to say you can't use either of these languages for either of these things, but that they were designed in fairly specific ways to address specific concerns.

Both of these languages also show how development continues as more varied use-cases and users get involved, for example, Go transitioning to modules instead of GOPATH as users outside of monorepos need to take advantage of it, and Rust's development of async/await and Tokio to simplify concurrent programming.

Going forward, critically examine the technology you use not just from a purely technical lens, but also from a historical and cultural one. Who created it, why did they create it, how concentrated is the ownership of it. Doing this will start to reveal patterns in the technological world that gives you more insight into decisions, and provide you with better heuristics for evaluation.