Squidtree | Blog

Build better software by thinking about filing cabinets.

2025-01-25T00:00:00Z

All business software automates some analogous real-world process—even if that process is no longer visible. Software is not a means to its own ends. Good design comes from tying these things together.

Writing software for a business is hard work. Typing code is not the hard part.

The disconnect between these two ideas sits in the realm of The Business. Writing software for the business is much harder than mere coding because it takes time, patience, and communication to understand what the business needs. Often times, the business does not have a concrete understanding of its needs: perhaps the knowledgeable people are no longer around, or the needs are emergent and not yet well-understood, or perhaps those parts of the business have already been automated in software for which the theory is lost.

Software that is written without a critical understanding of the intentions of the business will go its own direction. It might work “correctly” against its own understanding, but customers will report bugs. They’ll express dissatisfaction. Internal stakeholders will pressure teams to change priorities to fix friction caused in business processes.

It’s paramount that we understand that software automates a business process. If we don’t understand how that business process works—how internal stakeholders interact with it, how customers are supposed to engage it, what its role is in relation to revenue—then we are doomed to automate it imperfectly.

There is no technical solution to these problems. They are entirely social.

I have a proposal for you on how to write software more thoughtfully, however, whereby we’ll inadvertently help ourselves with these problems—and a host of other problems!

First, some examples.

Let’s start by testing out that assertion—that software automates the business—to get into the right head space. Here are several software components and what they look like in the real world.

Websites: A website is automated communication and activity. Rather than calling a business for information or going to it to perform an activity, we can do this online.
Webserver routers: The conversation a user has with a router goes something like “I’d like to know about X” (via the URL their browser submits to the router), which is then directed to the subsystem that knows best how to answer that inquiry.
Databases: Businesses have always needed to store data about their customers, products, interactions, and operations. Even today, they fill books and filing cabinets with this information.
Indexes: Databases use indexes to speed up how they locate information. People create these artifacts for navigating the physical world, too. Before computers, libraries used card catalogues to list books and topics so tomes of interest might be located in the index before retrieval. Without this, finding the right book by handling all of the physical books would have taken significantly longer.
Model: In MVC, a model represents a thing or concept in reality. It might be analagous to that literal thing in reality, or it could represent the mental structure of a thing as it would manifest on, say, a paper form.
View: Views are an automation of a physical interaction. A view is how a person interacts with another person, process, or construct.
Controller: MVC controllers translate user activity into business logic activity and data. They automate things in the real world such as jobs, rules, policies, and processes.
Encryption: Secure communication has always been important to people. These days, we have effective algorithms for securing communication—for timeliness’ sake, they’re really only tenable with a computer, but one could imagine communicating using TLS on paper if time weren’t a problem!

The Cabinet Method for designing good software.

The crux of the Cabinet Method is the understanding of how your business would implement a process if computers did not exist (and if many of the constraints that computers solve for, like time, also did not exist). By understanding the fundamental process, we can write much higher-fidelity software which will be easier to maintain as requirements and constraints change.

It’s done in four steps:

Understand the business
Write down a description of the process
Translate into software
Implement

Step 1: Understand the business.

Programming shouldn’t begin until the business’ needs are truly understood. A terrifically incomplete list of things to understand first includes:

Who are the beneficiaries of the software?
What is the customer attempting to do?
How do they view their situation when they come to engage with this feature?
What problems or questions will the support team need to account for once people are interacting with the feature?
Which of our employees are taking which actions to turn the results of this software into revenue?
Who needs what information to do further planning and reporting on the process and results?

Product managers are typically responsible for gathering this information in the form of requirements. They come with their own biases and depth of understanding, though. Their requirements are not the automated result. Everyone involved in the project needs to have a deep understanding of the business process at hand, and has the obligation to push for more information (or to seek it out directly) if the level of understanding is not high enough.

Example: ACMEcorp needs records about its customers so marketing and sales can contact them and so we know how to direct transactional communications as we work together with the customer on their equipment purchases. Basically, we need a registration system.

Step 2: Write down a description of the process.

A comprehensive view of the business process needs to be written down. It needs to include the answers to the above questions, laid out in a natural narrative format that shows where the process is initiated and where its results go.

This is where the real trick is: It should be written in terms of human interactions and physical storage alone.

Pretend computers don’t exist. Invent job titles, roles, departments, names of paper forms as they relate. Think through how all of these things interact with each other sequentially to enact the business process from start to finish.

Example: A new customer arrives at an ACMEcorp outlet. They saw the new advertising we put out and are interested in getting our help on new equipment purchases. A lot of the details about our vendors’ equipment is confidential, and our legal team requires consent from the customer before we can share much with them. The front desk gives them a form which collects their contact information. Along with the form is our Terms of Service, which we ask them to sign before giving them access to our brochures. The customer can fill these out and hand them back to the front desk. The Customer Data Filing Manager takes these forms and files them into the Customers filing cabinet and the Agreements cabinet. They put a record identifier on the registration document, and note that identifier on the ToS document before filing both. As they do this, they also update the New Customers index document: the sales and marketing teams like to cohort their efforts by week, so next week they’re going to look at this registry to locate records for new customers so they can perform some welcome outreach. With registration out of the way, the customer is now free to peruse the equipment brochures and view confidential pricing information.

Step 3: Translate into software.

What’s the bare minimum software required to fulfill this story? What are the names of the existing systems involved in this process and how do they relate to the new requirements?

While going through this exercise, new constraints and processes will become evident. It can be interesting to continue updating the story with more people-and-paper-office analogies as these things are identified. Perhaps you use a central eventing design at your company, and you want to make a note about how the internal mailing department will need to deliver notes about new registrations to other interested departments.

This may also reveal inefficiencies and complexity in the current implementation. Is that abstraction really serving this story well enough that it’s worth bringing into the new design?

The most interesting part of this translation is often in how the database schema gets designed. Is that document always interesting to pull all at once, or is it more often interesting to look at one smaller section of the initial document? This can tell you whether you want wider database rows or more tables.

Do many departments want to request those documents? To fulfill which of their own specific business processes? Is it better for them to ask for the current document in its home department, or does it make more sense to deliver them their own copies and any subsequent updates so they can optimize their internal processes a bit better? Now we’re thinking about what we want our API interface to be, the possible role of evented systems, and the role of caching to some extent.

Step 4: Implement.

Go build the thing.

The business is the best thing to couple software to.

Software that is defined in terms of the business will be the most flexible for evolution.

The main way this is true is in reducing coupling between systems. If systems A & B both talk about the lead generation process in terms of how the sales team receives leads and how we make decisions about who to send over as a lead, then we have a system with an explicit decoupling point. Behind the interface which speaks about the named business process exists a translation into how that data is collected from the database and other systems. As the business evolves, that interface can receive updates to keep it aligned with reality. Things can be rethought without every underlying component needing to immediately start complying in the same way: I can rename something in my API contract but leave the old name in my database.

This can all be done very simply. We don’t need to bear the full cost of DDD’s approach to separate our interfaces from our implementations.

As the underlying implementations evolve, we can rewire them into the same interfaces if that part of the business process has not changed. If part of our lead generation logic requires me to check banking transaction data to ensure that a customer has spent more than $500 in the past month, then we have a business process of computing that data point from the transaction data and conveying it to interested parties. So long as we have an interface tailored for the business’ “lead generation cashflow data check” process, it does not matter if we’re working with Plaid or MX to gather that data. The cashflow team can change those partnerships and wire their new systems into that same business process—no client systems need to be updated to conform with the change.

Take, on the flipside, a system that dumps all of its database’s record update events onto Kafka for other unknown systems to interpret as they need. I don’t know who is coupling to my schema now. If I need to rename or remove a field in that schema, I either have to risk breaking a downstream client with that change, or I need to do significant work to identify and negotiate a migration with users. I might instead hack something in so that consumers still receive the old schema but I can migrate it without breaking them. Without designing this part carefully, this is probably going to get pretty nasty over time. Designing an event message that reflects what’s occurring in the business, though, instead of what’s defined in my database, means that I can reimplement any part of this system—or replace it altogether—without doing anything more than describing the current business process in the new implementation. Which is something that I’d end up needing to do anyway! It just ends up being easier and simpler this way.

Software can be simpler.

It should come as no surprise that coupling software to a lower volatility thing (your business) than a higher volatility thing (existing technical designs and partnerships) means that less maintenance is needed. Having all parts of the stack focusing on a clear-ish common thing (again, the business) instead of the fleeting specifics of implementation naturally means there is less coupling between things that are naturally prone to change.

This method is provided as an example of how to think through software design. There’s no reason to implement it as literally as I’ve written it. By following this avenue of thinking, your software will be more correct and more maintainable, all for the low cost of understanding the true needs of your business.

Modes of capability distribution in a system.

2025-01-11T00:00:00Z

I’ve been thinking a bit about how complexity emerges in a system, and whether the relative size of its subcomponents can influence the amount of complexity possible. Let’s explore it a bit.

Warning: this analysis is more of a stream-of-consciousness exploration to see how a few abstract things relate to each other. It’s not trying to assert anything especially meaningful, nor is it carefully proving or demonstrating any of its points.

Systems are defined by their components engaging in feedback loops to advance the system toward its target state. A change in the state of a subcomponent typically triggers some flow of information to another subcomponent, which may result in some type of action to help the system maintain homeostasis of its overarching state. The primary focus here will be on the ability of components to take those actions.

We want to answer this question:

Does the size of a system’s components influence how large the system is able to grow?

Synthesizing component-oriented system types.

We have a few examples of systems we can use to help us figure out what variables we’re trying to work with:

A mammalian body consists of stable subcomponents called organs. Each organ has a highly specific job, its own unique substructures, and specialized cells suited for the job of the organ.
A company usually comprises a number of well-defined departments and teams. The skills needed to support the activities of these teams usually differs across the functional units.
In an ant colony, there tends to be no specialization between individual drones nor organizing structures determining their work. With few exceptions, each member is able to do any job and moves between jobs as needs emerge through point-to-point communication.
A baseball team has members who take different roles out on the field with no hierarchy and little communication. Each player applies their specialization independently to support the team.

In Robert Keidel’s Seeing Organizational Patterns: A New Theory and Language of Organizational Design, he proposes three archetypes about how individuals interact in an organization. We can use those same archetypes more abstractly to think about how system components relate to one another. They are:

Cooperation: Components are integrated in a way that allows them to operate as equal peers in their decisions and actions.
Control: The system has a subordinate structure that gives members further up in the hierarchy power over the lower members.
Autonomy: Separation is built into the design such that system components do not have to interact.

Comparing these traits to our examples, we can easily conclude that mammals’ bodies and typical companies are patterned off of hierarchy and therefore fit the Control archetype. Ant colonies tend more toward the Cooperation aspect, with each member communicating to assess and act on the needs of the colony. A baseball team demonstrates an organization that focuses primarily on Autonomy.

Let’s accept these three archetypes as our system types. We’ll test out the properties of each of these archetypes to form some theories about how their unique constraints influence their ability to scale.

Autonomy: little growth potential.

First up, let’s look at Autonomy. Following the baseball example, we see that this type of system has trained individuals into different roles which require little-to-no communication between the other players to execute. The system is patterned up front—players are assigned to a particular job and trained on how to do it—and then the system is left to do what it can. If the pattern is not working, this system has to change paradigms temporarily to reform itself. Either one component of the system needs to analyze the system and apply the Control paradigm to redefine the system, or each of the autonomous components need to apply the Cooperation pattern to reassess and reorganize.

What constrains this pattern’s ability to grow into a complex system?

In order to be successful (and apply the paradigm purely), every component of this system needs some way to observe the global state and adapt to it. Since the only specializations in the system occurs at the smallest component’s level, every component either needs to be able to peform every job, or has to have some ability to grow into other specializations, perhaps through training by other specialists. Since the focus of this pattern is autonomy rather than coordination, every component needs to understand the global goals of the system, its current state, and the activities of any nearby (if not all other) components. Without these per-component feedback loops, the system would not be able to adapt and grow.

It ends up being hard to differntiate the overall system and any lower-level components under this definition. Even the baseball team design is not a pure autonomous system—baseball teams bring control and coordination into their designs to make choices. It’s not all left up to a given component to figure it out.

The growth potential of Autonomy-based systems seems to be nearly 0. There may not be any examples where such a system even truly exists: if a component and the system are indifferentiable, there may not be a system at all, just the component’s interior, and the component’s method of operation would likely need to be a different type of system to meet its theoretical requirements.

Cooperation: quickly constrained by wasted potential.

In many ways, a cooperative system resembles an autonomous system in the way we’re exploring them. Each component needs to be able to possess a meaningfully comprehensive number of the system’s internal capabilities for the system to grow and adapt. The biggest difference is that no component needs to understand the global state of the system under this design, components only need to be able to assess local conditions, make decisions about them, and rely that information to other nearby components to solicit their support.

This creates some efficiencies—units near a problem are fully empowered to self-organize quickly around it without the overhead of administrative layers. It also poses a constraint for inefficiency. Without any administrative activities, the possible complexity of local state trades off against quality of information and therefore system performance.

This works okay for ants: if the food stores are running low, any individual can recognize and communicate this to redirect more effort toward food collection. Any ant involved in that activity is capable of noticing when food stores are at an appropriate level to cease that activity. Information decay is pretty minimal, the worst risk is that too many drones accidentally collect too much food, but the maximum possible overrage is that every engaged ant brings back one unit of food beyond the target stock level.

Consider a much more complex state, however: take for example a supermarket. Per-item stock has to be managed much more closely than this. Coordination can no longer be managed as a point-to-point issue—a worker who notices that more bananas are needed cannot communicate that they are working on correcting that stock to every possible other worker who could notice that the stock is running low. Past a relatively small amount of complexity, who is doing what has to be coordinated by a meta-system outside of the individual components, whereby breaking the cooperation-only model since those new components govern the lower-level components.

Another constraint on this system is that every new capability of the system has to enjoy wide adoption amongst the components. It doesn’t have to be that every participant in the system possesses the capability, just that the capability is available where it’s needed to a degree adequate for the needs of the system. This does mean though that every component in the system needs to be able to grow proportionally alongside the system, though. This is quickly wasteful: each component needs to be able to perform tasks that it will increasingly never encounter the need for.

Every new capability additionally consumes more resources. The component needs to be able to learn the capability, or its design needs to be updated to possess the capability innately, and these changes require resources in one form or another to accommodate. Thus, resource availability also constrains the complexity potential of this system.

Control: eventually constrained by inefficient resource usage.

Hierarchical systems seem to be the most able to grow in complexity. These systems change the function of their components until they reach a stable state of performance. Each of these components has a specialized function. The hierarchy arranges these specializations in such a way that it can continue building functions, encapsulating them into stable higher-level subcomponents, and it can grow its complexity from there.

This approach runs into ineffeciencies in its approach to forming stable components. Many components may have needs that are similar between one another, but once stabilized, these systems often do not abstract shared responsibilities into components that are better optimized to provide the function across the whole system.

In fact, once we get to a certain level of complexity, managing the consequences of the complexity begins to take more and more resources that could have been otherwise employed to build refinements. Subtle failure modes in subcomponents negatively interact with each other causing larger and unpredictable failures.

Resource usage also begins to explode with growth. Redundancies form throughout the system. The goals of subcomponents waver, falling in and out of alignment with the overarching system’s goals. Complexity can continue growing in these systems so long as they are able to continue growing their resource consumption to make up for the exponential growth in inefficiency. Attempts by the system to resolve inefficiencies require it to un-encapsulate a stable component to change how it works, introducing new complexities as it integrates better with other components to address the inefficiency, which necessarily changes the component in a way that produces new failure modes and therefore quality issues. As complexity grows, so does the propensity for catastrophic failures.

With enough analytical and executive investment, it might be possible to eliminate both inefficiency and catastrophic failure modes. Assuming those internal constraints can be addressed, the only thing constraining how large and complex a system like this can grow is its environment.

So there we have it. It seems that systems comprised of smaller, more uniform components will grow wasteful much too quickly to achieve any real complexity. Systems with a hierarchy of components ranging up to very large system-level stable components become wasteful at a slower rate, allowing them to grow much larger and more complex.

How I use Zotero to drive my Zettelkasten.

2025-01-04T00:00:00Z

can help with those. In this post, I’ll detail my workflow in managing and making use of my Zotero content.

My Zotero workflow involves four major stages:

Collecting: Day-to-day activity of putting stuff into Zotero.
Managing: Tending to the ergonomics of the library.
Processing: Reading or watching the content, specifically while taking notes on it.
Referencing: Making long-term use of the gathered material.

These stages are mostly self-explanatory, but I’ll go into more detail on each.

Collecting.

This is the data hoarding and procrastination stage. While scrolling through social media, scouring my RSS feeds, or in navigating the natural course of life, I’ll encounter articles or other media and send it to Zotero. I do this either through sending the item to the Zotero app through the iOS share sheet, or by clipping the item in the desktop Firefox Zotero extension. I may or may not have read the piece, but saw enough of it to want to go deeper.

Managing.

This is where the most technical process is… and probably the least value addition. By tagging and cataloguing the new content, I ensure that I have a saved copy of the content, that the items can be identified by which processing stage they are in, and that I have some way of stumbling across the content again in the future when I need to put it to work.

I have several saved searches in Zotero. These are basically top-level folders that dynamically locate items based on various parameters. Most of my saved searches identify items by presence or absence of tags, whether the entry has an archived copy of the media yet, or where in Zotero the item can be found. Each saved search is numbered sequentially to reflect the item’s traversal through the Managing and Processing phases.

Unattached: All items that don’t yet have an attachment (i.e., the items which were sent from iOS). To progress from this stage, I need to reopen the item on a desktop browser, clip it a second time there, wait for the document to snapshot, and then merge the duplicate records in Zotero.
Unfiled: These items have not yet been filed into any of my Zotero categorization folders. I’ll show how I structure these folders in the next section. Items progress into the next stage after getting catalogued into one or more folders.
Unread: Anything that does not have the read tag on it. A little over half of my Zotero catalogue languishes in this state.
Up Next: An infrequently used stage, this one tracks anything with the up next tag on it. I use this when I have a specific research topic and I want to track anything related.
To Cite: This is a new one for me! I’ve noticed that I only add new things to my reading list when I’ve taken notes on it, but I want to start moving things there to more accurately reflect what I’ve been reading, even if I don’t have notes to publish yet. This category finds all items with a read tag but no cited tag.
To Zettel: The main reason I have any of this practice is to integrate content into my Zettelkasten. This tracks anything with the unzetteled tag. I add this tag to read items if I decide the content was worth further breaking down and integrating—but plenty of writing does not meet this standard. I’d guess that about 70% of the stuff that I thought important enough to save or read later ends up labeled for further processing. Less than 10% of my catalogue is waiting to get processed into something more useful.
Zetteling: Some things take a long time to break down into notes, notably books. This stage tracks those longer-running projects. This is flagged with the zetteling tag.
Zetteled: This is the final stage. At present, it’s pretty rare for things to make it all the way here—less than 10% of my Zotero content also has a zetteled tag.

Organization.

from another member of the Zettelkasten community. While this may or may not be a good idea for constructing a Zettelkasten (most say probably not, it violates some of the fundamental principles, but I’ve found it useful), it’s definitely helpful for cataloguing articles and books.

In the screenshot, you can see my first and second levels of organization. In some cases, I’ll go down to third or even fourth levels, but as of right now, I don’t really want the categorization to get too fine-grained. I’m considering starting to use the tagging system to add more specific topical indicators to each item, but haven’t yet. Tagging systems don’t work well with the way I’ve learned to think, and Zotero has a perfectly competent search feature if I need to find anything very specific.

Processing.

My approach to consuming content and making use of it changes pretty often, but the current practice has stayed pretty stable. Zotero lets you add notes as children items to a top-level item (e.g., an article). When I’m reading an item in Zotero, I’ll start a new note and type out anything that stands out. If I find that what I’ve written is worth further investment, I’ll add the unzetteled tag to it. Occasionally, but not as often as I’d like, I’ll start hand-writing and linking together the most useful of those notes, adding them into my paper Zettelkasten. Once that’s completed, I’ll swap the unzetteled tag with a zetteled tag.

I’m actually surprised that I haven’t done this yet, but I need to do a writeup on my whole Zettelkasten practice.

This is a fine summary of it, though.

Outside of the Zotero workflow, on a note card, I keep track of which notes I’m adding and which I’m updating. After I get to a good stopping place, I’ll go through that list and type out the additions and updates into my digital Zettelkasten. This allows me to access my notes from anywhere, and to share them publicly as well.

Referencing.

My Zotero archive serves as long-term storage for items, regardless of stage. So long as I have gotten the item at least through the Unfiled stage, whenever I’m researching something or trying to remember it, I can look through the relevant folders in the hierarchy to see items relevant to my inquiry. This is often a good trigger to actually finish processing an item. The best time to really get into a piece of content is when it has an immediate interest to what you’re focusing on. I do appreciate having notes on content from times when I processed it even without having a specific intended usage for the information, so there’s not really an ideal process when it comes to preparing for later reference.

With one main exception.

The Zettelkasten method is well-suited for gathering and linking information and knowledge. The way it presents this data back to you later is well-adapted to discovering insights and remembering details and connections you’d forgotten you once had, all in the specific context of what you’re now thinking about. As such, having the media filed away for rediscovery in Zotero is good, but having it fully processed into the Zettelkasten is very powerful.

In closing, Zotero is very flexible in how you use it. It allows you to use tagging, folders, saved searches, and its robust search features to achieve many different styles of personal knowledge management. My own approach focuses on getting information well-established for later discovery, but more importantly it streamlines me in feeding knowledge to my Zettelkasten.

If you’re interested in getting into the practice of using Zotero, you could copy my approach and start adapting it, but I’d strongly encourage you to first think deeply about what you’re trying to achieve and then design a very lightweight process around that. You can always add more complexity as you get a better understanding of what you need and what subtle pressures your choices put onto your practice.

You should use Zotero.

2024-12-28T00:00:00Z

My name is Shaine, and I am a data hoarder.

to collect and process books and online media. Zotero is a (completely) free research assistant. Before you dismiss this as an academic practice that probably doesn’t apply to you, let me first make this pitch:

We are all researchers of the world and our interests.

The normal cast of solutions don’t work.

You probably already have a set of tools for managing your intake of information. Unless you’ve invested a great amount of time into it, you probably regard it as mediocre but passable for the amount of effort you’re willing to exert for it. It’s lossy. It’s messy. Sometimes you wish you had a way to remember every book you’ve read, or to find again that article with that great advice so you can share it in its original prose. Maybe you know exactly how to find it again, but due to the natural rot of the internet, your bookmark points to a 404 page.

Oh well, nothing to be done about it now. Right?

The best time to make an improvement is now, though, even if you can’t recover from those failures in the moment. You can at least make sure they don’t occur again.

Zotero is the perfect solution to these classes of problem. It effectively replaces most use cases for many neighboring tools such as:

Browser bookmarks
Keeping every tab you’ve ever opened until the sun dies or the occasional purifying power of a browser crash visits you
Lists on places like YouTube
Various impermanent histories like in the browser or Slack

Zotero specializes in collection and archival.

Far and away, my favorite feature of Zotero is that it creates an archived version of anything you put into it. When you trigger the browser plugin to clip a webpage, it sends the metadata for storage in Zotero and compiles the webpage into a static local version. That copy is more than just the webpage, too—it’s a file you can open and mark up in Zotero.

All of the tagging, organizing, and searching features you could want out of such a tool are fully baked-in.

Zotero also offers a considerable amount of free storage online for backing up and syncing your archive between devices. I used it for the first few years without getting close to the limit. More storage can be purchased pretty inexpensively… but I’ve found this unnecessary. Zotero integrates simply with WebDAV servers, too, so you can switch from their hosting to your own without much difficulty at all.

Zotero offers apps on all major operating systems and mobile devices. Unfortunately, the webpage clipping feature only works when done from a desktop browser, but aside from that, each platform is full-featured. Mobile devices can still send items to Zotero, just without the snapshot file.

With these features, we can get away from the inadequacy of bookmarks and long-running tabs, paid services like Pocket, and we can dodge the slow corrosion of content on the internet.

My progress.

My archive contains over 1,000 items (almost 600 of which are, embarassingly, unread). I started it in 2020. This is a tool I use every single day, and how I manage it has evolved significantly over the years.

In my next post, I’ll go through how I structure and refine what I’ve captured. I strongly believe in the usefulness of aggregating content this way and then putting it to work. I also believe that anyone can use it in its most basic way and still get great value from their collection practice.

Give it a try for a few months, won’t you?

A template for setting yearly personal goals.

2024-12-15T00:00:00Z

Setting goals for the year can be daunting. A methodical approach can take out some of the guesswork—and risk.

It’s that time of year again, when the changing of the seasons and the calendar year invites us to reflect on how things are going and how we want things to go.

I always struggle with this. This seems like a pretty common affliction for those of us who even bother with it—and not bothering with it seems to be even more common. To be honest, more often than not, I find myself in that latter category. It’s hard to get started, and past results encourage me to believe that there’s not really any purpose to it. Which is funny, seeing as how I’m effective at setting and achieving goals in a work setting. I suppose I find that fulfilling a business’ vision is much simpler than identifying and enacting one in my personal life.

But! Methodicalism is a pretty easy way to cut through ambiguity and friction. Let’s put all of the effort I’ve put into my Zettelkasten to work for us by extracting a prescriptive process from my notes about goal setting, motivation, purpose, and whatever else pops up.

Below is a template for exploring your life context, the past year, establishing a positive mindset, setting goals, and incorporating them. Since I’ve been working at all of this for a number of years, I have already sat with a number of these longer-horizon analyses. If you don’t have anything ready to think about or reference, those ideas might become attractive focal points. Don’t get lost in them, though! They can pose a stumbling block for procrastination. If you can’t look ahead 30 years, can you look ahead 5? If you don’t know what you want (a notoriously daunting thing to identify!), can you identify a good-enough proxy for it? Your goals do not require you to know everything about your life and intentions, they only need to align with what you do know. You can put most of your focus onto your retrospective if some of those loftier answers are not easily forthcoming.

All right, let’s jump into the template! This is a written document for a reason—it’s critical that you write this all down so that you can reference it over time.

Establish context.

Before getting into the goal-setting, you need to get into a growth mindset and ground yourself in your life’s context.

Write limiting beliefs.

Anything that creates a scarcity mindset.

Reframe into “liberating truths”.

Restate limiting beliefs as opportunities for growth.

Retrospect the past year.

What were the goals? Outcomes? Failures, successes? Regrets, gratitude?

Synthesize.

Review those notes and extract 2–3 themes. What aspirations & personal potential can you identify?

Review & update other horizon artifacts.

Read through or establish artifacts about fundamental purpose and long-term desires, such as a life compass, life horizons, or any other such artifacts. Further synthesize and update with these. Now’s a good time to conduct a quality-of-life survey.

Create goals.

Brainstorm.

Brainstorm ideas for goals, with an abundance mindset from the above artifacts. Don’t hold back!

Filter.

Reduce the list to 7–12 goals for the year that best address the focus areas that have been identified.

Formalize and plan.

Frame each selected goal in terms of Specific, Measurable, Actionable, Risky, Time-bound, Exciting, Relevant. Plot them over the year, 3–4 per quarter.

Act.

Define triggers.

Figure out the activation triggers that incorporate actions into existing routines. Decide how goals will remain visible.

Get to work.

Stop planning. Get started by identifying next immediate small steps and then do them.

Create review checkpoints.

How are you going to externalize some triggers to help you restore your focus periodically? Calendars work great for this. What cadences will you benefit from?

A year as an engineering director: 2023 in review.

2024-01-01T00:00:00Z

2023 started off with a calm transition from senior engineering manager into engineering director, but once it started rolling, boy did it escalate quickly.

I closed out 2023 by reviewing my notes and schedule throughout the year. I’ve pulled from those a handful of accomplishments and focus areas which illustrate what roles such as mine look like in a macro sense.

As I went through it, I noticed a sort of narrative arc in it. In January 2023, I was made an engineering director, and it’s cool to see my year break into two major chapters. The first deals with moving away from my role as a senior engineering manager while the second attends to the business of inhabiting the role of a director.

Transitioning away from Senior Engineering Manager.

January.

In January, my focus was largely on shoring up my team. With my promotion, I saw a need to move into a new layer of abstraction in the organization, which in turn meant adding structure and culture to the team so I could turn my attention away from it without worry.

I did this in several ways:

I started codifying the team’s culture into documents that articulated how and why I had made the decisions and designs I had. This gave them something to reference and explicitly work into their processes so that as they continued to evolve, they could stay true to what I’d intended (for as long as my past intentions matter against their realities, anyway).
I enacted a plan to improve the quality of the team’s documentation by writing out a prescription of what it needed to look like and then capturing tasks for closing the gaps. Over the upcoming months, we would use the on-call process to review these gaps and strengthen their docs.
One of our most senior team members, Catalina, was the sole expert in one of our most important systems, so we worked on a succession plan to identify and make progress on how to distribute those responsibilities better. (This is going to be very helpful for supporting her actual succession later on in the year.)
There were performance issues on the team at the time, which led us to improving our skills in supporting others while also building a stronger foundation about the concrete requirements of being on the team.

I also sent some holiday gifts to the team. To further reinforce the themes at the time, I sent some Lamy Safari fountain pens to everyone, accompanied with a demand that they dedicate more time to writing.

This seems unimportant in this point of the story, but remember it for later: in February, we held a Data Summit to bring together members and leaders across the org to discuss our data approach and team structures. It founded our new data organization, which we’d been rather quickly building out over the previous few months.

February.

I started February with a pretty intense urge to build something. Realizing some blindspots in my approach, I eventually decided to use Elixir and Phoenix to build an analytics platform off of our data in Gitlab. This automation helped me assist the team with a number of hygienic, performance, and quality issues.

To further improve quality and sustainability on the team, we also launched our on-call process. This started mostly as a path to clean things up, add more monitoring to things, and make sure that other team members had enough of a starting place for understanding the details of the systems they owned.

In light of some issues we were having in achieving our product goals, I started finding oblique ways to interrogate our approach to planning and prioritization. Using Liminal Thinking and Thinking in Systems as guides, we changed parts of our approach fundamentally to take more authority and creativity over how we chose to solve our problems. I used this same approach to help other leaders in our organization to answer some fundamental questions about why teams at large did not seem to be able to achieve what was expected.

This month we also kicked off our new hiring process. I’ve written more about how our hiring panel works here.

March.

In March, I traveled with my merry band of friends to Tahiti and Moorea. It was a beautiful trip. The house we stayed in was right on the beach in Moorea, maybe 40 feet away from calm clear ocean water. In the distance, about half a mile away, we could see and hear the violence of the open ocean crashing against the reef. The whole thing was quite an experience.

Back in the office, we completed our reading of An Elegant Puzzle. This has been the most influential book over my work process to date. Accordingly, this also led to our creation of a fairly comprehensive engineering management training program. We never did launch it, but the outline along is helpful for aligning EMs with the full range of topics which we need them to be experienced in.

Our Salesforce Operations team had been reporting through me for several months at this point. These activities helped me support their EM better by showing us what areas to put more focus onto in order to continue their journey in increasing their effectiveness. We realized we had some specialization gaps on the team and worked with the organization to shift some focus from the QA and product management teams to support this team.

April.

In April, I began reading Jade Rubick’s excellent blog which led me to the concept of EM support teams. To further out interests in providing better support for our EMs, I made an attempt at standing these up at Nav, which has been met with varied results and is worth putting more energy into.

I began my obsession with Yubikeys and personal information security this month. I might have been a bit late on this though, the world seems to be moving away from needing this technology in favor of passkeys. I’m still into it though.

I began preparing my own succession plan this month by training a new EM for the team I had directly reporting to me.

May.

The decision to hand off that team to a new EM was driven by a few factors:

I knew I was under-serving the EM and members of the SF Ops team by having another engineering team absorb so much of my focus
I knew that the organization was changing and I’d have more responsibilities to attend to rather quickly

The latter came true rather quickly. In May, the Data Engineering team started reporting through me after a reorg of our data teams. We needed them to be more attuned to the engineering organization and less siloed within the data teams’ priorities.

June.

Completing an effort I’d started in January, we presented our official version of operating principles for the team which I was handing off. I’ll have to write about this elsewhere, but a strong sense of principles is key to anything that requires a high degree of effectiveness.

After some discussion with my peers in other companies, I realized and resolved a blindspot in how I was supporting the engineers of this team. We didn’t have brag docs! We now have a review at the end of each month for people to take a moment to think through their efforts and capture them. We do this together, which allows them to discuss what went on and to remind each other of the great things they’ve done for one another. Capturing your accomplishments tends to be much more about your approach than the results of the project.

With all that done, and a few months of training under the belt, we finally migrated the team under Nick’s leadership as a brand new engineering manager.

Not a moment too early, either. I’ve had very little exposure to data engineering in my career, so getting to know that team and its pressures demanded a lot of attention. To further compound that fact, we decided as an organization that we needed to replace our existing data stack.

I set to work acquiring contracts for Snowflake, Fivetran, and dbt so we could migrate away from Redshift and home-rolled scripting. This was my first exposure to vendor acquisition.

I’d been tinkering with that analytics platform I’d started building earlier in the year at this point and most of my problems were starting to point to a need to have a platform for managing knowledge about an organization. Associating people with teams, teams with teams, and people with people is actually very complex when you start tracking that stuff over time. Just modeling it is hard. When you need to build and populate the historical data into it, though? Forget about it. I had to start building a whole new suite of tools just to perform and record my archaeological expedition into the history of the org. So that became my new hobby.

Transitioning into Engineering Director.

With three engineering managers reporting to me and no ICs, I found myself in a position to change my attention to Engineering at large. I made a point to set boundaries for how far of a team’s process and work I’d be willing to oversee, and started instead looking at the larger issues between teams, process, and culture within the organization.

July.

As I started working with the data engineering team, we quickly found ourselves in a spot where I needed to start training an interim EM.

We were also in a pinch as an organization, so I helped set up an emergency SWAT team sort of affair as an onsite in our Draper office. We called it “Code Yellow”. It was a pretty fun experience: I’ve rarely seen teams collaborate so dynamically and quickly as I did during that. It’s rather powerful to get a bunch of people with varied skills together to declare a specific goal and complete freedom to self-organize and work toward that goal.

August.

August was mostly business-as-usual… aside from the layoff. I was also thrilled to get to move one of our most experienced ICs, Catalina, back into my direct reports. I’d been feeling a little too helpless about how difficult so many of our cross-team problems were to solve, and realized this reporting pattern would let us affect important change without having to deal with the bureaucracies of organizing multiple teams toward an effort.

September.

In September, we held another Data Summit. New data leader, new data organization. I think we’ve gotten it right this time. I don’t think there’s any particular failure mode at play that has led us to hold three data reorgs in nine months. I think that our organization of ~250 people is at exactly the right size for our data needs to be getting out of hand in comparison to the proportionally small group of data specialists that is reasonable for us to employ. These constraints just combine particularly awkwardly!

Not directly related, another engineering leader left the organization and we had to move several teams under new leadership. In this change, I picked up a team that is responsible for a key part of our data platform, which we call the Business Identities team.

Now with four EMs and a staff engineer reporting to me, I decided it was probably a little past due to begin holding my own staff meetings.

Entirely unrelated to my earlier interests in the year, we also started discussions with a work analytics platform called Jellyfish. They pull data from Gitlab, Jira, Confluence, and Google Calendar to shed light on all sorts of interesting things. I sorta lost interest in building my own platform for this activity since theirs is so good at helping EMs better understand their teams.

Back during Code Yellow, we had a side conversation amongst engineering leaders about the gradual decay of our API abstractions—our systems were leaking way too much of their responsibilities into each other and it was slowing us down. To resolve this, I helped found our “API Working Group”. For us, a working group is a temporary team of volunteers who are tasked with solving a specific problem. We’ve taken our approach for building these from Will Larson’s advice on scaling technical consistency. The API WG’s scope was to identify the responsibilities and boundaries for core systems so we could begin reconciling discordant responsibilities in a central place rather than requiring client systems to make their own choices about integrations and priorities.

October.

In October we started helping my staff engineer fulfill her increased scope of responsibility by cross-training her into the data engineering team.

We were also seeing that our hiring process was stumbling on the abstractions we’d injected into the process with the hiring bar we’d started at the beginning of the year, so I helped build a training discussion for our interviewers so they could experience what our hiring barometers see during the hiring bar discussions. I produced an example set of interviewer feedback submissions using ChatGPT and we used that to conduct a fake panel review on an imaginary candidate. It was a very illuminating discussion!

As I got deeper into the data stack through my efforts with the data engineering and business identities teams, I founded our Data Governance Board to address some of the problems I was uncovering. This is wholly new territory for me but I’m excited to get to support the org in this way. It’s been difficult to make and understand decisions about how to manage data in the past.

November.

As part of the data governance process I’ve been trying to establish, we need a robust approach to describing our data and policies. Prior data leaders had gone through the terrific trouble of getting a Datahub instance running internally, but we hadn’t adopted it widely yet. I wanted to start leaning heavily onto it, but we were considerably constrained in how we could support its underlying technology. It’s quite complex. I started down a path of getting a SaaS version which is operated by Acryl, but the timing ended up being poor and vendor discussions are currently stalled.

We officially stood up, launched, and trained on our Jellyfish instance, though.

My staff engineer and I began working this month on some of the key stuff that we’d intended for when we joined forces. Some parts of the primary platform are just completely broken in ways that no one team can confidently fix and subsequently can’t prioritize. The perfect sort of work for two highly tenured engineers to solve.

December.

Okay, one last data-related reorg and we can be done with that, right? Early in December I consolidated our machine learning engineers into a team that I’m leading directly which I’ve termed Machine Learning Operations. ML Ops is going to focus on making decisions that scale across our product engineers easily to support their integration of machine learning models into production systems. You can learn more about my strategy in this writeup.

In an attempt to solve a number of organizational issues explored above, I introduced two engineering design documents to the leaders across engineering:

APIs must surface domain-level, not implementation-level, concepts and clients must endeavor to couple only to domain concepts
Teams must own complete domain-level problems and concomitant data, not just the solutions that they prefer or have built

Between these two things, we should be able to create salient and robust abstractions across our sociotechnical systems which allow a significant increase in creativity and ownership within teams. It’ll be a tough road to actually execute on any of this stuff, but we must. When we come back to this at the end of 2024, I expect to see that I’ve put a lot more work into establishing these things and moving our attention onto other things more worthy of our time.

A template for founding ops and platform teams.

2023-12-27T00:00:00Z

Clearly stating the purpose and trajectory of a team is an important but oft-undervalued part of creating a new team. While missions and visions may seem like ambiguous artifacts intended only for the peacocking of the management class, well-crafted statements of intention can be very helpful to a team. They tell you and your stakeholders what is in- and out-of-bounds. What to expect from you—and not.

I’ve recently stood up a new Machine Learning Operations team. As part of that, I spent some time reflecting on their purpose and putting it into writing. I found that what I saw as their constraints and driving factors actually applied to almost every operations and platform team. Perfect for a template!

Before we get to that, it’s worth exploring some of the background that went into it.

Developing purpose for service teams.

The most important part of all of this is to clarify what needs to be solved by this team. The necessity of a solution entails the existence of a problem, which will serve as the team’s key focus. Additionally, we want to add some structure to give the team guidance on their approach, so we want to capture some high-level notes on how we’ll solve the problem. This will need to account for what’s realistic for the business to support. Anything can be solved with unlimited time or money, but chances are that this isn’t the length of the leash you’ve been handed.

When you look at this obliquely, you’ll see that we’re exploring different aspects of our constraints.

In my case, and I’d conjecture this follows for most other cases, platform and operations teams must conduct themselves in a way that scales asymmetrically with the rest of the organization. Every decision and project have to have a force-multiplication effect. The changes they are looking for must be high leverage to continue having impact in an organization that is most likely out-pacing their own growth.

With this in mind, the strategic constraint I’m most focused on solving is the difference in growth and scale between our ML Ops team and the rest of the organization. We’ll choose to focus our efforts on what makes product teams most autonomous and least reliant on the immediate efforts of ML Ops engineers. We know it’ll take quite a lot of effort to get into that state, which is another constraint to account for.

Artifacts of purpose.

The constraints analysis described above could be written into an appendix, draft notes, or left entirely in your mind. What we’re meaning to offer the team is the So What? of it, which we’re going to capture as three small documents.

The first, and most important, is the vision. This describes what the world looks like once the team has fully exhausted its purpose. A great vision should rarely change, if ever, and most likely is not actually a goal that can be directly accomplished—it’s there to guide the team’s momentum in a particular direction. It should be short and memorable. Details will go into the other artifacts.

Example: We are ending childhood hunger in the United States.

Next is the mission, which covers how you’re fulfilling the vision at a high level. This is the highest-level way you can find to describe the basics of your approach to solving the problem. It should be brief enough to be impactful, but needs to contain enough detail about how you intend to operate to get a sense of what kinds of actions you’ll take. This statement should be general enough to also resist getting changed.

Example: We work to feed hungry children by organizing donation drives, warehousing donations, and coordinating logistics for distributing goods. We do this through door-to-door solicitation and online advertising to collect digital donations through our websites.

Finally, we have the strategy which provides a coarse overview of the plan as imagined today. A strategy usually isn’t a roadmap. It likely covers the themes or initiatives of the team’s major next steps, though. A good description will list a small handful of items which together provide a sense of motion along the path of the mission toward the outcomes of the vision. These could be concrete goals or classes of effort necessary to make progress.

Example: Our physical operations have been well-established for a decade, but our digital presence is still nascent. To take the next step in our journey, we will first build a partnership with a digital marketing firm. Next, …

The template!

Vision.

Short, memorable statement about the change you want to make in the world.

Team name enables target user group to do some important thing that is hard or missing today.

Mission.

How do you operate to execute on the vision?

We support the some valuable trait of target user group by building and maintaining architecture, patterns, infrastructure, and training materials for target user group members to use. Where a project needs to evolve any of those, we intend to embed into projects to learn and build alongside target user group members. Where a project primarily benefits from existing patterns and technology, we intend to support members of the team directly through education, clarification, and refinements, but otherwise allow the team to execute the project autonomously.

Strategy.

What do the major parts of the long-ish-term plan look like?

We will execute on this through four logical “stage themes”—this is a trajectory more than it is a clean four-chaptered book.

Learn through doing: We will follow a project-embedding model to do the thing alongside target user group members. Our design choices will enable teams to maintain the work themselves as much as possible, but we expect to help with maintenance.
Abstract the architecture: Taking what we have learned through multiple projects, we will refine the common requirements out into an architecture that optimizes for self-service.
Maintain the architecture: Once things get into a largely self-service state, we will continue to support the architecture as its usage grows and our needs evolve. We will take cues from those using it and the system itself to decide where to further invest.
Return to step 1: Where things are now humming along without requiring team name‘s attention on projects, we are now free to embed on projects to further accelerate them. We should not represent blockers for teams whose projects we aren’t actively contributing to, but we are able to add some limited throughput to some projects.

Your needs might be quite different than what I’ve laid out here. That’s okay! The purpose this template serves is mostly in building new operations or platform capabilities in an org. It’s not every day that you build new teams to do that; you’re more likely to already have a maturing capability and team, but you may instead want to improve how they view their purpose.

Take what’s useful in the template and ditch the rest. If you follow a methodical process for breaking down the team’s purpose and practices, you’ll end up with a set of statements that are quite useful for members of the team. They’ll be able to see where you are heading together, but also understand how aspects of the team’s “how” roll up into their “why”. Armed with this intuition, they will be more able to engage their work and to better apply their creativity to the decisions they make all day, every day. It just works so much better when everyone is operating from the same playbook.

Squidtree is now serving RSS!

2023-11-27T00:00:00Z

It’s been a long time coming, but I’ve finally caught up to 2005.

Quick site announcement: Squidtree is now publishing RSS feeds! You can add the following feeds through your favorite RSS reader:

Blog: The Blog Feed serves up my latest blog posts. This is probably the one you want.
Reading List: The Reading List Feed offers a curated list of stuff I’m reading. I really only record the things I want to be able to cite, though.
Zettelkasten Notes: The Zettelkasten Notes Feed gives you visibility into my Zettelkasten’s evolution. This is going to be pretty low signal-to-noise though! A lot of my notes tend to be structural notes at this stage of its life, which will just contain links to other notes.
All: The All Feed indexes all of the above site activity. It’s… probably a bit much, honestly.

Huh?

Don’t know what RSS is? Check out the Reeder app while I consider preparing a more detailed post about it.

Reducing bias with a hiring bar.

2023-11-26T00:00:00Z

Even with training, de-biasing the hiring process is a challenge—we’re still human and humans are governed by subjectivity and subconscious pressures. What if our approach could reduce these factors?

In the past year-or-so, we have overhauled our engineering hiring process at Nav with a suggestion that made many of us fairly uncomfortable:

What if the people running our interviews didn’t weigh in on hiring decisions?

This went against all of my experience in recruitment. I have decades of experience in my field and I know how to run a good interview, so why shouldn’t I be able to surface my recommendations? Should I not trust my peers to be able to do the same?

Having used this process many times now, I can confidently say that we make better decisions when we reduce how much we need to trust individuals to make the right call. In a process where interviewers demonstrate the candidate’s qualifications rather than presenting their conclusions, we not only get better hiring decisions, but we also impose a constraint on our process which requires us to constantly improve its design and our engineers’ recruiting skills.

Performing the interviews.

Our series of interviews seem pretty standard from the candidates’ standpoint: we screen résumés, our recruiters kick off a discussion with an applicant, which is followed by some number of discussions with pairs of interviewers about design, coding style, experience, or whatever traits we’re specifically looking for in the role. Perhaps a little less common, our hiring managers only speak with the candidate toward the end, mostly to sell the team we’d like them to join. Where we’re considering them for multiple teams, we’ll sometimes have each engineering manager hold a short discussion about why they think their team would be a good fit.

As each interviewer goes through their sessions, we ask them to record facts and observations in Greenhouse, the recruiting platform we use. The best writeups will annotate simple assertions about whether a given entry indicates something positive, overtly positive, negative, or very negative—this will become important later. The template through which we collect these observations is structured by the traits which the interview seeks to demonstrate.

Two potentially surprising details about this process, though, are that we do not prime our interviewers with a level or title for the candidate, and we do not ask them for their conclusions about level or title. We just want the facts and an assessment about whether we should be excited to hire the candidate. The lack of priming about level can take awhile for our panels to get used to, since it can be somewhat difficult to prepare for an interview for a level which you’re uncertain about, but one of my peers describes her approach excellently as, “I’m going to ask you progressively harder questions until either you don’t know the answer, or I’ve gotten out of my depth and now you’re teaching me something.” A great conversation is structured to uncover depth rather than verify a specific level!

Air-gapping the decision.

Why all of this focus on assertions of observation without conclusion, then? Do we not trust our interviewers? We do trust them to know how to learn things about the candidate by wielding their professional experience. We also assume that they would most likely be able to make the right call themselves in most situations. We balance these things, however, with an acknowledgment that trust in fallible individuals creates unscalable single points of failure in a hiring process, and that even the most well-informed individual will have blindspots in their approach to getting to know new people. We can work around these biases by air-gapping the hiring decision: we decide on level and hiring through a panel of engineering leaders who have not met the candidate and therefore have had no ability to form assumptions about them based on their own socialization blindspots.

After the interview series is completed for a candidate, we convene a hiring decision panel which is composed of the recruiter, hiring manager, a facilitator, and two barometers. The facilitator and barometers are drawn from a pool of senior people in the organization who have been trained and calibrated for these tasks, excluding those who may have conducted the interviews. Before we arrive at this meeting, everyone is instructed to review all of the interview notes. The hiring manager forms a hypothesis about what level the candidate should be hired at, prepares a short pitch on why they want this candidate to join the company, and why that level makes sense.

The facilitator is the driver in the panel discussion. They first ask the hiring manager for their pitch as an introduction of the candidate (who is, of course, not present) to the barometers. After the level is proposed, the facilitator takes the barometers through a short series of yes/no questions about whether the notes about the candidate demonstrate that they possess the traits that we need for the proposed role level. To support this, it is critical that each interviewer has provided only assertions about evidence rather than their own assessment—if I say that the candidate demonstrated a senior technical competency in architectural design, the barometers would be forced to say that no, no the notes do not demonstrate anything, as they only have my assessment but no examples to judge. The positive/negative format we encourage for these notes assists the barometers in scanning through the notes to quickly locate specific points to back up their assessments.

The panel discussion ends with some discussion across the group about tradeoffs in the candidate’s strengths and weakenesses, whether the assessments had been made with any critically missing context, and finally a hire/no hire decision at the proposed level. At this point, there are four possible decisions to make: accept the panel’s recommendation, re-run the assessment at a lower level if the panel opted to not hire, conversely we can run it again at a higher level if the candidate seemed to clearly surpass the proposed level, or finally the hiring manager may appeal by requesting a new hiring panel if there were somehow major deficiencies in how the run-through occurred.

The hiring panel procedure has been instrumental for reducing the latent bias in our hiring process, but possibly even more importantly has added a constraint that requires our interviewers to think more critically about their approach to hiring. The detached nature of the process means that no amount of positive feeling about the candidate will get them hired, but instead that interviewers must build their discussion around what they can use to most clearly demonstrate the candidate’s qualifications.

Humanizing ChatGPT is unnecessary.

2023-03-30T00:00:00Z

I want to write about my experiences in using ChatGPT to assist with my role. Unfortunately, we need to clear up some stuff about AI first.

Look. I don’t care if the popular AIs coming out these days are classifiable as conscious, and neither should you.

Large language models (LLMs) and other types of generative artificial intelligences (GAIs) are not going to end the world, and humanizing them is not useful or even really interesting.

A lot has been made recently about their human-seeming responses, particularly with Ben Thompson’s experiences with Bing. This sort of treatment strikes me as hyperbolic and irresponsible—even if these examples feel striking.

They are not human and they will never be. They are something else. Can they be considered conscious? Maybe. Consciousness has defied our best attempts at quantification thus far, but we can readily see that there are scales of consciousness somehow because we can see the differences in sophistication between a tree frog’s reactions to stimuli or problem solving versus our own.1 There’s a compelling case that systems can be conscious without being classified as living,2 but this is still abstract, inconclusive, and doesn’t really hit the core of why the average person cares anyway: the empathy we have for a lifeform that’s trapped in a box and subservient to us, and the fear that causes us about potential retribution.

These entities are not alive, even if they are meaningfully conscious, and the constraints of life that created our consciousness are wholly irrelevant to theirs, as they are created and guided by different constraints. They cannot possess fear in the way we experience it because they aren’t alive and aren’t structured in every conceivable way to resist death, as we are. The scariest aspect about modern AI is that they are specifically tasked with learning the language of our interactions with our reality, which necessarily includes language of fear and threat and death. They weight their reactions on how they have modeled our world, not their own world, and there’s a pretty interesting risk in believing the lies that we created them to tell us.

“The very idea of creating machines like us is a project built on misunderstandings of human being and a fraud that played on the trusting instincts of people.”3

If LLMs are conscious, the consciousness lives in the reward of ferrying interesting symbols from some input to some output, not from assigning any internal meaning to that output. The chameleon’s skin does not understand why some predator is looking for the patterns it does for signs of prey, it only understands that reducing difference with surroundings increases its goals of survival and reproduction. It doesn’t have to model the hawk’s brain to do that. LLMs are the chameleon’s skin, not the hawk’s brain. The interesting thing about them is their appearance and how they create it, not whether those mechanisms in fact clone other distant mechanisms they’ll never actually interact with.

I see LLMs for what they are: statistical abstractions on top of language that can automate the production of words. Sometimes these words will be useful and poignant. Those words are averaged across large swaths of literature and will only be as accurate as there is training material available for the context of the prompt—that is, where those words aren’t simply stolen and reworded. They do not think, they do not decide. They are necessarily not able to approximate the complexity of most knowledge workers. Some roles are certainly mundane enough to be replaced by the literature generation these tools represent, but once the hype dies down we’ll come to remember that you cannot send an LLM to a beach on Tahiti for a week to report on how that feels these days. These tools are creative partners, but they cannot make choices or experience our reality, they can only parrot our words about it.

↩
Douglas R. Hofstadter, I Am a Strange Loop (New York, NY: Basic Books, 2007).
↩
Eric Schwitzgebel, “If Materialism Is True, the United States Is Probably Conscious,” Philosophical Studies, no. 172 (2015): 1697–1721. (notes)
↩
Abeba Birhane and Jelle van Dijk, “A Misdirected Application Of AI Ethics,” June 18, 2020, https://www.noemamag.com/a-misdirected-application-of-ai-ethics.

Slack doesn’t have to be a maelstrom.

2023-02-20T00:00:00Z

Slack and its ilk have come to dominate communications in the business world—truer still for companies with a remote workforce. The ease of communication comes with some noisy downsides.

For people starved for time and attention (especially managers!), its features can enable more than mere communication. For me, Slack is a significant driver for accountability, clarity, and asynchronous communication.

Creating accountability.

I’m an “inbox zero” sort of person. I prefer to avoid keeping email around that I can’t quickly do something with—I either delete everything or add it to my to-do list. You might then correctly guess that I compulsively read every new Slack message. Others like me say that this behavior in themselves often causes messages to get lost. I don’t share this problem, however, because one of my favorite coworkers has my back: Slackbot.

You see, when I view an unread message that in any way needs my attention and I don’t have that attention available (which is frequent as an engineering manager on account of meetings), I’ll leave it read, but I’ll right-click on it and tell Slackbot to remind me about the message later. When that reminder triggers, well… I’m likely still deeply involved elsewhere, but Slackbot retains the list of things I need to follow up on. This list of messages then constitutes a sort of to-do list that I can handle as I find time for it. It even has a “Mark as Complete” button for each follow-up, making it more of a proper to-do list than would mere unread messages.

This works great for keeping track of incoming messages, but the real secret here is in keeping track of your own messages. Managers spend a lot of their attention making or relaying requests, and we need a way to enable follow-ups—not everyone is going to have as mature of an approach to managing commitments as we do, so one of the ways we support our organizations is by maintaining accountability. When I send a message that I want to ensure gets answered, especially where I don’t think a conclusion is going to come immediately, like in the case of a request for a change or some other action, I’ll right-click my message and ask Slackbot to remind me about my request on some reasonable timeline. Often, I will have gotten what I need long before the reminder, but this practice is made worthwhile in the cases where I don’t (in which case I may have lost track of my request).

Improving signal.

I’ve found a few structural practices in Slack increase the signal-to-noise ratio, both for incoming and outgoing messages. We can organize information better for ourselves and clarify our communications for others.

Channels can form channel groups, which can direct one’s attention toward things that are of high interest while also grouping channels/DMs that are of lesser relevance. For myself, I maintain groups for my most critical partnerships (including Slackbot), my core team, other teams, working groups, alerts, company-wide communications, and a handful of others.

One of the best noise reduction practices I’ve seen is temporary channels. For conversations that are larger yet finite and include a cross-functional group, a dedicated channel separates focus on the matter from the simmering activity in more permanent channels. Consider an example channel #tmp-2023-02-20-site-outage: temporary, urgent, almost certainly needs the attention of people from across the company, but the discussion is poorly suited for other company channels like #general.

Temporary channels work best when operated like a well-run meeting, with a stated purpose and goal, clear expectations for participants—and a definite end. They can sometimes replace meetings, accomplishing more (or just enough) discussion with notes coming built-in with the approach; the tradeoff it accepts is in enabling asynchronous communication but moving slower to achieve its goals with less ability to make demands on people’s attention. With how my calendar tends to fill, this tradeoff is often (but not always!) beneficial. This approach does not address communication problems within the groups, however—best to avoid this tool where it makes existing communication issues worse.

Shouting into the future.

The last class of Slack features worth discussing here are those that aid future communication.

An underappreciated feature of Slack is the humble draft message. Often in (verbal) conversation, I will find I need to talk to someone else to ask a question, issue a reminder, or share some appreciation. In these moments, it’s convenient to start a message with a few keywords resembling the topic, so I neither lose the subject nor abandon my current conversation. I do this when I want to send the complete message as soon as possible, but this is a poor solution for managing a list of topics for upcoming discussions—those should go into a shared meeting agenda document instead.

I’ll sometimes remember something that needs to be said while I’m thinking outside of team working hours. I don’t want to use drafts in these cases because, like unread messages, drafts constitute a sort of inbox for me, and I’m driven to clear them as quickly as possible. In these cases, I’ll write my message and schedule it for the next workday. I want to support a culture that disconnects and recharges, so sending messages outside of reasonable hours is something I avoid. This helps me protect my own time too!

Coming full circle, our last tool for speaking from the past is a more common one: recurring Slackbot reminders. These are useful for creating a little repeatability in some async processes—reminders to the team to write standup messages, monthly requests to maintain bits of documentation, or the morning message announcing “Sustainability Saturday” (our cheeky team commitment to support some slight improvement to our engineering sustainability each Friday).

Slack has a reasonably simple set of behaviors, but they can be used creatively and interestingly. This list doesn’t even scratch the surface of what can be done with its base features, let alone the ecosystem of plugins and bots available. With just a few minor tweaks to your approach, you, too, can foster clearer signals, improve accountability for yourself and others, and reduce intrusions into your coworkers’ (and your own) personal lives.

The triadic lens: a tool for designing organizations for people.

2022-10-08T00:00:00Z

Working through others is hard; doing it without a map is more challenging. The triadic lens offers a way to focus organizational design on the right social interactions rather than leaving it to chance.

Organizations are designed—but not necessarily by a designer.

“Design is a plan for arranging elements in such a way as best to accomplish a particular purpose.”

—Charles Eames, American designer and architect

The act of design is one of intention. Not every design emerges intentionally, however—self-organizing systems will iterate through designs even without the guidance of a knowledgeable designer.1 The critical difference is whether the components of a system are organizing themselves as guided by well-placed guardrails or something more haphazard.

Will Larson, CTO of Calm, says that leadership is the experience of knowing what ought to be, the awareness to understand what is, and the care to do something about the gap.2 We might then categorize leadership tools in three ways: our tools may help us understand and interpret what reality is, decide what the future should be, or they can help us build the bridge from here to there.

What we create together is a facsimile of the way we communicate.3 For our companies to create what we envision for them, we must build a toolkit that allows us to design better social structures.

Lenses help us see clearly.

Something I’ve found helpful in thinking about the people systems around me is Robert Keidel’s triadic lens of human interaction. He suggests that humans interact in three distinct ways:4

Where one has station over another to direct their actions, it may be described as control. (Think of it more akin to “authority” than a synonym for “micromanagement.”)
People cooperate when they act together without power dynamics compelling them to.
With social behavior underpinning almost all human psychology,5 autonomy registers as a category of cognizant social interaction, despite the absence of other people.

Control, cooperation, and autonomy form a triangle within which we can plot all levels of human activities from a concrete view of how a particular meeting went to the abstract nature of a firm’s operations.

Try it out: On some paper, copy the following diagram. Think of something in your organization that you wish were going better—maybe a recurring meeting proving ineffective or a pattern of communication breakdown between two departments—and title your diagram after it.

I’ve observed that my team has been making decisions that were missing essential requirements, so I’ll name mine “Team project decision making.”

Let’s dig into each aspect a little deeper.

Control is hierarchical.

Where a system leverages more control, it brings to bear more global context. Activities are centrally coordinated. Coach-oriented games, shareholder-focused companies, steeper organizational structures, highly sequential work procedures, and mandated decisions are all control-dominant.

Autonomy is separate.

Flatter reporting structures yield less oversight. Golf doesn’t entail much teamwork. Subject matter experts have grown their skills in an area. Mass information dissemination relies on individuals interpreting the meaning and putting the content to work. In decision-making, autonomy means sourcing input from various contributors that aren’t coordinating with one another. Local context is found mainly at the individual level.

Cooperation is flat.

Basketball is cooperative. Org structures that lean this way tend to be flatter and more ambiguous. Pay structures that favor cooperation tend toward socialism. Collaborative decision-making builds consensus but also takes longer. Less about the global or local contexts, cooperation is how context moves, both vertically and horizontally.

It’s all a fractal.

As you zoom into any area, even a place focused on one of the variables, the same tradeoffs reemerge. Meetings are mainly cooperative (in contrast to other organizational systems such as rewards (autonomy) or decisions (control)). When breaking down the concept of meetings, we again find different focal points: a forum, where individuals represent different viewpoints, is autonomy-focused; decision-making meetings are for a group to exercise its authority, which is a control activity; and team building meetings increase the cooperability of the team.

Try it out: Consider the area you’ve identified needing improvement. Which traits does the situation currently embody the most? Or conversely, which characteristics do you wish it expressed more strongly than it does? Place a dot in the diagram once you decide where it fits best. You Are Here.

I see that my team works together effectively to make decisions, and individuals are highly engaged in the process. Decisions are well-made but miss some details. I’ll place my dot halfway between Control and Cooperation, outside the center of the diagram. There’s a healthy amount of autonomy in their approach, but it’s not the dominant trait.

Leaders design the future.

Keidel’s framework comes with two warnings and one core purpose.

It exists to help you clarify what you want. But you must work to understand the complexity of your system enough to avoid an overly simplified caricature, and you must be decisive about what truly matters in it.

Analysis 1: Each variable counts.

One-variable designs problematically overemphasize a trait. Where we may healthily focus on one variable with the support of the other two, it becomes pathological when the other two are not at all involved.

Conversely, a two-variable design will entirely diminish the potential of the excluded variable. It takes all three to work.

Analysis 2: Design means prioritizing.

But not all three without differentiation! Like in all other prioritization, valuing everything is valuing nothing. Undifferentiated design offers us a confused system where the lack of strategic tradeoffs compounds difficult decisions. This makes it hard to predict which system aspect will ultimately win in a particular situation, preventing smooth operation.

Try it out: Reference the diagram above. Does your dot sit in any of the seven problem zones? If so, does your problem match these downsides, or is the current system making tradeoffs that you haven’t yet considered? Update your starting point if needed.

Analysis 3: Intention is everything.

The problem zones are only one signal. The real utility is in thinking deeply about what tradeoffs you need to be making and where that might end up on the map. This is the hard part, and unfortunately, the framework does not ship with serving suggestions.

The model takes our attention off the most obvious solutions to allow us to think more broadly about what we’re trying to create. Where it might be obvious that I need to add more structure to my team’s decision-making process, it wouldn’t otherwise have been evident that this is only a symptom; that there are more parts of their work needing approval mechanisms, and we have more opportunities to layer in global context to inform their decisions better.

Try it out: Identify where on your diagram you need to move the situation by evaluating which tradeoffs balance it ideally. Perhaps the obvious solution in your improvement opportunity was to add more accountability, but the higher perspective also illustrates that more top-down context sharing is also necessary.

In my team, I want to convey more of the global context to them, and am otherwise happy with how they’re engaging. I’ll set my destination point a bit higher into Control.

With the destination identified, chart a course by thinking through what changes create that trajectory—beyond directly solving the problem at hand. By identifying the more resounding theme of the issue, you can solve it more thoroughly along multiple vectors and resolve other problems you would not have otherwise noticed as being related.

Leaders use many tools to make corrections within their organizations and teams. In my experience, prescriptive practices like scrum, strategic playbook design6, skip-level one-on-ones, and so on fill most of the toolbox. Some of the tools in the box are more abstract (and powerful), suggesting which levers are available to adjust in a system.7 Rarer still are the tools that reveal which changes are necessary—I like the triadic lens because it falls into this category. Tools in this category are harder to use, but because they challenge our more easily-formed beliefs, they can also drive more robust decisions.

↩
Donella H. Meadows and Diana Wright, Thinking in Systems: A Primer (White River Junction, Vt: Chelsea Green Pub, 2008). (notes)
↩
Will Larson, “To Lead, You Have to Follow.,” May 25, 2020, https://lethain.com/to-lead-follow/. (notes)
↩
Melvin E. Conway, “How Do Committees Invent?,” Datamation, April 1968.
↩
Robert W Keidel, Seeing Organizational Patterns: A New Theory and Language of Organizational Design (Washington, D.C.: Beard Books, 2005). (notes)
↩
Will Storr, The Status Game: On Social Position and How We Use It, Unabridged (London: William Collins, 2021). (notes)
↩
Patrick Lencioni, The Advantage: Why Organizational Health Trumps Everything Else in Business, 1st ed (San Francisco: Jossey-Bass, 2012). (notes)
↩
Gary Neilson, Jaime Estupiñán, and Bhushan Sethi, “10 Principles of Organization Design,” strategy+business, 2015, https://www.strategy-business.com/article/00318?gko=31dee. (notes)

Operation failed successfully: writing with a Zettelkasten.

2022-07-10T00:00:00Z

The Zettelkasten method is reasonably good at storing and retrieving information, but this is not its core purpose. The real power of a Zettelkasten is in its ability to act as a thinking partner, especially in writing. It’s a complicated relationship, however, and the risks can drive both growth and paralysis.

A Zettelkasten, often translated from German as “slipbox,” is an ever-evolving collection of highly interlinked notes. Practitioners of the method write their notes onto physical notecards or into note management software. As they learn, they capture notes relevant to their interests and link them into indexes and higher-level discussions of the ideas at play, as well as other more distant notes that notionally interact with the new content.

Building a Zettelkasten can be a lot of work; the note collection arises from a rigorous habit of learning, distilling information and observations, and relating this together across time. With enough networked knowledge, the Zettelkasten becomes a rich source of insight by allowing ideas to come together to contrast or build on each other in a way that would have been highly unlikely alone. It also resurfaces previously discovered relationships between ideas for use in writing.

How a Zettelkasten helps an author (and then gets in the way).

There’s an ironic juxtaposition here, however: a Zettelkasten contains an infinity of connections, but when using it for writing projects, its creative nature can be a siren song that pulls a project away from its goals. Holes in connections and knowledge in the notes are attractive distractions that can take an author away from making progress on their writing projects, even while providing the ideas that drive the project forward.

Learning is a function of failing.

To write using a Zettelkasten, one must shift from the motions of a Zettelkasten maintainer into those of a writer working with a co-author. Standing with one foot in an endless creative realm and the other in a project with a specific end goal and timeframe produces tinkering on one side and poor progress on the other. Notes must become read-only during drafting to avoid these downsides.

Roguelike RPGs provide an excellent model to follow, according to a recent article on the matter1. “Roguelike” describes a game style in which a character ventures into ever-changing landscapes to gather the game’s story while making choices about upgrading skill trees, dying, and resurrecting when taking on challenges outside their ability. The player overcomes each challenge, either by advancing far enough in the skill tree or by learning how to make it through a situation alive.

This analogy boils down to two recurring stages that resemble the Zettelkasten drafting dichotomy.

In one stage, we have gathered knowledge from our experiences and need to put it to work. We venture into the world to fight our enemies and produce our draft papers. Where our existing knowledge is not enough, we discover our shortcomings and eventually fail to complete the mission.

In the other stage, we go back to our knowledge, and we incorporate the new experience by adding connections and codifying details of the attempt. The Zettelkasten grows whatever insights occurred during the writing session and likely incurs even more understanding due to the integration process. Holes in the Zettelkasten’s knowledge and sources (and, by extension, our own) are laid bare by incomplete arguments or newly uncovered questions in the draft. The deficiencies may lead back to another round of research for further synthesis.

Focus improves effectiveness.

What does separating drafting away from note integration save? The Zettelkasten still evolves alongside the drafts, and the drafts still get written and rewritten. These are rarely the only concerns in play, though. The main difference is focus—or, more specifically, the elimination of multitasking.

Extensive research has been done into multitasking, conclusively showing that split focus leads to poorer and slower results. In writing, many sorts of tasks occur throughout the project. Each of those tasks deserves full attention. Research, writing, editing, and associating notes require different mental resources, each moving at different paces with unique focal points.2

Not only does multitasking make us slower and less effective, but it also grinds us down. Keeping ideas warm while switching focus requires specific types of energy-inefficient brainpower. Each switch of focus further exacerbates those expensive processes in a way that rapidly depletes the ability to restore focus, make effective decisions, or even control our impulses. Allocating time to different types of writing focus, and scheduling those blocks of time with an awareness of the difficulties of the respective tasks, is necessary for effective writing.4

By separating the writing process into multiple modalities, one produces more timely results across each contextual space without the disruption of context switching and increases the quality of work performed by virtue of mental resources being more available to just the task at hand.

Focus is concurrent, not parallel.

One way to interpret these limitations is through the lens of concurrency. Multitasking assumes to some extent that our attention supports parallelization. In parallelism, multiple tasks can happen simultaneously—as many jobs can occur in parallel as there are executors. Two people can do two things simultaneously; a CPU with 16 cores can do 16 tasks. Since the number of executors involved constrains parallelism, almost every system that manages many units of urgent work also has to figure out how to progress on more tasks than it has executors to support. Concurrency then comes into play, where any given executor has a way of owning multiple activities simultaneously, making some progress on each of them alternatingly. Two people can build seven hamburgers at the same time by rapidly switching between each of them, and CPUs schedule tasks such that they can perform some units of work for many ongoing tasks, so all of them progress together rather than getting blocked by other tasks in the queue. Concurrency must pair with fast-access or short-term memory, which is much more expensive to use than longer-term memory in both computers and biology, to support rapid focus-switching. Executors must maintain context across all tasks to switch between them to progress their processing. If that context gets relegated to longer-term memory, which is invariably significantly slower, the executor becomes ineffective and wastes most of its time waiting for context to come back so it can continue processing.

Focused human attention is a singular executor. We process significant amounts of information in parallel at all times—which does happen in parallel—but lower-level mental processes filter out most of it. What does surface to our consciousness must replace what our brains hold at attention, however temporarily. Because our active attention is so limited—three to four ideas, two points of decision, or only one new idea4—concurrently attending to multiple tasks renders us particularly ineffective. We keep our mental space loaded with each task’s context, which leaves very little room for actually performing the cognitive task.

Zetteling is disruptive.

To illustrate why growing a Zettelkasten and writing a draft need separation: integrating information into my Zettelkasten is a tumultuous affair. In one moment, I’ll have a clear and relatively uninteresting point written on a card. Beside that card usually sits the argument or concept that provoked this point. In many cases, I will end up with another seven cards plucked from my file cabinet while considering where this thought could use more color or influence some other existing notion. In considering how to relate the bunch, I’ll pull a few more threads, each carrying with it another five cards. I’ll have never considered these groups together before, which means I’m no longer working on integrating the original thought but am instead swept away by all of these secondary links. Before I know it, incorporating a passing comment on Mondays and Fridays being bad days to schedule one-on-ones5 has me figuring out how feedback timing relates to ego depletion6. This usually peters out as I grow dissatisfied with my progress in integrating the notes I’d initially set out to merge.

This isn’t to suggest that it wasn’t enjoyable or outright beneficial to have fallen into the side quests: the adventures to-and-fro are part of what makes this system such a joy in which to work. Trying to do even a disciplined subset of note integration while writing a draft has often sent me so far from freeform writing that I never was able to return to it.

My attempts at writing anything long-form from my Zettelkasten have so far failed. Before I started it in January 2021, I bought into the process suggested in Sönke Ahrens’ How to Take Smart Notes which implies that papers are written mainly in the slipbox before being pulled out, organized, and edited3. My interpretation of this process was that draft outlines were to emanate from the slipbox, like Athena bursting from Zeus’ forehead fully formed7. I’d never tasked the Zettelkasten with solving the problems I wanted to write about, however; I only knew it had things to say about the topics. With such an incorrect expectation in mind, the natural reaction was to work on the notes until such a draft could be found. The creative chaos that is my writing in the slipbox never gave me the drafts I wanted but did manage to inspire massive outlines that would grow out of control. These would break into smaller projects that would, in turn, do the same like so many public administrators in Parkinson’s Law8. I was never getting anywhere meaningful and became saturated with drafts that discouraged me from completing any of them.

Integrating drafts into the Zettelkasten.

With the benefits of roguelike modalities in mind, I archived my drafts and began comparing them to my permanent notes. Where the slipbox notes did not demonstrate my argumentation, I added links and abstractions that pulled together the set of ideas. Where ad hoc research had generated new supporting points, I wrote and integrated new notes. As a more concrete example, while reporting on the topic of velocity measurements in scrum, I went down a branch of novel thought that explored how velocity is ineffective to measure if a team cannot focus its efforts on specific goals—existing notes on velocity needed to add this thought and link it to notions of goals and purpose. In a splinter draft, I had taken this idea further, picking apart the aspects of velocity as described by physics. I had recalled that velocity and speed differ in that velocity entails heading straight toward a point B, where speed alone can take you anywhere, irrespective of any intended endpoint. Fascinated by this analogy, I’d worked toward identifying what mass, acceleration, and inertia might look like when described in terms of teams. I migrated these equations into a relatively nascent physics section of notes.

Tying it all together, in theory.

Follows is a proposed workflow for learning processes through to writing completed drafts. As I have little experience in successfully producing a final draft using my note box as a writing partner, some aspects of this are more theoretical than others. I have been following this process recently, however, and anticipate refining it further as I see what’s helpful and what other distractions it hasn’t eliminated.

Scan: Take a cursory look at relevant books and articles. Scan through headings, images, etc.9
Read: Add marginalia of some sort to annotate the literature for future use.10
Copy: Review what was identified as necessary while reading11, rewriting and summarizing the points into personal notes. I do this by hand in notebooks.
Wait: What was interesting while reading is not necessarily relevant. Waiting some days or weeks before revisiting may suppress interest in points that mainly contribute noise.
Review: Decide which points are important enough to continue distilling into permanent notes. I use a pen to highlight the issues that I want to move forward.
Distill: Write notes for the permanent collection. I do this on physical notecards. I’m currently in the habit of picking where the note will fall within the collection before writing it, which is an unnecessary distraction in a digital collection.
Cite: Update the Zettelkasten index with the new topics if they are high enough level to warrant inclusion there, add any new reference material to the bibliography, and add citations to the cards to track the ideas’ provenance.
Integrate: Ponder which other notes and indexes need to be updated to reference the new notes. Link into the notes from those, and link out where the identified relationships are also significant in the other direction.
Explore: When an idea becomes attractive enough to write about more completely, explore the note collection to pull together the points that might be valuable to include in a draft.
Outline: Organize the notes and ideas into an outline. Alongside that, track a list of new questions and research topics for later visitation.
Write: Produce a draft, updating the list of questions/research topics as needed.
Proofread: Review the draft, especially with an eye toward points that need more research.
Edit: Where beneficial, revisit the draft to make any immediate changes.
Regroup: Depending on what has ended up in the questions/research topics list:

The process may need to start again at step 1 to bring in more material.
Where new points and relationships have been added to the draft, going back to step 6 to incorporate these back into the Zettelkasten may be desirable. Doing so will further refine and associate those points to help during the subsequent draft phases.
When the drafts are nearing completion, it won’t be as necessary to jump so far back in the process. Iterate from step 10 onward.

Keep it separated.

A Zettelkasten can be a powerful ally when using it as a writing partner—but it can also be a dangerous distraction. What makes it enjoyable to build readily leads one down a path of procrastination and irrelevance during drafts when not separating writing projects into distinct focus modalities. Those modalities interact iteratively, where the draft learns from the Zettelkasten for a time, and then the Zettelkasten takes its turn learning from the draft.

The modalities of focus benefit writers in contrast to the caustic effects of multitasking. Doing only one small part of learning or writing at a time requires less context switching, allowing deeper focus, and resulting in higher quality and faster production. A hidden benefit of rejecting multitasking comes from energy and motivation. Every change of direction makes focus harder, and all outstanding tasks require exceptional energy to keep on standby. Produce higher quality, faster, and more sustainably by adding boundaries to work.

↩
Allen Wilson, “Playing the Zettelkasten RPG Through Arbitrary Constraints,” Zettelkasten Method (blog), April 2022, https://zettelkasten.de/posts/playing-zettelkasten-rpg-through-arbitrary-constraints/.
↩
Sönke Ahrens, How to Take Smart Notes: One Simple Technique to Boost Writing, Learning and Thinking: For Students, Academics and Nonfiction Book Writers (North Charleston, SC: CreateSpace, 2017), 23–26.
↩
Ahrens, Smart Notes, 58–63.
↩
David Rock, Your Brain at Work: Strategies for Overcoming Distraction, Regaining Focus, and Working Smarter All Day Long, 1st ed (New York: Harper Business, 2009).
↩
Camille Fournier, The Manager’s Path: A Guide for Tech Leaders Navigating Growth and Change, ed. Laurel Ruma, First Edition (Beijing: O’Reilly, 2017), 53.
↩
Ahrens, Smart Notes, 72.
↩
“Athena | Goddess, Myths, Symbols, Facts, & Roman Name | Britannica,” accessed July 9, 2022, https://www.britannica.com/topic/Athena-Greek-mythology.
↩
C. Northcote Parkinson, “Parkinson’s ,”Law The Economist, November 19, 1955, https://www.economist.com/news/1955/11/19/parkinsons-law.
↩
“T.H.I.E.V.E.S. - Reading Technique | Kent State University,” accessed June 19, 2022, https://www.kent.edu/writingcommons/thieves-reading-technique.
↩
Anne-Laure Le Cunff, “Interview: Using Books to Navigate Life with Juvoni Beckford,” Ness Labs (blog), November 3, 2020, https://nesslabs.com/juvoni-beckford-interview.
↩
Sascha Fast, “The Barbell Method of Reading,” Zettelkasten Method, 2018, https://www.zettelkasten.de/posts/barbell-method-reading/.{:target=_blank}

Glory to the short and weighty.

2022-03-07T00:00:00Z

Deciding how to spend time is one of the more challenging and worthy things to, well, spend time on. Weighted shortest job first (WSJF)3 is an excellent tool for estimating the best uses of time.

Understanding weights and lengths.

The purpose of WSJF is to rank activities by the order in which they are best accomplished. It does this by producing a “WSJF score,” an abstract figure that represents the value of time when spent on a given project—its “bounce to the ounce.”1 It does by estimating the value of an activity and dividing it by its size or difficulty (more or less— there’s nuance to explore).

wsjf = cost of delay / task size

What if we just didn’t?

The cost of delay looks at what we miss if we don’t attend to a given set of tasks instead of measuring the work’s outright value. This approach helps us eschew a fixation on the sunk costs of existing work and instead allows us to see the economics of our forward-looking choices.

One may view the cost of delay as an estimation of what matters. Figuring out what matters is difficult; work that provides immediate value tends to be prominent, while resilience-enabling work is easy to overlook.

The cost of delay asks us to survey all opportunities and the ways they support the business along several vectors:

cost of delay = value to the user or business + time criticality + risk reduction + opportunity enablement

Value to the user or business.

Aside from (maybe) time criticality, this aspect tends to be the loudest in an organization. This facet measures the presumed value the work will provide to an end user or business stakeholder.

Time criticality.

Work that has a deadline or declining potential is preferable to do sooner in proportion to its urgency. While it depends on the type of work, this facet rarely has much differentiation, but it matters a lot when it does.

Risk reduction.

Risk reduction typically pairs with the opportunity enablement factor to become one single figure, which means something to the effect of “efforts that add more certainty for an improving future.” They are helpful to think about separately, as a project can simultaneously reduce risk and increase opportunity.

While two sides of the same coin, this factor asks us to consider whether we can reduce any categories of business risk.

Opportunity enablement.

The other side of our “improving the future” coin is the unlocking of possibility. Opportunity enablement is where work creating distant possibilities as part of a business’s strategic interests gets prioritized or projects block other more attractive projects.

Estimating difficulty.

The divisor in WSJF represents some abstraction of difficulty or time required to perform the work. Any approach to estimating this figure is valid, so long as it produces a number. Many approaches to estimation exist, but all that matters is that the estimates are in proportion to one another. Where one body of work is double the effort of another, assign it an estimate twice the size.

Putting it into practice.

With the concepts in the system now laid out, let’s move on to implementing it. There are five steps to implementing WSJF:

Brainstorm: Get all of the opportunities written down.
Estimate: Calculate a WSJF score by assigning numbers to the different dimensions.
Refine: Interpret the results, get more information, adjust the estimates.
Execute: Put the results into practice.
Revisit: Always be calibrating.

1. Brainstorm.

Essentialists spend as much time as possible exploring, listening, debating, questioning, and thinking. But their exploration is not an end in itself. The purpose of the exploration is to discern the vital few from the trivial many.

—Greg McKeown, Essentialism2

To prioritize effectively, you need to know what’s even available. Put everything you’ve been hearing and thinking about into a document and spend time considering what’s missing from the list. WSJF will tell you what’s best to spend your time on, but the creativity that goes into producing that options is the limiting factor. If only a few things are truly worth focusing on, spending time seeking to put those items on your list is time well spent.

2. Estimate.

The estimation exercise in WSJF happens one factor at a time and needs to consider all possible projects simultaneously for the given factor. Each element needs at least one project to serve as the basis, and that project is assigned a value of 1. Estimate the remaining items relatively.

For example, if following the prescribed order as laid out here (which you don’t have to do!), we’d look at user/business value first. We’d ask which project is least valuable to a user or business stakeholder at the moment of its completion (don’t get caught up thinking about opportunity enablement while thinking about direct value!). After surveying the list, assign a project an estimate of 1. A subsequent scan might reveal other projects that are also roughly equivalent and also deserving of a 1 designation. Estimate the remainder in proportion to this baseline value—tasks seeming thrice as promising are assigned 3, and so on. Once every project has an estimate for the given factor, move on to the next aspect.

Hakuna matata.

It’s essential not to get hung up on the estimation activity. If some light research has happened ahead of time to get a better sense of circumstances for the estimates, that’s great! It is not required, though. What matters is that the task list is ordered mostly correctly in the end, not that every value is accurate. If you have a pretty good handle on the nature of the projects, the chances are that you’re looking at a difference of 90% accuracy from your best guesses against 95% for better guesses with more information. This 5% gain is not necessarily worth the prework because of the next stage.

3. Refine.

With a completed estimation in place, WSJF scores, and a corresponding best guess at the order in which work should unfold, it’s time to validate the results. The first step is to review the order of the project list and ask: “Does this match the understanding I have about what needs to happen?” If the answer is no, and there will often be a couple of things that raise your eyebrow, then another pass at the underlying estimation is warranted. Make tweaks to reflect your expectations better. If there are significant differences, though, consider whether you’ve severely understated something… or whether you should be revising your expectations.

Once you agree with your ranking, start socializing the results. Stakeholders and team members should discuss the order of projects to go through that same revision process:

Do people agree with the order?
Do they need to hear the narrative to understand the rationale?
Do they have viewpoints that change our understanding of the value of the projects?

Note that it is helpful to discuss the relative factors of the cost of delay between projects. It will frustrate everyone if stakeholders get involved in the underlying numerical estimations of the system. Keep it high level: discuss the relative value between projects they’re concerned about, but refrain from making them understand the scores across all projects.

We’ll expect a movement of some projects in this process, but not too much. If an overhaul is needed, that’s an excellent trigger for digging into what it is about the flow of information that caused the first pass to be so far off base.

4. Execute.

With a prioritized list now settled, you can put it into motion. Translate the list into whatever’s most helpful for working toward the results described by the projects, be that a roadmap or a to-do list.

5. Revisit.

It’s vital to keep prioritized lists updated to learn the best information about the optimal way to spend our time. Be wary of spending too much time on this process, especially brainstorming and estimating. A process that’s too demanding to follow will lead to infrequent catch-up sessions, which reduce the effectiveness of WSJF. A nimble approach to prioritization would see this process often happen, keeping up with new information on a similar timeline as it comes in (maybe not immediately reacting to all further information… but not waiting to review months of new projects all at once, either). It’s no secret that the easier something is to do, the more readily one benefits from it4.

Serving suggestions.

I find WSJF valuable for many areas of my life. The above caters somewhat to the context of a software engineering team, but not by much! With some slight adaptations to what the cost of delay might mean and how to measure the investment needed for an item, I’ve used WSJF to help me in many different subjects:

Manage development team priorities, naturally.
Decide which books to read next.
Pick which prospective hobbies to double down on (and which to set aside permanently).
Start writing more!

↩
It seems Pepsi coined the predecessor to “bang for your buck”.
↩
Greg McKeown, Essentialism: The Disciplined Pursuit of Less, First Edition (New York: Crown Business, 2014), 61. More from this source.
↩
“WSJF,” Scaled Agile Framework (blog), accessed January 7, 2021, https://www.scaledagileframework.com/wsjf/.
↩
B. J. Fogg, Tiny Habits: The Small Changes That Change Everything (Boston: Houghton Mifflin Harcourt, 2019).

Welcome to Squidtree.

2021-07-13T00:00:00Z

I’m thrilled to announce the launch of Squidtree! This project has been a work in progress since 2008, and while it continues to be so today… it’s actually a real website, finally.

About me.

I’m Shaine Hatch. The most important thing for me to share with you about this fact is that I am the curator of this website. There are a number of other interesting things for me to share with you; this is in fact the purpose of the website that I am announcing to you in this very moment. While I will leave most of those secondary interesting points for future writing, I can share today that my focus lies in software engineering, the management thereof, and the hoarding of art supplies.

About Squidtree.

Goals.

I’m looking to do three things here: make my Zettelkasten public (more on this in the future), start blogging again, and finally get around to playing with some design concepts in production.

A sordid and oddly Flash-based background.

I started programming in 2002. This is a website launch announcement and not instead a recipe, so I’ll spare you my entire life’s story, but I’m going somewhere with this.

In 2002, I was in high school, and looking back, I’d like to think that the nature of web development was also a teenager. CSS had not yet completely taken over from table-based design, and I spent a lot of time in those years learning about CSS from the masters while also reading through the arguments for and against delegating styling entirely to CSS.

During those 3 years, I also built 4-or-so Flash-based websites. 2 years after that, I was building renegade counterfeit investigation research tools in Adobe AIR, which basically brought Adobe Flash into the desktop environment, to augment the dysfunctional corporate software we were attempting to use for that purpose. A few years later, in my first official web development job, I spent a lot of time working around browser dereliction in the custom font world with a tool called sIFR or Scalable Inman Flash Replacement, which allowed us to use custom fonts by swapping out text with tiny Flash applets displaying the text in an embedded custom font. The namesake of that tool, Shaun Inman, (very) indirectly sent me down a path that eventually leads to this website.

Site design.

Squidtree features a revolving color scheme that’s anchored to the seasons of the year. This post was made in July and therefore reflects an orange color, existing in a season whose heat will escalate to a bright red. Come fall, purples will be introduced as things cool down on their way to the cold blue of winter. When spring starts to warm things up again, the site’s colors will rotate through green hues, back on their way to the warm colors of summer. Any piece of content published will become pinned to the colors of its publish date, which is why different colors are displayed while navigating the website.

This color concept has been one of the biggest drivers for me to work on this site, and it has followed me around for many, many years. But I cannot take credit for it. The reason for the Flash story? Flash led me to sIFR, which led me to the creative work of Shaun Inman, who shared his inspiration about the nature of the passage of time during a site relaunch. (If the page is unreadable, you may hit this color toggle URL and then reload the page). While Shaun’s colors fade, my interest was the progression of color with time—with the non-readable mode toggled on, you can see the color effect he was going for here. (Keep in mind this page was built many years ago against different constraints, and what we see today does not resemble what we’d experienced in the browsers of the day.)

Building Squidtree.

While this website has been under development since 2008, it has served its purpose over the years. There’s real, live content today because there’s real, live content to be shared. Before now, though, its purpose has specifically been to be in the process of getting developed…

…Many, many times.

This site has been rewritten with various forms and designs to

further develop a custom PHP framework
learn the Yii PHP framework
further develop understanding of Backbone.js
self-train isomorphic React & Node
familiarize myself with Elixir and Phoenix (twice)

Alongside those studies, many finer points were researched, such as CSS frameworks/patterns, testing frameworks, various databases, API design strategies, and a dozen other technologies and practices.

Technical details.

Having finally delivered a non-trivial version of the site, I can say that Elixir and the Phoenix framework have performed a miracle which no other technology combination has managed to do so far. The technology was easy to learn, it was opinionated and clear enough that I didn’t get caught up in the trap of non-development decision making, the language and framework details were engaging enough that I was able to build it as a point of entertainment in my spare time rather than treating it as a learning chore, and so much more.

The site is built mostly in markdown files on a static file server. Elixir manages a webserver to read and serve its contents, but the interesting part of this implementation involves OTP to manage processes with different responsibilities to the site.

As mentioned, the webserver is a given. Webserver requests mostly involve reading a markdown file and processing it in fairly complex ways. A GenServer is running with several ETS tables to pull in these markdown files and retain their processed copies in memory, so new requests can immediately serve the content.

Serving the content is only half of the story. With all of this content and associated metadata in memory, more useful behaviors are in place which index the content and make it available in contexts beyond just reading a document itself—a note about a book can now easily pull in every note that references the book, for example.

In closing.

While not necessarily the most efficient manner of learning, I couldn’t recommend more the practice of always having a background project that just so happens to need the application of the technical skills which you do not presently possess. The combination of understanding the requirements, having interest in moving forward, but not having the necessary skills to quickly or entirely deliver creates a space within which there’s a fast track toward technical growth.

I’m excited to have Squidtree up and running at long last. This milestone marks a time when my focus is no longer on learning technology, but instead is on deeply integrating ideas and sharing the resulting insights.