Systems Thinking, Project Management, and Agile – Part 6: “Abandon All Hope,…”

[For this series, it will help to have read “System Dynamics and Causal Loop Diagrams 101.”]

“…ye who enter here.” So reads the inscription to the Gates of Hell in Dante Alighieri’s epic poem, “Divine Comedy.” Who among us hasn’t felt on occasion that stepping across the threshold to our place of employment is like passing through the gates of Dante’s Inferno? But as the poets have told us, the way to peace is to find the path through our troubles. In this article, we’ll look into just how deeply project system dynamics can adversely affect progress and even whether or not the project is successful.

But I do want to arm the reader with a couple of rays of hope. The concluding article in this series will focus on how this system model1 can be used to good effect, how it can be used to identify problems before they grow out of control. Therein lies the path to peace. Before we get there, we need to understand several more influential feedback loops.

As the Delay to Completion becomes critical, management begins to panic. Not wanting to push the deadline out they work to influence the other three options focused on modifying the behavior of the delivery team. The end result is a team that is caught in the Work Faster, Work More, and Add People loops along with all the other associated downstream loops. The effect is compounded by the emergence of other feedback loops if teams are placed in this position for an extended period of time.

Over time, the shortcuts, hacks, and quick fixes put in place to keep the pace of progress as high as possible settle in as technical debt. They work – for now – so they don’t surface as errors for quality assurance to discover. Down the road, however, solutions hastily put in place as stop-gaps fail when later solutions require existing solutions to be more robust then they are. For example, a software method that doesn’t take advantage of multi-threading may break when a later solution needs that method to scale beyond it’s single thread capacity. The shortcut is now a defect.

Figure 1

If the technical debt remains in place for an extended period of time, it may be covered by several release layers. When it does flip to defect status due to some later stress, it can be much more time consuming and expensive to uncover. The original developer of the code may not be available or even if she is, it could take her quite a bit of time to become reacquainted with the code. This can be thought of as a form of dark debt and is reflected in the Errors Build Errors Loop (Figure 1, J).

As the teams struggle to keep up the pace of progress and reduce the Delay to Completion, work streams start to become out of sequence. One team has an easier time at crafting their solution while another, to which they are dependent on the output, hits a significant snag and is delayed several weeks. In order to stay busy, the first team starts work on something else while the second team finishes their work. When the second team delivers, the first team is not prepared to immediately shift back to their original work stream and so their deliverable is delayed even further. Meanwhile, a third team, that was dependent on the first team’s deliverable has now been delayed by the cumulative delay of the first two teams. Teams and individuals begin to take shortcuts as delivery of interim work products become out of sync with each other. The diminished focus and desynchronization of work streams leads to an increase in the Error Fraction, which in turn leads to a further Delay to Completion. This is the Haste Makes Out-of-Sequence Work Loop (Figure 1, K).

Figure 2

As the effects of the Haste Makes Out-of-Sequence Work Loop build,  team begin switching back-and-forth between work streams depending on who is making the most noise for the completion of any particular deliverable. This is the Thrash and Churn Loop (Figure 2, L). Switching from stream to stream or, in worst cases, task to task, places a tremendous burden on development teams and can do more to slow progress than almost anything else I’ve encountered in team management. Not covered in this model is the type of churn that occurs when parts of the project undergo redesign after work has begun on the existing design. Long term projects are particularly susceptible to adverse impacts from redesign as the changes are often farther reaching. The drivers behind a redesign can range from trivial (a new CTO has a personal dislike for a platform vendor) to critical (a security flaw uncovered in a core technical component.)

If all the loops described to this point in the article series are allowed to run uncorrected the system is likely to crash as the project becomes one massive firefighting effort. A key indicator for when this is happening is employee morale.

Figure 3

The increased Fatigue, the growing burden of Work/Rework to Do, the unsatisfying Task Switching between work assignments all combine to causes a decrease in team Morale. This is the Hopelessness Loop (Figure 3, M). Teams are left with a powerless feeling of being caught on a never ending treadmill. And so, stepping across the threshold to the office is like passing through the gates of Dante’s Inferno.

The ripple effect from a decrease in Morale leads to a decrease in the Workforce as employees leave the organization in search of less stressful, more satisfying work. This is the Turnover Loop (Figure 3, N). The remaining demoralized employees are even less productive and unhappy employees make more mistakes, thus increasing the Error Fraction in the system. The downstream result is that the Delay to Completion increases yet again.

If corrective action isn’t taken the law of diminishing returns becomes evident and the system collapses. The cost overruns become prohibitive and the project is cancelled. Worst case, the organization runs out of resources (money, time, or both) and goes out of business. Those are bad things. In the concluding article to this series, we look at how this model can be used to read the current state of a project’s system dynamics and explore some ways we can intervene such that the system doesn’t run out of control.

References

1The core of the model I use to assess team and organization health is based on the work of James Lyneis and David Ford: System Dynamics Applied to Project Management, System Dynamics Review Volume 23 Number 2/3 Summer/Fall 2007


Image by Myriams-Fotos from Pixabay

Systems Thinking, Project Management, and Agile – Part 5: It Lives! But it’s Out of Control!

[For this series, it will help to have read “System Dynamics and Causal Loop Diagrams 101.”]

In the previous article for this series, I described three options managers could consider if moving the project deadline was out of the question.

  1. Increase employee work intensity
  2. Call for overtime
  3. Hire people

On the face of it, they each appeared to offer a path toward returning a drifting schedule to be on time. Now let’s look a little further down the road to see what happens when the juice is applied to each of these options in turn. If we implement any of these options, what are the likely consequences?

We know that errors in the work flow are unavoidable. If we encourage or pressure the development team to finish more work in less time (the Work Faster Loop1, Figure 1, C) this will result in an increase in the errors along with an increase in the amount of Work Done.

Figure 1

This is the Haste Makes Waste Loop (Figure 1, F). In other words, the increase in Work Intensity will have a concomitant increase in the Error Fraction which means there is an increase in Errors generated. The extended consequence of pulling the Work Intensity lever is an increase in Work to Do in the form of extra Rework to Do.

OK. So Option 1 isn’t a get-out-of-jail-free card. There are strings attached. How about Option 2, call for the development team to work overtime?

Figure 2

By increasing Overtime, the risk of Fatigue increases sharply. This results in yet another increase in the Error Fraction (tired people make more mistakes than rested people) and a decrease in Productivity (tired people don’t work as efficiently as rested people.) Both slow down Progress and increases the amount of Rework to Do in the system. This is the Burnout Loop (Figure 2, G).

OK. So Option 2 doesn’t lead to sunshine and roses. There are dark clouds and weeds in the mix. Let’s give Option 3 a go, hire more people!

Figure 3

So we’ve beefed up the Workforce by hiring a bunch of people to join the team. With all those extra people in the mix we’ve also increased the overall Congestion and Communication Difficulties. The email traffic increases, everyone’s Inbox fills up faster, meeting attendee size increases along with the number of meetings. The signal to noise ratio decreases and miscommunication increases. This increases the Error Fraction, decreases Productive, and decreased Progress. End result: the Too Big to Manage Loop (Figure 3, H).

But that’s not all. By hiring extra people, we’ve activated the Expertise Dilution Loop (Figure 5, I).

Figure 5

All those new hires don’t come in off the street ready to go. They decrease the depth of Experience available to focus on making progress. Experienced employees have to slow down and assist new employees in understanding the technical systems, the architecture, and development standards. New employees will need some period of time to become familiar with the work environment, project objectives,  who’s who, and where the coffee is.

As they work to understand and gain experience with the systems, new hires will necessarily make mistakes and increase the Error Fraction. While there are more workers available to focus on the product backlog, the available expertise is spread much more thinly and is collectively less experienced until such time the new workers are up to speed with what needs to be done and how. So the errors go up and Productivity goes down. The down stream effect is often a further increase in the Delay to Completion. As the saying goes, throwing more people at the problem more often than not makes the problem worse.

OK. So no unicorns and rainbows here either. More like a lot of warthogs and rain.

Looks like the first level effects were negated by the second level consequences. That’s bad enough, but the third level consequences can be even worse in that they are often much longer lasting and much more difficult to resolve. We’ll look at those in the next article in this series.

References

1The core of the model I use to assess team and organization health is based on the work of James Lyneis and David Ford: System Dynamics Applied to Project Management, System Dynamics Review Volume 23 Number 2/3 Summer/Fall 2007


Image by Gerd Altmann from Pixabay

Systems Thinking, Project Management, and Agile – Part 4: Welcome to the Labyrinth

[For this series, it will help to have read “System Dynamics and Causal Loop Diagrams 101.”]

The capable product owners I know have at least an intuitive understanding that the challenge of guiding a project through to completion is more than a bit like Theseus on his way to defeat the Minotaur. The great product owners have a much more present awareness of the labyrinth before them. Depending on the project, the team, and the work environment, the product backlog just might be the easy piece. It’s more knowable then the myriad of ways a system can work against project success.

The purpose of this series of articles is to shine light on those wily ways of the system, to make more known what capable product owners intuit, to help you become a great product owner.1

In the previous article, we covered how a project can end up with a growing delay before completion. The obvious fix was to push out the deadline, thus erasing the delay (The Shift Deadline Loop, Figure 1, B.) Management has a strong dislike for this and often avoids changing deadlines even when faced with minimal consequences. It’s more likely there are other factors that make the consequences significantly greater. Perhaps there are budget constraints or a delivery date that is tied to a major event like the launch of a suite of related products or a conference.

So if management is faced with an unmovable deadline, the Delay to Completion must be resolved by some other means.

Figure 1

With more work to do and less time to do it, there is now a Talent Resource Deficit. X number of employees working 40 hours a week will no longer get the work done on time. Management’s next set of options lie with changing the behaviors of the development team. We’ll consider three of these options.

The first option is to put pressure on the development team to focus on work more during the time they are working. Maybe this involves tightening the work hours people are expected to be available. Or restricting remote work so team members are in close proximity for longer periods of the day in the hope of shorting the delays inherent in remote communication and problem solving. Or working to eliminate distractions in the workplace. There are many possibilities here.

Figure 2

This is the Work Faster Loop (Figure 2, C) – complete more work in less time. If the development team is more focused, the thinking goes, Productivity will increase and in turn drive an increase in Progress. More Progress leads to less Work to Do which leads to less Total Known Remaining Work which leads to less Time Required to Complete Work and a decrease in the Delay to Completion. Eventually, the Talent Resource Deficit is reduced and the development team can relax a bit.

This looks great in principle. Will get to the messy reality in a future article, but for now, we just need to understand how management typically thinks things should work.

The second option is to ask the development team work Overtime.

Figure 3

Officially, management asks. Unofficially, it isn’t presented as an option. If the development team is putting in more hours, the thinking goes, then the amount of Effort being applied to the work stream increases. As with an increase in Work Intensity, this works its way through the system to reduce the Delay to Completion and ultimately, the development team will no longer need to put in extra hours. This is the Work More Loop (Figure 3, D).

The third option is to simply hire more people to work on the development team.

Figure 4

By deciding to Hire Talent, management will increase the Workforce and once again increase the Effort aimed at increasing progress. As with the increase in Work Intensity and Overtime, this eventually manifests as a decrease in the Delay to Completion. This is the Add People Loop (Figure 4, E).

There you have it. Schedule slipping? Flip one or more of the following switches…

  1. Extend the deadline
  2. Increase employee work intensity
  3. Call for overtime
  4. Hire people

…and in short order the system will be back in balance and the project on schedule. Problem solved.

Not so fast there, young Theseus. Remember, there’s a Minotaur on the hunt for you somewhere in this labyrinth. In the next article of this series we’ll begin looking a some of the ways this simplistic machine thinking can go sideways…fast.

References

1The core of the model I use to assess team and organization health is based on the work of James Lyneis and David Ford: System Dynamics Applied to Project Management, System Dynamics Review Volume 23 Number 2/3 Summer/Fall 2007


Image by Daniel Roberts from Pixabay

Systems Thinking, Project Management, and Agile – Part 3: Let the Interactions Begin!

[For this series, it will help to have read “System Dynamics and Causal Loop Diagrams 101.”]

In the previous article we learned how to read an important feature of system diagrams. Namely, the interactions – the direction and whether or not the effect of the interaction was direct or indirect. With that understanding in hand, we can begin to look at real-life interactions. Well, real in the sense they are reflections of real-world interactions. These are interactions that take place outside the Work Loop but nonetheless affect the performance of the Work Loop.

By the time we’re done building out the model, you’ll be aware of just how many brake and gas peddles there on in this project management automobile (building on the metaphor used in the previous post for this series!)

Revisiting the Work Loop1 (indicated by the icon)…

Figure 1

We see there are several things that can interact with progress: How productive an individual or team is and how much effort they apply to their work. The green open-head arrow indicates that the relationship between each of these interactions and progress is direct. An increase in Productivity, applied Effort, or both will increase progress. Decrease Productivity or applied Effort and progress slows down.

That seems straightforward. But it isn’t all good news. Being more productive and applying more effort will also generate an unknown increase in Errors. Consequently, the amount of Undiscovered Rework will also increase.

Figure 2

This means that more effort needs to be applied toward discovering the Undiscovered Rework, so the relationship between Undiscovered Rework and the effort to actually discover the rework is direct (the green open-head arrow.) An increase in the amount of Undiscovered Rework results in an increase in the effort needed to actually discover all the hidden rework.

There is an inverse relationship in the mix here, too (the red closed-head arrow.) As the time it takes to discover defects and bugs increases, the rate of rework discovery decreases. This is particularly true with dark debt issues and defects that have been hidden in the system for months or even years. Finding gnarly bugs often takes a lot of time and effort. UI typos and misaligned text box labels, not so much.

So far, so good. But what affect does the additional work from the Rework to Do bucket have on the project schedule?

Figure 3

The system as it stands can only handle so much throughput. (Later in the article series we’ll cover ways to influence this throughput.) Adding Rework to Do to the flow of overall work that needs to be done will also slow down the rate at which original Work to Do gets to Work Done.

If project life is good the amount of Work to Do and Rework to Do decreases so that the amount of total Known Remaining Work decreases. If the amount of Work to Do and Rework to Do are increasing, the amount of total Known Remaining Work increases and project life is bad. (Figure 3)

There could be any number of causes driving the project down the bad road, hopefully only for a short while. Since we don’t know what we don’t know,  after work begins on a project discoveries are made about additional work simply by working on known work. It could also be that additional work is added to the project intentionally. Perhaps marketing has discovered a feature that could place the end product in a stronger position or an existing feature needs to be strengthened to help close a sale or a planned approach turns out to be technically unfeasible or…the list is endless.

With the increase in the amount of Known Remaining Work, and all other aspects of the project remaining unchanged, at the very least the Time Required to Complete Work will increase. This in turn pushes out the projected delivery date and therefore increases the Delay to Completion. It’s at this point management starts getting grumpy.

Call out any project management methodology devised by man and it’s a safe bet that it drives toward establishing a predictable completion or delivery date. Agile methodologies are no different. Delivery dates are the interface between work teams and management. When faced with the news that a scheduled delivery date is at risk, management has two basic choices available to them. Either change the delivery date to match the performance of the delivery team or change the behavior of the delivery team such that the originally scheduled delivery date can be met. (A blend of the two is certainly possible but not particularly common in practice.)

The most obvious choice is to make changes that directly impact the Delay to Completion. That is, change the delivery date to accommodate the delivery team’s performance.

Figure 4

This introduces our first feedback loop – the Shift Deadline Loop (Figure 4, B.)

Let’s say the amount of Total Known Remaining Work has increased such that the Delay to Completion has grown to four months. If the decision is made to push the Deadline out by four months the effect is to increase the amount of Time Remaining which in turn decreases the Delay to Completion to zero. (Savvy Agile team members recognize that the shelf life of a zero completion delay is something less than 24 hours.)

But remember, schedule delays make management and other stakeholders grumpy. They’re loath to choose this path unless it is forced upon them by having exhausted all other options. And those options usually involve putting pressure on the system at other points.

If management chooses to follow the path of changing the delivery team’s behavior, the effects can be as far reaching as they can be significant. Depending on the choices made, the effects could be either very good or very bad. Very good results are hard. Very bad results are easy and therefore much more common. We will begin to explore these in the next article for this series.

References

1The core of the model I use to assess team and organization health is based on the work of James Lyneis and David Ford: System Dynamics Applied to Project Management, System Dynamics Review Volume 23 Number 2/3 Summer/Fall 2007


Image by Gerd Altmann from Pixabay

Systems Thinking, Project Management, and Agile – Part 2: Work, Work, Work…

[For this series, it will help to have read “System Dynamics and Causal Loop Diagrams 101.”]

…work, work, work. It’s what we do.

“I have to go to work.”

“What do you do for work?”

“OK, team. Let’s get to work.”

“Where do you work?

“Is that working for you?”

“That’ll never work.”

“Let’s work together.”

“Time to roll up our sleeves and get to work.”

The notion of work is so pervasive it underpins my belief that Agile principles and practices can be applied to a variety of human endeavors beyond the narrow focus on software development. In fact, the case can be made that Agile principles and practices have been around for millennia and only very recently were codified for a software development context. Agile simply feels more natural, more aligned with how humans think and interact to solve problems. From the way we explore and learn as children to the way we solve problems at home as adults, it’s much easier to recognize Agile patterns than waterfall patterns. In spite of this, when we go to work we’re subject to the behaviors and measures of machines and Taylorism.

It doesn’t have to be that way. Agile has been shown to more effective at increasing productivity and decreasing costs in contexts beyond software. So why isn’t it practiced everywhere all the time?

I can think of a couple of broad generalizations that answer this question. First, Agile isn’t a panacea. Nothing is. To paraphrase Winston Churchill, Agile is the worst form of project management, except for all the others. Second, in the light of Conway’s Law and Shalloway’s Corollary, the systemic monster pushing back on change is a formidable one.

I have no aspirations of making Agile a panacea and will never claim it to be one. But until something more promising comes along, I can work to improve the practices for applying Agile values and principles. As for the systemic monster, that’s what this series of articles are about.

Monsters are scary because we don’t know them, we can’t see them, they’re hidden from us, they’re “out there, somewhere.” We’ll begin the process of understanding the systemic workplace monster by shining a light on work. What is it? How do we define it?

With each new day, in one form or another, we face a newly filled box of Work to Do. On the far side of the day, there is an empty box of Work Done.

In a perfect world, by the end of the day, Work to Do is empty and Work Done is full.

This transition doesn’t happen by itself. Magic won’t get work moved to done. There’s effort involved. More effort means more progress. Less effort, less progress. On Agile software projects, Work to Do is described in the product backlog and Work Done manifests as a deliverable product or service.

Typically, there is some form of measure on progress toward the goal of getting work to done. In scrum, this might be story points completed or business value delivered.

But we don’t live in a perfect world. Whatever the endeavor, errors and mistakes are part of the work effort. Instructions were unclear or incomplete, time constraints caused the work to be rushed, the person doing the work was apathetic or otherwise unfocused – there are thousands of reasons for why some of the work fails to meet expectations.

Since our efforts to complete work are always less than perfect by some percentage, part of the effort that creates progress is also an effort that generates errors. Anyone managing a project – especially a technical project – should expect that there is a box of Undiscovered Rework hiding somewhere. How big that box is or how fast it’s filling are unknown. All we know at this point is the box of Undiscovered Rework exists. In software development, the contents of this box are referred to as defects or bugs.

We know the box of Undiscovered Rework is there somewhere. So now we need a deliberate effort aimed at discovering that rework. This is the job of quality assurance and testing professionals. Their efforts at rework discovery bring the defects and errors to light so that they can be documented and added to the flow of Work to Do.

This is the work loop.1 Human interactions and behaviors aimed at achieving some larger goal provide the energy for driving this loop. The quality of those interactions determine how fast work moves through this loop.

In subsequent posts, we’ll begin to explore several specific human interactions and behaviors the can either support or inhibit the flow of work through this loop. Be sure to read “System Dynamics and Causal Loop Diagrams 101” to prepare for the rest of the articles in the series.

Before you go, however, read through the following phrases and make a mental note of those that resonate with you – either because you have heard them before or perhaps because you have uttered them yourself while working on a project.

  • “We’re not moving the delivery date.”
  • “We’ll just have to work harder.”
  • “The team will have to put in more time until we’re caught up.”
  • “We’ll need more people on the project.”
  • “The team will have to work faster.”
  • “We’re to the point of exhaustion.”
  • “I’m losing track of all the pieces.”
  • “There’s no time for training.”
  • “Where did those errors come from?”
  • “We’re waiting on another team.”
  • “Another person quit the company?!?!”
  • “I don’t care. I get done what I get done when I get it done.”

References

1The core of the model I use to assess team and organization health is based on the work of James Lyneis and David Ford: System Dynamics Applied to Project Management, System Dynamics Review Volume 23 Number 2/3 Summer/Fall 2007


Photo by Elisa Ventur on Unsplash

Systems Thinking, Project Management, and Agile – Part 1: The Revenge of Frankenagile

[For this series, it will help to havve read “System Dynamics and Causal Loop Diagrams 101.”]

The ubiquitous employee handbook is filled with rules, regulations, and descriptions of how employees are expected to behave. The larger the organization, the thicker the handbook. Handbooks and associated policies like this have been described as “corporate scar tissue.” Someone somewhere at sometime made a serious mistake, whether intentional or not, and so a policy was created to prevent that something from ever happening again. The same effect can be seen with many consumer products that have lengthy warning labels and manual pages stating things not too far from “This toaster is not suitable as a flotation device in the event of a boating accident.”

In an effort to prevent a re-occurrence, the policies effectively limit – much like physical scar tissue – the ability of people within the organization to adapt, improvise, and innovate. They limit an employee’s range of motion within the organization’s solution space. The goal is to save the organization from human error and create the perfect business machine. But in the end, excessive policies condemn the organization to a slow but certain death.

Anyone who has worked within a large organization recognizes this. For some, it’s a comfort. Knowing the rules. Knowing where the fences are. And knowing where to place blame. The less ambiguity around how a situation can be interpreted the better. For others, maybe after an attempt to change things, the environment becomes too stifling and they leave for greener, wider pastures.

Given enough time, the policies become the document of record for the organization’s culture. Any attempts to change the way work gets done within an organization that has deep scar tissue will have to confront Shalloway’s Corollary:

When development groups change how their development staff are organized, their current application architecture will work against them.

I’ve learned this corollary is not limited to software companies. In every case I’ve experienced, whether in a software company or not, the system will push back. Hard. Every Agile practitioner needs to know and respect this. Riding into work on a unicorn with a bag of rainbows and pixie dust is a gig that will not end well. At best, the organization will have made an incomplete effort at implementing Agile and “Frankenagile” will be roaming the halls – a collection of project management methodological parts that by themselves served a valuable purpose in a larger or different context, but have been stitched together to form a monster in Agile name only.

In a small company, particularly if it is working to create a software product, the monster may be small. So performing corrective surgery, while still a lot of work, is quite possible in a relatively short amount of time. For larger organizations, particularly those with deep roots in traditional project management, it can be a scary sized beast indeed. Something not to be trifled with, rather something that needs a well thought out strategy and plan of action.

It is the latter scenario I’d like to address in this series of posts (this being Part 1,  the introduction) over the next several weeks. I’ll present is a method I’ve used quite successfully over the past 13+ years for assessing the extent to which Conway’s Law and Shalloway’s Corollary are in play. It is a method for determining both team and organization health within the larger management context. The extent to which Agile can be successfully implemented in an organization is dependent on how aware management, the Agile coach, and scrum masters are of the system dynamics driving organizational behavior.

Next post in the series: Assessing and Tracking Team Performance – Part 2: Work, Work, Work…


Photo by freestocks on Unsplash

Improving the Signal to Noise Ratio – Revisited

 

Additional thoughts about signals and noise that have been rattling around in my brain since first posting on this topic.

At the risk of becoming too ethereal about all this, before there is signal and before there is noise, there is data. Cold, harsh, cruelly indifferent data. It is after raw data encounters some sort of filter or boundary, something that triggers a calculation to evaluate what that data means or whether it is relevant to whomever is on the other side of the filter, that it begins to be characterized as “signal” or “noise.”

Since we’re talking about humans in this series of posts, that filter is an amazingly complex system built from both physiological and psychological elements. The small amount of physical data that hits our senses and actually makes it to our brains is then filtered by beliefs, values, biases, attitudes, emotions, and those pesky unicorns that can’t seem to stop talking while I’m trying to think! It’s after all this processing that data has now been sorted according to “signal” (what’s relevant) and “noise” (what’s irrelevant) for any particular individual. Our individual systems of filters impart value judgments on the data such that each of us, essentially, creates “signal” and “noise” from the raw data.

That’s a long winded way to say:

data -> [filter] -> signal, noise

Now apply this to everyone on the planet.

data -> [filter 1] -> signal 1, noise 1

data -> [filter 2] -> signal 2, noise 2

data -> [filter n] -> signal n, noise n

As an example, Google (itself a filter) is a useful one. Let’s assume for a moment that Google is some naturally occurring phenomenon and not a filter created by humans with their own set of filters driving what it means to create a let’s be evil good search engine. To retrieve 1,000,000 pieces of information, my friend, Bob, entered search criteria of interest to him, i.e. “filter 1.” Maybe he searched for “healthy keto diet recipes”. Scanning those search results, I determine (using my “filter 2”) 100% of the search results are useless because my filter is “how do i force the noisy unicorns in my head to shut the hell up”. The Venn diagram of those two search results is likely to show a vanishingly small set of relationships between the two. (Disclaimer: I have no knowledge of the carbohydrate content of unicorns nor how tasty they may be when served with capers and a lemon dill sauce.)

Google may return 1,000,000 search results. But only a small subset is viewable at a time. What of the rest of the result set that I know nothing about? Is it signal? Is it noise? Is it just data that has yet to be subjected to anyone’s system of filters? Because Google found stuff, does that make it signal? Accepting all 1,000,000 search results as signal seems to require a willingness to believe that Google knows best when it comes to determining what’s important to me. This would apply to any filter not our own.

All systems for distinguishing signal from noise are imperfect and some of us on the Intertubes are seeking ways to better tune our particular systems. The system I use lets non-relevant data fall through the sieve so that the gold nuggets are easier to find. Perhaps at some future date I’ll unwittingly re-pan the same chunk of data through an experienced-refined sieve and a newly relevant gem will emerge from the dirt. But until that time, I’ll trust my filters, let the dirt go as noise, and lurch forward.


Photo by Nathan Dumlao on Unsplash

The Shopping Mall in Your Inbox

Ian Bogost writes in the Atlantic:

It feels like every company and organization I’ve ever transacted with sends me email every week. Some every day, even. Some multiple times a day.

I see this in action with my wife’s “free” email account hosted by one of the major players in the email provider space. She has another email account that sees none of this. Not because that address is unknown but because it’s hosted on a private email server that I run. Nothing gets through that isn’t specifically allowed by the postfix/spamassassin filters and lists. It’s quite simple, really. If either of us wants to do business with a company that requires an email, I create an email alias with the company’s name as the username. That alias is then assigned as a value to a “whitelist_to” key for spamassassin and entered into an virtual aliases database table for postfix to know the actual email account where the message should be delivered.

If we start getting messages from companies that don’t match the username, we know the original company either sold their list or has been compromised. When that happens, the alias assigned to the “whitelist_to” key is reassigned to a “blacklist_to” key and the alias removed from the virtual aliases database table. We never see another message sent to that alias.

In a very few cases, this also means changing the email address registered with the original vendor. But more often than not, the alias was set up as a one-time purchase and can safely be excluded. I have hundreds of these aliases set up.

The major email providers could implement a system like this. But their business model relies on advertising revenue and such a system isn’t in their interests. The people who have signed up for a “free” email account are, after all, part of their product. And why would they listen to their product and not their paying customers.

Speculating about a perfect world, Ian continues…

The algorithms churn until you’ve interacted with enough promotional emails that every store you like delivers perfectly timed messages that cater to your every need and desire. Fulfilled, happy, you purchase every item advertised to you. Of course, this is not actually what happens. The irony of people’s supposed desire to receive emails from their favorite companies is that more than half of consumers in the United States and Canada say they receive too much promotional email. Personalization is supposed to make relevant messages get through and irrelevant ones falter. But what “relevance” means is constantly changing. If I need new pants, an apparel ad might be welcome. If I don’t, it’s just annoying.

This is the downside of a push system. My email server is more aligned with a pull system. If w need or want something, we go look for it, usually at some place we’ve shopped before. It’s a minimalist mindset thing. The personal habits around this approach greatly minimize impulse purchases. The lack of stuff in our life reflects the success of this strategy – we have no debt, drive 20 year old cars, and replace phones only when the apps we need no longer work because Android no long supports them.

Ian continues…

The fundamental reason marketing emails are clogging up everyone’s inboxes is that email marketing represents the collective outcome of a company’s interactions with its customers and the mailbox services; it does not, in other words, represent me.

Indeed. Virtually all the email marketing from a vendor is opt-out once you’ve given them your email address. Research suggests there would be significant shift in the amount of follow-on marketing if everything was opt-in.

Quoting Helena Tse, the vice president of marketing at Bonobos, Ian observes:

She also told me that the company “continuously explores how to better our toolkit to automate and optimize the frequency business rules.” A professional proclamation such as this might elicit nods at an industry conference, but it casts me in the role of a generator of data rather than a paying customer—let alone a living person who doesn’t need so many invitations to partake of Riviera shorts.

As I said, when you give your “free” email address to a business, you’ve become part of the overall product.


Photo by Matthew Henry on Unsplash

What’s in YOUR manual?

 

You go to see a movie with a friend. You sit side-by-side and watch the same movie projected on the screen. Afterward, in discussing the movie, you both disagree on the motives of the lead character and even quibble over the sequence of events in the movie you just watched together.

How is it that two people having just watched the same movie could come to different conclusions and even disagree over the sequence of events that – objectively speaking – could have only happened in one way?

It’s what brains do. Memory is imperfect and every one of us has a unique set of filters and lenses through which we view the world. At best, we have a mostly useful but distorted model of the world around us. Not everyone understands this. Perhaps most people don’t understand this. It’s far more common for people – especially smart people – to believe and behave as if their model of the world is 1) accurate and 2) shared with everybody else on the planet.

Which gets me to the notion of the user manuals we all carry around in our heads about OTHER people.

Imagine a tall stack of books, some thin others very thick. On the spine of each book is the name of someone you know. The book with your partner’s name on it is particularly thick. The book with the name of your favorite barista on the spine is quite a bit thinner. Each of these books represents a manual that you have written on how the other person is supposed to behave. Your partner, for example, should know what they’re supposed to be doing to seamlessly match your model of the world. And when they don’t follow the manual, there can be hell to pay.

Same for your coworkers, other family members, even acquaintances. The manual is right there in plain sight in your head. How could they not know that they’re supposed to return your phone call within 30 minutes? It’s right there in the manual!

It seems cartoonish. But play with this point of view for a few days. Notice how many things – both positive and negative – you project onto others that are based on your version of how they should be behaving. What expectations do you have, based on the manual you wrote, for how they’re supposed to behave?

Now ask yourself, in that big stack of manuals you’ve authored for how others’ brains should work, where is your manual? If you want to improve all your relationships, toss out all of those manuals and keep only one. The one with your name on the spine. Now focus on improving that one manual.


Photo by Ying Ge on Unsplash

Improving the Signal to Noise Ratio

Ellen Fishbein asks, “Why NOT be an information sponge?” In the Age of Information, I think her answer is a good one: “I currently believe that each of us must move away from the ‘information sponge’ mindset and try to develop a more nuanced relationship with information.”

I’d have to characterize myself as more of an information amoeba – (IIRC, the amoeba is, by weight, the most vicious life form on earth) – on the hunt for information and after internalizing it, going into rest mode while I decompose and reassemble it into something of use to my understanding of the world. Yum.

More generally, to be an effective and successful consumer of information these days, the Way of the Sponge (WotS, passive, information washes through them and they absorb everything) is no longer tenable and the Way of the Amoeba (WotA, active, information washes over them and they hunt down what they need) is likely to be the more successful strategy. The WotA requires considerable energy but the rewards are commensurate with the effort. WotS…well, there’s your obsessive processed food eating TV binge-watcher right there. Mr. Square Bob Sponge Pants.

What’s implied by the WotA vs the WotS is that the former has a more active role in optimizing the informational signal to noise ratio than the latter. So a few thoughts to begin with on signals and noise.

Depending on the moment and the context, one person’s signal is another person’s noise. Across the moments that make up a lifetime, one person’s noise may become the same person’s signal and vice versa. When I was in high school, I found Frank Sinatra’s voice annoying and not something to be mingled with my collection of Mozart, Bach, and Vivaldi. Today…well, to disparage the Chairman of the Board is fightin’ words in my house. Over time, at least, noise can become signal and signal become noise.

But I’m speaking here of the signal quality and not it’s quantity (i.e. volume)

Some years ago I came across Stuart Kauffman’s idea of the adjacent possible:

It may be that biospheres, as a secular trend, maximize the rate of exploration of the adjacent possible. If they did it too fast, they would destroy their own internal organization, so there may be internal gating mechanisms. This is why I call this an average secular trend, since they explore the adjacent possible as fast as they can get away with it.

This has been interpreted in a variety of ways. I carry this around in my head as a distillation from several of the more faithful versions: Expand the edge of what I know by studying the things that are close by. Over time, there is an accumulation of loosely coupled ideas and facts that begin to coalesce into a deeper meaning, a signal, if you will, relevant to my life.

With this insight, I’ve been able to be more deliberate and directed about what I want or need to know. I’ve learned to be a good custodian of the edge and what I allow to occupy space on that edge. These are my “internal gating mechanisms.” It isn’t an easy task, but there are some easy wins. For starters, learning to unplug completely. Especially from social media and what tragically passes for “news reporting” or “journalism”these days.

The task is largely one of filtering. I very rarely directly visit information sources. Rather, I leverage RSS feeds and employ filtering rules. I pull information of interest rather than have it pushed at me by “news” web sites, cable or TV channels, or newspapers. While this means I will occasionally miss some cool stuff, it’s more than compensated by the boost in signal quality achieved by excluding all the sludge from the edge. I suspect I still get the cool stuff, just in a slightly different form or revealed by a different source that makes it through the filter. In this way, it’s a matter of modulating the quantity such that the signal is easier to find.

There is a caution to consider while optimizing a signal-to-noise ratio, something reflected in Kauffman’s comments around the rate of exploration for new ideas: “If they did it too fast, they would destroy their own internal organization…”

Before the Internet, before PCs were a commodity, before television was popular it was much, much easier to find time to think. In fact, it was never something that had to be looked for or sought out. I think that’s what is different today. It takes WORK to find a quiet space and time to think. While my humble little RSS filters do a great job of keeping a high signal-to-noise ratio with all things Internet, accomplishing the same thing in the physical world is becoming more and more difficult.

The “attention economy,” or whatever it’s being called today, is reaching a truly disturbing level of invasion. It seems I’ve used more electrician’s tape to cover up camera lenses and microphones in the past year than I’ve used on actual electrical wires. The number of appliances and gadgets in the home with glowing screens crying out for bluetooth or wifi access like leaches seeking blood are their own source of noise. This is my current battleground for finding the signal within the noise.

Enough about filtering. What about boundaries. Fences make for good neighbors, said someone wise and experienced. And there’s a good chance that applies to information organization, too. Keeping the spiritual information in my head separate from my shopping list probably helps me stop short of forming some sort of cult around Costco. ( “All praise ‘Bulk,’ the God of Stuff!)

An amoeba has a much more develop boundary between self and other than a sponge and that’s probably a net gain even with the drawback of extra energy required to fuel that arrangement. Intellectually, we have our beliefs and values that mark where those edges between self and other are defined.

So I’ll stop for now with the question, “What are the strategies and mental models that promote permeability for desired or needed information while keeping, as much as possible, the garbage ‘out there?’”


Image by Pete Linforth from Pixabay