Creating productivity tools at the intersection of product strategy, mobile software development, and user-centered design

Posts tagged Analytics

We Are All Makers



When you code, you either produce a working program or you don’t. There isn’t partial credit for a program that mostly works. Even if only one line is written incorrectly within a hundred lines of working code, the entire feature is considered broken. Oftentimes, these types of small errors creep in when you are only able to give your work half your attention. Therefore, a common pre-requisite to producing great code is having uninterrupted “maker time” when you can dive deeply into a problem to figure out its solution.

In contrast, there is the concept of “manager time” where work can be done in piecemeal fashion and still lead to results. However, I believe that being effective in any endeavor, including management, requires a period of intense focus. In product management, the feedback loop simply isn’t quite as immediate, which has lead to the flawed conclusion that long periods of concentration are unnecessary for good work.

When a product manager digs through data spreadsheets, users insights and competitive analysis, a meaningful amount of raw processing power is required to synthesize and organize all of the information. The results of this process are usually tickets, specs, and requirements, which can be categorized as the action steps for the team. Next, a product manager has to lead the team in executing on these action steps to build product features and enhancements (and to fix bugs). After that, code gets shipped and consumers start to react to the new changes. Then, data starts to pour back in about how the product is doing – How do customers like feature X? How is engagement with feature Y trending? Finally, the product manager analyzes that data and uses it to inform the next round of decisions.

Sometimes, this full cycle can take weeks, or even months, before the company can make any type of conclusive judgment the on how well the manager did in deciding on the plan of action. It is clear though that the plan of action plays an integral part in determining the success or failure of the product. If the right feature wasn’t prioritized or if the wrong feature was cut from the roadmap, then there are consequences. Similarly, if the wrong view is being rendered or if the wrong variable was being passed to a method, there are also consequences.

However, in the latter situation, the feedback is almost immediate. If you run a broken function, an exception is thrown.  Maybe the error is caught later on by QA. In either case, you as the programmer know pretty quickly that something isn’t working. And a common cause of these errors is writing complex code when you’re not focused.

Granted, managers can check email or sit through meetings in varying bursts of time, but if they are to come up with effective roadmaps and clear action steps, long periods of concentration are absolutely necessary. It may not be immediately apparent that product managers are doing maker-type work due to the long feedback cycle, but in reality, even managers need a maker’s mentality in order to be effective at their jobs. Or, perhaps another way to look it is just that we are all makers.

Posted in Code, Discussion, Top Articles - Tagged

Goals of A/B Testing



Response to: http://37signals.com/svn/posts/3169-ab-testing-its-not-about-the-results-and-its-definitely-not-about-the-why

When a person’s only job is to administer A/B tests, he can take the luxurious position to “only judge a test based on whether we designed and administered it properly.”  However, most of us do not live in ivory towers where following the right process is all that matters.  When a split-tester is part of a company, their job is to make profit.   Therefore, a test’s results definitely matter inasmuch as it helps improve the bottom line.

However, there are a couple of caveats.  First off, winning tests should still follow ethical business practices.  If a design wins based a psychological trick, such as a false countdown timer, it should not be allowed to stay on the website.  Frankly, it shouldn’t have been tested to begin with.  Additionally, a test should not have its results judged until it has, at a minimum, passed the bar of being properly administered.  In other words, having a properly administered test is a prerequisite for all the benefits that A/B testing potentially brings.  Finally, there are certain tests that might not directly optimize for profit, but instead go at alternative metrics like virality or retention.  In the end, companies should still demand more out of their conversion department than just following proper testing methodology.

The author’s point that we shouldn’t judge our success on increased learning or improved conversion because they are outside of our control is just ridiculous.  As one of the commenters noted, even when a test fails, the team has increased its knowledge base because they now know more about what does not work.  Furthermore, this improved understanding of your customers, coupled with business strategy and common sense, should allow the tester to conceive of designs that win at better than random odds.  This is the whole point of hiring Conversion Specialists over trained monkeys.  Ultimately, the author concedes on these points, so I have to give him credit there.

 

Posted in Articles - Tagged