Reinvigorating a large Android code-base

The problem

We faced a classic problem on a recent project. Stop me if you've heard this one before. We are supporting a well established product with millions of users, running on a large underlying code base, but the code base has been building technical debt for years, slowly turning into a Frankenstein, causing support problems. The types of problems we were facing include:

• Separation of concerns violations making unit test maintenance and creation more challenging, with lower than ideal coverage
• Threading complexity in places leading to defects that are difficult to find and repair
• Numerous special-case customizations distributed throughout the code, each addressing specific customer needs that only activate for that customer
• Reduction in team velocity due to code complexity and the other issues listed above

Challenges of Addressing Technical Debt

In one of my first posts I touched on the challenges of balancing time spent paying back technical debt vs time spent adding new features. It is always difficult to find opportunities to address these types of concerns in a code base because the investment of a large refactoring effort is very high and the direct customer value can be perceived as low (especially outside of engineering) and difficult to quantify when compared to ongoing targeted feature enhancements and defect fixes.

I find the best approach is usually to address refactoring in an iterative approach rather than tackling an entire code base all at once. In this fashion you can balance new feature development with the need to keep your foundation stable.

Solution Overview

On this recent project, however, we were given a task to refresh the entire UI. This was an unusual opportunity to re-evaluate the code base holistically and introduce new concepts and techniques.

We targeted three primary technologies to aid in this effort:

• Dependency injection using Dagger2
• Decomposition of Android activities into Model/View/Controllers
• Usage of RxJava and retrolambda for component communication and thread management

Over the next several posts I will delve into the approach we took, how each contributed to solving the problems identified above, and some of the challenges we faced. 

Integration Test - Case Study

In the previous post I discussed an integration-test framework that I developed for testing system  interactions with a java  master controller. Recall that this system consists of separate View, Master Controller, and Model/ModelController components.

A basic flow through the system would start with an action in the View, pass through the Master Controller to the Model/Model Controller, pass back into the Master Controller, and then terminate with responses back to the view and the model controller.

For this case I configured Dagger2 to create a mock of the manager that interacts with the view (see mocked view in the diagram). I used the mock to verify that the correct method and method parameters are called at the end of the flow. I also wanted to verify that the correct methods and method parameters are called on the Model Controller, but I could not mock the Model Controller because it is integral to the flow we are testing. Instead, the test spied on the model controller to verify the expected method and method parameter calls.

Test flow (from diagram):

1. Initiate an action into the controller to mimic a user interaction from the GUI (View)
2. The Master Controller interprets the input action as an action needing interpretation and dispatch to the Model Controller and performs these actions. I used Mockito to verify that the correct Model Controller method was called and that the correct parameters were supplied.
3. The Model Controller interacts with the Model, processes the action and sends its own command back to the controller. I verify that the correct Master Controller method is called with the correct parameters. If the correct signature is seen, it means that the following components are working properly for the flow under test:
1. The JNI layer for passing between the Java Master Controller and the C Model Controller for these interactions
2. The Model Controller logic for this action 
4. The Master Controller then interprets the action from the Model Controller, dispatches an action to the View to inform it what to display, and replies to the Model Controller with a confirmation able to successfully interpret and handle the command. Both these terminal responses are also verified.

One added complexity with testing this integration is that there are several locations where the flow crosses thread boundaries. I needed a mechanism for delaying our verifies until a callback is received. The integration test will take much longer to run than a typical unit test (on the order of 15 seconds), but this does not mean that we can get sloppy and use sleep statements. To get reliable results, sleep statements would add more delay than necessary, and could still result in tests which are not repeatable. It’s almost never a good idea to sleep in an automated test...or in any code.

Mockito provides the doAnswer() method which is very helpful for mocking asynchronous responses from an object, but this is not helpful to us because we are testing objects which we can’t mock. What we need is a way of synchronizing with the asynchronous callback, because the code under test relies on that callback completing before it can proceed. Java provides such a mechanism with the CountdownLatch. The CountdownLatch will block until its counter reaches 0. We can setup the CountdownLatch in the test thread and give it a value of 1. In the callback thread, we reduce the count to 0 to unblock the test thread when we perform the callback, allowing it to proceed with its verification. We attach a timeout to the latch in case the callback is never hit (15 seconds in this case). This is much better than a sleep statement because the timeout condition is only hit on a failure condition. For this reason we can choose a large number that we know will pass under all conditions, without actually delaying the test execution time except for the exceptional condition of an actual failing test. 



This solution is still not ideal. We had to instrument the test with a handler that overrides the implementation we are testing to call the implementation’s body and then countdown the latch. I could have used DI to inject this test handler, but in this case the framework already had a mechanism for registering handlers directly, so we were able to register the test handler this way.

Developing an Integration Test Framework - Case Study

In the last two posts I described a test strategy and unit tests for a controller written mostly in java. The Master Controller interfaces with an http server hosting the View and a Model/Model Controller written in C. This Model/Model Controller was developed by a different software group and is like a black box for our purposes. I wanted the capability to test both our C/JNI layer and also test drops of the Model/Model Controller from the other software team. Integration tests provided me with the capability to accomplish both these goals, while also testing interactions between the modules unit-tested in isolation. I could test these interactions from the Master Controller with java test frameworks by observing expected system responses to stimuli, removing the need for an entirely separate C-based test framework. 

The Master Controller code splits its core functionality between "managers" with specific responsibilities, including handling boundary crossings to the View and to the Model/Model Controller. Each manager is housed in a container class. This is an excellent spot to introduce dependency injection for test components, as shown in the ManagerControllerSample.java below: 

I created a test framework that allowed for each integration test to determine for each manager whether it would be implemented as a mock, using the standard implementation, or using a spied version of the standard implementation. With Dagger2, I was able to inject the appropriate component (test or production) at run-time. In Graph.java we see the dagger graph. IntegrationTestBase.java  specifies the list of managers to mock and spy as seen in the call to initGraph(), and ManagerTestDataModule injects the appropriate manager type (standard, mock, or spy):


In the next post I will discuss an integration test that used this framework and a technique for synchronizing callbacks spanning thread boundaries.

Creating an Automated Test Strategy - Use Case

Background

On a recent project I was working on a team developing a software component as part of a larger system. This component can be thought of as a controller in a distributed MVC system, if you think of the model as having its own embedded controller. Our controller (the Master Controller) was written in Java. We had a JNI layer for interfacing with the C-based Model/Model Controller assembly and communicated with a web server hosting the view. I wanted to develop an automated test strategy that would:

• Validate the Master Controller
• Guard against regressions in in the Master Controller
• Validate system interactions
• Allow us to develop functionality independent from schedules/delivery of new functionality in the View and Model/Model Controller components

While most of this could be accomplished with unit tests, I decided we also needed some level of integration test for validating the system interactions. In the following description, bear in mind that the Master Controller is the device under test (DUT) and it interfaces with the View and Model/Model Controller but does not test them directly.

Strategy

It is always a good idea to discuss and document your test strategy. It was even more important for this project, as the other developers on the team were not as familiar with standard automated test tenets, coming from environments where automated test was not a priority. I went about this by performing the following steps:

• Develop and document the proposed strategy on internal project wiki
• Gain team buyin
• Implement integration test framework
• Create tests exemplifying usage for a variety of different types of modules
• Provide training

Our overall strategy was to rely on unit tests for the majority of functionality test, but provide a powerful integration test environment and a small number of tests validating interactions between these components.

Unit Test

There are some widely adopted principles for creating unit tests. For the wiki and the training, I put together a few simple patterns/anti-patterns. If you are at all familiar with unit test, there will be no surprises here:

Do

• Test each module in isolation
• Test edge conditions
• bad or null inputs
• etc
• Keep tests fast
• Ideally milliseconds
• Keep cyclomatic complexity low

Don't

• Allow timing dependencies other than timeouts
• No sleeps/timers
• Span threads in one test
• Depend on other tests or order of execution
• Leave artifacts
• Use @After methods (jUnit) to ensure that artifact cleanup will happen independent of test failure

I chose jUnit, Mockito, and PowerMock as the tools for our unit test. This decision was based on familiarity with the tools, their popularity, and suitability for usage within our system. Mockito performs most of the mocking functionality we needed. Mockito allowed us to:

• Handle external dependencies easily
• Good unit tests only test the module under test, none of its dependencies
• Mock responses from these dependencies
• Verify the method calls on these dependencies, including
• Parameters passed
• With stock matching algorithms or custom validators
• Number of invocations
• Verify that method that shouldn't be called are not called
• Mimic asynchronous callbacks from mocked objects

PowerMock provided us with the key additional capability to mock static method calls.

Integration Test

Although unit testing covers the bulk of the test for the product, it is also useful to validate interactions between the components. In the next post I will describe the integration test strategy I implemented for this project in detail.

System Test

The unit and integration tests created and supported by the development team were only one piece of the overall validation strategy. The QA team tested the overall product manually and with Selenium for creating automated tests driven from the html5/javascript View component.

CI

For tests to be effective, of course, they need to actually be run frequently. Ideally developers would all run the unit test suite before checking in, but this is not enforceable. We had a Jenkins CI environment, where we setup automated tests running on a fixed interval whenever code changes were checked in. Failures were reported and logged and emailed to the team for resolution. We also tied in a code coverage tool for reporting progress against our goals.

TDD

For those who are not familiar with test-driven development, the basic idea is that you create your tests before you implement your code using a recipe like:

1. Create your interfaces
2. Create your tests to these interfaces
3. Implement your code
4. Run your tests
5. Rinse/repeat as necessary until all tests are green (pass)

Some of the benefits of TDD are:

1. Enforces up-front accurate requirements and up-front interface design
2. Improves code readability, interface design, architecture, quality (clearly much less likely to make untestable code :))
3. Ensures that tests don't fall behind implementation

As part of this project I adopted a TDD approach, although found it difficult to adopt whole-heartedly. I found "concurrent" test/development a better fit rather than strict adherance to the recipe. It definitely took longer to take a TDD approach than it would have to simply perform the code, but no longer than it would have to develop the code and then the tests later. I advocated for similar approaches by other team members as part of the team training, but we did not enforce it.

Up Next

As already mentioned, my next post will delve into the integration test strategy and methodology that I adopted for the team on this project.

Advocating for Robust Automated Test

The conundrum

In virtually all software engineering organizations I have worked, management is aware of the benefits and importance of a comprehensive automated software test suite and the pitfalls of scrimping on test. Yet, it's actual application varies widely across projects and organizations. Often I find that automated test is adopted to some extent, but with insufficient rigor, thought, and attention. Occasionally I have seen teams that don't do automated test at all. In almost every team I have worked, when there is a crunch to get something done, testing rigor is relaxed, dropped outright, or at least postponed until after the crunch is over.

Clearly there are pain points in the process which is leading to this situation.

Pain Points

• External schedule constraints
• • Deadlines, deadlines, deadlines
  • Customer demo tomorrow
  • Emergency of the day
• 
  

  Schedule constraints are often be the biggest reason for reduced rigor in testing methodology. It is especially difficult when they are immutable deadlines coming from external sources such as: 

  
  

  • Software delivery to a hardware devices or systems whose release schedules are wholly dictated by hardware availability
  • • For example, when working on pre-installed app for a phone, the apps usually either make the delivery date or are pulled
  • Trade shows
• Challenge of quantifying the cost-benefit trade-off
• How do we know that we aren't spending more than we are getting in return?
• Test maintenance burden
• Difficulty of retrofitting tests into a legacy code base 
• Up-front cost
• • Test Framework identification
  • Developer training
  • Developer mindset shift
  • CI integration
  • Difficulty in getting "valuable" test metrics

Addressing the Pain

• External schedule constraints

One strategy to deal with a situation where there is insufficient remaining time to both deliver the software and a full set of developer automated tests is to:

1. Identify all the required tests and add them to the backlog
2. Deliver as many of the most critical tests as possible
3. Supplement the reduced set of automated tests with a more extensive QA test (both manual and automated) for the immediate deliverable
4. Schedule the remaining unfinished tests for delivery in the following sprint, as soon as the “fire drill” is over

In some especially reactive environments it can be challenging to break out of this “fire drill” mode, as they cascade upon each other. This could be a sign of issues that need to be addressed at the management level,including:

1. Unwillingness to turn away new business
2. Unrealistic expectations
3. Lack of understanding of the impact of these decisions
4. Insufficient development resources

It is a responsibility of the senior members of a developer team to point this out to management and help work out a plan to remediate the issue. Depending on team charter and business conditions this could be a very difficult problem to solve and isn’t necessarily a management failure. Developers should take an active role in sharing the responsibility associated with improving the process and steering the process back to sanity.

Even in the most aggressive environments, there will always be “some” down-time, which could be dedicated to catching up on test. This is a good time to advocate for sprints focusing on automated test get-well.

• Quantifying the cost-benefit trade-off

     Perhaps the best way to get upper-management support for test is to demonstrate that the cost in delivery schedule and engineering resource consumption is outweighed by the benefit. This can be challenging to quantify, to say the least. Metrics such as:

1. Severity and frequency of bug reports
2. Savings in support development time
3. Savings in refactoring time
4. Increased revenue due to release of a higher-quality product
5. Less need for refactoring 

are difficult enough to measure when you have the data. But, of course you can’t have internal data until you have a well-tested code-base to compare against, so it’s a bit of a chicken and the egg. One approach would be to present these benefits qualitatively, not quantitatively and reference cost-benefit tradeoffs published by other companies who have made this investment.

There is a severe penalty for “catching a code-base up” so the benefit is greater when applied at the beginning of a project.

• Test maintenance burden

     There is clearly overhead associated with maintaining tests. The amount of this overhead can vary dramatically on factors such as:

1. Proper test scoping
2. Test repeatability
3. Test complexity and supportability

When the tests are properly scoped, repeatable, and straightforward maintenance cost is not so great. Systemic violation of one or more of these factors can easily push the maintenance cost so high that the cost exceeds the benefit. Sometimes a developer or manager will have had past experience working in environments that have embraced automated test, but improperly applied some of these constraints. This can lead to a belief that automated test “is not worth it”. It can be very difficult to challenge a belief system when it is based on experience! It could help, when encountering someone who doesn’t believe in automated developer test to push into the experiences they had, understand where it may have failed, and explain how it might have worked better if applied differently.

• Difficulty of retrofitting into a legacy code base 

The value of adding good test coverage to a legacy code base is less and the cost greater than if it were applied from the beginning of the project. However, in such environments it may still be valuable when applied judiciously and iteratively in small chunks. For example, before refactoring a bit of buggy, complex, and/or obdurate code, it is helpful to provide strong test coverage for the methods in question and use this to validate the refactored code. Similarly, any time that new functionality is added, good test coverage for that new functionality can be easily justified. Finally, there is always “some” down time in a project, which can be used to bolster tests. I like to target areas of the code that are:

1. Most problematic (highest bug reports)
2. Functionally critical
3. Core (used by many components)
4. Most complex
5. Most likely to change

• Up-front cost

While there are initial costs associated with identifying and implementing a test framework, integrating with CI, and training developers, these costs are manageable and largely scale across multiple projects.

Next up

In upcoming posts I will discuss an automated test strategy that I advocated and adopted for a recent project.

Use Case: Being a Code Custodian, Manual Inspections

This is the third and final post of this series. Please see the last two posts for my thoughts on what it means to be a code custodian and the application of automated inspection tools in this role.

While automated tests have a place in detecting potential bugs or maintenance issues for a code-base, I find that only manual inspection can detect issues in proper usage of the classes, methods, and design patterns in use for a particular code base. At a recent positions, some of the stylistic things that I looked for in a manual inspection relate to reducing accidental complexity, and include:

• Crisp and clear interfaces, especially customer-facing API's
• Well-documented API calls
• In this case, we used Javadoc
• Many of the developers were non-native English speakers, making this more critical for our team
•  Well-named methods
• Proper usage of existing design patterns within the code base
• Avoidance of internal packages
• In our case, we had libraries that had external and internal packages. Sometimes developers would accidentally reach down into an internal package of another library
• A better approach would have been to find an automated approach for verifying compliance
• Proper application of standard object-oriented practices
• Stylistic consistency
• At a gross-level, without being intrusive

Of course, I also checked for more substantive issues like potential threading issues and other defects, but often the stylistic issues are more noticeable.

To avoid getting behind, I would review checkins on a daily basis on average. With about 10 active developers on the team, it wasn't practical to review every line of code that was checked in. To limit the time commitment I applied a few filters to guide my engagement level:

• Individual developer expertise
• Does this a developer have a track record of committing high-quality software?
• Does this developer have experience in the area of the code that is being changed?
• Type of code under change
• Is this a core or performance-sensitive area of the code, requiring more attention?
• Are the modules under change complex and/or already in need of refactoring
• Customer-facing
• Is this a customer-facing API or an internal method that will be obfuscated away?

We had a policy whereby developers were encouraged to ask for reviews of changes to critical and/or complex sections of code. The best developers in the group used this policy judiciously, and I would seldom review checkins by them unless they requested a review.  On the flip-side, if a developer had a track record of sloppy checkins and/or was working in an area of the code with which s/he had little familiarity I was likely to take more notice.

Having a role of technical lead, does not make anyone the expert on everything. I interpret the code custodian role to mean that I should coordinate with other experts on the team to help identify areas of risk and coordinate collaboration, not try to be the sole authority on everything. It's always helpful to get another set of eyes on your code, and I would frequently solicit code reviews for trickier areas of my own code submissions from one or two other senior members of the team. Additionally, I would sometimes ask for help from these same members in performing code reviews of submissions from others on the team that I find to be in a critical portion of the code that I have less familiarity with.

Of course it is very helpful to be intimately familiar with the software before taking on these types of reviews. In this case I had designed and coded much of the initial code base, which made the job easier. Were I to drop into a new code base, I could still look for general code and styling issues, but would be less apt to give any valuable feedback relevant to fitting into existing design patterns and proper usage of libraries, etc.

In an effort to preserve developer freedom, if the issues I detected were all small or stylistic, often I wouldn't say anything, but rather lock it away in case the advice might come in handy in the future as part of more substantive feedback. If the issue is in naming of methods and/or javadoc and the user is not a native-English speaker I might make the change directly in the code base and check it in since the changes are non-controversial and straightforward. For most other types of feedback I would usually make suggestions in an email. Developer woud either follow the suggestions or provide good reasons for not following them. Lastly, I find it can be helpful to reference well-known books by experts in the field when making a point to give it more credence and make it less of a single person's opinion.

In this series of posts I discussed the role of a code custodian in maintaining a clean base. I stress that this is not always an appropriate role and there are many different approaches and techniques. I don't necessarily advocate for the approaches laid out here, but list them by way of an example that worked well in one environment. I found the manual and automated inspections to be helpful but perhaps they would be too intrusive or unnecessary in another environment. In a third environment, perhaps more stringent techniques would be appropriate.

Use Case: Being a Code Custodian, Automated Inspections

As stated in my last post, the roles of an architect and/or team lead can vary dramatically depending on team dynamics. Being a code custodian may or may not be appropriate for your organization. On the one hand, it's hard to argue against the importance of having a cohesive code base with similar coding patterns and standards for quality. On the other hand, a heavy-handed environment can be very disabling for developers and reduce team members' feelings of empowerment. So how best to balance these competing factors?

A recent long-running project I worked on had the following team dynamics:

• Varied developer skill levels
• Fresh from university to senior
• Combination of remote and local developers
• Most remote
• English skills that also varied in quality between team members
• Mix of time working in the code base

Although we had some written coding guidelines, I find this to be a largely ineffective way of managing a code-base. We did have some standards, but the best standards are already available and published in books such as "Effective Java" by Joshua Bloch and "Clean Code" by Robert Martin.

Instead, I chose to rely on a mix of automated and manual inspection approaches. See my last post for a comparison between these approaches.

Perhaps the most important tenet to developing any of these strategies is to get active participation from other members of the team in their usage and application. Some of the inspections by these tools can feel intrusive and even arbitrary at times. Getting the team or a subset of senior developers on the team to buy-in to the list of checks is important to foster a feeling of ownership to the policy and to avoid tying developers down and hampering creativity. it might take a couple of iterations to refine the rules.

For this situation, we used CheckStyle, FindBugs, and lint in out suite of tools that were run on Jenkins. Checking stylistic rules can be particularly treacherous if applied too heavy-handedly. When looking at CheckStyle, the first thing I did was to immediately drop silly rules such as where semicolons or spaces are placed. There is nothing worse than breaking the build because you missed a spacing constraint. I think that it's important to maintain consistency in the code-base, especially within an individual module. But this consistency does not need to be enforced via an automated tool.

However, that doesn't make the likes of CheckStyle worthless. There are a number of checks within CheckStyle that, when enforced, aid in keeping the complexity down in the code base. We used a number of CheckStyle rules, but the following were somewhat helpful in keeping code complexity manageable:

• FileLength
• MethodLength
• ParameterNumber
• AnonInnerLength
• AvoidNestedBlocks
• MagicNumber
• Nested* (various Nested rules)

Some of the other methods, like HiddenField, were also useful in avoiding bugs.

FindBugs and lint (and occasionally CheckStyle) were good at identifying potential defects and leaks. At first the tools detected a number of issues, but over time there were fewer and fewer violations.

Having good reporting mechanisms was very important. At first our reporting mechanisms were very poor, requiring a lot of digging to find the source of the build error. This was largely due to the limitations of the central build CI system we were using. When we shifted the tests to run on Jenkins, the reporting was much crisper, making for a much better experience.

Design and Architecture in an Agile Environment

I have worked in several "scrum-like" agile environments over the years. I say scrum-like because each adopted some of the principles, but was unable to embrace all. In almost every case the biggest casualty tended to be an inability to totally isolate the development team from shifting priorities in the middle of a scrum. This is a difficult problem to address, particularly when working with exceptionally demanding customers, and perhaps a topic for another day. Another possible discussion item is whether or not "partial-scrum" is scrum at all.

One of the industry trends with agile environments is to tradeoff up-front design and architecture for an "emergent design" iterative environment where coding starts immediately. However, I would argue that agile does not, and should not, imply that there is no up-front design. Iterative development does not mean that you should rewrite the entire system in every iteration. Nor could anyone reasonably argue that this is an efficient use of the team's time. Yet, if you jump into implementation without any design the chances of "coding into a corner" greatly increases, potentially producing exactly this type of situation.

So we shouldn't give up design and up-front thought any more than we should give up good coding standards. Fast development does not have to equal reckless development. A technique I have used in some organizations is to have one or two senior developers/architects sign up for design and architectural tasks as deliverables for the sprint. These deliverables can be tracked much like any other deliverable, and are as integral to the sprint's success as code and QA deliverables. The end-result should be a light design in some electronic format, with buyin from the team and stakeholders.

The only real disconnect with the scrum process of adopting this technique is that typically the one-or-two people working on this task have a different focus from the rest of the team. This work is synergistic, however, as it results in a better definition of development backlog tasks for subsequent sprints. From this perspective, the team performing these tasks could be their own scrum, but often this is not practical because of how small the team is (could be one member) and because the members will often also take other tasks from the ongoing development sprint.

Another approach that I have seen work is to break up chains of sprints with planned one-or-two week intervals for the team to do design and architecture of  backlog items and cleanup work from prior sprints as needed. Whether this work is considered a sprint or not is up to the team, but I have found that occasional "unstructured" time is also useful for a team.

"Does design and architecture play a role in scrum? If so, what techniques have you seen work and which have you seen not work?"

Balancing Performance with Code Readability : Case Study (Android, Java)

Google has a number of blog posts on how to write code for Android performance. They make recommendations that conflict with normal Java standards such as avoiding enumerated types and using counter-type  loops over their collection types. They make good arguments for comparative performance improvements for each, but I think it's good to step back and evaluate Google's motivations before blindly applying all their recommendations.

• Motivation 1: Bad apps make Android look bad
  
  From Google's perspective, a few misbehaving apps can make the entire ecosystem look bad when compared to the highly controlled ecosystem of Apple. This is especially true for apps that are bad citizens (e.g.- consume more resources than needed for longer than needed, services that run longer than needed, activation of power-hungry resources like GPS hardware for longer than needed). Other recommendations are performance-oriented. There is little motivation for them to recommend well-written code over code that grabs every nanosecond of performance. They aren't the ones supporting the apps, after all.

• Motivation 2: Wide range of hardware capabilities
  
  Often times developers will test only on a high-speed device over WiFi or 4G and not understand the implications of how their app will run on older or less expensive hardware or over slow Edge networks in China or elsewhere. With the advent of smart devices like watches, the limitations can be even more severe, especially for  battery life.

While I strongly endorse writing programs that are good citizens, other recommendations to grab every nanosecond of performance I often find as not always necessary. I also have the luxury of architecting, designing, and developing apps for specific high-end Android devices, which makes motivation 2 less relevant. Even if this were not the case, I still prefer to write first for readability and, only when proven necessary, refactor for performance.

An example of where I recently found this  useful was in creating code whose function is coordinating an external set of contacts with the Android native contact database. There were a number of fields that were to be synchronized and the list of exactly which would be synchronized was subject to change. I found code in another application that someone wrote to do something similar as a reference. The author of this code created a string projection  where each ordinal was referenced by number into that projection. Insertions or deletions of values in this projection would change the meaning of each ordinal, making the code very brittle.

I introduced an enumerated type for storing the elements of the Android database string projection and relied on the ordinal position for later retrieving columns from this projection. The structure was thus "self-aligned" making for easy removal and addition of elements to the projection. The enum looked something like this:

Access to the columns, could then use the ordinal of the enumerated values like so:

instead of using "magic numbers" which rely on the order of the elements in the projection (in this case it would have been "1").

The new self-aligned approach is both much more readable and less susceptible to bugs as the code changes and there was no noticeable performance hit.

Balancing Performance with Code Readability

This is a topic much discussed in the industry and the position I take is a common one: Code first for readability and support, refactor for performance as necessary. There are notable exceptions to the wisdom of taking a readability-first approach. In general you will know if your code is one of these exceptions. It could be graphics-intensive operations, operations widely used and distributed in libraries, and similar situations where you know critical paths through the code a priori. For the 90% case, where squeaking out every nanosecond of performance is not critical I like to use the following steps:

1. Write code to be as readable and supportable as possible
2. Validate that the performance is acceptable
3. If performance is not acceptable, profile to find the hot spots
4. Identify techniques for improving performance of these hot spots
5. Using the Strategy Pattern
1. Extract each module to  be refactored into a separate component and introduce a simple factory to serve up the existing implementation
2. Introduce a higher performing implementation and update the factory to use this new strategy
3. Add comments as necessary to the new strategy implementation

The factory gives us much more flexibility. It allows us to restore the original strategy if we later find that we really don't gain enough overall performance to warrant the new version. It also allows us to experiment with different strategies for improving the performance before deciding on one.

I am of the school of thought that a lot of comments in code is a design smell that the code wasn't as well thought out and decomposed as it should be. I think that most code should be written to be self-documenting with appropriately named methods with small scope. There are certain exceptions to this. Writing for performance is a common way of adding unintentional complexity. Often this added complexity is unavoidable. We should add sufficient comments to explain the code and the reasoning behind the extra complexity so that someone further down the road does not reverse our decisions (perhaps for readability) without understanding the implications of that reversal.

Next week I will push into a use-case where I opted for readability over performance in conflict with published performance recommendations from the platform vendor.

Effective Code Reviews: Case Study

As mentioned in previous posts, mentoring is an important and rewarding aspect to software leadership. One of the more effective ways to teach is via thought-out code review feedback. Performing code reviews is also an excellent way for the reviewer to learn and pick up new tricks. I often find it helpful to be on each end of the code review.

I solicit code reviews of my work when coding something particularly mission-critical and/or complex and encourage the same from other members of the team. There are many ways to run code reviews and perhaps I will discuss this in a future topic. Code reviews can involve a team or be one-on-one. The case study discussed here was a one-on-one case where I was the reviewer. Usually this has worked well for me. This situation had some unique challenges, however. Specifics of this situation:

• Working with a senior, talented, open-minded developer
• Developer is a non-native English speaker
• Developer is very open to feedback
• Developer's style resulted in overly complex code
• Developer frequently requested reviews
• Most of the request reviews span numerous modules

Problems this posed for me:

• Often times the code was just too complex for me to perform an effective review and root out potential side effects
• The language barrier reflected itself in the code in the naming of methods and variables
• The time demands of the frequent requests were too high

To address the time constraints, I applied a common technique, which was to ask to limit the scope to areas of highest concern. This did not work, as the developer was not able to identify specific areas. The inability to identify "chunks" that could be easily identified to treat in isolation was another symptom of the complexity.

This situation continued over a period of weeks. The bulk of my feedback centered on addressing the complexity issues and was in the form of email. However, each time I did this I felt I did not grasp the nuances of the changes well enough to provide adequate feedback. I found that the developer would make the specific requested changes and then the next day make the same complexity mistakes in the next piece of code that he submitted. To make matters worse, we were in a critical part of the project where time pressures were significant.

The real fix is to fully refactor the functionality. This is something we scheduled, however, it was not realistic in the short-term, so it was important to stabilize the code as much as possible. Equally important was my desire to help this talented developer with his complexity issues so that the problems would not be repeated in future projects.

Finally I decided that, despite the time-zone challenges and the additional time demands, the most effective way to perform reviews in this situation was to do them interactively with the developer. While voice is normally my first choice, in this situation I found that interactive IM sessions were more effective. There are two reasons for this:

1. It gave both parties an opportunity to think carefully before each reply
• From the developer's side this extra time gave him the opportunity to translate as necessary. 
• From my side it gave me time to analyze and provide recommendations as we went through the code.
2. It provided a written transcription for reference during and after the discussion. 

My biggest hurdles were understanding the motivations and side effects of each of the changes. So the review then became something like this:

• Me: Method X, lines Y-Z: What are you doing here and why?
• Dev: Explanation
• Me: Did you consider factors A and B?
• Dev: Actually, I think there may be a bug C 

( One of the things I like best about code reviews, whether I'm reviewing or being reviewed is that in discussing the code often the coder himself will stumble on issues)

• Me: What if you took this 4-line return statement and simplified it to a method call

etc.

The end result was that I felt I was finally able to provide effective feedback and identify specifics of how to reduce the complexity of the code.

The lesson learned here for me is to not wait so long to consider alternate communication mechanisms if I find one mechanism is not working well. This is a lesson I try to apply to many other aspects of communications.

Key traits of a software leader - Dedication, Love for Mentoring, Balance and Perspective

• Dedication

Leaders and senior developers often know what to do, without the need for external direction. Of course you must do what your manager asks of  you, but once you start on it, you should usually be able to complete the task without requiring management oversight. Are the requirements clear and complete? If not, find out who to talk to get them clear and complete. If you can't get clear requirements from the prodouct owner for this task, come up with your own clear and complete requirements (written or in the form of tests) and then give these to the product manager or manager as a starting point. Hold a meeting with the stakeholders and leave with the requirements clarified. Take a similar approach for every other aspect of the delivery. Push for answers you need, make proposals along the way, always get buyin from others of the team, and don't be afraid to hold meetings when email, issue tracking systems, or other communication mechanisms fail you. Ensure that requirements end up in a written form that is signed off on by all stakeholders.
Whether it's the near-term objectives of a sprint or longer-term objectives, a leader is strongly invested both in his/her own commitments and in the teams' commitments. Are others on the team in need of your help? How can you ensure the success of the entire team? Like everyone on the team, leaders will put in extra hours during legitimate "crunch" times to meet important milestones. This does not mean that you should be expected to work 80 hour weeks ad infinitum. Learn when it is truly important to put in the extra hours and when it is unreasonable. If you are like many developers, you enjoy your work enough to want to put in longer hours. Instead of putting in an 80 hour week for your current sprint, consider spending time learning on your own and/or playing around with new technologies if not in a crunch period. This is also important to the team, as you can bring what you learn to the table to help in design of future corporate initiatives.

• Love for Mentoring

One of the most rewarding aspects of working on a development team can be the opportunity to help with the growth and learning of others on your team and teams you work with. There are many aspects to mentoring. Consistent application of the traits described in this blog series on leadership alone is a good way. From an early age on, people emulate the behavior of people they respect. Find other ways to personally connect with individuals on the team. Choose one or two on the team who are most engaged with you and work closely together. Often this will be more junior people on the team, often you will select people who show the most promise, but neither of these is a requirement. Remember that anyone with more experience in any one area can act as a mentor for anyone else in that area. You could easily end up helping someone older or with more overall experience, but less in one or more specific areas. Lastly, there are always opportunities to find others to help mentor you. Nobody is ever done learning and there are always people to learn from, whether on your team or externally. Try to spend as much time finding people whom you respect in an area you would like to grow in as you do in trying to find others who can learn from you.

• Balance and Perspective - Understanding of when and how much technical debt to accept

In constant conflict with your goal of delivering personal excellence will be the organization's need to deliver software in a timely fashion. A true leader will understand enough of the market dynamics to know when it is appropriate to "rush a solution" and when the constraints are artificial.
Organizations tend to constantly apply time pressure. You should understand when there is a true market force for a rush and when it is self-imposed by the organization. A clear-cut example of a true market need is delivery of software to a hardware product timeline owned by another organization or company. If there is a true market need, often you will be forced to create some "technical debt" and deliver software that is not as well written as you would like. In these situations, a leader will still strive for quality and test coverage in the solution, and sacrifice architectural and/or design integrity. In other situations, the time pressure may be internally-driven. You should act as the code-base custodian. Make sure that management understands the trade-offs of technical debt for this decision and weighs its long-term supportabilty costs. Importantly, a code-base custodian looks for appropriate times when external pressures are not as high to lobby for an investment in paying down technical debt from previous "rush jobs".

Key traits of a software leader - Personal Excellence, Integrity, Innovation, and Technology Evangelism

•  Personal Excellence

Software is continuously evolving and so must software professionals evolve. Find ways to continue to grow. There are numerous ways, including podcasts, books, blogs, meetups, technical conferences, code camps, and innovation challenges. Find a couple that fit your interest and style. Use resources like github and stack overflow to learn and experience other approaches to writing code and solving problems.

Leaders develop well-thought-out, well-architected, and well-tested software. Make everything you write something that you would feel comfortable publishing to open-source. and others on the team will emulate your style, template from your code, and learn from you.

• Integrity

Integrity is an important quality often overlooked in soft-skills discussions. Yet, integrity is arguably one of the most important aspects of leadership in any field. Never do anything unprofessional at work our outside of work with any of your colleagues. Treat all members of the team with respect and consideration...always. If you disagree with someone you should always make your voice heard, but your voice should be the voice of logic and reason, not the voice of emotion and ego. You don't need to be loved by everyone on the team but it helps to be well-liked, and you need to be highly respected.

Apply this integrity to your work ethic. You should provide aggressive schedule estimates that you can achieve with high confidence. You shouldn't have hidden agendas. Make choices and recommendations that are best for the team and organization. If this doesn't align with your goals, choose another team or organization that does.

•  Innovation

Innovation is perhaps one of the most difficult traits to learn. For an existing problem, it is easy to get caught up in what is familiar and to fit problems to existing toolsets and skills that you have always developed. Try to push beyond what is comfortable and known. Allocate some time before jumping into any task to thinking about what might be the best solution for your problem. Do this before providing schedule estimates.

Innovation also extends well beyond solving existing problems. Look around at the expertise of your group and your company. Periodically step back and think about other things that your group or company could invest in that would be a good fit for the company. Discuss some of these ideas with your manager or product manager and track them somewhere appropriate at your company, such as a wiki. If you strongly believe in your idea, don't be afraid to champion it or even spend some of your own time to develop a prototype.

•  Technology Evangelism

A strong leader will keep abreast of software technologies and advocate adoption within the team where appropriate. This should not become an exercise in constantly adopting the newest "hot" technology. Look for technologies which will improve the development cycle of your team. Would TDD improve the readability and maintainability of your team's code? Do you  have a complex system with frequent threading issues? Would a functional programming language improve the quality of your system? What about your revision control, bug tracking, and/or life-cycle tracking systems? Could your team benefit from improvements to these areas? Consider cost. Some of the best solutions are free. Remember that everything your team does is to improve the bottom line of the company. Often this is indirect and hard to quantify. Examples are investments in test, design, and architecture, which rarely benefit time-to-market, but often have a solid payoff in maintenance costs and ability to evolve the software over the lifetime of the product. Where possible, frame your recommendations in terms of cost savings.

Key traits of a software leader - Design, Quality, and Communication Tips

• Devotion and Knack for Design and Architecture

I am a strong advocate of refactoring throughout the lifecycle of a product. However, the cost of "fixing" a poor design can be very high and increases over time. It is always a good idea to do some level of design and architecture up-front, before any coding begins. There is a great deal of literature available on the benefits of good design that essentially boil down to software that is of higher quality, is easier to maintain, and has the flexibility to change over time. I believe it 's a mistake to think that Agile development precludes design and architecture. It is still important to include design and architecture in your agile processes. This may be a topic for a future post.

One example of something that must always be carefully considered is public interfaces. If your software project has a public-facing API, changing this API after release has a major impact on your customers. Of course you can version your APIs, but you will still be left supporting the old APIs for a long time, if not indefinitely. Poorly designed APIs can, thus, severely limit the growth and architectural flexibility of your code.

Bear in mind that software often lives in the field much longer than originally anticipated. I like to multiply the expected lifetime of the software by at least two when considering how much attention should  be given to design. Often times the end goal of a project is a prototype. Obviously for prototype work design is less important. Resist the surprisingly strong pull to allow a prototype to morph into a customer-facing product. If you work in an organization where prototypes frequently turn into product due to forces outside your control, you may need to start putting more design into your prototypes.

Lastly, while it is always more costly to do too little up-front design, there is also a cost of devoting too much time on design. This is a balance discussed in a future topic.

• Attention to Quality

This item speaks for itself. Apply modern coding techniques, and always remember the KISS principle. The highest quality approaches over the long-term are almost always the simplest approaches. Be an advocate for quality and simplicity within your team. As Hal Ableson and Gerald Sussman famously wrote, "Programs must be written for people to read, and only incidentally for machines to execute."

If you can't pick up something you wrote 3 months ago and understand what you wrote, chances are that code is also buggy. Two simple rules for code support that I find have the biggest impact are:

1. Name every method and class precisely
2. Decompose your methods as much as possible

These two techniques alone will make your code easier to read and understand.

Ensure quality through integration and unit tests. Measure test coverage and monitor tests frequently, if not with every checkin. Consider adopting a Test-Driven-Development(TDD) approach.

• Relentless and Effective Communicator

I find that face-to-face communications are always the most effective, but are not always practical, particularly with geologically diverse teams. Don't be afraid to hold meetings to discuss important design decisions, but don't waste people's times with frivolous ones. Pay attention to your attendee list. Invite all the stakeholders, but resist the urge to expand beyond that. If the list with stakeholders exceeds 5 or 6, consider if a smaller number of stakeholders would work better, and then later follow-up with the other stakeholders on the proceedings.

Where face-to-face is not possible, use video or phone. Next on my list would be instant messaging. Email tends to be one of the least effective means of communication. A good tactic to deal with an email chain greater than 5 or 10 replies is to setup a meeting to resolve the issues.

Being a leader does not mean directing everyone on the team on every aspect of design and coding. Recognize that there are many strong developers on your team and that everyone's view is valuable. The best leaders have well-respected voices in the discourse, providing strong influence on the group, not dictating to the group. Often others on the team have better or more practical solutions. A good leader will actively recruit other opinions on important matters and be quick to support good ones that are superior or equal to his/her own preconceived notions of solving the problem at hand. Many of us in software design have strong egos. I actually think this is important, because with ego comes confidence and a higher participation level. Be careful not to trample on others' egos and be careful to keep your own in check.

This extends even more importantly to cross-group communications. Always get input and buy-in from members of all affected teams on any decision which impacts or involves them. The best solutions tend to be the ones developed with a small group of people with differing backgrounds. There is no easier way to hit late roadblocks than to make a decision without including a major stakeholder.

Advance from team member to leader on your software development team - 10 Key Traits

How do you become a leader in a software organization?

In my experience, the qualities of a true leader don't vary much between work environments. This holds true from small startups with less than 10 employees to large companies with 100,000+ employees. I have worked in both extremes and had the pleasure of learning from many of these leaders.

I'd like to share my observation of ten of these qualities. While not every leader can be strong in all categories, I have seen all to be strong in a large subset. 

So, my top-ten list is...

• Devotion and Knack for Design and Architecture
• Attention to Quality
• Relentless and Effective Communicator
• Personal Excellence - constantly learning and growing
• Personal Integrity
• Innovative Thinker - bring unique perspectives to problems
• Technology Advocate
• Dedication
• Love for Mentoring
• Balance and Perspective - Understanding of when and how much technical debt to tradeoff vs market pressures

I have found that people who exhibit these attributes are recognized by members of their team and become de-facto "group leaders", increasing influence within and outside the organization. Other members of the team  look to these leaders for input, thoughts, and ideas.

In upcoming posts I will delve more deeply into these characteristics.