shower thoughts on agility

unexpected

we've seen developers writing unit tests, and it has worked out well for them.

we've seen developers writing those unit tests before writing any production code, and it has worked out well for them.

we've seen developers spending time in improving the code, without getting any new feature done, and it has worked out well for them.

we've seen developers doing pair programming - costing twice as much to solve the same problem? - and it has worked out well for them.

we've seen developers doing mob programming, having a small group of people working at the same time, on the same problem and on the same computer, and it has worked out well for them.

we've seen developers spending their work time in learning and practice, and it has worked out well for them.

we've seen developers quit doing estimates, and it has worked out well for them.

we've seen developers managing to deliver useful versions early and often.

we've seen developers managing to deliver value to their customers every single day.

we've seen organizations giving power to their development teams, and it has worked out well for them.

we didn't expect those things to work out this way - yet here we are.

this is very refreshing in my opinion, and i am very interested in exploring why those things worked out the way they did.
it seems like we really suck at predicting what would work and what wouldn’t.
but if we are so bad at those predictions, shouldn’t we just try things out?

the industrial age

first of all, i believe that our thinking has been stuck in the industrial age of taylorism and militarism, and that these ideas have shaped our way of thinking, teaching, working and just everything, up until today. taylorism might have worked well in factories, where every new produced thing was exactly the same, and it was finished at some point. this however seems not to be the case with software. software is never the same, and if it was you’d reuse it. software is never finished, it keeps changing and improving. software development is just fundamentally different to factory work. it is about knowledge, communication, experimentation, discovery and learning.

knowledge refineries

software starts with somebody having a need. this need unfolds in an idea, a vision. in a traditional world this vision leads to the creation of requirements. the requirements are used to create specifications. the specifications guide the writing of the code. each of these steps could be seen as refinements of knowledge. the knowledge starts very vague and becomes gradually more specific until it ends in code. the knowledge is transferred, refined, stored, and it decays (thanks to Alex Bolboacă @alexboly for this observation).

the risk of bugs

in a traditional world, those refinements are done by different people of different skills with distance among them. we know that communication is lossy, and that the devil is in the details. needs are often lost in communication, resulting in instructions without context. instructions without context are likely the source of speculative assumptions. bugs may not only occur in code, but in specification, requirements, and ideas as well. thus each refinement step has a risk of bugs. the amount of people involved, and the distance between those people increase the risk of bugs. the amount of refinement steps in the process, and the length of interval between them, increase the risk of bugs. the damage of bugs in early stages of the development process grows if left undiscovered. the damage of bugs is reduced when the code is in a maintainable state. one way to discover bugs early, and to prevent them from doing lots of damage, is continuous feedback. the faster the feedback, the smaller the risk. a waterfall process hides bugs until the last possible moment, it maximizes the risk of bugs, and the damage those bugs inflict.

code as a byproduct

if software development is so much about discovering bugs, maybe code is just a byproduct. the real product would be the learning, and the discovery of a bug-free vision.

the many minds problem

too many cooks spoil the broth, and it’s the same with code. different programmers have different opinions, styles and knowledge. many disconnected minds working on the same system can only make it unnecessarily complex. code is duplicated, when the existing implementation is not known or disliked. styles of design in areas of the code deviate from one another. programmers of all levels of experience have a hard time creating quality output, while quality output has a lot of attributes. mob programming helps solving these issues in an attempt to create a single mind of many.

storing knowledge

storing knowledge in brains is risky as brains are bad at remembering details. knowledge stored in just a single brain is even more risky, as it gets lost in fluctuation (bus factor). documentation quickly becomes dated. facts change, but no one updates the documentation. the only trustworthy long term store for knowledge is the code, as it never gets old. same with executable specification in the form of tests that change with the code.

transferring knowledge: meetings

when programmers get in the zone, their short-term memories become stuffed with details about a specific problem and the code. meetings often cause damage by tearing programmers out of this context, which took a long time to setup in the first place. to make things worse, the details discussed in those meetings are typically unrelated to the programmers recent context. so, not only will the programmer spend a lot of time getting back into the zone after the meeting, also parts of the knowledge gained within the meeting will get lost in that process anyhow.

transferring knowledge: email

a good way to avoid lossy communication is to have the recipient repeat in their own words what they have understood. the sender then has immediate feedback if there was a misunderstanding or not. email does not provide this way of feedback, it is typically ‘send and forget’, sometimes even disregarded.

knowledge refactoring

refactoring is a vital practice to make code more comprehensible, and to discover bugs. but what about requirements and specification? if they are similar to code, just not as refined - shouldn’t we refactor them, too?

estimation

the things we estimate are typically not the things that cause most of our effort. the things we estimate are naturally known things. but it is about the things we don’t know, that make the effort.

derived principles?

  • less people involved -> smaller teams.
  • less distance between people.
  • write code together.
  • continuous collaboration, share knowledge.
  • continuous and fast feedback using smaller batches.
  • idea to code as soon as possible.
  • avoid meetings and emails.
  • avoid estimation.
  • have the courage to question input.


to the comments

Who Needs Lombok Anyhow

Java is quite a verbose language and annotations can solve almost any problem. Right? - What a dangerous combination that is.

I understand it, totally. I’m lazy too. But other than saving all that boilerplate, have you considered the repercussions?

Magic is harmful

I don’t want any magic in my code. It is the thing that makes behaviour non-transparent. It makes you fire up your debugger, because you don’t actually understand what is going on behind the scenes. There should be no behind the scenes. Everything should be as explicit as possible, for economic reasons.

Magic is a feature with non-compositional semantics that succeeds in making the common case easy, at the cost of making the uncommon cases surprising, impossible, or ridiculously complex.

~ John A De Goes

@Data, @Getter and @Setter are a design smell

When i design the objects of my domain i want to avoid accessors. I want those objects to represent a set of behaviours on nicely encapsulated data. I’ll occasionally have some public fields on my Value Objects, but that’s about it. Why would i want getters or setters on these objects? They are a backdoor to the data that ought to be modified through specific methods. This smells like Feature Envy and Anemic Domain Model.

Also, setters work against my intention of making most of my objects immutable. After all, immutable objects are easier to refactor.

So where do i need accessors then? Maybe on my data transfer objects? Why would i want to go the length of making fields private and creating accessors on a structure that is solely data? Does this kind of information hiding provide any value? To me, it is just clutter. public final fields without accessors are a much cleaner solution here imho. And there i am, left without a need for accessors.

@AllArgsConstructor introduces connascence of position

This awful annotation couples the order of your fields to the order of its generated constructor arguments. So when you reorder your fields, you effectively break your code. That’s an awful connascence of position right there.

Bi-directional object relations cause stack overflows

So when you use Lomboks equals, hashCode or toString implementations, it is interresting that bi-directional associations will cause a stack overflow. I’m not saying bi-directional associations are desirable, or that those stack overflows aren’t obvious. All I am saying is, that if Lombok was just a little bit smarter, it could probably have avoided those.

The bad thing is not the stack overflow. It’s the fact that it is hidden behind an annotation.

The Lombok plugin might hit your cpu

If you do use lombok annotations excessively and have a lot of pojos in your code base, the lombok plugin might hit your cpu. I had one project where it caused the autocomplete drop down to load several seconds in specific areas of the code. The problem was gone once i deactivated the plugin. To be fair: Maybe it was just a temporary problem with the plugin that has already been addressed.

Compatibility issues with Java versions greater than 8

When i tried migrating a java 1.8 project that used lombok to java 11, i ran into issues. And despite the known issues and fixes that i found on the internet, i was not able to get it to work again within 2 hours of trying. Delombok to the rescue.

update: seems like i ran into a similar issue like this, trying to upgrade to java 11.

Without lombok, what are the alternatives?

We can just fall back to explicit pojo code. Some libs, ide features and plugins will help us out with the boilerplate.

equals and hashcode

For value objects, where you want all fields included, the apache commons lang EqualsBuilder and HashCodeBuilder come in handy.

@Override
public boolean equals(Object that) {
    return EqualsBuilder.reflectionEquals(this, that);
}
 
@Override
public int hashCode() {
    return HashCodeBuilder.reflectionHashCode(this);
}

maybe even use an intellij live template like this:

@Override
public boolean equals(Object that) {
    return org.apache.commons.lang3.builder.EqualsBuilder.reflectionEquals(this, that);
}

@Override
public int hashCode() {
    return org.apache.commons.lang3.builder.HashCodeBuilder.reflectionHashCode(this);
}

@Slf4j

I recommend using an intellij live template again.

// just put in 'className()' for the $CLASS$ variable
private static final org.slf4j.Logger LOG = org.slf4j.LoggerFactory.getLogger($CLASS$.class);

Logger via live template

@Builder

I really like the builder generation plugins that are freely available on the intellij plugin market.

Also, there is a new Replace Constructor with Builder refactoring available now. However, i prefer the Inner Builder plugin.

A case for Delombok

Thankfully, there is this nice little feature called delombok. It will transform all your former lombok annotations into the ugly boilerplate they created behind the scenes, so you can completely get rid of it. Don’t expect a beautiful outcome though, the generated code will contain a lot of @SuppressWarnings("all"). Also, the delombok feature of my intellij plugin caused my ide to freeze. So i had to use the Lombok Maven Plugin instead, it worked just fine.


to the comments

seamer: a refactoring tool for java

Just recently i was refactoring some very messy legacy java code and i stumbled upon this really long and complex method. I had to refactor it but there were no tests, no safety net, what so ever. It had so many different paths that adding unit tests would have been a tedious endeavour.

“Would be nice if had something like suture”, i thought. So i decided to just code it real quick.

Here is the result: https://github.com/gregorriegler/seamer.

Seamer is a refactoring tool for java that helps you get complex methods under test in order to make safe refactorings.

How does it work?

Basically you just wrap a complex method with a lambda and the seamer will then record all your invocations including input arguments and return values, serialize and persist them. You can later replay those invocations to make sure everything still works.

Seamer provides an api for you to make suggestions on the arguments you want to pass in. The tool will shuffle your suggestions and execute given arguments in all possible combinations. Or you can just let it run, click around in your application, and record real values.

When you are confident with the data you have recorded, go ahead and start your refactorings. Replay the seamer regularly to make sure everything still works.

If you have ideas for further improvements, feel free to do so, or just make a pull request.


to the comments

Between the lines

As coders we are constantly making decisions. This is what we do, it is our duty. Our decisions have a lot of impact. They affect the success of the project or product. They affect all stakes of the software, the users that are going to use it, and in enterprise even the company behind the employees that are going to use it. And of course they affect our own and/or our companies reputation. They affect what problems we face when we engage upcoming tasks, our co-workers future mood and even our own. Bad decisions can easily make us suffer.

What we do is not a routine, ever. We might have patterns, practices, principles and experience. Yet still, every new decision that we make has a unique context. This is why our work is very creative.

Sometimes we make good decisions and sometimes we make bad ones. This is true for all types of coders from juniors to masters. Sometimes we make bad decisions because we just don't know any better, due to a lack of programming-, technology- or domainknowledge. And sometimes we make bad decisions because we are being put under pressure.

Bad decisions are expensive and dangerous. They produce technical debt, and it is not uncommon that they lead us to even more bad decisions. Slowly but surely this burden becomes heavier. It cost us time, energy and money. In extreme cases it gets so expensive that it would be favorable to completely start over.

Willingly or not, because of the decisions we make, we coders are in great power. We are leaders, and we can easily steer towards failure. We are not just the ones to write lines. We are not just code monkeys. We are the ones who design the heart of the Software and bear great responsibility.

When everyone gets mad because of a problem, maybe a date or even a deadline, we coders are the ones to stay strong. We are the ones to keep calm, and disillusion the others. We know the code, so we know the truth. If we don't stand up, whoelse should?

It is hard to act professionally and responsible at any time, but it's mostly a good choice. Thankfully, code is very forgivingly in its nature. We can change it, thus allowing us to revert our decisions. We should value this changeability in the code. We should even take action to improve it. Maybe that'd be a good choice!?

I would like to end this post with a quote by @WoodyZuill:
Be the programmer you want to see in the world.


to the comments

Improve your Feedbackloop with Continuous Testing

Have you ever though about what the most valueable thing in software development was for you? And im not talking about things that value for you personally, but for the success of the development itself. Well i have thought about it, and for me it was Feedback - in any form. It is so important, because it enables steering. Development practices are made for the purpose of better feedback. TDD, Continuous Integration, Iterations, to name only a view. Many of the agile methods and XP are basically about better feedback.

It begins with customer interaction. I need as much feedback as possible, as frequent as possible. If i don't get feedback, i'm likely to get on the wrong track, resulting in a product that is not going to be used, because it's not what the customer needed. The more feedback i get, the better the outcome will be. If i get feedback rarely, i am unable to steer. I'm forced to make assumptions which are likely just obstacles. The quicker i get feedback, the faster i can deliver value to my customer. Feedback allows me to steer.

Feedback is as important for programming. I want it early and often. If i write hundreds of lines of code without running them, they will most likely result in a very long and painful debugging session, and a lot of changes. I don't want that, so i take baby steps. They make me go safer, faster and happier. There are two phases that define my programming feedback loop.
  1. The phase where i write code. Lets call it alpha, like so 'α'.
  2. The phase where i evaluate my code, and eventually fix errors. Lets call it beta, like so 'β'.
You could also see those phases as modes. It is important to understand here, that these phases have nothing todo with the alpha/beta definition of a software cycle. I just invented them to describe my programming feedbackloop.In the following graphics you'll notice that the lines get shorter and shorter by example which is intentional and should point out how i got faster using new strategies.
    When i first started coding, i did not write any tests. I wrote lots of code before i tried it out manually. Obviously it didn't work when i first ran it. I ended up in rather long α-phases, where i just wrote code, and also long β-phases, where i evaluated (got my feedback), and fixed it. Like this:



    I was very slow back then.

    I soon started with an interpreted language, which was very cool because i could run the scripts immediately. No need to compile or anything. Just write and run. It shortened my feedback loop, and i became faster overall:



    Sooner or later i eventually started tdd. And regardless of the language that i was using, interpreted or not, it again shortened my feedback loop and made me go faster. The loop was shortened to a single 'unit', which is obviously smaller than anything manually executable. It allowed me to evaluate small behaviours, long before the program was even runnable. It is important to understand, that the α-phase in the following graphic contains both writing tests and implementation. The β-phase is much shorter, since unittests run very fast.


    I thought this was just great, and it could not get any better. Wrong i was!! Later, i tried something that made me go like this:



    "What the ...?" You might ask. No, i did not break space-time. The thing i tried was Continuous Testing. Which basically means, that i do tdd, but i don't run my tests by pressing a button and then wait. I just have them run all the time in the background automatically... Everytime i change my code, my tests immediately run automatically, and show me "OK" or "NOT OK" on a small icon on my screen. Since the tests only take about a second to run, this feedback is instant. And since my IDE saves my files onchange automatically, i do not have to press ctrl+s or anything. I just code... and as i code my files get saved.... and as the files get saved my tests get run... fluently, immediately. This is HUGE. I am now progressing without disruption. I completely broke out of the phases, or if you want to call them 'modes'. I love it.

    I have used infinitest for this. It is a Continuous Testing plugin for Intellij/Eclipse. If you are doing javascript, i can also recommend grunt for Continuous Testing.


    to the comments

    About the „Is TDD Dead?“ discussion

    It did not take long after Agile died that TDD joined its party. If you havent noticed, David Heinemeier Hansson put up quite a rant against TDD on his blog leading to a lot of discussion - also on Twitter. Finally they ended up doing a live online discussion regularely featuring Kent Beck, Martin Fowler and David not (yet) involving the TDD advocate Robert C. Martin aka uncle bob. If you haven't watched it live, here is the initial 30min discussion of the series on Youtube. I really recommend the whole series.

    I think this topic and the ongoing discussion is very valueable to our craft because it is the kind of questioning and critical thinking that evolve what we do and how good we are at it. Even if we have a strong faith in something, we should not be afraid to question it, ever.

    I get davids points, however i do not entirely agree with all of his concerns. I want to share my thoughts in this blogpost.

    „Test induced design damage“
    A good design is a design that makes the software easy to change. I think this is something we all agree about. While TDD generally puts a lot of good pressure on the design, if you're not careful you're going to couple the tests tightly to the structure of your code. In this case even refactorings might force us to touch a lot of test-code aswell. You don't want that to happen. Tests should be the enabler of refactoring. I've happened to experience lots of applications that were so small that sociable unittests with reduced mocking and integration tests were just perfect. The value of this approach was that it was very easy to change/refactor production code without touching test-code. TDD does not solve these design problems, TDD does not design for you. TDD is just a design tool, but you are still the designer. Testing is engineering. Look for advantages, make tradeoffs. Trying to minimize mocking can definitely pay off.
    To Test first, or not to Test first
    The point is, you do not want to verify if the code is working manually. You'd have to do it way too often, and it would just take too long. You want to be able to fiddle with the code, and do small changes frequently, without having to try everything out manually. You want to do it automated and the feedback should be instant. So i see no point in 1. writing code, 2. trying it out, and then 3. writing a test. Why not write the test in the first place? Also, we tend to be very flaccid on writing a test for something that we have found to be working, because we tend to become euphoric about our success quickly which makes us tired of digging in details later. Other than that, the actual writing of the test first forces us to think about the problem to an extent that makes us recognize hidden details. These details are better be known before we start writing the code.

    TDD is so widespread nowadays
    I was laughing when i noticed that there are actually people who believe that everyone does tdd nowadays. Sadly, TDD, or lets say developer testing is in fact unused and/or undiscovered in many places. I've happened to see it missed in programming education and also software companies. Sometimes, it was even unwelcome. If you were using any kind of developer testing in these places, you were a pioneer. There is still plenty of work to do on this topic.

    The bottom line is, automated developer testing is good. We need it. We were silly if we didn't.


    to the comments

    An Annotation Nightmare

    @XmlElementWrapper(name="orders")
    @XmlJavaTypeAdapter(OrderJaxbAdapter.class)
    @XmlElements({
    @XmlElement(name="order_2",type=Order2.class),
    @XmlElement(name="old_order",type=OldOrder.class)
    })
    @JsonIgnore
    @JsonProperty
    @NotNull
    @ManyToMany
    @Fetch(FetchMode.SUBSELECT)
    @JoinTable(
    name = "customer_order",
    joinColumns = {
    @JoinColumn(name = "customer_id", referencedColumnName = "id")
    },
    inverseJoinColumns = {
    @JoinColumn(name = "order_id", referencedColumnName = "id")
    }
    )
    private List orders;

    Wait. What? Is this really what we have come to? I can't even see the damn property under this bloat. How did this happen? Yeah ok - we had to get rid of the old xml configuration horror somehow. But this? This is even WORSE. This class is supposed to be a goddamn pojo with a bunch of properties. Short and concise, easy to read. I, as a reader of this cass, am not interested at all how the database table is joining customers to orders. I'm neither interested how its being serialized. This is just implementation details. Reading this class, i am living in an object world and i want to know what data and behaviour the object has. Not more, not less. I don't care about column names, fetchtypes or json serialization for the moment. And i don't want to read, change or recompile this class for the sake of a tablename change. I don't want to add another annotation for storing this entity in a mongoDB neither. The entity should not have responsibility for these details. We are not only violating the Single Responsibility Principle here, we are doing a f****** responsibility party.

    Ok ok, enough of the rage. How do we deal with this issue? Some duplicate the entity for various layers with different annotation purposes. They map the entity onto the next layers related entity using an automated mapper like Dozer. Some even write that mapping themselves. But this is by no means a solution. It is just replacing one code smell with another one: Duplication.

    So please, focus on frameworks that don't force you to clutter your code. jOOQ is a nice solution to map database records to entities without annotations. Also, hibernate allows you to define your mappings in XML.

    Private Field Injection


    @Inject
    private MyService myService

    This is used quite often, while it shouldn't even be possible. The myService field is private, thus it's inaccessible from outside the class. Nevertheless it's possible and people do it. In reality it is a hack. The DI-framework sets the field using reflections doing setAccessible(true). You don't want hacks in your code, do you? Lets have a look at the alternatives:

    Setter Injection
    Well, at least its better than the private field injection, since its using a public method instead of hacking a private field. But still, ask yourself this: 'Is this class even supposed to live without the injected value?' Because if it's not, there is no reason what so ever for the class to get constructed without an instance of MyService. You want to implement this constraint on the class level and inside the constructor, not on the framework level.

    Constructor Injection
    This is usually the way to go. It allows you to
    • make the field immutable (there is usually no need to change it).
    • implement the constraint, that the class is not instantiatable without a given MyService in the right place.
    Of course it means that you cannot inject by annotation. But why would you want to? The class doesn't need to know, if it gets injections by a DI-Container or a Factory Class. It should not know anything of this. No @Autowired, no @Qualifier. All it needs to know is its own behaviour. Everything else should be handled outside of the class.
    One could use a configuration class or file for the actual injection.

    A DI-Container is a usefull tool that helps you wire your classes together. Use it for this purpose, but don't let it dictate your code. Uncle Bob wrote a great post where he explained how to use DI-Frameworks without having them dictate your code.

    @RunWith(SpringJUnit4ClassRunner.class) in UnitTests

    Why would you need this in unittests? Because it is automatically generated by your IDE/app template? No! You want to test the behaviour of a class, living in isolation in unittests. Not if the DI-Conainer is injecting a fields accordingly. Just inject yourself in a setup method. No DI-Container needed. By the way, all this testrunner does is these 3 lines of code.

    private TestContextManager testContextManager;
    //..
    this.testContextManager = new TestContextManager(getClass());
    this.testContextManager.prepareTestInstance(this);

    They are not worth blocking your only TestRunner slot. You want to keep it free for something like parameterized @RunWith(JUnitParamsRunner.class) or concurrency @RunWith(ConcurrentJunitRunner.class) tests.

    tl;dr
    Annotations have become more harmful than helpful these days. We should get back to pojos and focus on keeping our code as clutterless and framework-agnostic as possible to make it more readable and reuseable. Don't let frameworks dictate your codebases, since they should be exchangable tools. Beware of what a class should know, and what not. Some annotations are useful, most aren't.

    @DevNull({
     @SuppressWarnings
     @Autowired, 
    @Inject, 
     @Override 
     @XmlElementWrapper,
    @XmlJavaTypeAdapter,
    @XmlElement,
    @JsonIgnore,
    @JsonProperty,
    @ManyToMany,
    @Fetch,
    @JoinTable
    })


    to the comments