If someone were to ask you who the consumer of your service/app/program/whatever I’m sure you would have a clear answer (or at least I hope you know exactly who the consumer is) but have you ever thought about who is the consumer of the actual code you’re writing and the repository that it lives in? This is something that I think most developers don’t ever really consider until they’re just starting on a new code base and have complaints about the documentation or build process. Everyone has been through that but it demonstrates clearly who the consumer is. You and any other developer that is going to be work on or building the code.

The reason that this is important is because we don’t really spend much time thinking about the developer experience but we have whole teams who are paid to think about what the experience is for the people who use the final product of what we’re building. We have tools (like Lighthouse) that exist solely to measure how accessible and performant your web page is. We also have tons of different monitoring tools (SignalFX, Prometheus, Grafana, CloudWatch, Cloud Monitoring, etc) to tell us about how our services are performing and alert us to anomalous behavior.

Why does it matter?⌗

It matters because developer time costs money (usually a lot of it). The longer it takes to get a feature out, a bug fixed, or experiments running the more expensive the whole thing will be. Don’t we want features to be as cheap to deliver as possible?

There’s also an opportunity cost. Longer lead times on development the shorter the window is to potentially capitalize on the new feature.

It’s also just a really crumby experience to be working on a project and have to struggle through setup or deal with a terrible dev loop or (even worse) have a really bad deployment process. Developer experience can make or break a team’s morale and increase developer churn. No one wants to work on a project that is painful to do simple things.

And if this is an open source project then imagine that each of the issues I’m pointing out are points a path and at each point there’s an increasing chance that the new developer will just give up and move on rather than suffer through whatever contribution they were thinking of making.

Where our tooling seems to be OK⌗

There are a few places were we have some OK tooling that are worth talking about quickly.

Coveralls - Code coverage analysis
SonarQube - Security and code quality measurement
Shields - Code badges to communicate different statuses in Markdown

There are some pretty good CI/CD services but we’ll talk about those more later.

Where we could be doing better⌗

There are a bunch of areas where I think we are either missing tooling or the tooling exists but it’s not utilized as well as it could be. There are also a number of areas where tooling isn’t the problem and its more what is a priority or not.

Documentation⌗

I think the biggest area where just about everyone ends up complaining at some point is the code documentation. This is your Readme.md’s, your Contributing.md, and all those other text files that are meant to describe how to work with your code.

This is usually cranked out with minimal care at the start of a project and then never touched again until some poor soul is onboarded to the project and realizes that none of it is up to date and there’s so many things that are just blatantly wrong. They’ll then update at least some small section of it but there’s never a real story for maintaining any of it. It’s all left as tribal knowledge or an exercise for the reader.

This is an absolutely awful way to get started with a project.

But it’s so obvious what you’re supposed to do! And you’re a developer, you should be able to figure it out.

I’ve heard things of this sort before in response to why the documentation isn’t very good. Sure a developer could probably figure it out but there’s a few problems with this approach.

It’s a waste of time. Why have someone rediscover the wheel when you could just tell them?
It assumes that everyone is on the same skill level which just isn’t true and that’s OK
It gives new team members the impression that you don’t care about them and that they’re an outsider for not knowing your process

But we don’t have time to fix it, we’re supposed to be shipping features. That’s what is most important.

So make time! I guarantee that if you really tried you could convince your Product Manager/Team Lead/Manager that the relatively minimal cost of keeping your documentation up to date will pay off in the end. It’ll make it faster to onboard new team members and get them shipping features faster as well as reducing the training overhead that the new team members mentor(s) will experience since the new dev should be able to walk themselves through the setup process with little to no help.

I’m also guessing that the development process on your code doesn’t actually change all that often so it’s not like you’re going to be paying this high, continuous cost.

But it’s so boring! I would much rather write code than documentation.

And? Not everything that we should be doing is going to be fun. It’s one of those things that’s possibly boring to do in the moment but easily pays for itself when you don’t have to spend time answering the same question about how something works or what this one error means.

Solutions⌗

TODO: guide on readme⌗

TODO: playbooks⌗

Developer workflow/the dev loop⌗

After documentation, this is the next most obvious place that new developers will see a lack of care when it comes to developer experience. At this point a new developer has struggled through the documentation and are actually trying to make some change.

I see a few phases in this part that are worth talking about individually and what we can do about it.

Bootstrapping⌗

Here the user is trying to make sure they have all the required dependencies installed, any required runtimes are present, and they have any developer time configuration required by the repository is setup.

Some times this is a super quick process but there is some level of knowledge required and depending on who your users are these may be new developers (or at least new to your tech stack) and will need some assistance. Even a season developer could be made more productive with better tooling here.

My recommended approach is a “boostrap” script. This is simply some kind of script or utility that the user can call whenever they need to make sure that their environment is fully configured for local development. They may even call this script somewhat often depending on how often the requirements of the codebase changes.

Let’s look at a super quick example script that could be used to setup a Terraform repository.

set -o errexit -o pipefail -o nounset

function main() {
    printf "${FG_CYAN}Checking for required tools...${RESET}\n"

    ensure_bin_available 'asdf' 'https://asdf-vm.com/'
    ensure_bin_available 'keyring' 'https://keyring.readthedocs.io/en/latest/'

    ensure_asdf_plugin 'pre-commit'
    ensure_asdf_plugin 'terraform'
    ensure_asdf_plugin 'tflint'

    install_asdf_tools
    install_hooks

    echo

    printf "${FG_CYAN}Checking user configuration...${RESET}\n"

    ensure_dev_configuration

    echo

    printf "${FG_CYAN}Performing sanity check...${RESET}\n"

    sanity_check

    echo

    printf "${FG_GREEN}Bootstrap complete!${RESET}\n"
    printf "${FG_YELLOW}Make sure to run 'source .env' to load user configuration${RESET}\n"
}

function ensure_bin_available() {
    local name="$1"; shift
    local help_url="$1"; shift

    printf "Checking for required tool: %-23s " "${name}"

    if ! command -v "${name}" >/dev/null 2>&1; then
        echo "❌"
        echo "ERROR: ${name} is not available"
        printf "Please see ${FG_YELLOW}%s${RESET} for installation instructions" "${help_url}"
        exit 1
    fi

    echo "✅"
}

function ensure_asdf_plugin() {
    local name="$1"; shift

    printf "Checking for required asdf plugin: %-16s " "${name}"
    asdf plugin list | grep --quiet "${name}" || asdf plugin add "${name}"
    echo "✅"
}

function install_hooks() {
    echo -n 'Installing git hooks...                             '
    pre-commit install --install-hooks >/dev/null
    echo '✅'
}

function ensure_dev_configuration() {
    set_secret 'api_token'
    set_or_update_config 'user_id' '"${USER}"'
}

function sanity_check() {
    source .env

    run_command_with_capture 'terraform init'
    run_command_with_capture 'pre-commit run --all-files'
}

function set_secret() {
    local name="$1"; shift

    echo "Checking for ${name} secret..."

    if [ ! -e .env ]; then
        touch .env
    fi

    if grep --quiet "${name}" .env; then
        echo "✅ Secret ${name} already set"

        return
    fi

    keyring set 'terraforrm' "${name}"

    set_config_value "${name}" "\$(keyring get 'terraform' "${name}")"
}

function set_or_update_config() {
    local name="$1"; shift
    local value="$1"; shift

    echo "Checking for ${name} config..."

    if [ ! -e .env ]; then
        touch .env
    fi

    if get_config_value "${name}" > /dev/null && [ "$(get_config_value "${name}")" == "${value}" ]; then
        echo "✅ Config ${name} already set"

        return
    fi

    if get_config_value "${name}" > /dev/null; then
        update_config_value "${name}" "${value}"
    else
        set_config_value "${name}" "${value}"
    fi
}

function get_config_value() {
    local name="$1"; shift

    grep --perl-regexp --only-matching "(?<=${name}=).+(?=)" .env
}

function set_config_value() {
    local name="$1"; shift
    local value="$1"; shift

    echo "export TF_VAR_${name}=${value}" >> .env
}

function update_config_value() {
    local name="$1"; shift
    local value="$1"; shift

    sed --in-place "s/${name}=.*/${name}=${value}/" .env
}

function run_command_with_capture() {
    local command="$1"; shift

    set +o errexit
    output="$(eval "${command}" 2>&1)"
    command_failed=$?
    set -o errexit

    if [ ${command_failed} -gt 0 ]; then
        printf "%s" "${output}"

        sleep 1
        exit 1
    fi
}

RESET='\033[0m'
FG_GREEN='\033[0;32m'
FG_CYAN='\033[0;36m'
FG_YELLOW='\033[0;33m'

main

This is a somewhat long script (it’s written in Bash, that’s gonna happen) but it does only a few basic tasks.

Ensure the required tools are installed and install them where it can
Make sure that the user has the correct configuration setup locally
Attempts to run a quick test to make sure everything works afterwards

It also is able to make sure team members are setting any configuration secrets safely. In this case it’s using keyring to put things in a secure area instead of directly on the filesystem.

If Bash isn’t your style then you can use any other tool just make sure that you have something.

Making a change⌗

Now that the user is onboarded to the code base they’re ready to make a code change. There are quite a few places you can go to make this experience better but here are a few options depending on the type of project you’re dealing with.

Linting/formatters⌗

Linters and formatters are a great way to make sure developers are following the same style/code conventions as well as any rules or best practices that have been learned over time.

ⓘ Definition of Linter: A static analysis tool with a set of rules to verify some amount of correctness in code. Might also verify style conventions. It may also (and ideally should) auto-fix problems that it detects.

ⓘ Definition of Formatter: A tool that analyzes code to make sure it follows any defined conventions (line length, spacing, etc). It may also (and ideally should) auto-fix problems that it detects.

Using these tools you can reduce the number of style/convention comments that need to be added to pull requests, shorten how long a contributor must spend on said pull request, and divert any annoyance the contributor has with the prescribed conventions from the reviewer(s) to the tool which is telling them what to do.

Further, most editors and IDE’s have integrations for the majority of linters which means they can get live feedback and even have the tool do some of the work for them while they’re writing code.

You can also keep adding custom rules (assuming your tool supports that) when a new issue or convention arises meaning that you’re constantly keeping everyone on the same page without having to circulate notices about upcoming changes to the coding expectations.

There’s also a great tool called editorconfig which you can use to make sure that everyone’s editors has similar settings for things like tabs vs spaces, line endings, character encoding, and more. I’ve found this really helpful to prevent those weird cases of indentation getting mixed.

Example code⌗

If there’s some complicated or very opinionated way that code in your repository is written you can provide simplified examples of what is expected. These examples can be provided in a separate location in your code or in documentation.

The idea here is to provide really basic code that could be copy/pasted whenever someone needs versus having to look at other examples of similar code samples in the code base and extracting out any of the common bits from all the implementation specific bits.

Code generation⌗

Code generation is another way to scale how contributions are made. You can use code generation tools to update existing code to keep consistency, to stub out code from examples (see previous section), create new code from configuration (think protobuf/grpc).

Any code that can be generated by a computer is code that no developer has to read or have deep knowledge on. As the number of developers on a team/project grow this kind of tooling is going to be increasingly more important.

Example tooling:

Verifying a change⌗

Once the code changes are all done the developer needs to confirm that the change is “correct”. This means making sure there are tests and that the tests pass, that all the linters/formatters are happy, and any other requirements for a contribution are met.

As with previous phases there are multiple layers you can apply to make this smoother.

Tests and testing tools⌗

This seems obvious but I’ve been on more than one project that had minimal to no tests or if there were tests they were incredibly hard to run.

There should be tests and tests should generally be easy to run (there may be some hard to run tests and that’s OK). Tests provide developers with confidence that their changes work and that they don’t break anything that was working before. They can also highlight when something that was broken is now working properly. See xfail as an example of this.

When you don’t have tests its going to take much longer to verify that a change works, doesn’t break anything, and its more likely to produce unexpected failures. Even if you’re an expert on the codebase, you’re going to forget about some edge case or condition.

If tests are hard to run than no one is going to run them. Developers don’t want to wait around for tests to finish running and they’re not going to jump through hoops to make them work if they don’t have to.

You can also improve the testing experience by including test coverage as a means to try and verify that there are actually tests for as much of the code as possible.

⚠️ Code coverage can also lead to a false sense of security. Just because you’re at 100% coverage doesn’t actually mean you’re testing everything. Your tests could be poorly written or use an excessive number of mocks meaning you aren’t actually testing anything. Try and get as much coverage as is feasible and you can definitely skip testing areas of code where it doesn’t really make sense to bend over backwards for it.

It can also be helpful to have documentation on what your expectations are for testing to make things explicit.

Git hooks⌗

Once you have testing in place as well as linting and formatting you can start automating the running of those tools. As I’ve mentioned a few times already, developers aren’t going to read things or run things unless they have to. We’re inherently lazy (that’s a good thing!) and sometimes we need to be forced to do things.

You can use Git commit hooks to force developers to run tooling and checks on their contributions before they even push anything. There are many tools that make this easier (my favorite being pre-commit). If you combine pre-commit with your bootstrap script you can make sure developers are following the appropriate process when making changes. You can also block commits which don’t meet certain thresholds for quality.

Workflow tooling⌗

Continuing on with better tooling, having something that will define the specific tasks that a developer might need to do and performing those tasks consistently can really make a developers life easier and reduce the number of things they need to know to be productive.

A really easy example of such a tool is make but personally I would suggest something a bit more modern. Personally I like using Task but there are tons of good options out there to pick from.

In my opinion its much easier to tell a developer to run something like task start than to get them to run docker compose up --build. Sure the second option isn’t complicated but I would be willing to bet that many developers are going to forget the --build part and then waste time trying to figure out why their changes don’t seem to be working.

Putting common operations behind a workflow tool also makes it easier to change how those operations are done later. Instead of having to say “Hey, we’re using docker compose now and not go run” you can just make a PR to change your configuration file and the next time the developer runs the start task they get the change for free.

Continuous Integration & Continuous Delivery⌗

OK so developers can onboard, make changes, verify that they’re correct, and know how to perform important tasks. How do we communicate that all those steps were followed AND make sure that the deployment/release of those changes goes smoothly? Automating all that with CI/CD.

Even though we spent all that time automating the developer side of things (and provided a big carrot to follow best practices) it’s still possible that someone skipped all those steps and did something different and is contributing something which doesn’t meet the bar. You can catch all this by running those same checks that you run on developer machines on a remote machine (preferably a managed one such as Github Actions, Bitbucket Pipelines, Circle CI, etc).

You can keep these builds much more consistent since humans aren’t in the picture anymore to make mistakes or do things differently. You can also use a much bigger machine and run a full test suite that may not run on a developer machine or run tests that may require extra access or resources developers may not have.

It’s also much easier for a reviewer of a change to have trust if you have a green build than just someone pinky swearing that they tested everything.

Wrapping up⌗

There’s a lot here (admittedly this was meant to be a short post but there’s so many things!) and you don’t need to do all of them and you don’t need to do all of them immediately but seriously consider what makes sense for your team/project and invest the time and energy into making the developer experience better.

Make the improvements in steps and try and measure the improvement. I would be more than willing to bet you’ll see improvements on multiple fronts.

Who is the Consumer of your Code?