Misconceptions
It's easy to conflate the ideas of programming and software engineering. As I was learning and beginning my career in software, I often thought that all I needed to succeed as a software engineer was a good grasp of programming languages and frameworks. In my second role as a developer, I was learning the React framework as well as the Ruby programming language with the Ruby on Rails framework. In theory, I already knew JavaScript, but that turned out to not be quite 100% true. My understanding was good enough, though.
As I started picking up Ruby on Rails, I began to see the 'magic' that the framework performed under the hood. Here I was, the diligent monkey bashing away at the keyboard, while behind the scenes, Rails was pulling all the strings and making my life easier. That said, it wasn't smooth sailing. The company I was working for, Klara, generously bought me a €10 course on Udemy that I was using to build my skills. The only downside was that it didn't explain the concepts in enough detail for me to really understand them.
"So, you create this variable @x and then you use it over here in this file" - Wait, what??
Even though I was following the example-driven course, I certainly wasn't learning how Rails actually worked. Even less than that, I wasn't getting to grips with why and in what circumstances I needed to do each thing.
After all, Rails is only good if you know what you're doing with it. I was simply following a tutorial, building trivial CRUD applications that had no practical use to me.
Aside from this being my new number 1 peeve with online tutorials, it meant that I was only becoming a programmer, not a software engineer. I was learning the tool I was going to use, but not what problems it actually solved.
At this point, I must add that it's necessary to learn your tools well. Building a Ruby on Rails application is a lot harder to do if you don't know Rails, so learning it is certainly a step towards being a software engineer, but it is far from being the only step.
My misunderstanding is clear to me now, years later: programming is not the same as software engineering, it is merely a part of it.
Software engineering is a multi-dimensional discipline. It consists of a multitude of skills, many of which are othogonal to each other. What do I mean by this? Programming is the ability to translate the solution to a problem into code. That is all. Another facet of software engineering, for example, is successfully analysing and familiarising yourself with user requirements. These two skills are independent of one another. Learning one will not help you learn the other. This is what I mean by software engineering being multi-dimensional - the skills extend away from each other, rather than building on top of each other.
So, what else goes into being a great software engineer?
Analysing user requirements
As mentioned above, being able to understand user requirements is a core skill to being a great engineer.
I remember during my first days at RIDE Capital, I had many discussions with my manager. He was an enthusiastic developer and he gave me a bucket-load of information about how real estate companies worked. How they could be owned by directors and shareholders, but also by other holding companies and so on and so forth.
I remember sitting there and wondering why this all mattered. Wasn't I there just to write code at the end of the day, why the hell did I care about how company structures worked?
I quickly realised that it was absolutely essential for me to fully understand the domain that I was working in. How could I build a database schema respresenting the company structures we needed to store for our users if I didn't even understand those structures myself? How could I code up the stages of company founding if I didn't know what they were, which tasks needed to be done by who and in which order? I couldn't!
Really, truly understanding the domain of the company's software was the only way for me to be an effective engineer helping to build that software.
Designing software architecture
For a while now I've been coaching some junior software developers in my spare time. One huge issue that really strikes me with these students is the inability to grasp the data flow within software applications. None of my students are stupid. Far from it, they're some incredibly intelligent people who are dedicating themselves to learning software engineering and buidling a career in this industry. But one thing that always tends to come up is dealing with the data that flows through an application. Increasingly, frontend development has to deal more and more with advanced and complex data sets.
Back in my day (man, I feel old), as I just started out as a frontend developer, I just needed some HTML, CSS and good old fashioned jQuery, but these days, with React and other much more advanced libraries and frameworks, the burden is falling more and more on frontend developers to deal with complex data structures.
Fetching paginated data, storing and manipulating data in the application state, caching and invalidating caches for data, none of it is trivial and it requires a good understanding of the type of data that is flowing through the application.
This, in turn, leads to some more complicated architectural decisions that developers have to make.
When do I want this data? Where do I need it? How do I want to pass it around? In which shape or format do I need to keep the data in order to present it and update it as required by the application?
Often there is no right answer, there are just better or worse solutions to each of these questions. Having a good mind for architecture is becoming an invaluable skill and certainly one that an expert software engineer needs to have. Without it, your code will always just break down into spaghetti code, the project will be a nightmare to maintain and it will most likely have to be deprecated.
Learning this skill is no mean feat, either. While there are many good books around the subject, the best education, in my view, is experience. Seeing good (and bad) architecture in the real world, and seeing the effects this can have, is the best way to acquire an instinct for what constitutes good and bad architecture and there's a lot of room for the grey area with some trade-offs for most big architectural decisions.
On a quick side note, I would say that learning and building the skills around Test-Driven Development as well as the principle of Dependency Injection helped me a lot to build better architected software overall.
Testing and debugging software
The world of software testing is huge, and many books have been written about it. I have some strong opinions around testing, but I'll leave those for another day.
Testing is the process of verifying, through manual or automated checks, that the software you're creating works the way it's supposed to. We could argue over the specification of software all day long, but let's assume you're working against some sort of task or ticket with Acceptance Criteria (AC). In order for the task to be considered complete, the AC must be met and a software tester, like a QA, or another developer on the team, needs to be satisfied of that. Automated tests go a long way to making that case.
Writing automated tests usually goes something like:
- set up some accompanying data or environment to run the system under test (SUT)
- invoke the SUT
- assert that the SUT does X
The assertion can be some kind of output, like a return value from a function. It could be that some UI element is shown. It could also be that some side effect is triggered. Whatever the case, the principle remains the same: does my piece of software behave the way I want and expect it to. On a fundamental level, automated tests are what give us confidence in the software we're building. I sleep easy at night knowing that my test suite is green (which indicates passing, as opposed to red for failing).
Testing, though, is only as good as your ability as an engineer to imagine all the difference scenarios and all the edge cases that can occur. If I have a function add(a, b) that simply adds a and b together, my test won't be very good if it always assumes that a and b are integers. What happens if they're floats (i.e. decimals), or does it handle strings or other data types? Does it crash in those cases, or does it give a useless output. In writing tests, I need to consider all the different possible cases that could happen. Only then will I consider my software properly tested.
Tests fall into different categories. Unit tests typically cover the individual class or function and ignore all else. These tests are fast and reasonably easy to follow, but bring less confidence and therefore less value. Integration tests test multiple parts of the system at the same time. They're often larger, don't cover every single use case (since the number of these increases exponentially as different pieces of the software are covered), take longer, but give more confidence. System tests usually test the entire system, or as much of it as is reasonably possible, take the longest amount of time but give the most confidence. For example, if I have a test that users can carry out the core functionality of my application in one continous flow, I'm going to be happy that it's working as expected as long as that test is green.
The ability to write tests is a fundamental skill that every modern software engineer must possess to write high-quality software. Tests prevent many types of bugs from entering the system. But what happens when bugs do get in, what do we do then?
This is where debugging enters the fray.
Debugging is the process of troubleshooting a bug that has occurred, either in a test or in the real application, and finding out what is causing it and fixing it.
Debugging goes well with testing - if you can write good tests that describe the bug accurately and what the correct behaviour should be, you'll know you're done when your test is passing. Simple as that!
There are many ways to debug software, and most of them are highly dependent on the situation, the language and the type of system that has a bug, but finding and practicing a solid debugging strategy is an invaluable skill that software engineers need to hone in order to be masters of their craft.
Collaborating with cross-functional teams
As a software engineer, you'll rarely work in isolation. Software engineers are typically part of a larger team or department containing product managers, who will create user stories by understanding the users' problems and working to find a solution for them, graphic designers, who take the solutions to the users' problems and create visual designs of how that problem can be solved in a user-friendly and aesthetically-pleasing way, and even project managers who oversee the projects and ensure they're bringing value to the company.
Being able to work with these other roles is a highly undervalued skill among software professionals. I've seen situations where the communication between developers and non-technical colleagues breaks down simply because they don't know how to talk to each other. By using overly complicated or technical language, it's easy to alienate members of the team, which makes collaboration harder.
As an engineer, you need to work hard to be approachable and to communicate your questions and feedback about graphic design or product requirements in a way that will bring about constructive teamwork between you and your colleagues.
Communicating with the rest of the organisation falls into the same category. Let's say you've just finished a large and complicated feature that the rest of your business will love, you need a way to communicate that to them effectively. I go into a meeting and I say to my colleagues:
"Great news everyone, I've just resdesigned the database schema to be more flexible with the entries for the flim-flams and the wiggle-worps", I would quite rightly be greeted with blank stares.
If, instead, I go in and say, "Great news, everyone, I've made some changes that allow you to enter the customer data more easily" and give them a quick demo of the new functionality, I'll likely be greeted with rapturous applause, chants of my name and a shower of champagne. (OK, maybe not, but it'll be better, that's my point).
Using version control systems
A technical skill, for sure, but again not the same as programming. Version control systems, in short, make it easier to collaborate on one codebase with one or more engineer.
Version control allows for a single source of truth about what the code should be in a particular file. If there is a 'disagreement' about that, also known as a conflict, the developers in conflict need to resolve this. Or, one of them does anyway. Usually, the earlier change wins.
Git is the most popular version control system, and learning how to use it and work with it in a team is a core skill that all modern software development teams will require.
The good news is that it's fast and easy to reach a point where you can confidently work with it. The bad news is that there is a still a word of complex confusion waiting just under the surface once you get past that point.
But fear not, it'll come with practice!
Software engineering principles and best practices
Junior developers write simple code.
Mid-level developers write complex code.
Senior developers avoid writing code at all costs.
As you advance as an engineer, you'll learn about the value of keeping code simple. Keeping it readable. Reducing it as much as you can.
And not writing it when it's not necessary.
DRY, KISS, SOLID, YAGNI, there are a bunch of acronyms to guide the path of a software developer, but building up your own principles about what to do will also come with time and experience.
"Don't deploy on Friday afternoons" is a common mantra in software teams all around the world. I'd argue it comes from fear. The fear of making a mistake that is hard to rectify. The fear of breaking the software for the weekend.
It probably just comes from not wanting to work overtime for free on the weekend trying to fix whatever was broken on Friday afternoon, but what do I know?
Software principles exist for a reason. They come from tried and tested experience of thousands of developers. Learning these principles is one challenge. Applying them is yet another.
All I can say here is: the more experience you get, the better you understand what a good practice is and what isn't. Which principles help you do your work, and which hinder it.
Code reviews
Most teams have some sort of Code Review process. Before code can get into production, it needs to be reviewed by a peer to confirm that it's good to go. A fellow developer on your team will take a look at your changes (using Pull Requests on GitHub, for example) and either request changes or give an "LGTM" - "Looks Good To Me" to confirm that they're happy for those changes to be merged into the main branch and deployed to production.
Code review is checking for certain criteria:
- correctness of the code - does it do what it needs to do to satisfy the acceptance criteria?
- style - does it confirm with the agreed-upon style convention for the team/project/company?
- test coverage - is the software adequately tested?
- performance - is this code potentially introducing a performance issue?
- security - will there be security vulnerabilities due to this code?
Your team might decide on other criteria as well, depending on the project, language or resources that are available.
Being able to conduct a thorough code review will make you an asset to the team because it unblocks your colleagues. Either they need to make changes, in which case they can go ahead. Or they can merge their changes, in which case the feature or bug fix has reached production, bringing value to the company, and that developer is then free to work on the next task.
Setting up a system which encourages and promotes rapid code review brings great efficiency gains to any software team. Just make sure your code review process is thorough enough, speed doesn't compensate for buggy and unreliable software.
Documenting software designs
Technical writing is a skill in and of itself. It's a skill I'm still working on myself. In earlier times of software engineering, companies had dedicated technical writers. These technical writers would be spread out across the tech team and would provide documentation for all the systems that were being developed.
This might have worked well for some companies, but it doesn't scale very well. Bringing in new technical writers as the team grows is inefficient, and expecting technical writers to keep up to date with the systems they're describing is a challenge.
In recent times, it's much more common for the developers of the software to document it themselves. This reduces head count, for one thing, as the role of technical writer is removed, but also reduces the overhead for the team. It's no longer necessary for developers to explain or detail how the system works or what it does. Since they are the ones creating and maintaining it, they already have the required knowledge to write the documentation for it.
One drawback I see to this solution, however, is that developers know their own work a little too well. Sometimes it's easy to get stuck in the details, which might not be required for the project's documentation. It's more important to convey what it does, rather than how it does it.
Learning to be a better technical writer is another skill that comes with time and experience. I've found that brevity is often better than verbosity. Keep it short, make it easy to read and understand, and you'll be in good shape. (Not like this post, right?).
Sketches and diagrams are also an easy way to improve the quality of a technical document. If you can draw a graph to show how a network of services interact, that's probably better than trying to explain it with words.
Error and performance monitoring
With the introduction of telemetry tools like Sentry, DataDog, NewRelic and countless others that are designed for monitoring and investigating errors and performance issues across applications, there is a whole branch of software engineering that goes mostly unnoticed in many developer teams and indeed organisations.
While performance monitoring is predominantly necessary for companies or applications that have scaled significantly, it's becoming the norm for even small teams to set up a base level of monitoring, through tools like DataDog APM or Sentry error tracking, giving engineers a quick and easy insight into any issues that might be happening in real time.
Combining these with more traditional marketing-focussed tools like Google Analytics, MixPanel or Segment.io, engineers have good high-level visibility of KPIs such as latency, throughput, error rates and metrics to make sure that the software is in good shape overall, as well as low-level visibility with logs, error messages and stack traces allowing for fine-grained control when debugging issues that users are facing in real time or shortly after the fact.
Setting up these tools requires a good level of know-how, and customising the dashboards for optimal visibility is a valuable skill that engineers can gain with exposure and practice.
Developing and writing code
Of course, writing code in one or more programming languages is still a core part of the work that a software engineer does. Writing code brings together many of the other facets of software engineering, like analysing user requirements, designing software architecture and applying software engineering principles and best practices.
That said, the day-to-day tasks of a software engineer can vary greatly. Some days might be heavily weighted towards writing code and testing it with automated or manual tests. Other days might entail no programming whatsoever, with monitoring of large-scale changes, architecting and designing additional services to be added into the system, or gathering, analysing and discussing new user requirements all potential tasks that can take a day or more in extreme cases.
A good example of this was during my time at Deliveroo. The project was to move some tables from the main database into a new, secondary database. Our team made some significant changes in where our data was being read from and we had to be sure that none of those queries were failing and giving our users an inconsistent experience. In the days leading up to the switch, we introduced metrics and error reporting to notify us of any discrepancies between the original database and the secondary one. On the day of the switch, we mostly monitored the dashboards that we'd prepared. I didn't write a single line of code that day, neither did any of my teammates. But it was one of the most impactful days I've had as an engineer. If you want to hear more about it, let's go for a drink some time and I'll tell you all about it!
While this seems to me a comprehensive list of engineering duties and tasks that distinguish the role of software engineer from programmer, I'm sure there are others that I omitted, presumbly because I haven't had the need to do them before. If you think that's the case, feel free to get in touch and I'll happily extend my list!