Small details make a big impact: a day in the life of a Distinguished Engineer

When people ask me about my distinguished engineer role at Sophos, I joke that I’m fifty percent researcher, fifty percent troubleshooter, and fifty percent hands-on engineer.

I spend some of my time looking at technology direction—delving into emerging technology to understand what’s useful, and how that might influence our products in the future. I also think further ahead, to see how we can go beyond the state of the art.

The troubleshooting involves assisting teams within the Network Security Group. Sometimes they just need a different or more experienced set of eyes, and I’m happy to help.

And of course, there’s the engineering aspect at the center of my role. Outside of design or coding, I spend a fair amount of time facilitating conversations between engineering teams, seeing things through together, all the way to a solution.

Why customers need a faster firewall

Recently, I’ve spent a lot of time on accelerating our next generation of firewalls. This involved a lot of work on the data plane—making sure the components move the data through the firewall as fast as possible. That’s important because it makes a real difference to users’ experience every day.

For example, if you’re getting traffic from a local network it has to be classified for the right kind of inspection, then it needs to be inspected, and it needs to be sent on. And if you want your web pages to load fast, or for your device to respond smartly when you tap on a button, the time to do all that needs to be minimal.

So, I keep close tabs on the performance—looking out for bottlenecks and other potential pitfalls. We always need to keep the future roadmap in mind while we’re working on what we’re building now.

Fewer decisions make a faster firewall

With the new Sophos Firewall, we rebuilt the whole architecture to make it faster. And that’s all about decisioning.

Every time the firewall sees a new piece of data, it has to make decisions about what to do with it—but a lot of those decisions have already been made before. The art is in figuring out which decisions have to be revisited and which don’t. If you can find your previous decision quickly, you can skip the process and speed things up.

Each individual connection going through the firewall gets a certain treatment. When the connection is first starting, you have the most decisions to make, because you don’t know what the connection is carrying, where it’s going, or who’s talking to whom.

Then once you’ve made those decisions, you have to figure out how many are going to hold true for the rest of the connection. If you can program those decisions into special purpose hardware, the CPUs don’t have to spend time making them over and over again.

We want to take the decisions that are slowest and minimize the number of times we have to make them. Then we find the ones that can be accelerated and push them to special hardware. And that lets the general-purpose CPUs focus on the security processing they do best.

We redesigned the new Sophos Firewall software to accelerate that whole process using a “Fastpath”, which moves trusted traffic much more quickly. And the new appliance moves that into custom hardware called Xstream Flow Processors, where it can go faster still.

One team across many countries

I work with tons of people I’ve never met in person. The group includes people from Vancouver, Karlsruhe, Ahmedabad and Bangalore, as well as Abingdon in the UK.

Increasingly, I’m collaborating with engineers from other groups too. Of course, we want any one of our products to stand alone as the best in its class, but where Sophos really shines is making them work together in combination. So, we spend a lot of time figuring out where projects overlap—and that means I get to work with plenty of different people, all around the world.

But working remotely from my colleagues is nothing new for me. I live in a small town, and my family has a great life here, so I didn’t want to relocate them. So, most of the time, I work here at home.

During the cooler months, I work in a basement office. It’s quiet, away from everybody else in the house—my wife’s a veterinarian and we have four kids, so things get busy—and my wife bought me a nice monitor with plenty of real estate for when I’m coding.

But once the weather gets warmer, I prefer to take my laptop and set up on the back patio. We have a pergola, and that gives good shade from the sun.

Small actions have a significant impact

Because we’re in a small town, we can be involved in the community in very direct and concrete ways. We’re involved with our local parish church, and my son is into scouting; I enjoy helping out. Our community work has shown me how things can seem trivial in the abstract, but if they’re making a difference in somebody’s life, they become important.

For example, when my son’s scout group helps out at the local food kitchen—they’re just making someone a meal and wiping down their table. When, once a year, we join with our church to help the rural poor in Appalachia, we’re building decks, reinforcing floors, and painting and cleaning houses.

These all seem like simple things, but the important part is being present and valuing people as human beings. It’s so impactful in the long term.

And in the same way at work, the time we spend on small details can make a big difference to our users in the real world.

So, when we look at the control plane for our products—like how a firewall is configured and managed, and how it communicates with endpoints and other solutions to increase resilience—we have to think about all the real-world ways it can be used.

We have products built on custom hardware, and even a custom operating system, but some people might want to run them on generic hardware, or on a generic operating system. And we also make sure we can scale up and scale down, designing an architecture that works for big enterprise appliances, and smaller business products as well, so that everybody can benefit.

That means we spend a lot of time experimenting with our designs in different contexts.  As engineers, we might wish we could ignore certain use cases, because it would be much easier. But we can’t just hope things will work out; we need to know.

Yes, it’s extra effort, but it makes a real difference for our customers in the long term.