How do you hire your startup’s first security engineer?

Over the last few months, I’ve been through half a dozen interview processes for founding security engineer roles. Here’s what I think actually works, and what’s a waste of everyone’s time.

In short:

• Use your standard engineering interviews.
• Tweak the system design interview more towards security.
• Use the behavioural interview to understand how they’ll work with (and not against) your engineers.

Don’t use “what’s insecure about this HTTP handler” puzzles.

I’ve built and maintained countless “janitor” systems in my career: cronjobs which go and clean up old resources like user accounts which should no longer be active. On multiple occasions, one of those janitors has gone off the rails and locked the entire company out of their accounts.

Now, whenever I build systems like this, I include a “fuse”.

The fuse defines a maximum value I expect the system to handle at any one time. The number of accounts to delete, or number of permissions to revoke. If the janitor ever tries to exceed this value, the fuse “blows” and requires a human to intervene.

Yes, I’ve made the classic mistake and re-started a writing habit by fiddling with the technology. Rather than, you know, actually writing something.

But I cringed every time I clicked a link to someone’s blogpost and saw they were using the same Hugo template as me (which, with ~13k stars, is a lot of people!)

When I bought this domain, I had zero frontend experience so had no choice but to just pick something full-featured off the shelf and hope it had everything I’d need. But over the past couple years building a startup, I’ve learnt enough web skills to be dangerous.

People expect companies/services to tell them when untoward things could be happening to their accounts:

• Their account has logged in on a new device
• Their password has been changed
• An export of their data was started

These examples are things you should probably be notified about. But, if you go overboard with these warnings, you might find phishers triggering them intentionally.

A bad security warning

Imagine receiving this warning email:

I once got a debit card with the CVC number 000.

Now, in theory, I know there’s nothing wrong with this. The number was chosen randomly and it being “special” is purely in my head.

In fact, if 000 were excluded from the picking process, there’d be fewer possible CVCs and so overall they’d be less secure.

But, 000 still feels really wrong as a CVC. Was it a bug? Did an entire batch of cards get generated with the same CVC?

LaunchDaemon (or LaunchAgent) Hijacking is a MacOS privilege escalation and persistence technique. It involves abusing insecure file/folder permissions to replace legitimately installed, misconfigured LaunchDaemons with malicious code.

I first spotted this issue affecting the OSQuery installer but went looking and found multiple other products with the same problem. This isn’t a novel technique (it’s briefly mentioned in T1543.004) but I was surprised to find it so rarely talked about.

Example – Hijacking the OSQuery LaunchDaemon

What is OSQuery?

OSQuery is a monitoring tool that lets you remotely query the configuration and status of hosts using SQL. On MacOS, this is accomplished by running a LaunchDaemon on each laptop that syncs with a backend server.

You’d expect phishing sites to be hard to detect and track, but actually, many of them contain HTML fragments that uniquely identify them. One example doing the rounds at the moment is a bunch of Royal Mail phishing sites which all contain the string css_4WjozGK8ccMNs2W9MfwvMVZNPzpmiyysOUq4_0NulQo.

These sorts of long, random strings are extremely good indicators for tracking phishing sites. Any webpage that contains css_4WjozGK8ccMNs2W9MfwvMVZNPzpmiyysOUq4_0NulQo is almost certainly an instance of that Royal Mail phishing kit.

A/B testing is a lifesaver for a solo SaaS developer. While it’s hard to predict whether a 14-day or 1-month trial is going to get a better signup rate, it’s really simple to test: show half of people the 14-day button and the other half the 1-month button and just measure the click-through rate.

There are loads of tools out there to help you do this, but rule #1 of running a micro-SaaS like QueryCal is to keep your technology stack simple—it’s better to focus your time on features, not integrating a bunch of different technologies.

Nobody likes alerts that feel like a waste of time to review. But, how do you tell whether an alert is actually a waste of time?

The go-to metric to use is false-positive rate—if an alert has high a false-positive rate then it’s “noisy” and might be getting rid of. But, trying to distill the performance of an alert down to a single number is hard and false-positive rate might be a particularly bad choice.

Making users more secure generally means annoying them. Whether it’s making them carry a hardware security key or just enforcing a short screensaver timeout, changing how people go about their work is annoying—and an annoyed user is not a secure user.

The effectiveness of a lot of security controls relies on the user cooperating. If they get frustrated with all the barriers and friction between them and doing their actual job, they might just find ways around the controls—their shortened screensaver timeout is soon “fixed” with a keep-awake app and now they’re less secure than they were before.

Intrusion detection honeypots are just plain cool. They’re incredibly simple to run and give you extremely accurate alerts about intruders in your systems.

A honeypot can be as simple as a fake server inside your network that alerts if anyone connects to it. There’s no reason to intentionally connect to this bogus server, so any attempts are probably an attacker already inside your network.

Unfortunately, that’s about as far as most people get with them. It’s a shame that so often honeypots get written off as just a novelty—no more than a neat trick to catch pentesters. Most people miss the point behind honeypots.

Snapshot testing is an extremely fast way to add regression testing to an existing project. You simply take some example inputs and then snapshot the resulting outputs. From then on, you can have a high degree of confidence that any changes you make have not affected backwards compatibility (as this would have been detected as a change in a snapshot).

However there are many pitfalls you can run into as I found when writing cupaloy, a snapshot testing library for Go.