When I started working here we were around 50 employees in two countries, and now we’re going past 300 in four.
I think we’ve outgrown our organically-evolved Slack usage and we’ve slowly bumped into a few problems, or at least things that aren’t as good as they could be.
There’s a few people who agree with me, and I plan here to write down what we intend to do, what we do, and how it goes.
I’m posting this a couple of years late because it took me some time to get around to the follow-up bit, but I think it’s still worthwhile.
The problems
We’ve identified a few problems:
- It’s often unclear where the right place to ask a question is: some channels are named for their topic, others for their team; our products are known by different names internally than externally, and our dev team names don’t line up with product names.
- Lots of people feel overwhelmed by messages or notifications: misdirected @heres, long tangential discussions, slackbot messages not-intended for them etc.
- Moving from text chat to video chat is often confusing: like many companies, we pay for an internal VC platform (Google Meet) an external for-customers one (Zoom) and also have Slack’s.
- The same questions are asked repeatedly, because their answers change over time: “who is on-call this weekend”, “where are we at with customer X’s migration”?
- Some topics span products/teams: where’s the best place to ask a question about MySQL in a company with several teams and products that each use it?
The Solutions
Multiple Team Channels
We used to have one channel per team, now we have four. Here are the platform team channels:
#platform-team is the old team channel for everything. It’s public, and is where people ask questions of the platform team – how work is going on a thing, if a found problem is known about, that sort of thing. It’s also a place for announcements from the platform team. Anyone interested in what the platform team is doing should join.
#platform-priv is a channel that contains only the platform team members. It’s where we talk about the work we’re actually doing, and things that probably interest nobody else. Pretty much all of the conversation that would have been DMs between a few Platform team members becomes threads in here.
#platform-bots is where all the botspam goes – reminders about open PRs, stats from JIRA, etc.
I think Slack is the wrong place for alerts, but I’ve lost this argument repeatedly :) So the #platform-alerts channel is where alerts go.
Product Channels
Each development team looks after multiple products. But also there’s a lot more to a company with software as a product than writing code, and there’s lots of things that are product-specific, so we also have product channels. These are named with the customer-facing name of the product.
The team channels are full of discussion about the development work to implement features or fix bugs in the products, but the product channels are more business-facing; how the product is or can be used, what features might be considered in future.
We expect the product, professional services, customer success and sales teams to sit in the product channels for the products they’re interested in, and hope to have at least some of the dev team in there.
This gives a good and obvious place to go to for someone who has a problem with a product but isn’t particularly interested in which pokemon the team that works on it is named for.
It also splits up the developer chat and the user chat on the products; sales don’t need to ask a forum of developers about the possibility of something working; they don’t even need to understand how the dev teams are arranged or named.
Special Interest Groups
Someone else pinched this from the Kubernetes project, but I think it’s great so I’m including it here.
A SIG is a group of people with a common interest that isn’t necessarily exactly, or even partially, their job. We have SIGs for MySQL, Kubernetes, Linux Laptops, Ruby, Java, Security, Free Software, and several more.
This creates an obvious place for chat about stuff – the security sig’s channel is not the same as the infosec team’s one, and so is often just pasted articles about security incidents and following chatter. 90% of our developers write Java all day every day, but rather less than that want to talk about it.
Nobody really owns the development of how we manage our Linux Laptops, so there’s a SIG for people interested enough in it to try to improve it, it’s quite quiet outside of someone hitting a problem.
We’ve not really settled on what the right set of SIGs is, what’s needed in order to create or remove one, or if they should be able to do anything – should the Java SIG set our Java coding standards, for instance?
This project is largely the outcome of sig-slack, so this is a change that’s already happened, and isn’t ours to claim, but I think it’s a great improvement to our slack use.
Set user expectations
One thing slack is famous for is its notification-overload, and many people find themselves muting several channels in order to get some work done. We want to set some minimum expectations:
Everyone who is on-call must get interrupted by their alerts. They don’t have to do this using slack (a phone call or the VictorOps app works too, for instance).
Being mentioned by name or by alias should always trigger a notification. Notifications don’t need to be interrupting, but would often be. A response doesn’t need to be immediate unless the message obviously and reasonably demands it.
Team private and public channels should be followed closely enough that they’re a good place to have conversations; they should be checked between tasks, perhaps. The same goes for developers sitting in the channels for products they are working on.
And nothing else should be onerous; for most channels a check once or twice a day is enough; treat them like emails.
These go both ways. Before using @here a message writer ought to consider the need to disturb the entirety of a channel; they can reasonably expect their message to be read over the course of a day in any case and that may be enough.
We can’t expect anyone to catch up with messages throughout the day if the channels are still full of bot output; the above channel changes have to be made before we can raise our expectations that messages will be read.
Use Emoji
They’re quite good and some of our teams (including mine) are already quite heavy users.
One common set of exchanges in Slack for us is someone pasting a link to a PR and hoping to see it getting merged. It can’t be merged until someone’s approved it (having reviewed it) and the pipelines have run.
Here, as soon as our reviewer saw the post, they added the :eyes: emoji and clicked the link. This way they immediately told everyone else that review is in-hand and underway.
Questions that crop-up in the review might end up on the Github PR page, but we’ve found that often for Gitops changes they’re quicker to deal with in a thread; the questions are more “does this app actually need 12Gi of RAM?” than about the science of programming.
Here’s that same link once it’s been reviewed and approved, and the pipelines are running:
At this point styles differ. Once it’s merged you can just add the :merged: emoji, but if you think the asker wants to know when that’s happened you can reply in a thread which will notify them.
I also like using Emoji for votes; these are :upvote: and :downvote: respectively, not just the up-arrow ones. Some teams would have more-appropriate cultural call-outs than Reddit:
And you can also encourage norms with emoji, as a less-agressive way of responding than typing out a telling-off. Here, if you start a conversation in the channel that ought to have been a thread you’d get a barrage of 🧵reactions, for instance.
We want to advocate for this style of post-and-review, and see what other people can use. We already have quite unrestricted access to create emoji and an ever-growing collection of them.
Set channel bookmarks
It’s a little-known fact that slack channels can have bookmarks, and this causes people to often neither create them nor look for them.
If we make their creation a norm, people will look. We hope.
With these we want to remove a lot of the common questions people pop into the channel for. And having a known-place to send askers for these answers should encourage us to keeping the destinations of those links up-to-date as well.
Find useful slackbots
We’ve introduced two slackbots that are fantastically useful.
One talks to VictorOps to tell us who is on-call; rather than popping into a channel to ask a human, I can ask the robot and no human need be disturbed. But I can also then use @oncall-plat to notify the platform team on-call engineer directly, so I often don’t even need to look it up!
The other we’ve only just started using, and it creates slack channels for incidents. It’s not quite the Monzo solution, but similar and so far it looks like it’ll be brilliant once we’re used to it.
There’s likely some other workflows we can easily automate here. Many slackbots just automatically dump more text into Slack, it’d be good to find the ones that cause fewer messages to be sent.
Embrace/Accept Huddles
It’s long been the case that nearly every engineering company has paid for two or three different video-conferencing platforms: Slack, one of Office 365’s Teams or GSuite’s Meet for internal meetings, and then also Zoom for external meetings.
Slack’s not going away and now we have huddles, so you might as well not bother to try to not-use them. I think they’d be great for informal “let’s just jump on a call and talk about this” chats, especially from a thread or a channel, and I love them for virtual-officing.
We’re not going to specifically encourage them, but are also not trying to force people to use something else.
How did the solutions go?
I wrote most of the above at the start of a project that’s now a couple of job-changes ago and I had planned to write a conclusion some time after the implementation, but I left the company before I got around to that, so this is a couple of years later.
The team and product channels definitely improved things, but during the project we were aware that almost anything would be better than the current mess. I’m not sure we’ve stumbled across the gold-standard, but we’ve not felt the need to fiddle further.
We did end up prefixing the team channels with #team-, and introduced a standard set of prefixes (#project- for projects big enough to have names, #inc- for incidents, #chat- for non-work chat) and I think that probably helps with discovery. It also makes the Slack Channels Grouping plugin relevant.
SIGs work fantastically, and we can claim no credit for them. I think one important thing that was got-right with them, much to my frustration a couple of times, is that it’s quite hard to get them created. There’s a relatively small fixed group of SIGs.
We really failed to change user expectations, and I think that was partially because we weren’t management; the messaging wasn’t coming from on-high. We noticed that the people least-likely to mute annoyances were more-junior and so probably needed more reassurance that they could mute things.
Emojis were already popular when we arrived; I only added that to this document to spread the support. We did nothing around there, and don’t really think we’re about to have a ‘too many emojis’ problem. 😀
Team channel bookmarks never really took off, but they’re great for the founding documents of the project teams, especially for the progress measures that people adjacent to the project might want from time to time.
We found no more slackbots, but did get to document a way to propose new ones and try them out quickly.
We didn’t really see huddles take off; the Google Meet slack integration got basically as-good as huddles.
Did we solve any problems?
- It’s often unclear where the right place to ask a question is
This got much better with the new channel names, and especially having product channels alongside team channels. A happy side-effect, too, was that the product channels became much less developer-focussed places to talk about our products and what our customers were doing with them.
- Lots of people feel overwhelmed by messages or notifications
We definitely improved this, though not as much as I’d have liked. We were already pretty good at not having scattergun @s, but putting the robots and alerts somewhere ignorable was a useful change. I think we need to do more to encourage people to turn down Slack’s notifying.
- Moving from text to video was often confusing.
It still is. I don’t think there’s a tech fix for this as long as there’s three options for every call. I’ve strong opinions on most of the big VC options, but not Slack, and I’d be interested to spend some time at some point figuring out why nobody seems to use Slack’s video calls. I still think huddles are the natural place to go from a slack thread, even if only because the in-call chat just joins that thread.
- The same questions are asked repeatedly
Robots, bookmarks, and perhaps a separate documentation tidy-up project really helped with this. Also we know the channel re-org meant people didn’t feel such a need to try asking different questions in different places. I think the SIGs had the biggest impact, though.