Common Workplace Problems for Data Scientists, and How to Address Them
There are lots of great things about working in data science. But like any job, being a data scientist can be frustrating — especially when your company isn’t taking the right approach to big data.
The good news is that some of these problems are manageable or avoidable! Let’s take a look at some common workplace complaints of data scientists (drawn from around the web) and how you might be able to avoid or manage them.
Unreasonable Management Expectations
It’s often said that data modeling is 90 percent data gathering/cleaning and 10 percent model building. So it’s a huge headache when someone has a bright idea for a last-minute insertion.
Sometimes this is the fault of the modelers, but usually it’s wishy-washy management deciding at the very last second that something they just thought of just now is very, very important. ‘Wait, will we be including social media history in our analysis of auto accident frequency? I didn’t see it in the list of variables. Let’s add it!’
The data modeling people sigh at these kinds of requests, because it usually means a few days of additional data gathering and a delay in a (perhaps already determined) modeling schedule. Data modelers should keep lines of communication open and set some kind of ‘no further adjustments’ date so that this doesn’t happen. But it probably will anyway.
Complaints about unreasonable requests and expectations from management are pretty common among data scientists. Thankfully, it’s often possible to improve these kinds of situations by improving your own communication skills, setting clear expectations, and doing a little bit of education.
In many cases, the problem stems from the fact that the manager or team member doesn’t understand the implications of what they’re asking. This is a common issue in most technical fields, where changes that seem trivial to the layperson may actually require much more involved work behind the scenes. Assuming your manager or coworker is not unreasonable, however, setting clear expectations before a project begins (including cut-off points after which making changes or additions will significantly delay results) can go a long way.
When you do get this kind of response, the best approach is to stay positive and solutions-oriented, but also be clear about what’s possible. Consider a response like “Yes, we can definitely add in those social media metrics. I expect that will add three to five days to our project completion time, because we’ll need to capture and clean that data, and then adjust our model to account for it.”
'We have last week’s data, can you predict the next 6 months?' This is a pet peeve of data scientists. Clients cobble together a few rows of data in spreadsheets and expect AI to do the magic of crystal ball gazing, deep into the future. At times this gets quite weird, when clients confess to not having any data, and then genuinely wonder if machine learning can fill in the gaps.
*- *Ganes Kesari, co-founder & head of analytics at Gramener, via Towards Data Science
Expecting data scientists to take bad data, little data, or no data and turn it into meaningful, actionable predictions is another expectations problem data scientists can face. Managers may have read articles about the power of machine learning and AI and concluded that any data can be fed into an algorithm and turned into valuable business intelligence.
Of course, data scientists know this isn’t true — your analysis and predictions can only be as good as the data you’re working with. Certainly there are statistical techniques that can help you plug gaps in a data set, but there’s no magical algorithm that’ll predict six months of sales accurately when it’s only fed a week of data to learn from.
The best way to address this is early on in your position. Join a company that’s already collecting large amounts of good data, or start working to improve your company’s data collection and storage as soon as you join. Beyond that, you can do your best to set realistic expectations at the outset of every project based on the data that you know will be available to you.
Misunderstanding Data’s Significance
People who don’t understand that data is not truth - it is only data. It isn’t even information until someone wraps some context around it!
*- *Alexander M Jackl, data scientist, technology strategist, and architect, via Quora
This is a problem that can affect anyone, including data scientists themselves, so it’s something you could encounter in a manager, in a teammate, or even in your own mindset if you’re not careful. Workplace attempts to foster a data-first culture can sometimes stray into the realm of data worship, and it can be easy to forget that data can only be properly understood with context.
Providing that context is part of a data scientist’s job. If a source of data collection could be biased, for example, that’s context you need to factor into your analysis from the get-go. Broader contexts, like market trends, also need to be factored in. When coworkers and managers are inclined to trust the numbers no matter what, it’s your job to understand the weaknesses, biases, and contexts that have shaped those numbers.
Being Blamed for Bad News
Depending on the culture at work if you are a data scientist and recommend actions based on insights you have come up with you can either get a promotion, a bonus, or get fired.
*- *Ammar Jawad, product manager at Hotels.com, via Quora
One of the dangers of being a data scientist is that you sometimes have to be the bearer of bad news. If your analysis uncovers serious problems at the company, or paints a less-than-rosy picture of where the firm is headed, presenting that information to management can be uncomfortable. And although data scientists are almost never the cause of these problems, a bad manager might take their dissatisfaction out on you anyway.
To some extent, this is a problem you may be able to mitigate with better communication and better expectation setting. But ultimately, if your boss is asking you to dig into the company data and then blaming you because they don’t like what you find, it’s probably time to update your resume. Working in an environment where you’re going to be attacked for doing your job is not something you need to or ought to put up with.
Having to Convince Management
Unless you’re working in a company that puts data science at the forefront of decision-making, every project will be an exercise in defending everything you do. You constantly need to convince decision makers that your work can have a real effect and isn’t just some make-believe hoax [...] I’d prefer to spend less time convincing people some data science project should be initiated and more time actually working on the project.
*- *Håkon Hapnes Strand, senior data science consultant at Webstep, via Quora
This is a very common complaint, and it’s something you’re likely to encounter in your data science career. According to a recent study, nearly two-thirds of managers don’t trust data, preferring to rely on intuition. And those that do trust data tend to be mid-level managers who don’t always have much power to affect broad-scale strategic decisions. Practically speaking, that means that data scientists can face a challenge when trying to convince management of the value of a new project, and they also can face challenges with getting management to actually act on their results.
Communication skills are so critical for any data-science-related role for precisely this reason. Your analytical results aren’t going to have any impact on your company’s bottom line unless you can get management to actually act on them. Being convincing means communicating clearly, visualizing your data well, and keeping it simple. If you’re not certain of how well you’re doing this, run your presentation by a friend or relative with no technical or statistical background. They’ll probably tell you it was great, but pay attention to what questions they ask (these are the things you haven’t made clear enough) and what conclusions they draw from the data. This should give you some idea of what areas of your presentation might need improvement.
That said, it’s also important to remember that management might have to weigh other factors against the data’s recommendations, and data won’t always win. The Wall Street Journal documented a high-profile example of this last year: Netflix’s data team found that Grace & Frankie promotional images worked best when they didn’t include the show’s star, Jane Fonda. Executives then had to weigh the potential benefit of that information (more clicks on the show) against the potential future costs of annoying Jane Fonda.
The good news here is that convincing management should get easier once you’ve done it once or twice, assuming those projects go well. The same study that showed most managers don’t trust big data also showed that, according to its study author Dr. Nazim Taskin, “once a manager experiences good outcomes with big data, it builds confidence in applying analytics tools more regularly.”
Communication is Key
The recurring theme here is communication. Your data science skills and your excellent resume and portfolio may be what got you the job, but great communication skills are key to keeping it, and making your day-to-day life as a data scientist more pleasant.
Here are some helpful resources for improving your communication skills as a data scientist:
- Dataquest’s data visualization courses in Python and R, and our data viz design tips.
- Kaggle’s guide to presenting data science work
- Quora’s How can data scientists improve their communication skills?
