3 Common Challenges that Senior Level Data Scientists Face

How you can use your current skills to level up

Jan 20, 2024

toddler's standing in front of beige concrete stair — Photo by Jukan Tateisi on Unsplash

Many data scientists today find that the job doesn’t match up with the reality. And they’re not quite sure how to handle it.

When I started my career as a data scientist, I had high hopes of creating cutting-edge ML models. The reality at the 2-year mark —I spent most of my time on data engineering and analyst tasks. I was a data engineer and analyst with a DS title.

Then we get stuck and are unsure where to go next in our careers.

Before anyone gets upset, I want to emphasize that I value my experience in data engineering and analysis immensely. These roles laid a strong foundation to develop high-quality ML models and the requirements for them.

But as data scientists, we need to really consider if its an data engineer or analyst role we want. Or if we can use those skills to be better data scientists when data infrastructure and culture catch up.

Let's discuss some expectation challenges data scientists face, and how we can use what we have to enhance our skills to better support the businesses and our career.

We Want to Build Models (But Don’t)

Most data science education paints a picture of us constantly creating new models. However, in reality, especially in large organizations, you’ll spend a lot of timerefining and tweaking existing ML/AI workflows.

Much of the gritty daily work, really involves refining existing ML/AI models, gathering requirements, and prototyping. Building is a huge team effort - rarely one data scientist does the entire thing. You’re not alone.

Launching a new model isn’t just about coding; it involves scoping, getting buy-ins, and building coalitions. The process is more complex for high ROI models or those that face external users.

As experienced data scientists, you’ll find yourself diving into existing code, chatting with users, and enhancing what's already there. If you thought you'd always be building new models, this might feel a bit letdown.

We need to focus on the basics we are doing already: use case gathering, feature engineering, data preprocessing, etc. That matters.

Mastering the basics is like the journey of the apprentice in the documentary Jiro Dreams of Sushi. He spent years perfecting the simple task of making rice before moving on to more complex tasks. It's a reminder for us: every small step matters.

Each basic skill you hone contributes to building a quality model. Each basic skill is a factor that affects another skill. If we don't do data cleaning well, we can see its ripple effects on feature engineering. Knowing the nuances of cause-and-effect relationship affects your models.

The building new models is difficult if we don’t understand these basic pieces. Each ML/AI model is only as good as we can learn to tweak the smaller parts. Foundational understanding matters.

You’ll need to deconstruct it and boil it down. Understand why you’re doing the task and ask your seniors and leads questions. It helps you do your job better when you eventually start building.

And it helps better help us develop your technical potential. A good lead or senior data scientist will try to give you tasks that help your end-to-end modeling ability.

This foundational understanding will be invaluable when you start building models. It's all about enhancing your overall modeling skills.

Mastering these foundational tasks not only improves your current work but also helps when you level up, and are given responsibility for leading a model building team.

We don’t deploy models to production (enough)

Why don’t you get a chance to deploy as much? This is something I hear often. There's a bigger picture here.

This isn’t uncommon for data scientists, since most of your current value lies enabling the foundation for future ML and AI. Analytics and data engineering use cases is the foundation for that. Many ML/AI use cases originate in processes that the business is already doing.

Cost is another factor. In lots of companies, we end up doing more engineering and analytics work because upgrading data infrastructure, managing legacy data, and handling all sorts of tech and strategy headaches can be difficult. Along with the product management around it.

These challenges mean your often doing foundational work, even if it feels less glamorous.

By the time we get to the ML modeling phase, we've spent a significant amount of time in roles like data engineers or analysts — it takes some time to reacquire our modeling skills. Then it takes time to figure out how to deploy. This is a hard learning curve.

You need to assess and sharpen the data engineering and analytics skills we already have.

Data engineering helps you tackle tech challenges head-on, from data sources to pipelines. They help you navigate the technical side, from handling data to finding the right sources. Its really helped me and my team root cause and find data issues that often affect models — as well as engineering better feature engineering and data pipelines.

On the analytics side, it's about the end-users. I've seen smart data scientists build models without thinking about user interaction or product. That's where analytics come in handy. They keep you focused on the users, ensuring the data and models align with their needs. After all, a lot of your ML work builds on what analysts already tackle.

A data scientist is like Swiss army knife, juggling multiple skills. Even if you’re not deploying models daily? You need to make sure that your skilsets are kept sharp. That’s the value in what you and I are doing now. And we need to keep at it.

We’re Unsure of Our Career Path

Managment isn’t for everyone. You might even want to be hands on, and keep building models. But you hit a point where you ask yourself: “Is there something more?”

This is completely normal.

By this point you’ve probably realized that ML/AI models aren’t islands. You realized there’s a larger ecosystem. Building the model is still fun, but understanding how to integrate it and drive value is much more challenging.

Data engineer is one path one option, but its not the only path. If that’s your passion follow it. But know this: there are other options.

One great way? Take on a product, strategy, or technical leader role.

Here’s some interesting roles you can level up to:

AI Product Manager: They act as the bridge between the business side and the tech world. Tuning into user needs, they shape the direction for AI-driven products. Collaborating with data, platform, and business teams, they ensure ML/AI products are both functional and in line with company goals. They continuously evaluate how products are received by users, making refinements based on feedback and performance metrics.

Lead Data Scientist: They possess a deep expertise in technical areas like recommendation systems, NLP, etc. While they maintain a strategic view of the company's ML direction, their core strength lies in the technical nuances of their domain. They guide the data science teams with this blend of strategy, tech skill, and technical leadership.

AI/ML Strategist: Their main focus? Setting the big-picture strategy for AI and ML. By teaming up with data scientists and company leaders, they ensure AI solutions hit business targets. They work with the PMs, lead data scientists, and management to make sure strategies work with the current tech and business teams. They work closely with AI PMs to build coalitions that drive ML/AI product adoption.

You don’t have to have the job title of ML project manager, AI strategist, or lead data scientist. Plenty of data scientists I met were doing this work without the title.

If you want to deliver more ROI and impact, and be invaluable as ML/AI matures, it might be worth looking at these roles. Prepare for the future, as this all starts to accelerate. There's a lot of future value in enabling the operations for AI/ML.

We Still Have a Job - Just not What we Think It Is.

For the last 5 years I’ve seen the role that data scientists take on, transform.

We've shifted from analysts to decision scientists, and then to varying forms of data scientists, engineers, or even ML engineers. I've felt the growing pains, seeing this in real time both as a consultant and employee,

Challenges persist, and expectations differ from reality. And all of us haven’t gotten to do data scientist role we thought we’d take on. Data scientists sometimes find themselves on the back burner to data infrastructure or data engineering tasks.

But there's a silver lining. As these systems and processes mature, there'll likely be a renewed demand for our roles - just in a different title or specialization. Data engineering and analytics engineering underwent a similar path from data analysts. We will be no different.

Position yourself for the future.