How should you be using this outline?

I have met many engineers and students who think that AWS or any platform-specific certification will help them secure jobs.

However, I have never hired an engineer based on AWS certification, though my team uses the AWS tech stack daily.

AWS certification is excellent if you're unfamiliar with their tools and services. However, it is not a testament to a data engineer's skill.

An experienced data engineer knows the best practices of data engineering and how to be adaptable to different business requirements. It is platform agnostic regardless of the cloud services or tools you use.

Certification in services and tools is a bonus and never a core requirement.

If you want to set a path to upskill, then the information below is for you. It is inexhaustive and may differ for different companies, but it will cover most job requirements in the industry.

Contents Overview

• A programming language that is commonly used for manipulating data.

• Batch-based data pipelines

• Streaming data pipelines (Can be optional)

• Big data tools and concepts (Can be optional)

• Data modeling

• Managing relational and non-relational databases

• Continuous Integration / Continuous Deployment (CI/CD) frameworks

• How do we ensure data quality and integrity from source to destination?

Programming language for manipulating data

Any language can be used to manipulate data, but Python is the current industry standard.

Other noteworthy languages include R programming. However, Python takes the cake for its maturity, community support, versatility with Jupyter notebooks, and reputation for production services.

R is excellent for quick prototypes and statistical analysis, but I have not seen any data or machine learning (ML) engineering models deployed in production as a service in R.

Batch-Based Pipelines Overview

A data pipe is a series of processes and logic where data is transferred from source to destination.

Batch-based pipelines are all about transferring data in chunks on a set schedule, and it is one of the easiest to learn because:

• It is usually for business objectives that are not time-critical.

• You control every aspect of the pipe where you can manage load and time; thus, given your resources, you can always experiment optimally.

• Batch-based pipelines are prevalent in every company.

Batch-based pipelines are foundational knowledge for any aspiring data engineer. At the very least, you need to know what is:

• Extract-Transform-Load (ETL) design

• Extract-Load-Transformation (ELT) design

There are other types of pipelines, such as machine learning (ML) pipelines, but if you're aspiring to be a quintessential data engineer, the above is good enough.

Orchestration Tools

Many companies use orchestration tools such as Apache Airflow or Dagster as their default platform.

Orchestration tools help a data engineer ensure that every part of the process is well-defined and decoupled as much as possible.

Pro Tip

• You don't have to master every orchestration tool to be good at batch-based processes. Many concepts, such as writing Directed Acyclic Graphs (DAG), are transferrable.

• Avoid building a deep tree with DAG. The deeper the DAG, the more time it takes for the service to recover if there is an issue whenever you are doing an end-to-end validation.

  • It can be very frustrating to deal with a business-critical issue and wait a long time to validate because the DAG is deep.

• Ensure that every step that you create within a DAG is testable on its own.

  • When a DAG step cannot be isolated for testing, it becomes difficult to troubleshoot when a bug arises.

• Always create repeatable patterns with DAGs. It saves you plenty of time when it comes to new pipes.

Streaming Data Pipelines Overview

When data freshness is critical to the business outcome, you'll need near real-time data capabilities to respond effectively.

For example, you'll need real-time streaming for fraudulent credit card charges because if bad actors are given time, they will cause more financial damage.

Streaming data pipelines usually come in 2 transmission forms:

• Synchronous

• Asynchronous

When I worked at CBS Sports for NFL real-time prediction, the plays we ingested had to reflect the actual live game activity. Thus, the transmission must be synchronous.

Generally, asynchronous transmission is preferred as it is a non-blocking operation. Since there is no need to capture the data in order, you can scale the pipes easily.

Data Streaming Tools

Apache Kafka, AWS Kinesis, and RabbitMQ are some of the tools companies commonly use.

It may be easy to set up a demo to run a data streaming pipe, but recovering a broken service can be very difficult when the business requires high-volume and high-throughput operations. It takes experience and the proper business environment to manage different volume and throughput levels well.

Pro Tip

• Don't let your lack of streaming experience prevent you from applying for jobs. Many companies do not have a good use case for real-time analytics and may not be hiring for this skill.

• If you want to start with data streaming, stream web server logs. It is one of the easiest places to start because it is easy to set up a website with a server and stream these logs into a centralized place.

Big Data Tools & Concepts

Suppose a company or business operates on a very high volume of data daily, in terabytes and petabytes. In that case, it is essential to know some of the big data tools and concepts to be relevant.

Tools such as (inexhaustive):

• Apache Hadoop and MapReduce

• Apache Spark

• Apache Flink

Cloud services provide these tools on-demand, such as (inexhaustive):

• AWS Elastic Map Reduce (EMR)

• Google Cloud Dataproc

Large companies usually have the resources to house and process high volumes of data in the regions on petabytes. At the same time, small and medium businesses typically wouldn't go beyond terabytes of data because it is expensive.

Thus, big data tools are not essential if the company you're working for or applying for does not have a business case for it.

Data Modeling

Data modeling is the process of organizing and representing data in a way that helps to answer business needs and provide actionable insights.

Data stored but unused is an expense. After data collection, data must be preserved and represented in such a way that:

• It reflects business and operational processes accurately on the ground.

• The data is trustworthy, and there are no integrity issues and errors that could impair our interpretation of the data.

• We can scale and manage the data schemas as we collect more data for the betterment of the business.

Data modeling is both an art and a science. There is no absolute way to do things, but there are best practices that can help us optimize our data schemas and structures, which are contextualized to the business.

Pro Tip

• Data modeling has existed for years, and established methodologies exist, such as Kimball data modeling and the Inmon method. Leveraging these resources as your baseline will help you improve quickly at data modeling.

• It takes practice to be good at data modeling, which is often required in tech interviews. To practice, you must familiarize yourself with different business contexts and domains and shape a warehouse or database based on these business contexts.

• No perfect answer exists, so you mustn't struggle with this topic.

  • If you're data modeling at work, the goal is to iterate your experiments and try different modeling methods until the model is optimized for your business use case.

  • If you're trying to ace an interview, the interviewers typically want to know if you have sound data modeling principles. It is not about whether you can give a perfect answer.

Managing relational and non-relational databases

As data is stored within databases, it is imperative to know:

• How to optimize and manage the load on databases, especially if your business case involves high volumes of transactions.

• How to ensure periodic backups so that if your database fails to run and has to be removed, you have a backup to ensure there is little to no data loss.

• The necessary steps needed to recover a failed database service.

While you may not be in a position to create new database clusters because you have a DevOps person to work within your company, it is essential to know what it takes to recover the service so that you're able to facilitate the recovery process.

• DevOps can recover a service from failure but may not have expertise in managing your data. Working with your DevOps person is essential to recover the data and the service.

• Not all DevOps are well-versed in managing database clusters, and as a data engineer, you'll need to provide advice and expertise.

There are two basic types of databases:

• Relational databases

• Non-relational databases

Other storage options or types can exist, but tech interviews typically focus on what is commonly available in the industry. Unless you're interviewing for a role that uses a certain type of database, such as graph databases, there's no need to learn and memorize the entire catalog.

I'll not cover the details of the different types of databases because there is a lot to digest, but to ace your interviews, you'll need to know:

• What are the best use cases of relational databases vs non-relational databases?

• The strengths and weaknesses of each type.

Tech interviews typically won't ask you questions that you can answer by memory through searching online. They will give you a business context and ask you to design a solution to store the data based on the business context.

Your approach and attention to detail reveal everything we need to know to understand your maturity level with these designs.

Pro Tip

• Learning how to set up and manage databases isn't tricky. You can always do it locally on your machine to try things out.

  • However, if you're doing it on a local machine, you won't be able to account for scale and volume in a professional setting.

• You don't need to memorize every use case regarding relational and non-relational databases. The point is to learn a few typical use cases and understand why they are used in their context. From these examples, you can think on your feet when you're asked to design a solution.

  • Suppose you know a few key concepts related to relational databases versus non-relational databases. Applying the most optimal design to a problem based on its principles will not be difficult.

• Many other types of databases are derived from relational and non-relational databases.

  • Once you know the basics, the knowledge and skills transfer to other kinds of databases.

CI/CD Frameworks

When working in a company, you must deploy changes and new features or services through a CI/CD process.

We must version-control these changes to mitigate problems from bad actors or resolve conflicts when too many engineers work on the same code repository.

The goal isn't for you to be able to be a subject-matter expert in CI/CD frameworks. Rather, the goal is to inspire confidence that your deployments to production are working as intended through testing and version control as a team.

For tech interviews, you won't be asked in detail about CI/CD because CI/CD processes are tailored to the company that you work with.

However, you may be asked to describe the high-level processes during an interview within a CI/CD deployment.

Pro Tip

• Companies can use different tools to manage deployments, such as CircleCI or Jenkins. The goal isn't to know every single tool out there. Rather, understand the baseline concepts so you can apply these principles in any company you work in.

• It is possible to simulate CI/CD processes on your local machine or in a sandbox environment in the cloud. If you're new to the topic, this will help you better understand the concepts.

How can data integrity, quality, and timeliness be ensured from source to destination?

Data integrity, quality, and timeliness are vital to any business as they help the company make informed decisions about its value and context. Thus, a testing framework is required to ensure these benchmarks are met.

These testing frameworks can be:

• Built in-house by a company

• Tools in the market, such as DBT

You need to know:

• How can the principles of data integrity, quality, and timeliness be applied to different architectural designs and business use cases?

• What can you do as a data engineer to uphold these principles from a day-to-day perspective?

In a tech interview, questions are uncommon to focus primarily on testing. However, it is relatively common for interviews to ask how you would ensure best engineering practices for data integrity, quality, and timeliness.

Pro Tip

• Anyone can build a testing framework from scratch if they know the tenets of data integrity, quality, and timeliness and have sufficient coding knowledge.

• Never over-test your work where you cannot apply relevant actions to address issues. It is a waste of time because testing does not inherently provide value to a business. Instead, it reduces the risk of your work from failure.

What separates someone great from the good?

These are some of the tell-tale signs that we know if you are excellent at your work:

• The amount of data you manage daily.

  • The larger the data, the more resources you need to process it from source to destination. You'll need to plan around scalability and maintainability to accommodate more overheads, which speaks of experience and skill.

• Able to optimize the architectural design and data processing steps according to the business context and efficiency.

  • Both scissors and chainsaws can cut. However, you wouldn't typically use a chainsaw to cut paper or wood. Experienced engineers know how to design data pipelines according to their context.

• You know the strengths and limitations of each design and framework.

Summary

Data engineering is an ever-evolving subject because more tools and concepts are constantly being developed for new use cases.

However, the fundamentals rarely change, and innovations are often built on top of established practices.

I didn't cover many other topics, such as database change management systems and system health monitoring. However, you'll learn these along the way, and you'll unlikely come across them in an interview.

This article highlights what I have learned in my career, which has lasted over 14 years. Feel free to provide feedback or comment if I missed anything.

• Amazon and the endangered future of the middle manager

• Google CEO Sundar Pichai says search giant has slashed manager roles by 10%

• Mark Zuckerberg's Meta to 'flatten' middle managers in cost-cutting push: report

The list goes on.

Big tech companies tend to set the trend for other companies to follow. In recent years, the Technical Program Manager (TPM) has replaced the Scrum Master (and sometimes even the Product Owner role) in several organizations in the name of efficiency.

It is a watershed moment for the tech industry. Companies are reorganizing their structures for leaner and quicker decision-making processes, and the quintessential engineering manager role is facing similar changes.

I do not foresee the engineering manager role being massively redefined because of the mass layoffs against middle management. However, it is not surprising that the demand for tech lead managers will increase rather than engineering managers in the future and that there will be more opportunities for managers who are relevant to the industry as individual contributors.

Why flatten the management hierarchy?

Fiefdom is one of the reasons why companies want to flatten the management hierarchy:

• Amazon CEO Andy Jassy Says the 'Way to Get Ahead' at Amazon Isn't By Overseeing a 'Giant Team and Fiefdom'

What is fiefdom? An area over which someone exercises control as or in the manner of a feudal lord (Merriam-Webster).

If you have been working in a large corporation for some years, you may have heard teams not prioritizing your requests because they have 'other priorities.'

These 'other priorities' sometimes have an underlying meaning. It is not in the other team's interest to serve your purpose, even though we are supposed to work together for a common goal.

I remember being in a happy hour session with my ex-colleagues, and one of them said, "Someone warned me about you that you might take our lunch because your work might replace ours."

I was working on an iOS and Android tracking package where I could centralize all tracking activity and telemetry into one unit. It made absolute sense for the business to reuse tracking variables across different analytical solutions such as Adobe Analytics and ComScore since the data we send them is the same anyway.

The outcome is to reduce repetitive code and data and to have a single point of failure with unified telemetry.

However, my work worried the mobile teams since it reduced reliance on them to manage and maintain different analytical solutions separately.

This is classic fiefdomism that negatively impacts companies. I cannot count how often these situations happened to me.

Thus, I can understand why the big tech companies are removing middle managers to reduce fiefdoms.

What is a Tech Lead Manager (TLM) and why?

A Tech Lead Manager is required to contribute both as a technical lead and an engineering manager, where you'll have to split your duties as an individual contributor and team leader.

As the name "tech lead" suggests, you're supposed to be the most experienced and skillful in the team to ensure quality implementation and engineering standards.

TLMs typically do not manage a big team because management duties grow with more people under your responsibility. It is designed to stay small for nimbleness and quick delivery.

Why are TLMs preferred over quintessential engineering managers? Quintessential engineering managers are known for being out of touch with code and design after a few years because management and collaboration duties require full-time attention.

As an engineering manager, it is typical for my work days to have at least five or more hours of meetings, discussions, and planning. This can be psychological and emotionally draining, and there were moments when I felt no more capacity to think about code at work after a series of meetings.

How do I plan to reinvent myself as an engineering manager in 2025?

The quintessential engineering manager is here to stay in the tech industry for the time being because they are needed to maintain focus and order and reduce organizational debt within teams.

However, it is clear to me that the trend to reduce middle managers will persist in the coming years as tech companies aim to induce nimbleness and quick decision-making processes.

As much as I want to grow my management portfolio, it seems obvious that if I invested all my energy in this area, I would be swimming against the tide. Thus, it will be important for me to ensure that my skills and experience as an individual contributor make me relevant to the industry.

Thus, in the next phase of my career, I seek to:

• Embark in personal projects for my enrichment and build repositories of code.

• Share the tutorials and learnings of each personal project through my newsletter and other platforms.

• Begin to build relationships and network through my newsletter and other platforms.

In the midst of it all, self-reflection and retrospectives are needed to ensure I'm still focused on what I set out to do and honest with my shortcomings and failures so that I can learn and grow.

Ultimately, the vision is to ensure I'm equally capable of being both an individual contributor and a team leader in data engineering, analytics, and science.

To achieve this vision, I have decided to reboot and trim some of my endeavors so that I can focus.

Rebooting my YouTube channel

I started my YouTube channel right after I completed my Masters degree in Business Analytics from NYU Stern.

At that time, several life milestones happened:

• I got a girlfriend and was engaged in 2021.

• In 2022, I joined PlayStation from CBS Sports and married in the next month.

• My son was born in 2023, and I became a father. It was one of the most rewarding events in my life, and I was glad I could spend time with my wife and son.

• In 2024, I frequently traveled between Singapore and Los Angeles with my family because my family is currently based in Singapore, while my work is fundamentally in the States.

• Amid it all, I taught remotely in two data analytics boot camps at Arizona State University (ASU) and Denver University (DU), respectively.

In short, I barely had time to commit to my YouTube channel simply because I had no capacity. Video creation is time-consuming, and I'm still learning to manage the process.

I decided not to teach for the time being because it did not give me enough time to work on my projects.

To reboot my channel, I will be starting anew by:

• Remove videos that look subpar.

• Reboot and restart tutorials on data-related topics.

I won't commit to a regular upload schedule because my focus is on writing more code as templates for my learning and reusability. Videos will only be considered when I have the time to create them.

However, the YouTube channel is meant for my branding and socialization. Once I sort out the more important matters, I will create a cadence for building content on the channel.

Decommissioning Gefyra.co Site

When I started my Gefyra site over ten years ago, it was meant to be a freelancing site. Later, I tried to brand it as a tech content site, and now I've lost focus on what the site means to me.

Amid it, I uploaded notes for my ex-students when I taught at Denver University, and I used them as part of my teaching material.

Unfortunately, I will bring down the website and content as part of the decommissioning.

If I review some of the materials again, I will include them here as part of my newsletter content.

To my students who are reading this post, I'm sorry that I am going to remove the teaching material I have provided for you. If you want the notes again, please drop me a message on LinkedIn, and I will consider how to post the content.

Focusing on Level Up With Data

Moving forward, Level Up With Data will be my focus on all data-related and tech career topics. The focus will be on tutorials and deep dives since my focus is building up my skills while I work on my career.

Sharing my content will be part of branding myself, and I hope to be as relevant to the industry as possible at every opportunity.

What can you expect out of the newsletter?

The goal is for you to benefit from my experience and share it within the newsletter.

I'm making it as structured as possible for my benefit because I refer to my material whenever I'm rusty on any topic.

Feel free to connect with me on Substack for any questions and comments. Though I primarily write for myself at this juncture, I'm happy to hear from you if you prefer what you want to see in my newsletter. I might learn something from it.

Why was I missing for the past few weeks?

I recently finished my part-time engagement with Denver University (DU) and 2U, where I taught a data analytics boot camp from Feb 2024 to August 2024.

It was my most challenging boot camp because of the relatively low motivation and attrition rate.

I taught at the University of Miami (UM) and Arizona State University (ASU) before, and the classes I have managed thus far were different. The content wasn't too varied, and I have not deviated from my teaching style.

Thus, I reassessed why and recalibrated what I wanted to do in the future from this experience.

I started teaching because I had an arrangement with a friend to teach coding and tech to youth-at-risk girls in Thailand. I had never taught academically before my engagement at UM, and I used that as an opportunity to learn what teaching is like.

It was an enlightening experience because I came in as a permanent instructor midway to another instructor who couldn't afford the time to teach. Since I couldn't wear another person's armor, the students had to adjust to my style.

Then, one of the students lamented why they should be learning to code for data analytics. Even today, coding is one of the significant hurdles most students struggle with as beginners. Some of their computer illiteracy can compound this problem.

I understand that the six-month data analytics boot camp was marketed to beginners who needed only a decent computer with an internet connection. As long as your computer can run the course materials and you can follow the course, you should be ready to embark on a journey to become a data analyst, scientist, or engineer.

However, if it only takes six months to get a six-figure salary in tech, then folks would be banging doors to get to tech boot camps to get started.

The six months cover the foundational materials for anyone to embark on the journey to success in a tech job. Would you think a chef who graduated from a culinary school in six months could manage a leading restaurant in New York?

How did I enter the tech world?

As I recall my journey into the tech world, I took three years to get a bachelor's degree in Computer Science, majoring in multimedia and game development from the University of Wollongong (UOW). In my three years, I built games in Java and C++, explored different ways to hone my craft, and was largely proficient in building an app at any time by the end of the course.

However, being able to build an app did not necessarily mean I was able to contribute value to a product or feature in a company. I joined CBS Interactive (now more commonly known as Paramount Global, and it is going to be acquired soon at the time of this writing), and while I was trying to collect data for my work, I brought the entire office's internet down because of the vast amount of data.

That was in 2011.

It took many trials and errors to finally reach a point where I contributed value to the company in dollars and cents. It was where I could say I was a major contributor to a USD 1.5 million and a USD 1.75 million project that I knew I was contributing above my pay grade.

Why am I recounting this journey? I think many aspiring students were hoping for this scenario to happen:

• Get a certificate or a degree to get a tech job, preferably above six figures, to start

• Learn on the job to be proficient

• Rise through the ranks and explore other opportunities for growth

However, the reality can be as follows:

• I got my certificate or degree.

• I can't find a job because I couldn't pass my tech interviews.

• I need to sign up for additional interview courses to ensure that I can pass that interview.

• However, the difficulty curve keeps growing. Some companies even require some experience for entry-level jobs.

• I have lost money and time on this, and I don't know how long I can last.

This is a common trend nowadays. Then, everyone blames the job market and companies for not offering job opportunities to folks.

Let me offer some perspective so that you know what you can do about it.

Companies are navigating towards candidates who can hit the ground running rather than training from scratch.

Whether we like it or not, companies need to make money to give us compensation and welfare benefits.

If the cost of managing an employee far outweighs the value that he or she brings to the table, the relationship will not work out since, fundamentally, the company needs the money to pay employees.

I managed senior engineers who knew little about automation or version control. They didn't seem to realize that this lack was negatively impacting the team in terms of value, and we ran the risk of scrapping the entire team.

Training was also provided in the form of paid coursework. However, the improvement rate of providing value against our running cost was slow and negative. I had folks placed on training for an entire year, and they still had doubts about whether they could build a data pipe from start to finish.

Obviously, companies understand these problems, and announcing layoffs is a bad PR move. The easier way to avoid this is to raise the quality bar for assessing aspiring job candidates.

Thus, we see a trend where folks mock job descriptions and exclaim that entry-level positions need a few years of experience. It wasn't what companies intended, and I don't believe you need a few years of experience for an entry-level job, but this is the market we live in.

Folks also mistake the term "entry-level" work. Entry-level does not mean you know nothing about the job. You may lack experience, but you should at least have enough coding skills and fundamental engineering principles so I do not have to run coursework to replace the lack of foundational training that was supposed to be provided by your certification or school.

How can you navigate through this hurdle?

Internships help a lot in this area. If you have decent internship experience, it can lower the barrier of entry in terms of experience.

Also, prepare for technical interviews specifically. Your grades don't matter if you don't pass the technical interviews.

It would be best if you had experience building your own apps and prototypes and had the opportunity to show it in an interview.

A tech career is more like a lifestyle rather than just a job.

A chef is often graded by their years of experience because the years of experience are a good indicator of consistency in producing good food. It is close to a tech career.

Often or not, folks become discouraged because of interview rejections. It does hit hard when our efforts to get into a job aren't recognized.

Sometimes, the root of motivation is a problem. Tech jobs are highly paid, and many aspiring candidates consider money the primary factor. If you're out of a job, the pressure to get paid amplifies the problem, and your motivation gets shipwrecked with every new rejection.

However, those who excel in their jobs often don't look only at the money. They spend time and resources beyond their call of duty to hone their craft. Stephen Curry could drill a three from any point on the court because he probably shot thousands of them already.

You have to believe in the craft beyond the monetary compensation. That will give you the motivation and the discipline to succeed, where you have the strength and will to go beyond what others would do to excel in your craft.

Thus, it is not a matter of "I'm feeling lousy because I didn't get it." It should be, "What areas of improvement can I make to be better at my craft?"

Your motivation should extend beyond your job interview. Even after over a decade of experience, I'm still looking at areas where I can improve my craft. It doesn't matter if they are technical skills or in management skills. Even today, I regularly:

• Look at all the recent LinkedIn job descriptions that interest me.

• Make notes on the requirements of what the job descriptions are asking for.

• Document the gaps where I'm lacking.

• Make plans to address those gaps.

Every job has its own requirements. Often, a chef is challenged by the long hours in a kitchen and the consistency he or she needs to bring to the table. A teacher is challenged by their student's behavior and the material they have to prepare for their work to succeed.

The everlasting quest for skills and knowledge challenges a tech career. The beauty of it is that it can give you time for your family and yourself if you know how to manage it. The bad part is that you can never rest on your laurels because what you know today can be obsolete tomorrow.

Self-awareness is the best place to start.

If you do not know where you are on a map, you can never reach your destination, even if you can describe it.

A recruiter asked me, "What is the biggest challenge in leading a team of engineers?"

I didn't hesitate for a second. "Self-awareness".

It is not only about others not having a sense of self-awareness. Even I have my blind spots. You can't improve what you don't know.

Be as honest and objective as you are with yourself. Sure, there are some bad companies that may try to short-change you.

However, blaming them doesn't necessarily help you in your situation. Many times, the ones who are blaming them are still wishing they were part of them.

Rather, find out how you can systematically achieve what you want so that you are where you want to be and there for the long haul. It starts with self-awareness and the willingness to be teachable about it.

What about the recent layoffs in tech?

For many years, the big tech companies were the biggest hirers of talent, and the smaller companies suffered because of this.

There are pockets of opportunities within small and medium-sized enterprises (SMEs) that you could position yourself to leverage.

If you're starting in the tech world, give yourself a few years in an SME and hone your craft. This experience will give your resume some credibility, and it will not be too late to go for a bigger opportunity in the future.

And there are opportunities. My ex-student just proved it by getting a job.

Why?

• If you want your team to be self-organized (this means they can plot and execute work towards business goals without your micromanagement), you want them to own their careers as a first step.

  • I managed engineers who expected to be promoted based on what I told them to do on a task-to-task basis. I will expand on why this is a bad idea for everyone in a future post, but in short, it is not sustainable in the long run, as I had to manage around ten people across the UK and the US at the time of this writing.

• It is to the individual contributor's benefit that they own their career. 

  • I care about them enough to coach them on their personal career vision, but I may not always be their manager forever.

• A career vision keeps everyone focused on what they want to achieve and not be derailed by distractions.

  • For example, I had an engineer who was too focused on coordinating and planning work instead of actual coding and execution. However, the same engineer wanted to be a subject matter expert and be relied upon as a lead.

    • The engineer's daily tasks contradicted his personal vision because his technical skills suffered, and he did more work as a technical program manager (TPM) than as an engineer.

  • It gives me room as a manager to calibrate his focus back to his career vision, and we know we can journey together regarding the team's business vision and goal.

I believe that when the individual contributors in my team align their personal career vision with the team's business goals and vision, the team's outcome and value will be greater than the sum of its parts because it gives the individual contributors an opportunity to express their talents and gifts with minimal supervision.

Thus, I began guiding my team to have their career vision so that they can hold themselves accountable and see for themselves if they are given the room and opportunity to realize their career vision within the team.

How did I craft my career vision?

A career vision is a personal headline to socialize or brand myself to others. It follows the three principles of crafting a career vision in my previous post:

• Understand what a career vision is and what it isn't.

• Anchor the reason for having your career vision.

• Ignore the how when crafting your career vision.

Thus, I created this personal headline, borrowing a template from my career coach (Andrew LaCivita). 

I'm a premier source of aid that helps companies create innovative teams so that through the team's products, they positively impact the community and be rewarded for their service.

In this template, there are three fundamentals in my headline:

• Who are you? 

  • A premier source of aid.

• For whom do I do it?

  • For the company. 

  • The team that I'm managing.

  • The community that we are serving.

• How do others benefit?

  • Positive impact on the community, which meets the company's goals.

  • The team should be rewarded for their service.

Why did I craft this headline for my career vision?

We can only achieve long-lasting success as a team because every individual has a limit.

My late dad was diagnosed with a brain tumor a short while before I was promoted to be a manager in CBS Sports. After finishing up the NFL Google AI assistant for USD 1.7 million, I had to slow my work down and focus on the family.

It was then that I realized that I could not build everything from the ground up to create innovative products. Like everyone else, I have family obligations, health, and other matters in life that I have to focus on in their season.

For the NFL Google AI Assistant project, I had to build the technical parts alone. We were short on time, and I had to make my prototype ready for production in one month. I would have preferred to bring others on board to build the app with me.

Thus, I set my sights on building teams so that we can develop innovative products that benefit the community, which in turn should bring profits and rewards for the company and team.

I enjoy the adrenaline rush of creating and publishing a product for the masses, and thus, I look forward to this aspect of my career.

How did my career vision help me personally?

When I was ready to move on from CBS Sports, I came across a managerial job description from PlayStation that involved managing a team in web analytics.

The interview experience was great, but the interviewers kept hinting that I needed more years to match my managerial profile, even though I was already ten years in the industry.

They liked me enough to offer me a role. Still, they downsized the role into an individual contributor lead and promised to find a way to promote me once I accrued enough managerial experience.

It was tempting because we could negotiate the salary offer to what I wanted, but I had to consider whether I wanted to be an individual contributor.

And the answer was no.

There wasn't a need for me to rush, as I was doing well in CBS Sports and could afford to stay to boost my managerial experience. Although I felt I needed a change after being in the same company for ten years, it was not worth sacrificing my vision for a short-term gain that may or may not lead to the outcomes that I want.

Thus, I turned down the offer and resumed my job search.

A few months later, I applied for another managerial role at PlayStation from another team and at Nike. Both jobs eventually offered me the respective roles, and I ended up in PlayStation as a data analytics and engineering manager.

Had I taken the individual contributor route, I might have delayed my progression to managing and leading a team.

As a professional for over ten years, I have found that developing my skills in any tech field is easy, given time and effort. However, it is only possible to grow my management skills with a team to lead.

A career vision helps you to begin with the end in mind.

The last thing you want to do is to spend all that time climbing a tree, but you discover it is the wrong tree to climb.

There are other ways that a career vision has helped me, which I will detail in future posts, such as:

• Using my career vision to shape the "how".

• Discover the gaps from my current state to my career vision.

• Create a plan of action to address the gaps in my growth toward my career vision.

I will also create data analysts, engineers, scientists, or other profiles in the data world so that you can begin with the end in mind. These profiles will answer the questions that include:

• What kind of core skills would you need to succeed in any company?

• How can you discover the gaps between your skills and experience and the above profiles?

In the future, I might offer one-to-one coaching on aligning your career vision with your job profile, but I'm strapped for time at the moment, so I will let you know through my newsletter. 

Until then, you can look forward to my post on how I used my career vision to address my current needs and develop a strategy and plan of action for growth.

If this is helpful, please share this post with someone who is in need. It will benefit both of us.

The premise focuses on who I want to be rather than the shiny rewards a job or a role can offer.

This resonated with me because I disliked YouTube videos with titles such as "A Day in the Life of a Data Scientist" or A Day in the Life of a High-profile Tech Role. It usually annoys me, especially when the titles include "At Google," "At Facebook," or any renowned tech company.

I am making a sweeping statement here. Still, the few videos that I have seen involve:

• A professional waking up in the morning

• Making coffee

• Going to the office

• Doing some work

• Taking a break here and there

• Walking the dog in the evening before going to bed

Why does it rub me the wrong way? Firstly, it has nothing to do with the job. Instead, it is a promotional piece of lifestyle that this tech role seemingly offers. Let's also consider that data scientists or tech roles are highly paid and sought after. It makes everything look so easy and comforting to be in a tech role.

I'm not suggesting that folks posting such videos misinform others about the role. It is YouTube, a place to get views so creators can make money. And I'm not suggesting that some people in these roles are not living the lifestyle presented in the video.

However, it is a far cry from what the tech role is all about.

Tech roles support businesses. In order to provide value to the business, an element of discipline, knowledge, and skill must occur.

Many folks enrolling in a tech boot camp didn't realize the discipline part of a tech role. I would use another example to illustrate my point. It is not difficult for anyone to learn to cook. However, there is a difference between cooking for a family, cooking for a local restaurant, and cooking for a top-tier restaurant in a hotel as a celebrity chef.

The difference is simply standards. We can accept subpar food from our friends and family, but we definitely cannot when we eat at a restaurant.

This is one of the reasons why many give up in a tech boot camp after some time or give up pursuing a data career. The idea of a tech role that promotes a seemingly easy lifestyle and the discipline to excel in that role is mismatched. There is always a price to pay to excel in any work, regardless of whether it is tech or not.

People told me I was working on their dream job when I led a team to predict sports analytics for CBS Sports. However, only some knew about the sacrifices and discipline I had to make to get there. There were moments when I worked until 4:30 a.m. and sacrificed weekends because I had to finish the product before the deadline. Nobody knew the thousands of dollars and time I spent to get skilled without the company's help.

When the dust settles after I've completed the job, I can sit back, smell the roses, and enjoy life, just like the YouTube videos depict.

The idea of a career vision is to ensure your reason for getting into tech will fuel your determination when your legs feel heavy on your journey because being in a tech role isn't a sprint but a marathon.

How do you craft a career vision?

First, understand what a career vision is and what it isn't.

A career vision is a purpose and outcome you set for your success and personal fulfillment. No one else but you can create it.

Folks typically confuse a position of influence as a purpose and an outcome. I have engineers who told me they want to be a tech lead or be promoted to a senior position. There's nothing wrong with wanting to have a position of influence, but positions are a means to an end and not merely an end by itself. Positions should serve a purpose and a calling in your season.

What happens if you're not promoted to the position that you want? Do you then lose your career vision as a result?

A career vision should be steadfast enough to help you navigate in times of ambiguity and uncertainty. Tying it to a position in a company may discourage you from the purpose that you set out to be when you're not where you want as presently constructed.

Also, people assume that career growth should always be positive and linear. However, most of us experience moments of rise and fall, just like the S&P 500 stock, but it should pan out positively at the end of a few years.

As how Jeff Bezos said, "We are stubborn on vision and flexible on details." Tying your career vision to a specific position in a company can hinder you from being flexible.

Second, anchor the reason for having your career vision.

Anchoring a career vision is about anchoring the reason why you're embarking on a data career or a tech role in the first place. It should be something that can withstand challenges, such as:

• Do you want it bad enough that you're willing to sacrifice for it?

• Are you willing to learn whatever you require to succeed, including learning to become self-sufficient before requiring additional support?

• Do you believe in yourself no matter what?

• Can you remain focused because your love for the purpose remains the same even if everything around you could change?

When Andrew LaCivita (I'm trying to give him credit here) posed these questions to me through his course, it was a resounding yes.

• I invested USD 80000 in my NYU Masters of Business Analytics. Even before investing in the master's program, I spent money on equipment, cloud services, and training programs to stay on top of my game.

• I moved from Singapore to the United States to ensure I anchor my career.

• I was OK with consistently putting in the effort and hours to self-study, prototype, and build workable artifacts so that I could anchor my role in this field.

Does it mean I don't have more important things in my life outside of my career? Of course, I do! I have my family, and I do have other matters that I'm more passionate about. Like everybody else, I'm working hard to support my family financially, and being in a tech role that pays well certainly helps.

However, if my reason is only on monetary rewards, it is easy to give up and find another role outside of tech that could provide financially well for me and my family.

Does it mean I'm working so hard that I'm neglecting the other essential things in my life? Certainly not! It is a matter of pace and consistency that I'm willing to take to excel. I don't see the need to burn every weekend to excel, but I do see the need to organize my life to reserve enough time to dedicate to my craft and not compromise on my other aspects of life.

The above hard-hitting questions are designed to ensure you are sure of your path, even though you may not have all the answers.

Third, ignore the how when crafting your career vision.

Too often, we let the "how" discourage us from our career vision because we simply cannot see how to achieve it from where we are.

On many occasions, the hurdles seemed insurmountable. Perhaps you're discouraged because others seem to have an easier learning curve than you are when you're trying to get started in your data career. Or the lack of finances or time makes you feel you're behind.

Bill Gates famously said, "People overestimate what they can do in one year and underestimate what they can do in 10 years."

With time, the "how" can be worked out eventually. Craft the vision without figuring out how to get there first, and once you believe in your vision enough, you'll be able to figure out the rest with tenacity and hard work.

That's why I started my newsletter: to help others achieve their data careers by providing the "how" and the details.

I started my career in Paramount Global in Singapore, formerly CBS Interactive, before the merger with Viacom on April 6th, 2011. I was hired as an engineer for data analytics and search engine optimization (SEO).

I will refer to my time in Paramount Global as CBS Interactive (CBSi) to avoid confusion.

One of the main perks of being in CBSi for my role was learning cutting-edge technologies. I was exposed to Hadoop, recommendation systems and search engines, and Adobe Analytics very early, which really helped in giving me a good breadth of what is out there.

However, being an offshore office in Singapore, growth opportunities in promotion were few. For some reason, even though I contributed and was involved with some big projects with good business impact, promotion wasn’t accessible as there wasn’t availability.

Couple that with the fact that there was a reorganization within the company every one or two years. I later realized that it hurt my career. That’s because on every occasion, I would have a new manager or vice president preside over my role, and it became apparent that I had to prove my worth in the new team.

I still remember when the vice president (VP), who presided over my team, visited the Singapore office. I had a Singaporean manager, and the team hired a director in the States to oversee Adobe Analytics rollouts.

It turns out that the new director didn’t know what he was doing, and because my manager was better in experience and skills, he had to redo my director’s work.

In my one-on-one with my VP then, I asked why she did not promote my manager as a director and had a less inexperienced director to run some of the projects instead.

She said, “We needed someone in the States to do the job.” It then dawned upon me that it didn’t matter how well my manager did then because as long as he was not in the States, he would not be offered the director role.

If even my excellent manager wasn’t given the opportunity to promote, then what about me?

From that day on, I knew I had two options. Either to move to the States to further my career within CBSi or I should leave for greener pastures.

I prayed to God about experiencing a business trip in CBSi. It was one of the perks that attracted me to the role, but I had already been in the company for four years, and I had seen my colleagues travel for work, but I had yet to have the experience.

Shortly after, I went on my first business trip to San Francisco, and at that time, there was another reorg. The VP I worked with became the VP of tech at CBS Sports, and that was when I asked if there were opportunities for an internal transfer.

I also highlighted that I wanted to dive into data science if I was given the opportunity to join CBS Sports since I was working in the central data team with CBSi then and had some experience.

It so happened that there were opportunities to expand into the gambling space within CBS Sports, and we could do some predictive products for work. I flew to Fort Lauderdale, Florida, to meet with the department, and the VP gave me an opportunity to work in Florida so that we could be near to work.

At that time, I was interviewing for a few jobs simultaneously. I already had an offer from a local telco company in Singapore while in the final interview for Facebook.

Eventually, I selected to go to the States, partly because while I was in the Facebook interview, the interviewers seemed perplexed about why I would want to be in Facebook Singapore if I had the chance to relocate to the States and have a more extensive portfolio then what I was interviewing at that time.

So, I left Singapore for Fort Lauderdale, Florida, in August 2016.

Slowly but surely, I was given opportunities to prove my worth within the company, and it was a good move because the reorgs that happened in CBSi later did not affect my role. I was tasked to build the Microsoft Golf Forecaster, a campaign for approximately USD 1.5 million, and the NFL Google Assistant for about USD 1.75 million.

After delivering the projects, during my one-on-one with my manager, I said I would quit if I didn’t get promoted. My manager quickly assured me a promotion was coming, but it took some time.

That’s because my jump was from a data engineer to a data engineering manager for data science; they had to work with HR since they were skipping a few levels for my promotion.

The NFL Google Assistant was very pivotal in getting me a promotion. At that time, the company needed to be more confident that we had the skills and expertise to deliver the project, and they wanted to outsource the work to a third-party vendor.

The third-party vendor asked for half a million.

I still remember being relatively upset because I was already working to build the voice assistant from scratch, and the other directors could sense my mood during our meeting with the third-party vendor.

In short, when they asked me to introduce myself, I said, “I’m the person who makes magic in CBS Sports.”

Please don’t do what I did, though. I don’t know what would happen to you if you were to show an attitude.

Ultimately, the company trusted my work as my prototype was nearly complete. I heard that teams from other companies were vying for the Google opportunity, but Google gave it to CBS Sports since they saw how close to completion my work was.

Once we clinched the deal, it didn’t matter if I had to work weekends and late weekdays to get the work out of the way. I had a USD 1.75 million target board behind my back, and I wasn’t going to sleep on it.

Once it was delivered, many of my colleagues and leaders commented that they wanted to build a team around me, which truly allowed me to become a data engineering manager for data science.

So, thank you to CBS Sports for the opportunities I had and my colleagues who believed in my work. If it weren’t for their belief in my work and allowed me to deliver, I wouldn’t have had the opportunity to become a data engineering manager.

Level Up With Data

Coaching and helping you grow in your data career.

By Jonathan Moo

The course workload is sometimes too intense, with a vast amount of information to be absorbed in a relatively short period.

The curriculum is broadly relevant to the industry, and anyone with a decent computer and internet connection can apply to the course. We covered basic instruments for data wrangling, such as Excel, and then moved on to more advanced topics, such as Python, data visualization, databases, statistics, machine learning, and Big Data.

The course sums itself up in 24 weeks, approximately six months.

It is a reasonable time frame to complete a course and quickly find a job if you have been looking for an opportunity for some time. After all, few have the luxury of going without decent pay for so long.

However, with only 24 weeks, it is an informational dump to the students, who often need help with the time pressure and grasping the essence of the course.

To add even more pressure, some had never used a command-line tool before the start of the course. Some needed help figuring out where to find the directories and files they had kept within their machines. In short, there were moments when computer literacy was genuinely lacking, and it demoralized some students even to follow the course content.

As a seasoned professional for over ten years, I knew the course material wasn't the be-all and end-all. Any data engineer or analyst who is worth their salt knows that the topics taught have depths they need to reach to be relevant for an entry-level position.

The actual litmus test is whether I would hire any of them as entry-level professionals on my team. I was an ex-manager for data engineering and science for sports predictive data in CBS Sports, and currently, I'm a data analytics and engineering manager at a global brand.

I longed for a solution to solve this predicament, and I identified a couple of problems that needed addressing:

• The students need more time to digest content and be able to internalize their skills as part of them.

• They need coaching to identify their gaps and be able to address them.

• They need a like-minded community for encouragement and persistence.

Thus, I'm keen to start a space and community for folks who want to kickstart their data careers more sustainably and organically so that they can make it part of their lives.

That's why I am starting a newsletter to drip properly prepared and curated content for anyone who wants to kickstart their data career. 

It is not designed to be fast. If you need speed, a boot camp can suffice.

It is designed to be consistent, and you hold the reins of whether you should be part of a journey or not. My success would be if you didn't need my content to further your career.

My service to the community would be more than just a newsletter in its final form. Jeff Bezos's famous maxim, "Obsess our customers," is one of the team maxims I have established within my analytical and engineering teams over the years. As I create a strategy and framework to kickstart your data career, look out for additional helpful services and resources in the future.