If me for some time, you in all probability know I began my profession as a QA engineer earlier than transitioning into the world of information analytics. I didn’t go to highschool for it, didn’t have a mentor, and didn’t land in a proper coaching program. Every part I do know in the present day—from SQL to modeling to storytelling with information—is self-taught. And consider me, it’s been a journey of trial, error, studying, and re-learning.
The Dilemma That Modified My Profession
Just a few years in the past, I began fascinated with switching organizations. Like many individuals in fast-evolving tech roles, I confronted a surprisingly tough query:
What function am I truly doing? Which roles ought to I apply for?
On paper, I used to be a Knowledge Analyst. However in actuality, my function straddled a number of features: writing SQL pipelines, constructing dashboards, defining KPIs, and digging into product analytics. I wasn’t certain whether or not I needs to be making use of for Analyst roles, BI roles, or one thing fully completely different.
To make issues worse, again then, job titles had been imprecise, and job descriptions had been bloated with buzzwords. You’d discover a posting titled “Knowledge Analyst” that listed necessities like:
- Construct ML pipelines
- Write complicated ETL scripts
- Preserve information lakes
- Create dashboards
- Current executive-level insights
- And oh, by the way in which, be nice at stakeholder administration
It was overwhelming and complicated. And I do know I’m not alone on this.
Quick ahead to in the present day: fortunately, issues are evolving. There’s nonetheless overlap between roles, however organizations have began to outline them extra clearly. On this article, I need to break down the actual variations between information roles, by the lens of a real-world instance.
A Actual-World State of affairs: Meet Quikee
Let’s think about a fictional quick-commerce startup known as Quikee, launching throughout a number of Indian cities. Their worth proposition? Ship groceries and necessities inside 10 minutes.
Prospects place orders by the app or web site. Behind the scenes, there are micro-warehouses (additionally known as “darkish shops”) throughout cities, and a fleet of supply companions who make these lightning-fast deliveries.
Now, let’s stroll by the info wants of this firm—from the second an order is positioned, to the dashboards executives use of their Monday morning conferences.
Step 1: Capturing and Storing Uncooked Knowledge
The second a buyer locations an order, transactional information is generated:
- Timestamps
- Order ID
- Objects ordered
- Value
- Low cost codes
- Buyer location
- Fee technique
- Assigned supply accomplice
Let’s assume Quikee makes use of Amazon Kinesis to stream this information in actual time to an S3 information lake. That stream is high-volume, time-sensitive, and essential for enterprise monitoring.
However right here’s the catch: uncooked information is messy. You’ll be able to’t use it immediately for decision-making.
So what occurs subsequent?
Step 2: Constructing Knowledge Pipelines
Enter the Knowledge Engineers.
They’re accountable for:
- Ingesting real-time information
- Validating schema consistency
- Dealing with failures and retries
- Writing pipelines to maneuver information from S3 into a knowledge warehouse (say, Snowflake or Redshift)
That is the place ETL (Extract, Rework, Load) or ELT pipelines come into play. Knowledge engineers clear, format, and construction the info to make it queryable.
For instance, an order desk would possibly get cut up into:
- Orders → One row per order
- Order_Items → One row per merchandise in an order
- Funds → One row per fee try
At this stage, uncooked logs are was structured tables that analysts can work with.
Step 3: Dimensional Modeling & OLAP
As management begins asking strategic questions like:
- “Which metropolis brings in essentially the most income?”
- “Which retailer is underperforming?”
- “What’s our common supply time by zone?”
…it turns into clear that querying transactional information immediately received’t scale.
That’s the place dimensional modeling is available in.
As an alternative of flat, uncooked tables, information is structured into Reality and Dimension Tables.
🔸 Reality Tables
- Massive, quantitative information tables which include overseas keys together with measures and metrics (Nicely, more often than not. There are factless reality tables as properly which should not have any measures).
- Examples:
fact_orders,fact_payments,fact_deliveries - Comprise metrics like income, order depend, supply time
🔹 Dimension Tables
- Smaller, descriptive tables that assist perceive the info in a reality desk
- Examples:
dim_store,dim_product,dim_customer,dim_delivery_agent - Assist filter, group, and be a part of information for deeper insights
This construction allows OLAP—quick, analytical querying throughout a number of dimensions. For instance, now you can run queries like:
“Present me common supply time by retailer and hour of day, over the past 7 days.”
This step is completed by Knowledge Engineers at many of the organisations however I did construct few Dim and Reality tables once I was working as a Enterprise Intelligence Engineer at Amazon.
Step 4: Defining KPIs and Metrics
That is the place Analytics Engineers (or BI Engineers) shine.
They sit between the technical information layer and enterprise customers. Their obligations usually embrace:
- Defining KPIs (e.g., churn charge, repeat buy %, time-to-fulfillment)
- Writing logic for complicated metrics (e.g., cohort retention, lively customers)
- Creating semantic fashions or metrics layers in instruments like dbt or Looker
- Guaranteeing constant definitions throughout the corporate
For instance, at Amazon, our group didn’t question uncooked information to calculate income each time. As an alternative, we created pre-aggregated reality tables at each day, weekly, and month-to-month grains. That manner, dashboards loaded sooner, and metrics stayed constant throughout groups.
Analytics Engineers act as translators between engineering and the enterprise—defining what we measure and how we measure it.
Step 5: Evaluation, Reporting & Storytelling
Now comes the function of the Knowledge Analyst.
Armed with clear, modeled information, they give attention to answering actual enterprise questions like:
- “Why did retention drop in Bangalore final month?”
- “Which coupon codes drive essentially the most new customers?”
- “What are the highest merchandise reordered within the first 30 days?”
They construct dashboards in instruments like Tableau, Energy BI, or Looker. They run ad-hoc SQL queries. They dive into A/B take a look at outcomes, consumer habits traits, and marketing campaign effectiveness.
However above all, they inform tales with information—making complicated numbers comprehensible and actionable for stakeholders.
Who’s Who?
TL;DR: The place Do You Match?
Right here’s how I give it some thought:
- Love constructing strong pipelines and fixing scalability issues? → You’re a Knowledge Engineer
- Love defining enterprise metrics and organizing complicated datasets? → You’re an Analytics Engineer
- Love uncovering insights and storytelling with information? → You’re a Knowledge Analyst
After all, real-world roles usually mix these. Particularly at smaller corporations, chances are you’ll put on a number of hats. And that’s okay.
The secret’s not the title—however the place you add essentially the most worth and what energizes you.
Last Ideas
It took me a very long time to know what I truly do—not simply what my job title says. And if you happen to’ve ever felt that confusion, you’re not alone.
As we speak, I can clearly say I function on the intersection of information modeling, enterprise logic, and storytelling—a candy spot between analytics and engineering. And I’ve realized that the flexibility to attach the dots is extra essential than becoming into an ideal field.
In the event you’ve walked an identical path—or put on a number of hats in your function—I’d love to listen to your story.
Drop a remark 👇 or share this with somebody figuring it out too.
