. Part of me began this journey as a result of knowledge engineering is among the hottest and highest-paying careers proper now. I’m not going to fake that wasn’t an element.
However there’s extra to it than that.
I’ve been studying knowledge analytics for some time now. SQL, Energy BI, Python (Pandas, NumPy, somewhat Polars), knowledge cleansing, EDA. You title it, I’ve been within the weeds with it. And I genuinely get pleasure from it. However someplace alongside the way in which, I began getting interested by what occurs earlier than the info lands on my desk. How does it transfer? Who builds these pipelines? What does the infrastructure behind all of this really appear like?
That curiosity planted a seed.
Then AI began making a variety of what I do quicker and simpler. Which is nice. Nevertheless it additionally made me assume: if AI can deal with the evaluation, what’s my edge? What can I construct and perceive that goes deeper? I work as an IT System Analyst at a startup, and whereas I benefit from the work, I noticed I wasn’t difficult myself the way in which I needed to. I used to be prepared for extra.
The ultimate push got here from a video by Knowledge With Baraa, the place he laid out a whole knowledge engineering roadmap. One thing about seeing it structured and damaged down made it really feel actual and doable. So right here I’m.
I’m studying knowledge engineering in public. And this text is the start of that journey.
Additionally, simply leaving a disclaimer that I’m not affiliated with Knowledge with Baraa. I’m simply sharing my private journey. Hope it helps.
Why Knowledge Engineering Particularly
I wish to spend a second right here as a result of I believe this query deserves an actual reply.
Knowledge analytics taught me easy methods to work with knowledge after it arrives. Clear it, discover it, visualize it, draw insights from it. That skillset is genuinely beneficial. However the extra I realized, the extra I stored bumping into the identical wall. The info I used to be working with had already been formed and moved by another person. Somebody had constructed the pipeline that introduced it to me. Somebody had determined the way it was saved, the way it was structured, how typically it refreshed.
I needed to be that particular person.
Knowledge engineering sits upstream from analytics. It’s about constructing the programs that make evaluation attainable within the first place. Knowledge pipelines, storage structure, workflow orchestration, large-scale knowledge processing. These are the foundations every thing else is constructed on. And truthfully, that form of infrastructure work appeals to me in a method that pure evaluation now not does.
There’s additionally a sensible argument. Knowledge engineering roles persistently rank among the many highest paying within the knowledge trade. As AI instruments get higher at automating the analytical layer, the demand for individuals who can construct and keep dependable knowledge infrastructure is barely going to develop. I’d moderately be constructing the pipes than simply utilizing them.
And yet one more factor. The startup I work at doesn’t use any of the instruments I’m about to be taught. Which implies each hour I put into that is totally self-directed. No workforce to be taught from, no work tasks to use it on. Simply me, the web, and no matter I can construct alone. That’s a problem I’m selecting on function.
Why I’m Doing This in Public
Writing about what I be taught is one thing I already consider in deeply. It forces you to really perceive one thing earlier than you clarify it. It retains you accountable. And over time, it builds one thing {that a} resume alone by no means might.
However I’ll be sincere about my fears too, as a result of I believe that’s the purpose of doing this publicly.
I’ve shiny object syndrome. There, I stated it. I’ve explored graphic design, animation, writing, advertising, and IT earlier than touchdown in knowledge. There’s at all times one thing new and thrilling pulling my consideration. Knowledge engineering might simply get changed by the subsequent flashy factor in my feed if I’m not intentional about it.
Consistency is one other one. I work a 9-5 the place I barely contact the instruments I’ll be studying. There’s no pure reinforcement at work, no colleague I can bounce Airflow questions off of. I’m constructing this totally alone time, outdoors of my job obligations.
And steadiness. Three to 4 hours a day is the aim. Some days that can really feel simple. Different days it’ll really feel unimaginable.
Publishing this journey is my accountability system. If I’m going quiet, you’ll know I slipped. And I’d moderately not slip.
What I’m Beginning With
I’m not ranging from zero, which helps. I have already got newbie to intermediate SQL information from my knowledge analytics work, fundamental Python fundamentals, and a few hands-on expertise with Pandas. That offers me a basis to construct on moderately than rebuild from scratch.
Right here’s the total studying stack, roughly within the order I’ll be tackling it.
1. SQL: Going Deeper Than Analytics
I do know SQL. However analytics SQL and engineering SQL are completely different animals. I’ll be going deeper into question optimization, indexing, working with very massive datasets, and writing SQL that’s constructed for efficiency moderately than simply exploration. For those who’ve solely ever used SQL to drag and filter knowledge, there’s an entire different layer beneath value understanding.
Why it’s first: Every part in knowledge engineering ultimately touches SQL. Getting sharp right here earlier than layering in additional advanced instruments makes the remainder of the journey simpler.
2. Python: From Exploratory to Manufacturing-Prepared
I’ve the fundamentals. Pandas, NumPy, some Polars. However the Python I’ve been writing lives largely in notebooks. Exploratory, messy, not constructed to final. The aim now could be to write down cleaner, extra structured, reusable code. Features, modules, error dealing with, scripting. The form of Python you’d really put in a pipeline.
Why it issues: Python is the glue that holds most fashionable knowledge engineering stacks collectively. Airflow makes use of it. PySpark is constructed on it. Getting comfy right here is non-negotiable.
3. Git and GitHub: Model Management Finished Correctly
I’ll be sincere. My Git information is at present “copy the command, hope it really works.” That has to vary. Model management is prime to working like an engineer moderately than simply an analyst. I’ll be studying branching, pull requests, and easy methods to handle code correctly throughout tasks.
Why it issues: Each challenge I construct from right here on goes on GitHub. It’s portfolio, it’s self-discipline, and it’s how actual groups work.
4. Apache Spark and PySpark: Large Knowledge Processing
That is the place issues get genuinely thrilling. Apache Spark is among the most generally used engines for processing large-scale knowledge. PySpark is the Python API for it, which suggests I can use a language I’m already considerably conversant in to work with distributed knowledge at scale.
The soar from Pandas to Spark is a mindset shift. Pandas works on a single machine. Spark is constructed to run throughout clusters. Studying to assume in that distributed method is among the expertise that separates knowledge engineers from analysts.
Why it issues: If you wish to work with huge knowledge in a manufacturing setting, Spark is sort of unavoidable. It reveals up in job descriptions consistently and is core to the Databricks ecosystem I’ll be constructing towards.
5. Apache Airflow: Orchestrating Knowledge Pipelines
Knowledge pipelines don’t run themselves. You want one thing to schedule them, monitor them, and deal with failures gracefully. That’s the place workflow orchestration instruments are available in, and Airflow is my choose.
I thought of a couple of choices right here. Databricks Workflows is nice should you’re already deep within the Databricks ecosystem. Azure Knowledge Manufacturing unit is sensible for Azure-heavy environments. However Airflow is free, open-source, cloud-agnostic, and extensively used throughout the trade. It additionally teaches you the core ideas of orchestration in a method that transfers to different instruments. Beginning with Airflow felt like the fitting name, particularly since I’m making an attempt to maintain prices low.
Why it issues: Orchestration is what turns a set of scripts into an precise pipeline. Understanding Airflow is knowing how manufacturing knowledge workflows are managed.
6. Databricks: The Knowledge Platform
In some unspecified time in the future it is advisable to choose an information platform and go deep on it. I’m going with Databricks. It’s constructed on high of Spark, it’s in excessive demand, and it has a free Group Version that permits you to follow with out paying for cloud credit.
The alternate options are strong too. Snowflake is a clear, quick SQL warehouse that a variety of corporations love. BigQuery is Google’s totally managed, serverless choice and genuinely glorious should you’re leaning towards Google Cloud. However Databricks sits on the intersection of massive knowledge, machine studying, and knowledge engineering in a method that matches the place I wish to go. It made probably the most sense for my targets.
Why it issues: Employers need you to have platform expertise. Going deep on one is extra beneficial than realizing somewhat about all of them.
How I’m Structuring the 12 Months
The sincere reply is that this would possibly take longer than 12 months. And I’m okay with that. I’d moderately take 15 months and really perceive what I’m doing than rush by way of in 12 and are available out shaky on the basics.
The final strategy is to maneuver by way of every ability so as and never advance till I’ve constructed one thing with what I simply realized. Tutorials are nice for orientation however tasks are the place actual studying occurs. My plan is to doc every section right here on In the direction of Knowledge Science: the ideas, the tasks, the frustrations, and the wins.
For monitoring progress, I’m utilizing the Notion roadmap from Knowledge With Baraa as my spine. It breaks down every ability into core subjects and lets me observe the place I’m with out getting overwhelmed by the total image all of sudden.
As for time dedication, three to 4 hours a day is the goal. A few of that shall be structured studying. Some shall be constructing. Some shall be writing about what I simply realized, which is its personal type of learning.
What Success Seems Like
Touchdown a high-paying knowledge engineering function is the aim. That’s actual and I’m not going to decorate it up.
However alongside that, I wish to change into a reputable voice on this house. Somebody who builds issues value speaking about, paperwork the journey with out filtering out the laborious components, and possibly makes the trail somewhat clearer for somebody arising behind me.
The writing and the educational feed one another. The portfolio turns into the proof. The proof builds the model. That’s the imaginative and prescient.
Beginning At the moment
This text is my official begin date. I’m not ready till I really feel prepared or till every thing is completely deliberate. I’m beginning now, writing as I’m going, and letting the method be public and somewhat messy.
For those who’re someplace on an identical path. Whether or not you’re in analytics fascinated by engineering, in IT questioning what’s subsequent, or simply somebody making an attempt to construct expertise that maintain their worth in an AI-accelerated world. Observe alongside.
I believe we’ll have loads to speak about. I’ll even be sharing my learnings on my YouTube channel. So be at liberty to subscribe beneath and observe alongside.
That is the primary article in an ongoing sequence documenting my knowledge engineering journey. I’ll be publishing recurrently on my progress, the tasks I’m constructing, and every thing I be taught alongside the way in which.
And if you wish to get entry to the Notion template, in case you’re on the identical journey as I’m, you’ll be able to entry it right here.
Observe alongside on my journey beneath.
