out to somebody on LinkedIn I’d by no means met, asking what he thought was the most effective path into information engineering, and whether or not certifications had been even price pursuing. Seems he’d already learn considered one of my articles. So as a substitute of giving me a recent reply, he handed me again my very own. “You’re on the suitable path,” he stated. “You’re following the 12 month roadmap, persist with it. Don’t take options from anybody, it’d confuse you.”
I learn that message twice. As a result of two weeks earlier than that, I had almost talked myself out of the very roadmap I’d publicly dedicated to.
Let me again up.
It’s been a month since I printed my first article on this journey, “From Knowledge Analyst to Knowledge Engineer: My 12-Month Self-Examine Roadmap.” Since then I’ve written three extra, each strolling by one thing I constructed. An ETL pipeline from scratch. That very same pipeline made manufacturing prepared with SQLite and idempotency. Then scheduled with GitHub Actions, which was a lesson on portability earlier than it turned a lesson on scheduling.
These 4 articles are a good report of what I did this month. They aren’t a good report of what the month really felt like. So this one is completely different. No code, no walkthroughs. Simply the elements that didn’t match wherever else.
The plan stated one factor, the month did one other
The roadmap laid issues out so as. SQL first, then Python, then Git, then Spark, then Airflow, then Databricks. Tidy. Sequential. The type of plan that appears nice on a Notion board.
I didn’t observe it. Not likely. As a substitute of shifting by the stack so as, I constructed one small pipeline and stored pushing on it till it broke in new methods. SQL confirmed up. Python confirmed up. Git confirmed up. However not within the order I deliberate, and never as a result of I scheduled it that approach. They confirmed up as a result of the pipeline demanded them.
I used to assume that meant I used to be off observe. Now I feel it simply means the plan was a place to begin, not a contract. The roadmap bought me shifting. The pipeline determined what I really wanted to be taught subsequent.
The partitions had been by no means actually in regards to the instruments
When you learn the construct articles, there’s a sample I didn’t name out on the time. Each wall I hit had a technical repair and a non-technical lesson sitting beneath it.
- Idempotency wasn’t actually about SQLite. It was about studying to mistrust my very own assumption that “it labored as soon as” meant “it should hold working.”
- Persistence wasn’t actually about Google Drive. It was about realizing my work might vanish the second I closed a tab, and that I’d already misplaced a file as soon as earlier than with out noticing how shut I’d come to shedding extra.
- Portability wasn’t actually about GitHub Actions. It was about discovering that one hardcoded path had quietly made my entire pipeline depending on a single setting, with out me ever deciding that on goal.
None of these are coding classes. They’re pondering classes that occurred to floor by code. That looks like the true form of knowledge engineering to this point, a minimum of from the place I’m standing. The instruments are how the teachings present up. They’re not the teachings themselves.
The half the construct articles didn’t present
Right here’s what they didn’t present. Someplace in the midst of this month, the power that got here from folks watching began to fade. Not dramatically. It simply bought quieter. I finished feeling the stress I wrote about in article two, the one the place strangers reached out excited to observe alongside. That stress didn’t disappear as a result of I finished caring. It light as a result of the purpose on the finish of it, touchdown a excessive paying information engineering position, began to really feel distant. Far sufficient that it stopped being motivating and began being exhausting to even take into consideration.
It didn’t assist that my precise job bought busier, and that I needed to be taught Laravel for work on prime of every thing else. Three to 4 hours a day sounds cheap till you’re drained out of your day job and the factor ready for you at house is one other unfamiliar stack.
There have been stretches the place I critically thought of slowing down. What stored me on the desk wasn’t the imaginative and prescient of the job anymore. It was smaller than that. I stored fascinated by the one particular person on the market who’s precisely the place I used to be in Might, caught someplace between analytics and engineering with no clear map, who is likely to be ready on the following article. Serving to that one particular person turned extra motivating than the summary thought of my very own future position. I additionally made a degree of searching for one thing genuinely difficult to fill my free time with, as a substitute of the straightforward, senseless stuff that fills time with out leaving something behind. Even on the times the job felt distant, a minimum of I wasn’t losing the hours.
Nearly getting pulled off beam
I additionally must be sincere about one thing I referred to as out earlier than I even began this journey. In my first article I admitted to having shiny object syndrome, leaping between design, animation, advertising and marketing, IT, and now information, earlier than any of them bought the prospect to stay. I stated I’d must be intentional to keep away from that occuring right here too.
It occurred anyway. Partway by the month I fell right into a rabbit gap about certifications, whether or not I wanted one, which one, whether or not I used to be even on the suitable path in any respect. That spiraled into critically contemplating a pivot towards AI engineering as a substitute, as a result of within the second it seemed shinier and extra in demand. I needed to consciously cease and remind myself of the trail I’d already chosen, and already informed an viewers I used to be dedicated to.
Which brings me again to that message.
The stranger who handed me my very own recommendation
That LinkedIn message I discussed at first got here proper after that wobble. I’d reached out to somebody for recommendation on the most effective path into information engineering, half searching for permission to second guess myself, half questioning out loud whether or not a certification would assist. As a substitute he informed me he’d already learn my roadmap article, and that the reply to my query was sitting in my very own writing. Stick with the plan. Cease amassing opinions from strangers, as a result of too many opinions simply confuse you.
There’s one thing nearly humorous about being talked again into your individual plan by somebody who solely knew about that plan as a result of I’d printed it. However that’s the entire level of doing this in public. The accountability doesn’t solely come from folks watching you construct. Typically it comes again round and fingers you your individual phrases at precisely the second you want them.
What I really realized about how I be taught
The opposite realization this month had nothing to do with motivation and every thing to do with how I work. I used to assume I wanted to construct one thing large to really be taught something. An actual mission, a number of weeks, one thing portfolio worthy from the beginning. That intuition almost killed my momentum greater than as soon as, as a result of large initiatives are straightforward to start out and straightforward to desert midway by.
What’s labored as a substitute is small. A pipeline I can construct and break in a weekend teaches me simply as a lot as a sprawling multi week mission, and I really end it. Going into month two, I’m buying and selling the concept of 1 large construct for a string of small ones. Mini initiatives, sized to suit round a 9 to five and a Laravel deadline, each geared toward a particular idea as a substitute of an entire resume line.
The place that leaves me
When you’d requested me on day one what success seemed like, I might have stated the excessive paying position, full cease. A month in, that’s nonetheless someplace within the image, nevertheless it’s not what’s really getting me to open my laptop computer after an extended day at work anymore.
What’s getting me there now’s smaller, and truthfully extra sustainable. I like constructing issues. I like writing about what breaks. And each time somebody tells me an article really helped them determine their very own subsequent step, that does extra for me than the considered some future job provide ever has.
The roadmap remains to be the roadmap. I’m nonetheless following it, cron job and all. However I’m beginning month two with a clearer sense of what’s really going to hold me by the following eleven, and it’s not the factor I believed it will be once I wrote that first article.
That is a part of my ongoing sequence documenting my transition from methods analyst to information engineer. When you’ve been following alongside, thanks. If that is your first article within the sequence, the sooner ones are linked above.
