, I made a decision I wished to transition from knowledge analyst to knowledge engineer.
Like many individuals beginning out, I used to be overwhelmed by the sheer variety of issues I assumed I wanted to study. Knowledge warehouses, orchestration instruments, distributed processing, streaming methods, cloud platforms, infrastructure. The listing appeared countless.
As a substitute of making an attempt to study every thing directly, I took a special strategy.
I created a 12-month self-study roadmap constructed round one easy concept: study by constructing.
Fairly than leaping from one tutorial to a different, I’d construct a collection of small initiatives. Every challenge would introduce a number of new ideas whereas reinforcing what I had realized earlier than. The purpose wasn’t to construct essentially the most refined purposes potential. It was to develop the behavior of considering like an engineer by fixing one downside at a time.
The primary challenge in that journey was a GitHub ETL pipeline. It started as a easy Python script that fetched repository knowledge and exported it to a CSV file. As I realized extra, I steadily improved it. I changed CSV information with SQLite, made the pipeline idempotent to forestall duplicate knowledge, and ultimately automated it with GitHub Actions.
By the point I completed, I spotted one thing that hadn’t been apparent after I began.
Constructing the ETL logic was truly the straightforward half.
The tougher questions appeared as soon as I ended excited about a script that runs as soon as and began excited about a system that should run again and again with out me watching it.
- How ought to or not it’s scheduled?
- What occurs if it fails midway via?
- The place ought to retry logic dwell?
- How do you package deal the appliance so it runs the identical manner all over the place?
These questions taught me way more about knowledge engineering than parsing JSON or writing SQL ever did.
That first challenge left me with one other problem.
GitHub Actions labored nicely for automating a small ETL pipeline, however I wished to grasp what adjustments once you use a workflow orchestrator constructed particularly for knowledge engineering. I wished to find out how engineers separate orchestration from execution, how containerized workloads match into that image, and what a production-minded pipeline truly appears like, even on a small scale.
So for my second challenge, I made a decision to construct an automatic RSS ingestion pipeline.
On the floor, it’s a reasonably easy utility. It fetches articles from an RSS feed, parses them into structured objects, and shops them in PostgreSQL.
However the purpose was by no means to construct an RSS reader.
The purpose was to discover the engineering choices that rework a Python script right into a dependable knowledge pipeline.
On this article, I’ll stroll via these choices, the errors I made alongside the way in which, and the teachings I realized constructing my first pipeline with Kestra.
Why Construct One other ETL Pipeline?
After ending my first ETL challenge, I thought-about transferring on to one thing fully totally different.
Perhaps an information warehouse challenge. Perhaps Apache Spark. Perhaps an API with a extra advanced transformation layer.
As a substitute, I constructed… one other ETL pipeline.
At first, that most likely appears like a step backwards.
In any case, I’d already constructed an extraction pipeline, made it idempotent, and scheduled it with GitHub Actions. Why repeat the identical train?
As a result of I wasn’t making an attempt to study a brand new dataset. I used to be making an attempt to study a brand new mind-set.
One lesson from my first challenge caught with me.
Writing the ETL logic wasn’t the tough half. The tough half was every thing surrounding it.
- How ought to the pipeline be executed?
- How ought to it get well from failures?
- How do you package deal it so it runs constantly on any machine?
- The place does scheduling belong?
And maybe the largest query of all:
The place ought to the obligations of the appliance finish, and the place ought to the obligations of the orchestration layer start?
These questions don’t have a lot to do with RSS feeds or GitHub repositories. They’re engineering questions, and I spotted I may discover them with nearly any knowledge supply.
That’s why I selected an RSS feed.
Not as a result of RSS is especially thrilling, however as a result of it’s deliberately easy.
The extraction logic solely takes a number of traces of Python. That meant I may spend much less time worrying about enterprise logic and extra time excited about structure.
For a similar purpose, I made a decision to maneuver away from GitHub Actions for this challenge.
GitHub Actions was an incredible introduction to scheduling. It confirmed me learn how to automate a workflow and gave me my first style of working an ETL pipeline with out handbook intervention.
However GitHub Actions isn’t designed particularly for orchestrating knowledge workflows.
I wished to grasp what adjustments once you use a device that’s constructed with knowledge pipelines in thoughts.
That’s what led me to Kestra.
Fairly than asking, “How do I run this Python script each hour?”, I discovered myself asking a special set of questions.
- How ought to retries be configured?
- How are workflow executions tracked?
- How ought to surroundings variables be handed right into a container?
- What does a failed execution seem like?
- How do you separate the appliance from the infrastructure that runs it?
These questions had been precisely what I wished to discover.
By selecting a easy ETL pipeline, I may give attention to the engineering choices as a substitute of getting distracted by difficult enterprise logic.
Wanting again, I believe that was the proper resolution.
This challenge isn’t attention-grabbing as a result of it processes RSS feeds.
It’s attention-grabbing as a result of constructing it pressured me to consider reliability, repeatability, and orchestration in a manner my first challenge by no means did.
The First Architectural Determination: Docker Earlier than Kestra
With the challenge outlined, my first intuition was to leap straight into Kestra.
In any case, orchestration was one of many most important causes I selected this challenge. Why not begin there?
As a substitute, I did one thing that turned out to avoid wasting me numerous frustration later.
I ignored Kestra fully.
That may sound counterintuitive, however I wished to keep away from introducing a number of transferring components earlier than I knew the core utility truly labored.
So I constructed the challenge in layers.
First, I wrote the Python ETL.
Its job was deliberately small. Fetch the RSS feed, parse every entry into an Article object, and save the outcomes to PostgreSQL. Nothing extra.
feed = feedparser.parse(RSS_URL)
articles = parse_feed(feed)
save_articles(articles)That’s actually all of the ETL did. I deliberately saved the appliance small as a result of the main target of this challenge wasn’t the transformation logic. It was every thing that occurred round it.
As soon as that labored reliably, I turned my consideration to the database.
I wished repeated executions to be protected, so I made the inserts idempotent utilizing PostgreSQL’s ON CONFLICT DO NOTHING. That manner, working the pipeline a number of instances wouldn’t create duplicate rows.
INSERT INTO articles (...)
VALUES (...)
ON CONFLICT (id) DO NOTHING;This one line made the pipeline protected to execute repeatedly. Whether or not Kestra ran the workflow as soon as or 100 instances, PostgreSQL grew to become chargeable for stopping duplicate data.
Solely after the ETL and database labored collectively did I introduce Docker.
That call modified how I assumed concerning the challenge.
Initially, Docker felt like one other device I wanted to study.
By the tip of the challenge, I spotted it had turn out to be one thing far more necessary.
It grew to become the unit of execution.
FROM python:3.13-slim
WORKDIR /app
COPY necessities.txt .
RUN pip set up --no-cache-dir -r necessities.txt
COPY . .
CMD ["python", "fetch_rss.py"]Packaging the ETL this manner meant the appliance grew to become self-contained. As a substitute of asking Kestra to grasp my Python challenge, I may merely ask it to run a container that already knew learn how to execute the pipeline.
As a substitute of considering, “Kestra will run my Python script,” I began considering, “Kestra will run my Docker picture.”
That distinction might sound delicate, but it surely fully adjustments the connection between your utility and your orchestration layer.
As soon as the ETL was packaged right into a container, it not mattered whether or not it ran on my laptop computer, inside Kestra, or on one other machine totally.
The runtime surroundings was all the time the identical.
That consistency gave me one thing I didn’t have earlier than: confidence.
Earlier than introducing Kestra, I may run the container manually and confirm that it fetched the RSS feed, linked to PostgreSQL, and continued the anticipated data.
If one thing failed, I knew the issue wasn’t hidden behind one other layer of orchestration.
That validation step turned out to be extremely beneficial later.
Throughout improvement, I bumped into points with networking, container configuration, and workflow execution. As a result of the Docker picture had already been validated independently, I may instantly rule out the ETL itself and give attention to the orchestration layer.
That made debugging dramatically simpler.
Wanting again, I believe this was one of many largest classes from the challenge.
It’s tempting to attach each part collectively as rapidly as potential and hope every thing works.
A greater strategy is to validate every layer earlier than introducing the following one.
On this challenge, the order seemed like this:
- Validate the Python ETL.
- Validate PostgreSQL persistence.
- Validate the Docker picture.
- Lastly, let Kestra orchestrate a container that I already trusted.
Every layer constructed on the earlier one.
By the point Kestra entered the image, I wasn’t making an attempt to debug Python, PostgreSQL, Docker, and orchestration on the identical time.
I used to be solely fixing one downside.
That incremental strategy made the complete challenge really feel far more manageable, and it’s a workflow I’ll most likely proceed utilizing on future knowledge engineering initiatives.
The Assumption That Turned Out to Be Unsuitable
With a working Docker picture, I used to be satisfied the exhausting half was over.
- I had a Python ETL that labored.
- I had PostgreSQL working in Docker.
- I had verified that the container may fetch RSS articles and save them to the database.
Now all Kestra needed to do was run it.
Or so I assumed.
My unique assumption was easy.
Kestra would level to my Python information, execute the script, and every thing would work precisely because it had from the command line.
It didn’t.
I rapidly found that there was an necessary distinction I hadn’t totally appreciated.
Kestra is an orchestrator.
It isn’t chargeable for constructing Python environments or managing utility dependencies. Its accountability is deciding when and the way workloads ought to run.
That realization modified the path of the challenge.
As a substitute of treating Kestra as one other place to execute Python code, I began treating it because the layer chargeable for orchestrating a workload that already existed.
That workload was my Docker picture.
As soon as I made that psychological shift, the structure grew to become a lot cleaner.
- The ETL grew to become a self-contained utility.
- Docker grew to become the deployment artifact.
- Kestra grew to become the orchestrator.
Every layer had a transparent accountability, and none of them wanted to know the way the others labored internally.
Apparently, reaching that time wasn’t fully easy.
My first intuition was to discover a manner for Kestra to execute the Python challenge straight. That strategy sounded easier, however the extra I experimented with it, the extra I spotted I used to be asking the orchestrator to take accountability for one thing the appliance ought to already present.
As soon as I embraced Docker because the execution artifact, the workflow grew to become a lot easier.
Some approaches seemed promising at first however launched pointless complexity. Others labored, however didn’t align with how Kestra encourages containerized workloads to be executed.
Ultimately, I landed on a workflow that felt surprisingly easy.
As a substitute of making an attempt to show Kestra learn how to run Python, I let Kestra do what it does greatest.
It launches a container.
duties:
- id: run_etl
sort: io.kestra.plugin.scripts.shell.Instructions
containerImage: rss-pipeline-etl:newest
taskRunner:
sort: io.kestra.plugin.scripts.runner.docker.Docker
instructions:
- python /app/fetch_rss.pyWanting on the workflow now, what stands out isn’t how a lot YAML it incorporates. It’s how little Kestra truly must find out about my utility. Its solely accountability is to launch a container that already is aware of learn how to execute the ETL.
Inside that container, my utility already is aware of precisely what to do.
That small architectural change solved extra than simply the fast execution downside.
It additionally bolstered an concept that’s changing into a recurring theme in my studying journey.
Good engineering typically isn’t about including one other layer.
It’s about giving every layer a single accountability and permitting it to try this job nicely.
- Python shouldn’t fear about orchestration.
- Kestra shouldn’t fear about Python dependencies.
- Docker shouldn’t know something about RSS feeds.
Every part solves a special downside.
As soon as I ended asking one device to resolve each downside, the complete system grew to become a lot simpler to purpose about.
Wanting again, this was most likely the largest mindset shift in the entire challenge.
I didn’t simply discover ways to use Kestra.
I realized what orchestration truly means.
As soon as a Pipeline Runs Routinely, Every part Modifications
Up till this level, I had been working the pipeline manually.
If one thing failed, I used to be sitting in entrance of my pc. I may learn the error, make a change, and take a look at once more.
That security web disappears the second a pipeline begins working by itself.
One of many first issues I configured in Kestra was a easy hourly schedule.
triggers:
- id: hourly_schedule
sort: io.kestra.plugin.core.set off.Schedule
cron: "0 * * * *"On paper, it was only a cron expression.
In apply, it represented a a lot greater shift.
The pipeline not trusted me remembering to run it.
Each hour, Kestra would begin a brand new execution, launch the ETL container, and course of the newest articles from the RSS feed.
That instantly raised one other query.
What occurs if a type of executions fails?
Throughout improvement, I intentionally launched failures to reply that query. I pointed the pipeline at an invalid database host and watched what occurred.
The primary execution failed, precisely as anticipated.
Extra importantly, it didn’t cease there.
As a result of the workflow was configured with retries, Kestra mechanically tried the execution once more after a brief delay.
retry:
sort: fixed
maxAttempts: 3
interval: PT30SThat was considered one of my favourite moments within the challenge.
As soon as I corrected the configuration, the workflow accomplished efficiently with out requiring any adjustments to the appliance itself.
That was considered one of my favourite moments within the challenge.
Not as a result of retries are significantly difficult, however as a result of they highlighted one other separation of obligations.
The ETL shouldn’t resolve whether or not it deserves one other probability.
That’s an orchestration concern.
By transferring retry logic into Kestra, the Python utility remained centered on a single accountability: course of the feed and persist the outcomes.
The orchestration layer dealt with resilience.
Scheduling launched one other problem that I’d already encountered in my first ETL challenge.
Repeated executions imply repeated makes an attempt to write down knowledge.
If the identical RSS article seems in a number of hourly runs, the pipeline shouldn’t insert it twice.
Happily, I had already solved an identical downside earlier than.
The database layer was designed to be idempotent utilizing PostgreSQL’s ON CONFLICT DO NOTHING.
INSERT INTO articles (...)
VALUES (...)
ON CONFLICT (id) DO NOTHING;That meant each execution may safely try to insert the identical data with out creating duplicates.
The mix of scheduling, retries, and idempotent writes made the pipeline far more forgiving.
If a run failed, Kestra may retry it.
If a profitable retry encountered knowledge that had already been written, PostgreSQL would merely ignore the duplicates.
Neither layer wanted to know what the opposite was doing.
They every dealt with their very own accountability.
The final enchancment was visibility.
Early in improvement, my logs seemed precisely such as you’d count on from a challenge that was nonetheless being debugged.
I printed total RSS objects to the console simply to verify the parser was working.
It wasn’t fairly, but it surely served its function.
Because the pipeline grew to become extra steady, these debug statements grew to become much less helpful.
I changed them with logs that described the execution as a substitute of dumping uncooked knowledge.
Earlier than
print(feed.entries[0])After
print("=== RSS PIPELINE START ===")
first = feed.entries[0]
print("First entry preview:")
print(f"Title: {first.get('title')}")
print(f"Hyperlink: {first.get('hyperlink')}")
print(f"Feed title: {feed.feed.title}")
print(f"Fetched articles: {len(articles)}")
print(f"Saved {len(articles)} articles to the database.")
print("=== RSS PIPELINE END ===")Every run now tells a easy story.
=== RSS PIPELINE START ===
First entry preview:
Title: Christian Ledermann: Migrate From mypy To ty And pyrefly
Hyperlink: https://dev.to/...
Feed title: Planet Python
Fetched articles: 25
Saved 25 articles to the database.
=== RSS PIPELINE END ===These adjustments didn’t make the appliance smarter. They made it simpler to grasp. And that’s an necessary distinction.
Good observability isn’t about producing extra logs. It’s about producing the proper logs.
By the tip of the challenge, I spotted one thing attention-grabbing. The Python code hadn’t grown dramatically. A lot of the work had occurred round it.
Wanting again, many of the engineering effort wasn’t spent making the ETL smarter. It was spent making it extra dependable.
- Scheduling ensured it ran with out me.
- Retries helped it get well from transient failures.
- Idempotency protected the database from duplicate writes.
- Logging made each execution simpler to grasp.
Individually, none of those adjustments had been significantly advanced. Collectively, they remodeled a easy Python script into one thing that behaved far more like a manufacturing system.
The Closing Structure
By the tip of the challenge, the structure had settled into one thing that felt surprisingly easy.
Wanting on the last structure, it’s tempting to suppose the challenge was all the time heading on this path.
It wasn’t.
Every layer was added solely after the earlier one had been validated.
- The ETL got here first.
- Then PostgreSQL.
- Then Docker.
- Lastly, Kestra.
That order mattered.
As a result of each part had already been examined independently, I by no means discovered myself debugging Python, Docker, PostgreSQL, and Kestra on the identical time. Every resolution decreased the variety of unknowns as a substitute of accelerating them.
Extra importantly, each part ended up with a transparent accountability.
- Python is aware of learn how to fetch, parse, and persist RSS articles.
- PostgreSQL is aware of learn how to retailer knowledge safely and stop duplicates.
- Docker gives a constant execution surroundings.
- Kestra decides when the workload ought to run and what ought to occur if it fails.
None of these parts try to do one another’s jobs.
Paradoxically, that’s what made the completed system really feel a lot easier than I anticipated.
What This Mission Modified Concerning the Approach I Assume
Once I began studying knowledge engineering, I assumed the tough half can be writing ETL code.
That’s what most newbie tutorials give attention to.
- You discover ways to name an API.
- You rework the information.
- You reserve it someplace.
Repeat.
These are beneficial expertise, however they’re just one a part of the image.
This challenge taught me that the engineering begins after the script works.
As soon as a pipeline is predicted to run each hour, survive transient failures, keep away from duplicate knowledge, and produce logs that specify what occurred, the questions turn out to be far more attention-grabbing.
You’re not excited about particular person features. You’re excited about methods.
One of many largest mindset shifts for me was understanding the distinction between execution and orchestration.
At first, these concepts felt nearly interchangeable. Now they really feel fully separate.
The Python utility ought to give attention to enterprise logic. The orchestrator ought to give attention to when, the place, and the way that utility runs.
Conserving these obligations separate made the complete challenge simpler to purpose about.
It additionally modified how I take into consideration Docker.
Earlier than this challenge, I noticed Docker primarily as a option to package deal purposes.
Now I see it as a deployment artifact.
As soon as the ETL had been packaged right into a container and validated independently, I may cease worrying about whether or not it will behave in another way inside Kestra.
That confidence turned out to be one of many largest benefits of containerizing the appliance.
Maybe an important lesson, although, had nothing to do with Kestra or Docker. It was the worth of constructing incrementally.
Each main resolution adopted the identical sample.
- Construct the smallest factor that works.
- Validate it.
- Solely then introduce the following layer.
That strategy made the challenge really feel a lot much less overwhelming than making an attempt to attach every thing collectively from the start.
Wanting again, I believe that’s a lesson I’ll carry into each future challenge, whatever the know-how.
Wanting Forward
This RSS pipeline is just the second challenge in my knowledge engineering studying journey. In comparison with my first ETL pipeline, the code itself isn’t dramatically extra advanced. What modified was the way in which I approached the issue.
As a substitute of asking, “How do I write this script?”, I discovered myself asking questions like:
- The place ought to this accountability dwell?
- What occurs if it fails?
- Can I run it repeatedly with out worrying about duplicate knowledge
- Can I belief it to run after I’m not watching?
These questions pushed me to suppose much less like somebody writing Python code and extra like somebody designing a system. And I think that’s the actual worth of constructing initiatives.
Each challenge teaches a brand new device. However the very best initiatives slowly change the way in which you suppose. This one definitely did.
I’m positive the following challenge will problem a totally totally different set of assumptions. Actually, I’m trying ahead to discovering out what they’re.
That is a part of my ongoing collection documenting my transition from methods analyst to knowledge engineer. In the event you’ve been following alongside, thanks.
