For the previous few months, I have been constructing a small knowledge pipeline completely on my Home windows machine. RSS ingestion operating in WSL2, Postgres in Docker, Kestra orchestrating the entire thing, dbt remodeling the information on high. Each piece labored. I had a working lineage graph, passing exams, a scheduled movement that fetched articles and saved them to a database.
All of it labored as a result of it was all sitting in a single place. My laptop computer.
This text is about what occurred after I tried to maneuver it off my laptop computer and onto an actual server, utilizing nothing however the CLI. I anticipated the laborious half to be AWS itself: accounts, cases, networking. It wasn’t. The laborious half was every little thing I did not know I might constructed on high of “it is all operating on the identical machine.” That assumption was doing extra work than I noticed, and taking it away broke issues separately, in ways in which taught me greater than the unique construct did.
Organising the field
Getting a server operating was, truthfully, the simple half.
I created an AWS account on the newer $100-credit free tier, arrange an IAM person for CLI work as an alternative of utilizing root (a small good behavior that saved me from myself greater than as soon as), and launched a t3.small EC2 occasion operating Ubuntu 22.04.
Just a few small surprises got here up alongside the way in which. Nothing main, however sufficient to remind me {that a} cloud server is not simply my pc in a unique location.
The default 8GB disk was too small earlier than I might even began. Resizing it meant operating growpart and resize2fs, and the directions I discovered on-line stored referencing a tool referred to as xvda. My occasion did not have one. Newer occasion varieties use the Nitro system, which names drives issues like `nvme0n1` as an alternative. Small naming distinction, but it surely’s the form of factor that makes you doubt every little thing else you are about to kind, since if the primary command would not even discover its goal, what else goes to quietly not apply.
I additionally noticed the occasion go “impaired” at one level whereas Kestra was pulling its Docker picture for the primary time. My finest guess was that it had run out of reminiscence. I added a 1GB swap file as a bit of additional respiratory room, and it hasn’t occurred since. Mainly, it provides the system someplace to dump reminiscence when the RAM will get full, utilizing disk area as a slower backup. It was a easy repair, but it surely was additionally my first reminder {that a} rented server doesn’t have fairly the identical headroom I’d gotten used to on my laptop computer.
I hooked up an Elastic IP so the server’s tackle would not change each time I ended and began it, and locked down the safety group so solely my very own IP might attain ports 22 (SSH), 8080 (Kestra’s UI), and 5432 (Postgres). That final half became its personal minor recurring chore: my residence web would not give me a hard and fast IP tackle, so most classes began with me re-authorizing no matter tackle I at the moment had earlier than the rest would even load.
None of this was laborious. It was simply a number of small, particular details about this explicit form of pc that nothing in my native setup had ever pressured me to be taught.
Getting the pipeline over
I put in Docker, introduced over my challenge with rsync as an alternative of git, since my repo is public and my .env file has actual secrets and techniques in it, and ran docker compose up -d. Postgres got here up wholesome. Kestra got here up too.
One factor caught me off guard right here: Kestra’s flows do not reside in recordsdata that sync mechanically while you copy a challenge over. They reside inside Kestra’s personal inner database. My one movement, fetch_rss, merely did not exist on the server till I pasted its YAML into the UI myself. It is a small factor, but it surely’s the primary trace of a sample that confirmed up time and again throughout this entire course of: issues that felt like “my challenge” had been truly break up throughout two completely different sorts of state, recordsdata I personal and duplicate round, and inner state that lives solely contained in the operating system.
I clicked Execute. It failed instantly.
The toolbox that wasn’t there
My movement used a Course of process runner for the precise work. It constructed the Python digital atmosphere, put in the dependencies, and ran my ETL script. That was precisely how I had it arrange regionally, and it had by no means precipitated me any issues.
The error was a couple of lacking bundle: python3.12-venv. Digging in, I realized one thing I genuinely hadn’t thought of. The Course of process runner would not run on the EC2 host. It runs inside Kestra’s personal container. Kestra’s container had Python, however not the piece wanted to construct a digital atmosphere.
I fastened it briefly by operating apt-get set up -y python3.12-venv straight contained in the container. It labored, however I knew it wouldn’t survive a restart as a result of the change wasn’t a part of the picture itself. I’d fastened the speedy downside, however not the underlying setup.
That is the place I needed to make an actual choice, and it is the primary genuinely helpful lesson from this entire challenge: the supervisor and the employee should not be the identical factor. Kestra’s job is to resolve when issues run and maintain observe of them. It’s not presupposed to even be the factor constructing Python environments and putting in packages by hand. Each time a process wanted one thing new, I might be rebuilding Kestra’s personal picture to suit, which is a wierd quantity of accountability at hand to your orchestrator.
As an alternative of making an attempt to maintain patching Kestra itself, I moved the precise work out of it and switched the duty to Kestra’s Docker process runner. The thought was fairly easy: Kestra would spin up a separate container only for the Python job, run the ETL script there, and eliminate the container when the job was finished. It was nearer to how I needed the setup to work, though I used to be about to run into a couple of extra issues.
Giving the supervisor a key it did not understand it wanted
For Kestra to launch a separate employee container, it wants to have the ability to speak to Docker itself, the precise daemon managing containers on the host. The way in which you grant that’s by mounting the Docker socket into Kestra’s personal container:
That is value sitting with for a second, as a result of it isn’t a impartial choice. Something with entry to that socket can successfully ask Docker to do something on the host, together with spinning up containers with way more entry than they need to have. Regionally, alone laptop computer, this by no means registered as an actual tradeoff. On a machine sitting on the open web, it is a real one. I made a decision it was acceptable for a studying challenge operating a single low-stakes movement, but it surely’s precisely the form of choice that deserves to be made on goal, not found by chance three steps right into a debugging session.
With the socket mounted, I up to date the movement to make use of the Docker process runner and pointed it at a python:3.12-slim picture. First actual take a look at: Permission denied.
Turned out the socket being mounted wasn’t sufficient. Kestra’s personal course of contained in the container wasn’t operating as root, and that specific door solely opens for root. Having the important thing is not the identical as being allowed to make use of it.
Including person: "0:0" to Kestra’s service definition fastened it, although not earlier than I spent a complicated round-trip discovering that Docker Compose would not at all times rebuild a container simply since you modified a line within the file. docker compose up -d --force-recreate kestra was the command that really made the change take impact. up -d alone quietly determined nothing vital had modified.
The quantity that wasn’t actually there
With permissions sorted, the movement ran additional, and hit a brand new wall: Couldn't open necessities file: No such file or listing.
I might mounted my challenge folder into the employee container so it might see the script and its dependencies. I might show the mount labored by operating the very same docker run command by hand, exterior of Kestra, and it labored completely. No matter Kestra was doing, it wasn’t the identical factor.
This ended up being the longest detour in the entire challenge, and truthfully, most likely the one which irritated me probably the most in hindsight. The worst half was how quietly it failed. I first tried a setting referred to as volume-enabled, however I had it within the unsuitable a part of the config. Then I attempted once more with the right property identify, volumeEnabled—no sprint—and put it in the fitting place, underneath plugins.configurations, focusing on the Docker process runner plugin by its full class identify. Nonetheless nothing.
No error. No warning. Nothing. It simply stored silently ignoring the setting.
Ultimately, I discovered the actual clarification: host-folder mounting for the Docker process runner seems to require Kestra’s Enterprise version. I’d been chasing a setting that merely doesn’t work on the free tier I’m utilizing. It wasn’t a typo or a misconfiguration. I used to be principally making an attempt to open a door that I didn’t even know was locked. The irritating half was that Kestra by no means made that clear—it simply quietly ignored the setting and left me questioning what I used to be doing unsuitable.
I need to flag this particularly for anybody following the same path: when a setting seems to do nothing regardless of how accurately you write it, cease assuming you have got the syntax unsuitable. Test whether or not the function exists within the version you are truly operating.
The way in which that is truly meant to work
The repair wasn’t to pressure the mount to work. It was to cease making an attempt to mount something in any respect.
Kestra has a function referred to as Namespace Recordsdata, primarily a small file retailer that lives inside Kestra itself. You add your challenge’s recordsdata into it as soon as, and Kestra arms them to employee containers mechanically at runtime, no host folder entry required. It is the fully-supported model of what I used to be making an attempt to hack along with volumes.
Importing them from the server became its personal small chain of errors, which by this level within the day felt virtually anticipated. I attempted a PUT request first; Kestra needed a POST with the file wrapped as multipart kind knowledge. I received that proper and received again silence, no affirmation, no error, which I initially learn as success. It wasn’t.
A curl -i flag to really present me the response headers revealed the actual downside: 401 Unauthorized. Fundamental auth had been on the entire time, my browser simply by no means advised me as a result of it was already logged in. curl has no reminiscence like that. Including -u username:password to each request fastened it.
As soon as the recordsdata had been importing, I made a behavior of checking each instantly after, since one early batch try had by some means paired the unsuitable file sizes with the unsuitable filenames. Slower, but it surely meant each mismatch received caught the second it occurred as an alternative of surfacing three steps later as a mysterious script error.
With all six recordsdata confirmed right, I rewrote the movement to make use of `namespaceFiles: enabled: true` as an alternative of a quantity mount, and switched the file paths from absolute (`/workspace/python/…`) to relative (`python/…`), since Namespace Recordsdata land straight within the container’s working listing moderately than wherever I might been mounting issues.
This time, it truly ran the script. Fetched 25 articles. Parsed them. After which:
The final assumption: localhost is not a spot
This was the smallest repair of the entire day, and in addition probably the most trustworthy one, within the sense that it uncovered an assumption I might by no means had purpose to query earlier than.
My database config had DB_HOST=localhost by default. And regionally, that was completely effective. The whole lot was operating on the identical machine, so saying “the database is correct right here” was truly true.
However on AWS, Kestra’s employee was operating in its personal separate container. So when it tried to hook up with localhost, it was principally trying inside its personal little container and saying, “The place’s the database?” Postgres was sitting in a unique container, one which it might attain by way of the shared Docker community utilizing the service identify postgres as an alternative.
I handed the actual connection particulars into the employee container as atmosphere variables, matching them in opposition to the container community we might arrange earlier (networkMode: rss-pipeline_default, the identical community Postgres and Kestra already shared), and the pipeline lastly ran begin to end. Fetched articles, saved them, finished.
The very last item I fastened, instantly after, was the password sitting in plain textual content in that very same config. Kestra has a KV retailer constructed for precisely this, a spot to retailer a price as soon as and reference it from the movement ({{ kv('DB_PASSWORD') }}) as an alternative of writing it out anyplace. Small factor, but it surely felt like the fitting observe to finish on: getting one thing working after which instantly asking whether or not the way in which it is working is one I might be comfy with another person seeing.
What truly broke, and why it issues
Zooming out, each single failure on this course of traced again to one among two issues:
Issues That Labored Regionally however Broke Throughout Containers
“The database is on localhost”, “The Python atmosphere I arrange by hand remains to be right here.”. Regionally, being on the identical machine does a number of invisible be just right for you. You don’t consider these issues as necessities since you by no means needed to. They’re simply there, and every little thing works. As soon as you progress to separate containers, these assumptions disappear, and all of the sudden you understand how a lot you had been counting on them with out even noticing.
Issues That Failed Silently As an alternative of Loudly.
A mistyped config property. A function that does not exist within the tier you are operating. An unauthenticated request that returns nothing as an alternative of an error. None of those crashed something. They simply quietly did lower than I requested, which is a a lot more durable factor to debug than an precise crash, as a result of there is no stack hint pointing on the hole. You simply have to note that one thing did not occur.
If I needed to compress this into one piece of recommendation for somebody doing this for the primary time: when one thing you constructed regionally strikes to an actual server, deal with each assumption about “issues being in the identical place” as a declare you could re-prove, not a truth you get to maintain. And when a repair appears to do nothing in any respect, nonetheless many occasions you double test the syntax, take into account that it would genuinely be doing nothing, as a result of the function is not accessible to you within the first place.
What’s Subsequent
The pipeline itself is finished, operating, and really saving actual knowledge to a cloud database. That was the aim for this piece, and it is genuinely satisfying to kind that sentence after the day I simply had.
I have been going backwards and forwards on what comes after this. A part of me needs to complete the total loop, join rss-transform, my dbt challenge, to this new cloud database, and shut out the RSS pipeline as one full, end-to-end story from uncooked feed to wash, examined knowledge. There’s one thing interesting about that form of closure.
But when I am trustworthy, this challenge has already taught me most of what it got down to train me. I went from “what even is a Docker container” to debugging Docker socket permissions, silent config failures, and container networking, on an actual server, with actual penalties after I received one thing unsuitable. That was the entire level of selecting this challenge within the first place, and sooner or later, squeezing extra classes out of the identical pipeline begins to really feel like staying someplace previous after I’ve truly outgrown it.
So I am genuinely undecided, and I believe that is an trustworthy place to go away this piece. Possibly the subsequent article is the dbt-to-cloud connection, wrapping this sequence up correctly. Possibly it is one thing fully new, a contemporary challenge, a unique set of expertise, a purpose to really feel like a newbie once more in a brand new manner. I do not know but, and I might moderately say that plainly than manufacture a tidy roadmap I am not truly dedicated to.
Both manner, I will be documenting it the identical manner I documented this: truthfully, errors included.
That is a part of my ongoing sequence documenting my transition from techniques analyst to knowledge engineer. Should you’ve been following alongside, thanks.
