feeling a continuing sense of AI FOMO. Each day, I see folks sharing AI suggestions, new brokers and abilities they constructed, and vibe-coded apps. I’m more and more realizing that adapting rapidly to AI is changing into a requirement for staying aggressive as an information scientist in the present day.
However I’m not solely speaking about brainstorming with ChatGPT, producing code with Cursor, or sprucing a report with Claude. The larger shift is that AI can now take part in a way more end-to-end information science workflow.
To make the thought concrete, I attempted it on an actual undertaking utilizing my Apple Well being information.
A Easy Instance — Apple Well being Evaluation
Context
I’ve been sporting an Apple Watch day by day since 2019 to trace my well being information, resembling coronary heart price, power burned, sleep high quality, and many others. This information incorporates years of behavioral indicators about my each day life, however the Apple Well being app principally surfaces it with easy pattern views.
I attempted to investigate a two-year Apple Well being export six years in the past. However it ended up changing into a type of facet tasks that you simply by no means completed… My purpose this time is to extract extra insights from the uncooked information rapidly with the assistance of AI.
What I needed to work with
Listed below are the related sources I’ve:
- Uncooked Apple Well being export information: 1.85GB in XML, uploaded to my Google Drive.
- Pattern code to parse the uncooked export to structured datasets in my GitHub repo from six years in the past. However the code could possibly be outdated.
Workflow with out AI
A typical workflow with out AI would look lots like what I attempted six years in the past: Examine the XML construction, write Python to parse it into structured native datasets, conduct EDA with Pandas and Numpy, and summarize the insights.
I’m positive each information scientist is acquainted with this course of — it isn’t rocket science, but it surely takes time to construct. To get to a cultured insights report, it could take at the least a full day. That’s why that 6-year-old repo remains to be marked as WIP…
AI end-to-end workflow
My up to date workflow with AI is:
- AI locates the uncooked information in my Google Drive and downloads it.
- AI references my outdated GitHub code and writes a Python script to parse the uncooked information.
- AI uploads the parsed datasets to Google BigQuery. After all, the evaluation may be finished domestically with out BigQuery, however I set it up this solution to higher resemble an actual work setting.
- AI runs SQL queries in opposition to BigQuery to conduct the evaluation and compile an evaluation report.
Primarily, AI handles practically each step from information engineering to evaluation, with me appearing extra as a reviewer and decision-maker.
AI-generated report
Now, let’s see what Codex was in a position to generate with my steering and a few back-and-forth in half-hour, excluding the time to arrange the setting and tooling.
I selected Codex as a result of I primarily use Claude Code at work, so I needed to discover a unique instrument. I used this opportunity to arrange my Codex setting from scratch so I can higher consider all the hassle required.
You may see that this report is nicely structured and visually polished. It summarized priceless insights into annual developments, train consistency, and the affect of journey on exercise ranges. It additionally offered suggestions and said limitations and assumptions. What impressed me most was not simply the velocity, however how rapidly the output started to appear to be a stakeholder-facing evaluation as a substitute of a tough pocket book.
Please notice that the report is sanitized for my information privateness.
How I Truly Did It
Now that we now have seen the spectacular work AI can generate in half-hour, let me break it down and present you all of the steps I took to make it occur. I used Codex for this experiment. Like Claude Code, it could possibly run within the desktop app, an IDE, or the CLI.
1. Arrange MCP
To allow Codex to entry instruments, together with Google Drive, GitHub, and Google BigQuery, the following step was to arrange Mannequin Context Protocol (MCP) servers.
The best solution to arrange MCP is to ask Codex to do it for you. For instance, once I requested it to arrange Google Drive MCP, it configured my native information rapidly with clear subsequent steps on the right way to create an OAuth shopper within the Google Cloud Console.
It doesn’t all the time succeed on the primary attempt, however persistence helps. After I requested it to arrange BigQuery MCP, it failed at the least 10 instances earlier than the connection succeeded. However every time, it offered me with clear directions on the right way to take a look at it and what data was useful for troubleshooting.
2. Make a plan with the Plan Mode
After establishing the MCPs, I moved to the precise undertaking. For an advanced undertaking that includes a number of information sources/instruments/questions, I often begin with the Plan Mode to choose the implementation steps. In each Claude Code and Codex, you may allow Plan Mode with /plan. It really works like this: you define the duty and your tough plan, the mannequin asks clarifying questions and proposes a extra detailed implementation plan so that you can overview and refine. Within the screenshots beneath, yow will discover my first iteration with it.
3. Execution and iteration
After I hit “Sure, implement this plan”, Codex began executing by itself, following the steps. It labored for 13 minutes and generated the primary evaluation beneath. It moved quick throughout totally different instruments, but it surely did the evaluation domestically because it encountered extra points with the BigQuery MCP. After one other spherical of troubleshooting, it was in a position to add the datasets and run queries in BigQuery correctly.
Nonetheless, the first-pass output was nonetheless shallow, so I guided it to go deeper with follow-up questions. For instance, I’ve flight tickets and journey plans from previous travels in my Google Drive. I requested it to seek out them and analyze my exercise patterns throughout journeys. It efficiently positioned these information, extracted my journey days, and ran the evaluation.
After just a few iterations, it was in a position to generate a way more complete report, as I shared initially, inside half-hour. You will discover its code right here. That was in all probability one of the vital vital classes from the train: AI moved quick, however depth nonetheless got here from iteration and higher questions.
Takeaways for Knowledge Scientists
What AI Adjustments
Above is a small instance of how I used Codex and MCPs to run an end-to-end evaluation with out manually writing a single line of code. What are the takeaways for information scientists at work?
- Assume past coding help. Slightly than utilizing AI just for coding and writing, it’s value increasing its function throughout the complete information science lifecycle. Right here, I used AI to find uncooked information in Google Drive and add parsed datasets to BigQuery. There are lots of extra AI use circumstances associated to information pipelining and mannequin deployment.
- Context turns into a drive multiplier. MCPs are what made this workflow far more highly effective. Codex scanned my Google Drive to find my journey dates and browse my outdated GitHub code to seek out pattern parsing code. Equally, you may allow different company-approved MCPs to assist your AI (and your self) higher perceive the context. For instance:
– Connect with Slack MCP and Gmail MCP to seek for previous related conversations.
– Use Atlassian MCP to entry the desk documentation on Confluence.
– Arrange Snowflake MCP to discover the information schema and run queries. - Guidelines and reusable abilities matter. Though I didn’t exhibit it explicitly on this instance, it’s best to customise guidelines and create abilities to information your AI and lengthen its capabilities. These subjects are value their very own article subsequent time 🙂
How the Position of Knowledge Scientists Will Evolve
However does this imply AI will substitute information scientists? This instance additionally sheds gentle on how information scientists’ roles will pivot sooner or later.
- Much less handbook execution, extra problem-solving. Within the instance above, the preliminary evaluation Codex generated was very primary. The standard of AI-generated evaluation relies upon closely on the standard of your downside framing. You should outline the query clearly, break it into actionable duties, establish the correct strategy, and push the evaluation deeper.
- Area data is essential. Area data remains to be very a lot required to interpret outcomes appropriately and supply suggestions. For instance, AI observed my exercise stage had declined considerably since 2020. It couldn’t discover a convincing clarification, however mentioned: “Potential causes embrace routine modifications, work schedule, life-style shifts, harm, motivation, or much less structured coaching, however these are inferences, not findings.” However the actual purpose behind it, as you may need realized, is the pandemic. I began working from dwelling in early 2020, so naturally, I burned fewer energy. It is a quite simple instance of why area data nonetheless issues — even when AI can entry all of the previous docs in your organization, it doesn’t imply it would perceive all of the enterprise nuances, and that’s your aggressive benefit.
- This instance was comparatively easy, however there are nonetheless many lessons of labor the place I might not belief AI to function independently in the present day, particularly tasks that require stronger technical and statistical judgment, resembling causal inference.
Necessary Caveats
Final however not least, there are some concerns you might have to bear in mind whereas utilizing AI:
- Knowledge safety. I’m positive you’ve heard this many instances already, however let me repeat it as soon as extra. The info safety danger of utilizing AI is actual. For a private facet undertaking, I can set issues up nonetheless I need and take my very own danger (actually, granting AI full entry to Google Drive looks like a dangerous transfer, so that is extra for illustration functions). However at work, all the time comply with your organization’s steering on which instruments are protected to make use of and the way. And ensure to learn by way of each single command earlier than clicking “approve”.
- Double-check the code. For my easy undertaking, AI can write correct SQL with out issues. However in additional difficult enterprise settings, I nonetheless see AI make errors in its code infrequently. Generally, it joins tables with totally different granularities, inflicting fanning out and double-counting. Different instances, it misses essential filters and circumstances.
- AI is handy, but it surely would possibly accomplish your ask with surprising unintended effects… Let me inform you a shaggy dog story to finish this text. This morning, I turned on my laptop computer and noticed an alert of no disk storage left — I’ve a 512GB SSD MacBook Professional, and I used to be fairly positive I had solely used round half of the storage. Since I used to be taking part in with Codex final evening, it grew to become my first suspect. So I really requested it, “hey did you do something? My ‘system information’ had grown by 150GB in a single day”. It responded, “No, Codex solely takes xx MB”. Then I dug up my information and noticed a 142GB “bigquery-mcp-wrapper.log”… Seemingly, Codex arrange this log when it was troubleshooting the BigQuery MCP setup. Later within the precise evaluation job, it exploded into a large file. So sure, this magical wishing machine comes at a price.
This expertise summed up the tradeoff nicely for me: AI can dramatically compress the gap between uncooked information and helpful evaluation, however getting essentially the most out of it nonetheless requires judgment, oversight, and a willingness to debug the workflow itself.
