Picture by Writer
# Introduction
Though in trendy knowledge science you’ll primarily discover Jupyter notebooks, Pandas, and graphical dashboards, they don’t at all times provide the stage of management you may want. Then again, command-line instruments will not be as intuitive as you want, however they’re highly effective, light-weight, and far quicker at executing the precise jobs they’re designed for.
For this text, I’ve tried to create a stability between utility, maturity, and energy. You’ll discover some classics which can be almost unavoidable, together with extra trendy additions that fill gaps or optimize efficiency. You may even name this a 2025 model of essential CLI instruments record. For many who aren’t aware of CLI instruments however wish to study, I’ve included a bonus part with assets within the conclusion, so scroll all the way in which down earlier than you begin together with these instruments in your workflow.
# 1. curl
curl is my go-to for making HTTP requests like GET, POST, or PUT; downloading recordsdata; and sending/receiving knowledge over protocols corresponding to HTTP or FTP. It’s preferrred for retrieving knowledge from APIs or downloading datasets, and you’ll simply combine it with data-ingestion pipelines to drag JSON, CSV, or different payloads. The very best factor about curl is that it’s pre-installed on most Unix techniques, so you can begin utilizing it immediately. Nevertheless, its syntax (particularly round headers, physique payloads, and authentication) might be verbose and error-prone. If you find yourself interacting with extra advanced APIs, it’s possible you’ll choose an easier-to-use wrapper or Python library, however understanding curl continues to be an important plus for fast testing and debugging.
# 2. jq
jq is a light-weight JSON processor that allows you to question, filter, remodel, and pretty-print JSON knowledge. With JSON being a dominant format for APIs, logs, and knowledge interchange, jq is indispensable for extracting and reshaping JSON in pipelines. It acts like “Pandas for JSON within the shell.” The most important benefit is that it gives a concise language for coping with advanced JSON, however studying its syntax can take time, and very giant JSON recordsdata could require further care with reminiscence administration.
# 3. csvkit
csvkit is a collection of CSV-centric command-line utilities for reworking, filtering, aggregating, becoming a member of, and exploring CSV recordsdata. You may choose and reorder columns, subset rows, mix a number of recordsdata, convert from one format to a different, and even run SQL-like queries in opposition to CSV knowledge. csvkit understands CSV quoting semantics and headers, making it safer than generic text-processing utilities for this format. Being Python-based means efficiency can lag on very giant datasets, and a few advanced queries could also be simpler in Pandas or SQL. In case you choose pace and environment friendly reminiscence utilization, think about the csvtk toolkit.
# 4. qwk / sed
Hyperlink (sed): https://www.gnu.org/software program/sed/handbook/sed.html
Basic Unix instruments like awk and sed stay irreplaceable for textual content manipulation. awk is highly effective for sample scanning, field-based transformations, and fast aggregations, whereas sed excels at textual content substitutions, deletions, and transformations. These instruments are quick and light-weight, making them excellent for fast pipeline work. Nevertheless, their syntax might be non-intuitive. As logic grows, readability suffers, and it’s possible you’ll migrate to a scripting language. Additionally, for nested or hierarchical knowledge (e.g., nested JSON), these instruments have restricted expressiveness.
# 5. parallel
GNU parallel accelerates workflows by working a number of processes in parallel. Many knowledge duties are “mappable” throughout chunks of information. Let’s say it’s a must to execute the identical transformation on lots of of recordsdata—parallel can unfold work throughout CPU cores, pace up processing, and handle job management. You need to, nevertheless, be conscious of I/O bottlenecks and system load, and quoting/escaping might be tough in advanced pipelines. For cluster-scale or distributed workloads, think about resource-aware schedulers (e.g., Spark, Dask, Kubernetes).
# 6. ripgrep (rg)
ripgrep (rg) is a quick recursive search device designed for pace and effectivity. It respects .gitignore by default and ignores hidden or binary recordsdata, making it considerably quicker than conventional grep. It’s excellent for fast searches throughout codebases, log directories, or config recordsdata. As a result of it defaults to ignoring sure paths, it’s possible you’ll want to regulate flags to look all the pieces, and it isn’t at all times out there by default on each platform.
# 7. datamash
datamash gives numeric, textual, and statistical operations (sum, imply, median, group-by, and many others.) immediately within the shell by way of stdin or recordsdata. It’s light-weight and helpful for fast aggregations with out launching a heavier device like Python or R, which makes it preferrred for shell-based ETL or exploratory evaluation. But it surely’s not designed for very giant datasets or advanced analytics, the place specialised instruments carry out higher. Additionally, grouping very excessive cardinalities could require substantial reminiscence.
# 8. htop
htop is an interactive system monitor and course of viewer that gives dwell insights into CPU, reminiscence, and I/O utilization per course of. When working heavy pipelines or mannequin coaching, htop is extraordinarily helpful for monitoring useful resource consumption and figuring out bottlenecks. It’s extra user-friendly than conventional prime, however being interactive means it doesn’t match effectively into automated scripts. It might even be lacking on minimal server setups, and it doesn’t substitute specialised efficiency instruments (profilers, metrics dashboards).
# 9. git
git is a distributed model management system important for monitoring modifications to code, scripts, and small knowledge property. For reproducibility, collaboration, branching experiments, and rollback, git is the usual. It integrates with deployment pipelines, CI/CD instruments, and notebooks. Its downside is that it’s not meant for versioning giant binary knowledge, for which Git LFS, DVC, or specialised techniques are higher suited. The branching and merging workflow additionally comes with a studying curve.
# 10. tmux / display screen
Terminal multiplexers like tmux and display screen allow you to run a number of terminal periods in a single window, detach and reattach periods, and resume work after an SSH disconnect. They’re important if it is advisable run lengthy experiments or pipelines remotely. Whereas tmux is really helpful resulting from its lively improvement and suppleness, its config and keybindings might be tough for newcomers, and minimal environments could not have it put in by default.
# Wrapping Up
In case you’re getting began, I’d suggest mastering the “core 4”: curl, jq, awk/sed, and git. These are used in every single place. Over time, you’ll uncover domain-specific CLIs like SQL shoppers, the DuckDB CLI, or Datasette to fit into your workflow. For additional studying, take a look at the next assets:
- Information Science on the Command Line by Jeroen Janssens
- The Artwork of Command Line on GitHub
- Mark Pearl’s Bash Cheatsheet
- Communities just like the unix & command-line subreddits usually floor helpful tips and new instruments that may develop your toolbox over time.
Kanwal Mehreen is a machine studying engineer and a technical author with a profound ardour for knowledge science and the intersection of AI with medication. She co-authored the e-book “Maximizing Productiveness with ChatGPT”. As a Google Era Scholar 2022 for APAC, she champions range and tutorial excellence. She’s additionally acknowledged as a Teradata Range in Tech Scholar, Mitacs Globalink Analysis Scholar, and Harvard WeCode Scholar. Kanwal is an ardent advocate for change, having based FEMCodes to empower ladies in STEM fields.
Picture by Writer
# Introduction
Though in trendy knowledge science you’ll primarily discover Jupyter notebooks, Pandas, and graphical dashboards, they don’t at all times provide the stage of management you may want. Then again, command-line instruments will not be as intuitive as you want, however they’re highly effective, light-weight, and far quicker at executing the precise jobs they’re designed for.
For this text, I’ve tried to create a stability between utility, maturity, and energy. You’ll discover some classics which can be almost unavoidable, together with extra trendy additions that fill gaps or optimize efficiency. You may even name this a 2025 model of essential CLI instruments record. For many who aren’t aware of CLI instruments however wish to study, I’ve included a bonus part with assets within the conclusion, so scroll all the way in which down earlier than you begin together with these instruments in your workflow.
# 1. curl
curl is my go-to for making HTTP requests like GET, POST, or PUT; downloading recordsdata; and sending/receiving knowledge over protocols corresponding to HTTP or FTP. It’s preferrred for retrieving knowledge from APIs or downloading datasets, and you’ll simply combine it with data-ingestion pipelines to drag JSON, CSV, or different payloads. The very best factor about curl is that it’s pre-installed on most Unix techniques, so you can begin utilizing it immediately. Nevertheless, its syntax (particularly round headers, physique payloads, and authentication) might be verbose and error-prone. If you find yourself interacting with extra advanced APIs, it’s possible you’ll choose an easier-to-use wrapper or Python library, however understanding curl continues to be an important plus for fast testing and debugging.
# 2. jq
jq is a light-weight JSON processor that allows you to question, filter, remodel, and pretty-print JSON knowledge. With JSON being a dominant format for APIs, logs, and knowledge interchange, jq is indispensable for extracting and reshaping JSON in pipelines. It acts like “Pandas for JSON within the shell.” The most important benefit is that it gives a concise language for coping with advanced JSON, however studying its syntax can take time, and very giant JSON recordsdata could require further care with reminiscence administration.
# 3. csvkit
csvkit is a collection of CSV-centric command-line utilities for reworking, filtering, aggregating, becoming a member of, and exploring CSV recordsdata. You may choose and reorder columns, subset rows, mix a number of recordsdata, convert from one format to a different, and even run SQL-like queries in opposition to CSV knowledge. csvkit understands CSV quoting semantics and headers, making it safer than generic text-processing utilities for this format. Being Python-based means efficiency can lag on very giant datasets, and a few advanced queries could also be simpler in Pandas or SQL. In case you choose pace and environment friendly reminiscence utilization, think about the csvtk toolkit.
# 4. qwk / sed
Hyperlink (sed): https://www.gnu.org/software program/sed/handbook/sed.html
Basic Unix instruments like awk and sed stay irreplaceable for textual content manipulation. awk is highly effective for sample scanning, field-based transformations, and fast aggregations, whereas sed excels at textual content substitutions, deletions, and transformations. These instruments are quick and light-weight, making them excellent for fast pipeline work. Nevertheless, their syntax might be non-intuitive. As logic grows, readability suffers, and it’s possible you’ll migrate to a scripting language. Additionally, for nested or hierarchical knowledge (e.g., nested JSON), these instruments have restricted expressiveness.
# 5. parallel
GNU parallel accelerates workflows by working a number of processes in parallel. Many knowledge duties are “mappable” throughout chunks of information. Let’s say it’s a must to execute the identical transformation on lots of of recordsdata—parallel can unfold work throughout CPU cores, pace up processing, and handle job management. You need to, nevertheless, be conscious of I/O bottlenecks and system load, and quoting/escaping might be tough in advanced pipelines. For cluster-scale or distributed workloads, think about resource-aware schedulers (e.g., Spark, Dask, Kubernetes).
# 6. ripgrep (rg)
ripgrep (rg) is a quick recursive search device designed for pace and effectivity. It respects .gitignore by default and ignores hidden or binary recordsdata, making it considerably quicker than conventional grep. It’s excellent for fast searches throughout codebases, log directories, or config recordsdata. As a result of it defaults to ignoring sure paths, it’s possible you’ll want to regulate flags to look all the pieces, and it isn’t at all times out there by default on each platform.
# 7. datamash
datamash gives numeric, textual, and statistical operations (sum, imply, median, group-by, and many others.) immediately within the shell by way of stdin or recordsdata. It’s light-weight and helpful for fast aggregations with out launching a heavier device like Python or R, which makes it preferrred for shell-based ETL or exploratory evaluation. But it surely’s not designed for very giant datasets or advanced analytics, the place specialised instruments carry out higher. Additionally, grouping very excessive cardinalities could require substantial reminiscence.
# 8. htop
htop is an interactive system monitor and course of viewer that gives dwell insights into CPU, reminiscence, and I/O utilization per course of. When working heavy pipelines or mannequin coaching, htop is extraordinarily helpful for monitoring useful resource consumption and figuring out bottlenecks. It’s extra user-friendly than conventional prime, however being interactive means it doesn’t match effectively into automated scripts. It might even be lacking on minimal server setups, and it doesn’t substitute specialised efficiency instruments (profilers, metrics dashboards).
# 9. git
git is a distributed model management system important for monitoring modifications to code, scripts, and small knowledge property. For reproducibility, collaboration, branching experiments, and rollback, git is the usual. It integrates with deployment pipelines, CI/CD instruments, and notebooks. Its downside is that it’s not meant for versioning giant binary knowledge, for which Git LFS, DVC, or specialised techniques are higher suited. The branching and merging workflow additionally comes with a studying curve.
# 10. tmux / display screen
Terminal multiplexers like tmux and display screen allow you to run a number of terminal periods in a single window, detach and reattach periods, and resume work after an SSH disconnect. They’re important if it is advisable run lengthy experiments or pipelines remotely. Whereas tmux is really helpful resulting from its lively improvement and suppleness, its config and keybindings might be tough for newcomers, and minimal environments could not have it put in by default.
# Wrapping Up
In case you’re getting began, I’d suggest mastering the “core 4”: curl, jq, awk/sed, and git. These are used in every single place. Over time, you’ll uncover domain-specific CLIs like SQL shoppers, the DuckDB CLI, or Datasette to fit into your workflow. For additional studying, take a look at the next assets:
- Information Science on the Command Line by Jeroen Janssens
- The Artwork of Command Line on GitHub
- Mark Pearl’s Bash Cheatsheet
- Communities just like the unix & command-line subreddits usually floor helpful tips and new instruments that may develop your toolbox over time.
Kanwal Mehreen is a machine studying engineer and a technical author with a profound ardour for knowledge science and the intersection of AI with medication. She co-authored the e-book “Maximizing Productiveness with ChatGPT”. As a Google Era Scholar 2022 for APAC, she champions range and tutorial excellence. She’s additionally acknowledged as a Teradata Range in Tech Scholar, Mitacs Globalink Analysis Scholar, and Harvard WeCode Scholar. Kanwal is an ardent advocate for change, having based FEMCodes to empower ladies in STEM fields.
