Don’t Let Conda Eat Your Exhausting Drive

In case you’re an Anaconda person, you already know that conda environments enable you to handle bundle dependencies, keep away from compatibility conflicts, and share your tasks with others. Sadly, they will additionally take over your laptop’s onerous drive.

I write plenty of laptop tutorials and to maintain them organized, every has a devoted folder construction full with a Conda Atmosphere. This labored nice at first, however quickly my laptop’s efficiency degraded, and I seen that my SSD was filling up. At one level I had solely 13 GB free.

Conda helps handle this drawback by storing downloaded bundle recordsdata in a single “cache” (pkgs_dirs). Whenever you set up a bundle, conda checks for it within the bundle cache earlier than downloading. If not discovered, conda will obtain and extract the bundle and hyperlink the recordsdata to the lively setting. As a result of the cache is “shared,” totally different environments can use the identical downloaded recordsdata with out duplication.

As a result of conda caches each downloaded bundle, pkgs_dirs can develop to many gigabytes. And whereas conda hyperlinks to shared packages within the cache, there may be nonetheless a have to retailer some packages within the setting folder. That is primarily to keep away from model conflicts, the place totally different environments want totally different variations of the identical dependency (a bundle required to run one other bundle).

As well as, giant, compiled binaries like OpenCV could require full copies within the setting’s listing, and every setting requires a replica of the Python interpreter (at 100–200 MB). All these points can bloat conda environments to a number of gigabytes.

On this Fast Success Information Science challenge, we’ll have a look at some methods for decreasing the storage necessities for conda environments, together with these saved in default areas and devoted folders.

Reminiscence Administration Methods

Under are some Reminiscence Administration methods that may enable you to scale back conda’s storage footprint in your machine. We’ll talk about every in flip.

1. Cache cleansing
2. Sharing task-based environments
3. Archiving with setting and specs recordsdata
4. Archiving environments with conda-pack
5. Storing environments on an exterior drive
6. Relocating the bundle cache
7. Utilizing digital environments (venv)

1. Cleansing the Bundle Cache

Cleansing the bundle cache is the primary and best step for liberating up reminiscence. Even after deleting environments, conda retains the associated bundle recordsdata within the cache. You may liberate area by eradicating these unused packages and their related tarballs (compressed bundle recordsdata), logs, index caches (metadata saved in conda), and momentary recordsdata.

Conda permits an non-compulsory “dry run” to see how a lot reminiscence will probably be reclaimed. You’ll need to run this from both the terminal or Anaconda Immediate in your base setting:

conda clear --all --dry-run

To commit, run:

conda clear --all

Right here’s how this appears on my machine:

This course of trimmed a wholesome 6.28 GB and took a number of minutes to run.

2. Sharing Activity-based Environments

Creating just a few environments for specialised duties — like laptop imaginative and prescient or geospatial work — is extra reminiscence environment friendly than utilizing devoted environments for every challenge. These environments would come with primary packages plus ones for the precise process (akin to OpenCV, scikit-image, and PIL for laptop imaginative and prescient).

A bonus of this method is which you could simply hold all of the packages updated and hyperlink the environments to a number of tasks. Nonetheless, this gained’t work if some tasks require totally different variations of the shared packages.

3. Archiving with Atmosphere and Specs Information

In case you don’t have sufficient storage websites or need to protect legacy tasks effectively, think about using setting or specs recordsdata. These small recordsdata file an setting’s contents, permitting you to rebuild it later.

Saving conda environments on this method reduces their measurement on disk from gigabytes to some kilobytes. After all, you’ll must recreate the setting to make use of it. So, you’ll need to keep away from this method if you happen to often revisit tasks that hyperlink to the archived environments.

NOTE: Think about using Mamba, a drop-in alternative for conda, for sooner rebuilds. Because the docs say, “If you already know conda, you already know Mamba!”

Utilizing Atmosphere Information: An environmental file is a small file that lists all of the packages and variations put in in an setting, together with these put in utilizing Python’s bundle installer (pip). This helps you each restore an setting and share it with others.

The setting file is written in YAML (.yml), a human-readable data-serialization format for knowledge storage. To generate an setting file, you need to activate after which export the setting. Right here’s make a file for an setting named my_env:

 conda activate my_env
 conda env export > my_env.yml

You may identify the file any legitimate filename however watch out as an current file with the identical identify will probably be overwritten.

By default, the setting file is written to the person listing. Right here’s a truncated instance of the file’s contents:

identify: C:Usershannaquick_successfed_hikesfed_env
channels:
  - defaults
  - conda-forge
dependencies:
  - asttokens=2.0.5=pyhd3eb1b0_0
  - backcall=0.2.0=pyhd3eb1b0_0
  - blas=1.0=mkl
  - bottleneck=1.3.4=py310h9128911_0
  - brotli=1.0.9=ha925a31_2
  - bzip2=1.0.8=he774522_0
  - ca-certificates=2022.4.26=haa95532_0
  - certifi=2022.5.18.1=py310haa95532_0
  - colorama=0.4.4=pyhd3eb1b0_0
  - cycler=0.11.0=pyhd3eb1b0_0
  - debugpy=1.5.1=py310hd77b12b_0
  - decorator=5.1.1=pyhd3eb1b0_0
  - entrypoints=0.4=py310haa95532_0

  ------SNIP------

Now you can take away your conda setting and reproduce it once more with this file. To take away an setting, first deactivate it after which run the take away command (the place ENVNAME is the identify of your setting):

conda deactivate
conda take away -n ENVNAME --all

If the conda setting exists exterior of Anaconda’s default envs folder, then embody the listing path to the setting, as so:

conda take away -p PATHENVNAME --all

Observe that this archiving method will solely work completely if you happen to proceed to make use of the identical working system, akin to Home windows or macOS. It is because fixing for dependencies can introduce packages that may not be appropriate throughout platforms.

To revive a conda setting utilizing a file, run the next, the place my_env represents your conda setting identify and setting.yml represents your setting file:

 conda env create -n my_env -f directorypathtoenvironment.yml

It’s also possible to use the setting file to recreate the setting in your D: drive. Simply present the brand new path when utilizing the file. Right here’s an instance:

conda create --prefix D:my_envsmy_new_env --file setting.yml

For extra on setting recordsdata, together with manually produce them, go to the docs.

Utilizing Specs Information: In case you haven’t put in any packages utilizing pip, you should utilize a specs file to breed a conda setting on the identical working system. To create a specification file, activate an setting, akin to my_env, and enter the next command:

 conda checklist --explicit > exp_spec_list.txt

This produces the next output, truncated for brevity:

 # This file could also be used to create an setting utilizing:
 # $ conda create --name  --file 
 # platform: win-64
 @EXPLICIT
 https://conda.anaconda.org/conda-forge/win-64/ca-certificates-202x.xx.x-h5b45459_0.tar.bz2
 https://conda.anaconda.org/conda-forge/noarch/tzdata-202xx-he74cb21_0.tar.bz2

------snip------

Observe that the --explicit flag ensures that the focused platform is annotated within the file, on this case, # platform: win-64 within the third line.

Now you can take away the setting as described within the earlier part.

To re-create my_env utilizing this textual content file, run the next with a correct listing path:

conda create -n my_env -f directorypathtoexp_spec_list.txt

4. Archiving Environments with conda-pack

The conda-pack command permits you to archive a conda setting earlier than eradicating it. It packs the complete setting right into a compressed archive with the extension: .tar.gz. It’s useful for backing up, sharing, and transferring environments with out the necessity to reinstall packages.

The next command will protect an setting however take away it out of your system (the place my_env represents the identify of your setting):

conda set up -c conda-forge conda-pack
conda pack -n my_env -o my_env.tar.gz

To revive the setting later run this command:

mkdir my_env && tar -xzf my_env.tar.gz -C my_env

This method gained’t save as a lot reminiscence because the textual content file possibility. Nonetheless, you gained’t have to re-download packages when restoring an setting, which implies it may be used with out web entry.

5. Storing Environments on an Exterior Drive

By default, conda shops all environments in a default location. For Home windows, that is beneath the …anaconda3envs folder. You may see these environments by operating the command conda information --envs in a immediate window or terminal. Right here’s the way it appears on my C: drive (this can be a truncated view):

Utilizing a Single Environments Folder: In case your system helps an exterior or secondary drive, you’ll be able to configure conda to retailer environments there to liberate area in your major disk. Right here’s the command; you’ll have to substitute your particular path:

conda config --set envs_dirs /path/to/exterior/drive

In case you enter a path to your D drive, akin to D:conda_envs, conda will create new environments at this location.

This method works effectively when your exterior drive is a quick SSD and once you’re storing packages with giant dependencies, like TensorFlow. The draw back is slower efficiency. In case your OS and notebooks stay on the first drive, you could expertise some learn/write latency when operating Python.

As well as, some OS settings could energy down idle exterior drives, including a delay once they spin again up. Instruments like Jupyter could battle to find conda environments if the drive letter adjustments, so that you’ll need to use a set drive letter and be certain that the right kernel paths are set.

Utilizing A number of Atmosphere Folders: As a substitute of utilizing a single envs_dirs listing for all environments, you’ll be able to retailer every setting inside its respective challenge folder. This allows you to retailer all the things associated to a challenge in a single place.

For instance, suppose you could have a challenge in your Home windows D: drive in a folder referred to as D:projectsgeospatial. To position the challenge’s conda setting on this folder, loaded with ipykernel for JupyterLab, you’d run:

conda create -p D:projectsgeospatialenv ipykernel

After all, you’ll be able to name env one thing extra descriptive, like geospatial_env.

As with the earlier instance, environments saved on a distinct disk may cause efficiency points.

Particular Observe on JupyterLab: Relying on the way you launch JupyterLab, its default conduct could also be to open in your person listing (akin to, C:Usersyour_user_name). Since its file browser is restricted to the listing from which it’s launched, you gained’t see directories on different drives like D:. There are a lot of methods to deal with this, however one of many easiest is to launch JupyterLab from the D: drive.

For instance, in Anaconda Immediate, kind:

D:

adopted by:

jupyter lab

Now, it is possible for you to to choose from kernels on the D: drive.

For extra choices on altering JupyterLab’s working listing, ask an AI about “ change Jupyter’s default working listing” or “ create a Symlink to D: in your person folder.”

Shifting Current Environments: You need to by no means manually transfer a conda setting, akin to by chopping and pasting to a brand new location. It is because conda depends on inner paths and metadata that may turn out to be invalid with location adjustments.

As a substitute, you must clone current environments to a different drive. This can duplicate the setting, so that you’ll have to manually take away it from its unique location.

Within the following instance, we use the --clone flag to provide an actual copy of a C: drive setting (referred to as my_env) on the D: drive:

conda create -p D:new_envsmy_env --clone C:pathtooldenv

NOTE: Contemplate exporting your setting to a YAML file (as described in Part 3 above) earlier than cloning. This lets you recreate the setting if one thing goes incorrect with the clone process.

Now, once you run conda env checklist, you’ll see the setting listed in each the C: and D: drives. You may take away the outdated setting by operating the next command within the base setting:

conda take away --name my_env --all -y

Once more, latency points could have an effect on these setups if you happen to’re working throughout two disks.

Chances are you’ll be questioning, is it higher to maneuver a conda setting utilizing an setting (YAML) file or to make use of--clone? The brief reply is that --clone is the most effective and quickest possibility for transferring an setting to a distinct drive on the identical machine. An setting file is greatest for recreating the identical setting on a totally different machine. Whereas the file ensures a constant setting throughout totally different programs, it may take for much longer to run, particularly with giant environments.

6. Relocating the Bundle Cache

In case your major drive is low on area, you’ll be able to transfer the bundle cache to a bigger exterior or secondary drive utilizing this command:

conda config --set pkgs_dirs D:conda_pkgs

On this instance, packages at the moment are saved on the D drive (D:conda_pkgs) as an alternative of the default location.

In case you’re working in your major drive and each drives are SSD, then latency points shouldn’t be vital. Nonetheless, if one of many drives is a slower HDD, you’ll be able to expertise slowdowns when creating or updating environments. If D: is an exterior drive related by USB, you might even see vital slowdowns for giant environments.

You may mitigate a few of these points by maintaining the bundle cache (pkgs_dirs) and often used environments on the sooner SSD, and different environments on the slower HDD.

One final thing to think about is backups. Main drives could have routine backups scheduled however secondary or exterior drives could not. This places you susceptible to shedding all of your environments.

7. Utilizing Digital Environments

In case your challenge doesn’t require conda’s in depth bundle administration system for dealing with heavy dependencies (like TensorFlow or GDAL), you’ll be able to considerably scale back disk utilization with a Python digital setting (venv). This represents a light-weight various to a conda setting.

To create a venv named my_env, run the next command:

Any such setting has a small base set up. A minimal conda setting takes up about 200 MB and consists of a number of utilities, akin to conda, pip, setuptools, and so forth. A venv is far lighter, with a minimal set up measurement of solely 5–10 MB.

Conda additionally caches bundle tarballs in pkgs_dirs. These tarballs can develop to a number of GBs over time. As a result of venv installs packages immediately into the setting, no additional copies are preserved.

Typically, you’ll need to think about venv once you solely want primary Python packages like NumPy, pandas, or Scikit-learn. Packages for which conda is strongly beneficial, like Geopandas, ought to nonetheless be positioned in a conda setting. In case you use plenty of environments, you’ll most likely need to stick to conda and profit from its bundle linking.

You will discover particulars on activate and use Python digital environments within the venv docs.

Recap

Excessive influence/low disruption reminiscence administration methods for conda environments embody cleansing the bundle cache and storing little-used environments as YAML or textual content recordsdata. These strategies can save many gigabytes of reminiscence whereas retaining Anaconda’s default listing construction.

Different excessive influence strategies embody transferring the bundle cache and/or conda environments to a secondary or exterior drive. This can resolve reminiscence issues however could introduce latency points, particularly if the brand new drive is a gradual HDD or makes use of a USB connection.

For easy environments, you should utilize a Python digital setting (venv) as a light-weight various to conda.