Anaconda and Package Management

Anaconda and Package Management #

Table of Contents #

Anaconda #

[ h ][ d ][ g ][ w ] Anaconda [ y ]

Docs
- [ d ] User Guide
  - [ d ] Packages
- [ d ] Command Reference
- [ d ] Updating the Anaconda Metapackage
[ b ] Blog
- [ b ] 12-07-2022. “Conda Is Fast Now. Like, Really Fast”.
- [ b ] 05-06-2022. “New Release: Anaconda Distribution Now Supporting M1”.
- [ b ] 04-30-2022. “New from Anaconda: Python in the Browser”.
- [ b ] 03-16-2022. “A Faster Solver for Conda: Libmamba”.
- [ b ] 11-19-2021. “Anaconda Individual Edition 2021.11”.
- [ b ] 07-15-2021. “A Python Data Scientist’s Guide to the Apple Silicon Transition”.
- [ b ] 08-13-2019. “What’s in a name? Clarifying the Anaconda Metapackage”.
- [ b ] 06-25-2019. “How We Made Conda Faster in 4.7”.
- [ b ] 03-11-2019. “Understanding and Improving Conda’s Performance”.
- [ b ] 11-28-2018. “Understanding Conda and Pip”.
- [ b ] 08-15-2017. “Keeping Anaconda Up To Date”.

[ h ][ d ][ g ][ w ] Conda [ PyPA ]

[ d ] User Guide
- [ d ] The Anaconda Metapackage
- [ d ] Metapackages
- [ d ] Conda performance
[ d ] Commands
[ d ] Developer Guide
[ b ] Blog
[ h ][ d ][ g ][ w ] Miniconda
[ h ][ d ][ g ][ w ] Conda-Forge
[ h ][ d ][ g ][ w ] Miniforge

[ d ] Conda-Build

[ d ] User Guide
recipes

[ d ] Anaconda Server

[ h ][ d ][ g ][ w ] Anaconda Cloud & Anaconda Notebooks

[ h ] Anaconda Learning
- [ h ] Get Started with Anaconda
- [ h ] Jupyter Notebook Basics
- [ h ] Conda Basics
- [ h ] Conda Essentials
- [ h ] Build and Deploy Data Apps in Anaconda Notebooks
- [ h ] Python for Spreadsheets
- [ h ] Data Analysis with Excel and Python
- [ h ] Get Started with Python in Excel
- [ h ] Anaconda Server Basics

Conda performance

[ d ] Conda > User Guide > Concepts > Performance
[ d ][ g ] conda-libmamba-solver
[ b ] 12-07-2022. “Conda Is Fast Now. Like, Really Fast”.
[ b ] 03-16-2022. “A Faster Solver for Conda: Libmamba”.
[ b ] 06-25-2019. “How We Made Conda Faster in 4.7”.
[ b ] 03-11-2019. “Understanding and Improving Conda’s performance”.

Conda and pip

[ d ] Conda > User Guide > Tasks > Managing environments > Using pip in an environment
[ d ] Conda > User Guide > Tasks > Managing packages > Installing non-conda packages
[ d ] Conda > User Guide > Configuration > Improving interoperability with pip
[ b ] 12-04-2018. “Using Pip in a Conda Environment”.
[ b ] 11-28-2018. “Understanding Conda and Pip”.

some Anaconda repos

[ h ] Apple
[ h ] Conda-Forge
[ h ] Defaults (used for Conda environment base)
[ h ] JavaScript
[ h ] Plotly
[ h ] PyViz

More #

[ h ][ d ][ g ][ PyPA ] Package Installer for Python (PIP)
- [ d ] Secure Installs
- Security
  - [ g ] pip-audit [ PyPI ]
  - [ g ] Python Packaging Advisory Database
  - [ h ] Open Source Vulberabilities Database
  - [ h ][ g ] The Update Framework (TUF)
  - https://talkpython.fm/episodes/show/435/pypi-security
  - Ars Technica - 10 malicious Python packages exposed in latest repository attack
  - PyPI Security Pitfalls and Steps Toward A Secure Python Ecosystem
[ h ][ d ][ g ][ PyPA ] Pipx
[ h ][ d ][ g ][ PyPA ] Mamba
[ h ][ d ][ g ][ PyPA ] Pipenv
[ h ][ d ][ g ][ PyPA ] Poetry
[ h ][ d ][ g ][ PyPA ] Venv
[ h ][ d ][ g ][ PyPA ] Virtualenv
[ h ][ g ] Python Package Index (PyPI)
[ h ][ g ] Python Packaging Authority (PyPA)
- [ d ] PyPA > Tutorials > Installing Packages
- [ d ] PyPA > Guides > Building and Publishing > Packaging and distributing projects
- [ d ] PyPA > Guides > Tool Recommendations
- [ d ] PyPA > Glossary > Virtual Environment

RealPython

[ RealPython ] 03-15-2023. Acsany, Philipp. “How to Evaluate the Quality of Python Packages”.
[ RealPython ] 12-20-2021. Acsany, Philipp. “Dependency Management with Python Poetry”.
[ RealPython ] ----------. Stratis, Kyle. “Python Application Layouts: A Reference”.

more

Getting Started #

Initialization #

In the terminal, run

conda init bash --dry-run

to see the changes that would take place for shell configuration. If you are satisfied, run

conda init bash

Close the current terminal instance and open a new one for the changes to take effect. If you need to undo shell configuration, run

conda init bash --reverse --dry-run

to see the changes that would take place to undo shell configuration. If you are satisfied, run

conda init bash --reverse

Close the current terminal instance and open a new one for the changes to take effect.

Check the version and update conda #

conda --version # or `conda -V`

View the help menu of command conda.

conda # or `conda -h` or `conda --help`

View the help menu of subcommand subcommand.

conda subcommand -h

Update conda.

conda update -n base -c defaults conda

Configuration #

conda config

conda config -a

Package and Environment Management #

List conda environments.

conda info -e # or `conda env list`

Create a new conda environment called data-science-env with Python 3.11, Pandas 2.1, etc., and their dependencies using the conda-forge channel.

conda create -n data-science-env -c conda-forge --strict-channel-priority \
'ipykernel =6.29' \
'jupyterlab=4.00' \
'numpy     =1.26' \
'pandas    =2.01' \
'python    =3.11'

Alternatively, create a conda environment associated with a new project directory.

mkdir data-science-project && conda create -p data-science-project \
-c conda-forge --strict-channel-priority \
'ipykernel =6.29' \
'jupyterlab=4.00' \
'numpy     =1.26' \
'pandas    =2.01' \
'python    =3.11'

Activate conda environment data-science-env.

conda activate data-science-env

Install package matplotlib in conda environment data-science-env.

conda install -n data-science-env matplotlib

Delete package pandas from conda environment data-science-env.

conda remove -n data-science-env pandas

Delete the conda environment called data-science-env.

conda activate base && conda remove -n data-science-env --all

List the current conda environment’s packages and dependencies; their versions; and their channels.

conda list --show-channel-urls

Deploying a conda environment in Jupyter #

deploy an environment in Jupyter

install packages jupyterlab and nb_conda_kernels into the environment that hosts Jupyter
install package ipykernel into the environment to be deployed

conda create -n jupyter jupyterlab nb_conda_kernels # env `jupyter` supports the deployment of other envs
conda create -n myenv ipykernel                     # env `myenv` can be deployed as a notebook in jupyter

List available Jupyter kernels.

jupyter kernelspec list

Create a new Jupyter kernel called myJupyterKernel.

python -m ipykernel install --user --name myJupyterKernel

Delete the Jupyter kernel called myJupyterKernel.

jupyter kernelspec remove myJupyterKernel

Lab extensions.

jupyter labextension -h

Notebook extensions.

jupyter nbextension -h

Building a conda package and uploading it to a channel #

conda activate base && conda list # look for packages `conda-build` and `anaconda-client` and install them if necessary
conda config --set anaconda_upload no
git clone https://github.com/Anaconda-Platform/anaconda-client/ && cd anaconda-client/example-packages/conda
conda build .
cd $CONDA_PREFIX/conda-bld/osx-64
tar zxvf package

anaconda login
anaconda upload package # upload package `package` to personal channel

Texts #

[ h ] Beuzen, Tomas & Tiffany Timbers. Python Packages. CRC Press.

Terms #

[ w ] Anaconda
[ w ] Conda
[ w ] Environment Management
[ w ] Package Management
[ w ] Software Distribution

Command Reference #

`anaconda`#

`conda`#

conda --version
conda --help

`conda activate`#

conda activate --help

conda activate <ENVIRONMENT_NAME> # activate env
conda activate <PATH>             #

`conda config`#

conda config --help

conda config --show
conda config --show-sources
conda config --add channels conda-forge
conda config --describe

`conda create`#

conda create -n     <ENVIRONMENT_NAME> PACKAGE(S) # create env
conda create --name <ENVIRONMENT_NAME> PACKAGE(S) #

conda create -p       <PATH> PACKAGE(S)           # create env at a specific location
conda create --prefix <PATH> PACKAGE(S)           #

`conda env`#

conda env --help

# create an environment from a file
conda env create --help
conda env create -f     <ENVIRONMENT_FILE>
conda env create --file <ENVIRONMENT_FILE>

conda env list --help
conda env list

# export an environment to a file
conda env export --help
conda env export                          # view the env
conda env export > env.yml                # save the env to file `env.yml`
conda env export --from-history > env.yml # save the env to file `env.yml` 

`conda info`#

conda info --help

# examples
conda info
conda info -a
conda info -e # --envs

`conda init`#

conda init --help

# template
conda init [ --reverse ] [ --dry-run ]

# examples
conda init
conda init --all
conda init bash
conda init zsh
conda init powershell
conda init cmd.exe

`conda install`#

conda install --help

# templates
conda install PACKAGE(S)                          # install packages into the active env
conda install -n <ENVIRONMENT_NAME> PACKAGE(S)    # isntall packages into env `<ENVIRONMENT_NAME>`
conda install PACKAGE(S) -c <CHANNEL_NAME>

# example
conda install -n pyfinance -c conda-forge pyfinance==1.2.7=py_0

`conda list`#

conda list --help

conda list                       # list packages in the active env
conda list -n <ENVIRONMENT_NAME> # list packages in env `<ENVIRONMENT_NAME>`

`conda remove`#

conda remove --help

# templates
conda remove PACKAGE(S)                       # remove packages `PACKAGE(S)` from the active env
conda remove -n <ENVIRONMENT_NAME> PACKAGE(S) # remove packages `PACKAGE(S)` from env `<ENVIRONMENT_NAME>`
conda remove -n <ENVIRONMENT_NAME> --all      # remove env `<ENVIRONMENT_NAME>`

`conda repo`#

conda repo --help

`conda search`#

conda search --help

# examples
conda search matplotlib
conda search 'matplotlib>=3.5.0'
conda search pyfinance --channel conda-forge
conda search pyfinance -c        conda-forge

`conda skeleton`#

# conda skeleton
#
# Python (PyPI)
# R      (CRAN)
# Perl   (cpan)
# Lua    (luarocks)
# RPM    (rpm)
conda skeleton --help
conda skeleton pypi -h
conda skeleton pypi click # download package `click` from PyPI and genrerate a recipe
conda skeleton cran fansi # download package `fansi` from CRAN and genrerate a recipe

`conda update`#

conda update --help
conda update conda

Notes #

Programs are distributed as packages.

Package Manager

automatically search, download, compile, build, install a program from source, and update
identifies compatible versions of the program and its dependencies
handles updates
examples of platform- or ecosystem-specific package managers
- Advanced Package Tool (APT) (Debian)
- Homebrew (macOS)
- pip (Pip Installs Packages, Package Installer for Python)

Environment Manager

easily create, save, load, and switch between environments

Conda

a platform-agnostic open-source package and environment management system
using a package and environment management tool facilitates portability and reproducibility of (data) science workflows

~/anaconda3/            # Anaconda root
├── bin/                # binary executables, human-readable scripts, compiled programs
│   └── conda           # command `conda`
├── conda-meta/
├── condabin/
├── envs/               # named environments
│   └── snowflake/      # env `snowflake`
│       ├── bin/
│       │   └── python  # python interpreter
│       ├── conda-meta/
│       ├── include/
│       ├── lib/
│       ├── man/
│       ├── share/
│       └── ssl/
├── etc/
│   └── conda/
│       ├── activate.d/   # activation script (everything in this directory is executed)
│       └── deactivate.d/ # deactivation script
├── include/              # header files
├── lib/                  # uncompressed libraries and modules (callables)
├── libexec/
├── man/                  # manual pages
├── pkgs/                 # package cache of compressed tarballs (which populates directories `bin`, `lib`)
├── share/
├── shell/
└── ssl/

Conda Environment

a self-contained directory that contains a set of packages of programs, libraries, scripts, etc.
optional name points to prefix (the absolute path)
conda environment activation modifies
- environment variable CONDA_PREFIX
- environment variable PATH

# Conda Package - a compressed file with a particular directory structure (`bin`, `info`, `lib`)

package_name-package_version-build
├── bin/
├── info/
│   ├── info/about.json
│   ├── info/files
│   ├── info/git
│   ├── info/hash_input.json
│   ├── info/index.json
│   ├── info/licenses/
│   ├── info/paths.json
│   ├── info/recipe/
│   └── info/test/
└── lib/

Conda Channel

a URL pointing to a directory containing a collection of packages
https://repo.anaconda.org/pkgs/main
local channels can be specified with file:// or a path
community-led public channel
- maintain feedstocks: repositories that contain recipes that are built into packages by automated infrastructure

.condarc

an optional runtime config file in the YAML format
created in the home directory the first time command conda config is run
possible config files and folders
- ~/.anaconda
- ~/.conda
- ~/.condarc
- ~/.continuum

~/.condarc # config file

channels  :
- conda-forge
envs_dirs :
- ~/anaconda3/envs
- /path/to/alternate/envs
pkgs_dirs :
- ~/anaconda3/pkgs
- /path/to/alternate/pkgs

Environment File

a plain text file in the YAML format that describes the content of an environment

name         : machine-learning-env
channels     :
- conda-forge
dependencies :
- ipython                      # interactive terminal
- pandas==1.5.2=py39hecff1ad_0 # data processing
- python
- scikit-learn                 # machine learning algorithms
- seaborn                      # visualization
variables    :
  VAR1 : valueA
  VAR2 : valueB
prefix       : /Users/user/anaconda3/envs/machine-learning-env

name     : data-viz
channels :
- javascript
dependencies :
- python=3.9
- bokeh=2.4.2
- numpy=1.21.*
- nodesjs=16.13.*
- flask
- pip
- pip:
  - Flask-Testing

# Conda Recipe - a directory that contains instructions on how to create a package (a meta.yaml file and additional files such as build.sh required to compile the package)
# recipes are compiled either by `conda-build` or by automated feedstocks

meta.yaml               # mandatory: a file that contains the recipe metadata (only package/name and package/version required)
build.sh                # optional: the bash-executable script that installs the files for the package (Linux/macOS)
bld.bat                 # optional: the cmd-executable script that installs the files for the package (Windows)
run_test.[py,pl,sh,bat] # optional: an optional Python test script that runs automatically if it is part of the recipe
                        # optional: patches that are applied to the source
                        # optional: other resources that are not included in the source and cannot be generated by the build scripts (icon files, readmes, build notes)

Anaconda Certified: conda Fundamentals #

[ h ] Anaconda Certified: conda Fundamentals

[ p ] Conda Basics
- [ p ] How to Use Anaconda Notebooks
- Getting Started with conda
  - [ p ] Setup and installation
  - [ p ] Overview of conda
  - [ p ] Using a package and environment management system
  - [ p ] Why use conda and Module 1 summary
- Working with Environments
  - [ p ] How to create and activate an environment
  - [ p ] Specifying a location for an environment
  - [ p ] Installing packages into an environment
  - [ p ] List packages and environments
  - [ p ] Deleting a package or environment
  - [ p ] Module 2 summary
- Channels and Packages
  - [ p ] Conda packages
  - [ p ] Conda channels
  - [ p ] Searching for packages
  - [ p ] Installing a package from a specific channel
  - [ p ] Adding a channel
  - [ p ] Module 3 summary
- Sharing and Deploying Environments
  - [ p ] Overview
  - [ p ] Environment files
  - [ p ] Creating an environment from a file
  - [ p ] Exporting an environment to a file
  - [ p ] Deploying an environment
  - [ p ] Setting up conda on JupyterLab
  - [ p ] Module 4 summary
- [ p ] Conclusion
[ p ] Conda Essentials
- [ p ] How to Use Anaconda Notebooks
- Introduction
  - [ p ] Conda Essentials: Introduction
- Module 1: The conda directory structure
  - [ p ] Conda Directory Structure: Learning Objectives
  - [ p ] How does conda work?
  - [ p ] What’s in the conda directory?
  - [ p ] Installing a new environment
  - [ p ] The conda directory structure
  - [ p ] What happens when an environment activates?
  - [ p ] Conda activation scripts
  - [ p ] Conda Directory Structure: Summary
  - [ p ] Exercise 1: Explore the environment directory
  - [ p ] Exercise 2: Locate Python in the new environment
  - [ p ] Exercise 3: Locating a newly installed Python package
- Module 2: Building packages
  - [ p ] Building Packages: Learning Objectives
  - [ p ] What are conda recipes?
  - [ p ] Public recipes: conda_gc_test and toolz
  - [ p ] Public recipes: prodigal and pandas
  - [ p ] Conda recipes summary
  - [ p ] How to build a package
  - [ p ] Building packages continued
  - [ p ] Uploading a package
  - [ p ] Building packages: Summary
  - [ p ] Exercise 1: Create a new recipe from scratch
  - [ p ] Exercise 2: Build a package from your recipe
  - [ p ] Exercise 3: Building packages review questions
- Module 3: Introduction to conda-build
  - [ p ] Introduction to conda-build: Learning Objectives
  - [ p ] What happens during the conda build process?
  - [ p ] Generating recipes for Python packages on PyPI
  - [ p ] Generating recipes for R packages on CRAN
  - [ p ] Writing a recipe and building a package from scratch
  - [ p ] Best practices for building packages
  - [ p ] Building packages: Summary
  - [ p ] Exercise 1: Add a source section to a recipe
  - [ p ] Exercise 2: Debugging a recipe
  - [ p ] Exercise 3: Installing a package
- [ p ] Conclusion
[ p ] Anaconda Server Basics
- [ p ] Creating a .condarc file and setting up the Anaconda CLI
- [ p ] Common Vulnerabilies and Exposures (CVEs)
- [ p ] Filtering Channels and Mirrors
- [ p ] Creating a Channel and Mirror in the UI
- [ p ] Creating a Mirror in the Command Line Interface (CLI)
- [ p ] Environments

Conda Essentials #

# Exercise 1: Explore the environment directory
conda activate base && echo $CONDA_PREFIX      # `~/anaconda3/`
ls $CONDA_PREFIX                               # 1. list the contents of the conda installation directory
ls $CONDA_PREFIX/envs                          # 2. list the contents of directory `envs`
conda create -n ce-exercises                   # 3. create a new environment named `ce-exercises`
ls $CONDA_PREFIX/envs                          # 4. list the contents of directory `envs` again

ls $CONDA_PREFIX                               # 5. list the contents of environment `ce-exercises`
conda activate $CONDA_PREFIX/envs/ce-exercises # 6. activate environment `ce-exercises` by its prefix, not its name
echo $CONDA_PREFIX                             # `~/anaconda3/envs/ce-exercises/`
conda install python                           # 7. install Python in environment `ce-exercises`
ls $CONDA_PREFIX                               # 8. list the contents of environment `ce-exercises` again

# Exercise 2: Locate Python in the new environment
ls $CONDA_PREFIX/bin/python                    # 1. locate the Python executable
ls $CONDA_PREFIX/lib/python3.11/               # 2. locate the Python package

# Exercise 3: Locating a newly installed Python package
ls $CONDA_PREFIX/lib/python3.11/site-packages       # 1. list the contents of directory `site-packages` within the Python installation directory
conda install numpy                                 # 2. install NumPy in environment `ce-exercises`
ls $CONDA_PREFIX/lib/python3.11/site-packages       # 3. list the contents of directory `site-packages` again
ls $CONDA_PREFIX/lib/python3.11/site-packages/numpy # 4. list the contents of the new directory and compare with https://github.com/numpy/numpy

mkdir my-recipe
echo -e 'package:\n  name: my-recipe\n  version: 0.0.0' > my-recipe/meta.yaml
conda config --set anaconda_upload no
conda build .
ls $CONDA_PREFIX/conda-bld/osx-64/
tar jxvf $CONDA_PREFIX/conda-bld/osx-64/my-recipe-0.0.0-0.tar.bz2

Anaconda Server Basics #

conda config                                                                 # generate file `~/.condarc` in the user home directory
conda config --set channel_alias http://gibbons-lms.cs.anaconda.com/api/repo # add Anaconda Server as a channel alias
conda config --prepend default_channels anaconda                             # set the default channel to channel `anaconda`

conda repo --version || conda install conda-repo-cli                         # check whether package `conda-repo-cli` is installed
conda repo config --set-site gibbons-lms.cs.anaconda.com                     # set the site for the conda-cli client (modify the Conda Repo CLI config file)
cat ~/.conda/repo-cli-config.yaml                                            # view the Conda Repo CLI config file
conda repo login                                                             # login to Anaconda Server via conda-cli
conda repo mirror --list-all

To Review #

How to import package package from the parent directory?

import sys
sys.path.append('..')
import package