Why do we use Python environments?

This is a blog about Python environments, what they are, and why they’re important. It starts with a long-winded analogy, which you can read, or just skip straight to The Explanation.

Imagine you work in a microbiology lab (maybe you don’t have to imagine). You specialise in expressing recombinant antibody fragments in E coli – that is, taking gene sequences from other species and putting them into E coli cells, then extracting the encoded proteins. Your E coli cells grow very happily in LB medium, which has the following recipe:

Your whole lab uses this, so it gets mixed in a single batch and divided into flasks as needed. Life is good.

You start working on a certain antibody fragment that’s particularly difficult to express. Whatever you try, your soluble yield is low – the cells don’t form vigorous cultures like you’re used to, they die earlier than you expect, and they produce a lot of insoluble sediment. You surmise that this new antibody is causing problems and is not expressing correctly. After some research, you find a paper (Wu et al., 2002) that claims some success in expressing a recombinant antibody fragment in a different bacteria, B subtillis, which the authors modified to contain molecular chaperones that help stop the protein from clumping together. Sounds promising! You decide to try the methods in the new paper, which mentions “super-rich broth” (unhelpfully giving no recipe – thanks Wu et al.). After a little research, you discover that “super-rich broth” probably refers to Terrific Broth, which is an enriched version of LB that looks like this:

You look at the recipes and decide it’s not too different. You have all these ingredients in your lab. Why not just modify the batch of LB? Add a couple of extra grams per litre of tryptone, bump up the yeast extract, pour in some glycerol, add the potassium salt and send it back to the autoclave. OK, there’s a little extra NaCl, but that’s probably fine. Full-steam-ahead with your B subtillis experiments!

Then your supervisor speaks to an old colleague in the same field, who’s making great progress using yeast as the host organism. “We will try yeast! S cerevisiae!” says your supervisor, in a peak of enthusiasm, demanding that you start right away. Very well. But all your flasks are now full of your bespoke Terrific Broth with extra NaCL. You look up the growth medium for yeast, and you find the recipe for YPD:

Yeast extract? Tick! Peptone? That’s basically the same as tryptone. Glucose? Well, you’ve already got glycerol in there, it’s just a different carbon source. We can work with this…we also have the extra salt, but yeast can handle a little salt, right? (It can, according to Illarionov et al., 2021). You go back to the big batch, dilute it by a factor of 2.4 to get the yeast extract right, add 15 g peptone per litre to bump up the diluted tryptone, and stir in 17.94 g of glucose per litre (which you reckon is about right when you’ve adjusted for the specific density of glycerol and difference in carbon concentration). Jobs a good ‘un!

Hopefully you see the issue here. Before long, your beloved E coli cultures won’t be growing as well as they used to (Kram et al., 2015), you won’t be able to replicate results, and your lab mates will hate you.

And now, dear reader, I will explain the analogy. When you first start working with Python, you install the Python interpreter. This lets you run simple scripts and code that only uses the core language. This is already extremely powerful, but if you work in science you probably outgrow it pretty quickly – maybe you came from R and you miss DataFrames, so you install Pandas. You’re interested in deep learning, so you follow a TensorFlow tutorial. You’re using code from a repository that accompanied a published paper, or that your labmate developed, and it has a bunch of dependencies – other code libraries that you need to install to get it to work. You follow the instructions in the README and install the dependencies, and everything works fine.

Problems arise when you start working on multiple projects. A new person joins your lab, and they prefer PyTorch to TensorFlow. You installed TensorFlow last year when you followed the tutorial, so it uses an old version cuDNN (NVidia’s GPU-accelerated deep learning library), but your labmate’s code uses the latest version of PyTorch, which needs brand new cuDNN. Now your TensorFlow code doesn’t run. This sort of problem could occur any time you install a new Python package; its dependencies might be compatible with what you already have, or they might not. If they’re not, you probably won’t know until things start going wrong, and it won’t be obvious why. On Unix systems, where some utilities depend on Python, you can end up breaking not just your own code but parts of the operating system too.

That’s where environments come in. Environments are a way of isolating dependencies between projects. You effectively install a separate version of Python, with it’s own software packages, for each project. Instead of going back and modifying the whole lab’s batch of medium every time you need to add something, potentially sabotaging other experiments in the process, you take a different vessel, add what you need, mix it up, and keep it separate.

Next time I’ll write a bit more about what Python environments look like on a machine, and how to use them in practice.

Wu, S. C., Yeung, J. C., Duan, Y., Ye, R., Szarka, S. J., Habibi, H. R., & Wong, S. L. (2002). Functional production and characterization of a fibrin-specific single-chain antibody fragment from Bacillus subtilis: effects of molecular chaperones and a wall-bound protease on antibody fragment production. Applied and environmental microbiology, 68(7), 3261–3269. https://doi.org/10.1128/AEM.68.7.3261-3269.2002

Illarionov, A., Lahtvee, P., & Kumar, R. (2021). Potassium and Sodium Salt Stress Characterization in the Yeasts Saccharomyces cerevisiae, Kluyveromyces marxianus, and Rhodotorula toruloides. Applied and environmental microbiology 87(13), e03100-20. https://doi.org/10.1128/AEM.03100-20

Kram, K. E., & Finkel, S. E. (2015). Rich Medium Composition Affects Escherichia coli Survival, Glycation, and Mutation Frequency during Long-Term Batch Culture. Applied and environmental microbiology, 81(13), 4442–4450. https://doi.org/10.1128/AEM.00722-15