Python Environment Variables, And How to Work with Them

Want to manage your config better? Learn how to work with environment variables in Python.

When I was teaching myself Python, I’d build projects to apply what I’d just learned. A subset of those involved connecting to a database and querying it using Python. Which meant I needed to store the database configuration and sensitive info such as username and password for authentication.

Hard coding such sensitive info in a Python script was not a good idea. I learned how to use config files and environment variables—along with Python’s built-in modules to work with them.

So whenever I need to use sensitive info such as passwords and API keys in my applications, I set them as environment variables and fetch them as needed. In this tutorial, I’ll walk you through environment variables and how to work with them in Python.

What Are Environment Variables?

Environment variables are variables external to your application that store configuration information, system settings, and the like. They are typically managed by the operating system or the application environment. Key characteristics of environment variables include:

• Name-Value Pairs: Environment variables consist of a name (also known as a key) and a corresponding value.
• System Scope: You can set environment variables at the system level, making them accessible to all processes running on the system. If needed, you can also modify or define them at the application level, affecting only that specific application.
• Dynamic and Mutable: You can modify environment variables during runtime, providing flexibility.

How Environment Variables Are Helpful

Environment variables offer several benefits for managing configuration and sensitive information in your Python applications:

• Separation of Concerns: By storing configuration outside of your code, you keep the concerns of configuration management separate from your application logic.
• Security: You can store sensitive data, such as API keys and database credentials, in environment variables—without exposing them in source code—reducing the risk of exposure. 
• Flexibility: With environment variables, updating configuration settings is simple, as you can update/make changes outside of the code base. Environment variables allow you to adjust configuration settings without modifying your code. This flexibility is especially useful for deploying applications to different environments or when updating credentials.

In the following sections of this tutorial, we’ll explore how to set, access, and manage environment variables in Python and how they enhance configuration management in your projects.

How to Set Environment Variables

You can set environment variables using the command line. The scope of such environment variables applies only to the current session—and they do not persist outside of the current session.

If you’re on a Mac or Linux machine, you can set an environment variable in your current terminal session like so:

export MY_VARIABLE=my_value

If you’re a Windows user, you can set an environment variable temporarily as shown:

set MY_VARIABLE=my_value

Access Environment Variables in Python

Python provides the os module for operating system-related functionality. And os.environ is a dictionary of environment variables. The names of the environment variables and their values are the keys and values of the dictionary, respectively.

So you can access the values of environment variables—using (their names as) the keys—just the way you’d access elements of a dictionary.

Here are a couple of examples:

import os
print(os.environ['HOME'])
# Output: /home/balapriya

print(os.environ['USER'])
# Output: balapriya

So far so good. But what happens if you try to access the value of an environment variable that was never set?

Let’s try to access API_KEY that we haven’t yet set:

print(os.environ['API_KEY'])

As expected, you’ll get a KeyError:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen os>", line 679, in __getitem__
KeyError: 'API_KEY'

Handling Key Errors

You can handle the KeyError as shown:

import os

try:
	api_key = os.environ['API_KEY']
	print(f'API_KEY is set to: {api_key}')
except KeyError:
	print('API_KEY is not set. Please configure it.')

This approach does not abruptly stop the execution of the program when a KeyError exception is raised. It, however, provides a descriptive error message:

# Output
API_KEY is not set. Please configure it.

So when the rest of the program does not execute as expected, we know that we’ve missed setting a required environment variable.

Accessing Environment Variables Using the get() Method

You can use the dictionary method get() to get the value of an environment variable. Instead of a KeyError, the get() method returns None if the variable is not found.

Accessing the NOT_SET variable which we haven’t set returns None:

print(os.environ.get('NOT_SET'))
# Output: None

I prefer raising a Key Error when the environment variable is not set. Then let it pass silently or be subsumed in the None that the get() method returns.

But the get() method is helpful when we can pass in a default value for a particular environment variable if it’s not set.

Here’s an example:

print(os.environ.get('HOME','/home/user'))
# Output: /home/balapriya

How to Manage Configuration with Environment Variables

Now let’s take a couple of practical examples where we use environment variables in our application.

Example 1: Configuring Database Connection Parameters

Say you want to connect to a PostgreSQL database from Python. To do so, you can install and use the psycopg2 connector:

pip install psycopg2

In this example, we use environment variables to configure the database connection parameters. If the environment variables are not set, we provide default values to use.

import os
import psycopg2  

# Retrieve database configuration from environment variables
db_host = os.environ.get('DB_HOST', 'localhost')
db_port = os.environ.get('DB_PORT', '5432')
db_user = os.environ.get('DB_USER', 'myuser')
db_password = os.environ.get('DB_PASSWORD', 'mypassword')

# Establish a database connection
try:
	connection = psycopg2.connect(
    	host=db_host,
    	port=db_port,
    	user=db_user,
    	password=db_password,
    	database='mydb'
	)
	print('Connected to the database!')
except Exception as e:
	print(f'Error connecting to the database: {e}')

Example 2: Managing API Keys

Let’s take another example that involves the use of API keys.

In addition to the ChatGPT interface, you can also use the OpenAI API to support OpenAI LLMs in your applications.

When you sign up for an OpenAI account, you’d (typically see) some free time-limited API credits. Grab your API key by navigating to Settings > View API Keys.

You can use the Open AI Python SDK and a framework like LangChain to build applications. To do so you need to install the libraries (in a virtual environment) using pip:

pip install openai
pip install langchain 

Here’s how you can set the OPENAI_API_KEY as an environment variable:

import os
os.environ["OPENAI_API_KEY"]='your-api-key'

You can now access the Open AI LLMs in your script like so:

from langchain.llms import OpenAI
model=OpenAI(model_name="gpt-3.5-turbo")

How to Modify Environment Variables in Python

You can tap into the os.environ dictionary from the os module to modify environment variables within the current Python process:

import os

# Modify an existing environment variable or create a new one
os.environ['MY_VARIABLE'] = 'new_value'

In Python, you can use the subprocess module to spawn subprocesses from existing Python script. Which is helpful when you want to run system programs in Python.

In the following example, we modify the PATH environment variable by tapping into the os.environ dictionary. We then run echo $PATH as a subprocess:

import os
import subprocess

# Set a custom environment variable for the subprocess
os.environ['PATH'] = '/custom/path'

# Run a subprocess that accesses the PATH environment variable
result = subprocess.run("echo $PATH", shell=True, stdout=subprocess.PIPE)
output = result.stdout.decode()
print(output)
print(f'Subprocess output: {output}')

We see that the PATH takes the value of /custom/path:

# Output
/custom/path

Scope of the Modified Environment Variables

It’s important to note that these environment variable updates are temporary and are valid only for the current Python process. Once the script terminates, the changes are discarded:

• Current Python Process: When you modify an environment variable using os.environ within your Python script, the change is local to the current Python process. It won’t affect other running processes or future Python sessions.
• Child Processes: Environment variable changes made within the current Python process are inherited by child processes created by your script. For example, if you spawn a subprocess from your Python script (parent process), the child process will have access to the modified environment variables (as seen in the example).
• Not System Wide: Environment variables set within a Python script will not persist outside of that script’s execution.

If you need to make persistent changes to environment variables at the system level, you typically need to do so using operating system-specific methods.

How to Load .env Files with python-dotenv

The python-dotenv library is a popular Python package that simplifies the process of loading environment variables from a .env file into your Python project. It is particularly helpful when you have multiple environments (e.g., development, production) with different configurations, and you want to keep these settings separate from your source code.

Installing python-dotenv

To use python-dotenv, you need to install it first. You can install it—inside a virtual environment—using pip, the Python package manager:

pip install python-dotenv

Loading Environment Variables from a .env File

You can now create a .env file in your project’s root directory and populate it with key-value pairs, just like regular environment variables. Let’s create the following .env file with placeholder values:

API_KEY=your_api_key_here
DB_PASSWORD=your_database_password_here

You can now load the environment variables from the .env file using python-dotenv like so:

import os
from dotenv import load_dotenv

# Load environment variables from .env file
load_dotenv()

# Access the environment variables
api_key = os.getenv("API_KEY")
database_password = os.getenv("DB_PASSWORD")

# Print out the env variables
print(f"API Key: {api_key}")
print(f"Database Password: {database_password}")

Notice that we’ve used os.getenv(VARIABLE_NAME) to get the values of the environment variables. This is also a valid (and less commonly used) way to access environment variables.

Here’s the output:

API Key: your-api-key-here
Database Password: your-database-url-here

In this example:

• We use load_dotenv() to load the environment variables defined in the .env file into the current environment.
• We then use os.getenv() to access the environment variables: API_KEY and DB_PASSWORD.

Conclusion

And that’s a wrap! I hope you’ve learned how to manage configuration and sensitive information using environment variables in Python applications. We’ve covered the basics of setting and accessing environment variables, as well as their practical use in configuring applications. 

While environment variables are certainly helpful in separating config from source code, you should store sensitive variables as secrets in production use cases.

For managing secrets, I recommend exploring tools like HashiCorp Vault or AWS Secrets Manager.

Next, you may also read about Python Floats and Python Flask.