25  Directory navigation

Navigting directories is essential for reading, saving, changing, adding, and removing data files. Python’s os and ‘glob’ modules provide plenty of tools for directory navigation and file operations.

os module

The operating, os, module provides a way of using operating system-dependent functionality like reading or writing to the file system, managing file and directory paths, and working with file permissions.

# Import module
import os

# Finding current working directory
current_dir = os.getcwd()
print("Current Working Directory:", current_dir)

Current Working Directory: /Users/andrespatrignani/Soil Water Lab Dropbox/Andres Patrignani/Teaching/Scientific programming/pynotes-agriscience/basic_concepts

# Move up one directory (exit current folder)
os.chdir('..')

# Changing Directory
os.chdir('/path/to/new/directory')  # Navigate to a new directory

# Create new directory
new_dir = os.path.join(current_dir, 'soil_data')
os.makedirs(new_dir, exist_ok=True)  # it will override if directory already exists

# Remove empty directory
dir_to_remove = os.path.join(os.getcwd(), 'soil_data')
os.rmdir(dir_to_remove)

# Remove directory containing files (need to import shutil module)
# shutil.rmtree(dir_to_remove)

# Splitting file names into parts
file_name = "soil_samples.csv"
file_path = os.path.join(current_dir, file_name)
dir_name, file_name = os.path.split(file_path)
file_name_only, file_extension = os.path.splitext(file_name)
print(f"Directory:{dir_name} \nFile Name:{file_name_only} \nExtension:{file_extension}")

# List specific files
files_list = os.listdir(current_dir)
for file in files_list:
    if file.endswith(".csv"):
        print(file)

Using the glob module

The glob module is similar to the os module, but is widely used for file name pattern matching and allows you to search for files and directories using a convenient wildcard matching that is particularly handy for listing files in a directory, especially when you’re interested in files of a specific type (such as csv, txt, jpg). To target files with a specific extension, we use a wildcard symbol (*), which acts as a placeholder for 'all files of a particular type'. For example, to list all the text files in a given directory, we use the pattern *.txt.

# Import module
import glob

# Get current working directory
print(glob.os.getcwd())

# Move up one directory (exit current folder)
glob.os.chdir('..')

# Get all files in directory
print(glob.os.listdir())

# List and store all files with specific file extension
txt_files = [file in glob.glob("*.csv")] # This is a list comprehension