1.2.7. Standard Library

Note

Reference document for this section:

1.2.7.1. os module: operating system functionality

“A portable way of using operating system dependent functionality.”

Directory and file manipulation

Current directory:

In [1]: import os
In [2]: os.getcwd()
Out[2]: '/tmp'

List a directory:

In [3]: os.listdir(os.curdir)
Out[3]:
['profile_jq0cbm05',
'profile_wmysubhk',
'profile_rnmmr4kj',
'clr-debug-pipe-593-882-out',
'profile_5adlsgo9',
'.X11-unix',
'www-data-temp-aspnet-0',
'profile_qbs222pi',
'profile_ed40tjl7',
'profile_c7ohtaiu',
'profile_ngm_n5p_',
'profile_7yw_8n6a',
'clr-debug-pipe-1632-21659-out',
'profile_1nk360if',
'profile_14068yov',
'systemd-private-e4ff52a3bafa496ebabc89929e9640ff-chrony.service-ny2Qn5',
'profile_tingcl5b',
'profile_e1ss0c1v',
'profile_6opmz5sq',
'profile_o81dw4_a',
'profile_i2d9y0oc',
'profile_h4y9spxa',
'profile_g5upcok1',
'systemd-private-e4ff52a3bafa496ebabc89929e9640ff-haveged.service-mCbVAg',
'profile_yof2so6a',
'profile_utm33h65',
'profile_bm1ptemh',
'profile_29e1pfyx',
'profile_bl6m4g2y',
'profile_18g2oy4f',
'profile_rae_wv42',
'profile_6tqwf8he',
'profile_pm5fbpk8',
'clr-debug-pipe-593-882-in',
'profile_whhao0sn',
'profile_27vmk9ui',
'profile_hrp7i0dm',
'profile_fgsvx9je',
'dotnet-diagnostic-593-882-socket',
'.Test-unix',
'profile_nzgej79l',
'systemd-private-e4ff52a3bafa496ebabc89929e9640ff-systemd-logind.service-qfka0l',
'profile_rngvq24a',
'profile_b_xhuhe8',
'profile_dpnn7d78',
'clr-debug-pipe-1614-21312-out',
'profile_vg0dfgad',
'profile_b2ri8kg6',
'profile_vovwjjeg',
'profile_gagucyuk',
'dotnet-diagnostic-1632-21659-socket',
'profile_5wh41b1_',
'profile_4o4m640d',
'dotnet-diagnostic-1614-21312-socket',
'.font-unix',
'profile_8v4zf266',
'profile_wbwopcqo',
'profile_iv0ref7k',
'profile_fjbbvrr9',
'profile_3znymg07',
'profile_c8i6omqz',
'profile_i22ocvio',
'.XIM-unix',
'profile_24qgu1zi',
'profile_3tr4v7i8',
'clr-debug-pipe-1614-21312-in',
'profile_8p132fjy',
'profile_4uoy88x8',
'profile_ts0en99j',
'profile_bv3hpybf',
'profile_wzz667co',
'profile_hm7kxgbb',
'clr-debug-pipe-1632-21659-in',
'systemd-private-e4ff52a3bafa496ebabc89929e9640ff-systemd-resolved.service-FLqBuE',
'profile_kg6626nx',
'profile_vig4_pvf',
'profile_q7r2lvl0',
'profile_v5_sqnnf',
'profile_7gjwupq_',
'profile_hri7manl',
'profile_cm8fbqgi',
'profile_rncm4ltf',
'profile_dtoiwgiu',
'profile_vgf_laby',
'profile_g1qh0dbp',
'profile_8nwv5w9d',
'.ICE-unix',
'profile_0l_mqplq',
'profile_2g9agbsx',
'profile_7kxoz1l7',
'snap-private-tmp']

Make a directory:

In [4]: os.mkdir('junkdir')
In [5]: 'junkdir' in os.listdir(os.curdir)
Out[5]: True

Rename the directory:

In [6]: os.rename('junkdir', 'foodir')
In [7]: 'junkdir' in os.listdir(os.curdir)
Out[7]: False
In [8]: 'foodir' in os.listdir(os.curdir)
Out[8]: True
In [9]: os.rmdir('foodir')
In [10]: 'foodir' in os.listdir(os.curdir)
Out[10]: False

Delete a file:

In [11]: fp = open('junk.txt', 'w')
In [12]: fp.close()
In [13]: 'junk.txt' in os.listdir(os.curdir)
Out[13]: True
In [14]: os.remove('junk.txt')
In [15]: 'junk.txt' in os.listdir(os.curdir)
Out[15]: False

os.path: path manipulations

os.path provides common operations on pathnames.

In [16]: fp = open('junk.txt', 'w')
In [17]: fp.close()
In [18]: a = os.path.abspath('junk.txt')
In [19]: a
Out[19]: '/tmp/junk.txt'
In [20]: os.path.split(a)
Out[20]: ('/tmp', 'junk.txt')
In [21]: os.path.dirname(a)
Out[21]: '/tmp'
In [22]: os.path.basename(a)
Out[22]: 'junk.txt'
In [23]: os.path.splitext(os.path.basename(a))
Out[23]: ('junk', '.txt')
In [24]: os.path.exists('junk.txt')
Out[24]: True
In [25]: os.path.isfile('junk.txt')
Out[25]: True
In [26]: os.path.isdir('junk.txt')
Out[26]: False
In [27]: os.path.expanduser('~/local')
Out[27]: '/home/runner/local'
In [28]: os.path.join(os.path.expanduser('~'), 'local', 'bin')
Out[28]: '/home/runner/local/bin'

Running an external command

In [29]: os.system('ls')
Out[29]: 0

Note

Alternative to os.system

A noteworthy alternative to os.system is the sh module. Which provides much more convenient ways to obtain the output, error stream and exit code of the external command.

In [30]: import sh
In [31]: com = sh.ls()
In [31]: print(com)
basic_types.rst exceptions.rst oop.rst standard_library.rst
control_flow.rst first_steps.rst python_language.rst
demo2.py functions.rst python-logo.png
demo.py io.rst reusing_code.rst
In [32]: type(com)
Out[32]: str

Walking a directory

os.path.walk generates a list of filenames in a directory tree.

In [33]: for dirpath, dirnames, filenames in os.walk(os.curdir):
....: for fp in filenames:
....: print(os.path.abspath(fp))
....:
....:
/tmp/junk.txt
/tmp/clr-debug-pipe-593-882-out
/tmp/clr-debug-pipe-1632-21659-out
/tmp/clr-debug-pipe-593-882-in
/tmp/dotnet-diagnostic-593-882-socket
/tmp/clr-debug-pipe-1614-21312-out
/tmp/dotnet-diagnostic-1632-21659-socket
/tmp/dotnet-diagnostic-1614-21312-socket
/tmp/clr-debug-pipe-1614-21312-in
/tmp/clr-debug-pipe-1632-21659-in
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/dhist
/tmp/bookmarks
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README

Environment variables:

In [34]: os.environ.keys()
Out[34]: KeysView(environ({'SHELL': '/bin/bash', 'COLORTERM': 'truecolor', ...}))
In [35]: os.environ['SHELL']
Out[35]: '/bin/bash'

1.2.7.2. shutil: high-level file operations

The shutil provides useful file operations:

  • shutil.rmtree: Recursively delete a directory tree.

  • shutil.move: Recursively move a file or directory to another location.

  • shutil.copy: Copy files or directories.

1.2.7.3. glob: Pattern matching on files

The glob module provides convenient file pattern matching.

Find all files ending in .txt:

In [36]: import glob
In [37]: glob.glob('*.txt')
Out[37]: ['junk.txt']

1.2.7.4. sys module: system-specific information

System-specific information related to the Python interpreter.

  • Which version of python are you running and where is it installed:

In [38]: import sys
In [39]: sys.platform
Out[39]: 'linux'
In [40]: sys.version
Out[40]: '3.11.9 (main, Apr 2 2024, 15:19:53) [GCC 11.4.0]'
In [41]: sys.prefix
Out[41]: '/opt/hostedtoolcache/Python/3.11.9/x64'
  • List of command line arguments passed to a Python script:

In [42]: sys.argv
Out[42]:
['/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/sphinx/__main__.py',
'-b',
'latex',
'-d',
'build/doctrees',
'.',
'build/latex']

sys.path is a list of strings that specifies the search path for modules. Initialized from PYTHONPATH:

In [43]: sys.path
Out[43]:
['/home/runner/work/scientific-python-lectures/scientific-python-lectures',
'/opt/hostedtoolcache/Python/3.11.9/x64/lib/python311.zip',
'/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11',
'/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/lib-dynload',
'/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages']

1.2.7.5. pickle: easy persistence

Useful to store arbitrary objects to a file. Not safe or fast!

In [44]: import pickle
In [45]: l = [1, None, 'Stan']
In [46]: with open('test.pkl', 'wb') as file:
....: pickle.dump(l, file)
....:
In [47]: with open('test.pkl', 'rb') as file:
....: out = pickle.load(file)
....:
In [48]: out
Out[48]: [1, None, 'Stan']

The PYTHONPATH Search Solution