1.2.7. Standard Library

Note

Reference document for this section:

1.2.7.1. os module: operating system functionality

“A portable way of using operating system dependent functionality.”

Directory and file manipulation

Current directory:

In [1]: os.getcwd()
Out[1]: '/tmp'

List a directory:

In [2]: os.listdir(os.curdir)
Out[2]:
['profile_i3xi3m6f',
'profile_yyn8bz65',
'profile_2fx6atix',
'profile_cil_76vt',
'profile__2g_qc5v',
'profile_7pdqo1q6',
'profile_gvlu3zgw',
'profile_ksup2j07',
'profile__rpa34ah',
'profile_bmftbsbt',
'profile__v_tn2g9',
'profile_4hghaamz',
'profile_ilyov4q2',
'profile_aendtq49',
'profile_mxiqgwq7',
'profile_1ss34u3n',
'profile_eathjl68',
'profile_esmf9xr_',
'profile_p1l2lo4k',
'dotnet-diagnostic-1641-10535-socket',
'profile_cvn_pilb',
'profile_7v8cpif2',
'profile_dhvc5ob4',
'dotnet-diagnostic-1625-10287-socket',
'profile_k7ph57lr',
'profile_g0me9dur',
'profile_yjz_6mum',
'profile_aqhr6dby',
'profile_uw0dacyv',
'profile_61cb913g',
'profile_2oiphfh2',
'profile_yt1_yzx9',
'profile_h590n7q_',
'profile_7lq96roz',
'profile_48c0v_uj',
'clr-debug-pipe-1641-10535-out',
'dotnet-diagnostic-594-862-socket',
'.font-unix',
'clr-debug-pipe-1625-10287-out',
'profile_lazdtr04',
'profile_8r48hx7g',
'profile_icb6id24',
'clr-debug-pipe-1641-10535-in',
'profile_liicnl4a',
'profile_mnj0qjla',
'systemd-private-6e72fa05ed2942df840c89b5a3a087a8-chrony.service-FH6gI2',
'profile_xki5z_tg',
'profile_n8n19_os',
'profile_thhfoe_t',
'profile_fifzwhxn',
'profile_uko9596w',
'profile_9gs68ja8',
'profile_0uvkhgpm',
'profile_udst2qmz',
'systemd-private-6e72fa05ed2942df840c89b5a3a087a8-systemd-resolved.service-qcj95S',
'.X11-unix',
'profile_fi09proe',
'profile_gsm6_89l',
'profile_6zfppuld',
'profile_uoszp233',
'.XIM-unix',
'profile_2fuxa4pv',
'snap-private-tmp',
'systemd-private-6e72fa05ed2942df840c89b5a3a087a8-systemd-logind.service-wZhxY1',
'clr-debug-pipe-1625-10287-in',
'profile_1ad88g5_',
'profile_z42ta0hs',
'profile_k26muwyp',
'profile_13olttdb',
'clr-debug-pipe-594-862-out',
'profile_34mfx0mp',
'profile_sfktz7bu',
'systemd-private-6e72fa05ed2942df840c89b5a3a087a8-haveged.service-48tDXq',
'www-data-temp-aspnet-0',
'.ICE-unix',
'profile_t7txzjdr',
'profile_su7n5cwo',
'profile_m2zlvycp',
'.Test-unix',
'profile_f5h7c5hi',
'profile_7wlvwisw',
'profile_1j5z0gcn',
'profile_w9nm2om4',
'profile_om3j3x9p',
'profile_uw1nyrma',
'profile_8jj8h8_z',
'profile_baht1qhb',
'profile_w34f6p1o',
'profile_1gictcqb',
'profile_1plczn8p',
'profile_bi8z_c_x',
'clr-debug-pipe-594-862-in']

Make a directory:

In [3]: os.mkdir('junkdir')
In [4]: 'junkdir' in os.listdir(os.curdir)
Out[4]: True

Rename the directory:

In [5]: os.rename('junkdir', 'foodir')
In [6]: 'junkdir' in os.listdir(os.curdir)
Out[6]: False
In [7]: 'foodir' in os.listdir(os.curdir)
Out[7]: True
In [8]: os.rmdir('foodir')
In [9]: 'foodir' in os.listdir(os.curdir)
Out[9]: False

Delete a file:

In [10]: fp = open('junk.txt', 'w')
In [11]: fp.close()
In [12]: 'junk.txt' in os.listdir(os.curdir)
Out[12]: True
In [13]: os.remove('junk.txt')
In [14]: 'junk.txt' in os.listdir(os.curdir)
Out[14]: False

os.path: path manipulations

os.path provides common operations on pathnames.

In [15]: fp = open('junk.txt', 'w')
In [16]: fp.close()
In [17]: a = os.path.abspath('junk.txt')
In [18]: a
Out[18]: '/tmp/junk.txt'
In [19]: os.path.split(a)
Out[19]: ('/tmp', 'junk.txt')
In [20]: os.path.dirname(a)
Out[20]: '/tmp'
In [21]: os.path.basename(a)
Out[21]: 'junk.txt'
In [22]: os.path.splitext(os.path.basename(a))
Out[22]: ('junk', '.txt')
In [23]: os.path.exists('junk.txt')
Out[23]: True
In [24]: os.path.isfile('junk.txt')
Out[24]: True
In [25]: os.path.isdir('junk.txt')
Out[25]: False
In [26]: os.path.expanduser('~/local')
Out[26]: '/home/runner/local'
In [27]: os.path.join(os.path.expanduser('~'), 'local', 'bin')
Out[27]: '/home/runner/local/bin'

Running an external command

In [28]: os.system('ls')
Out[28]: 0

Note

Alternative to os.system

A noteworthy alternative to os.system is the sh module. Which provides much more convenient ways to obtain the output, error stream and exit code of the external command.

In [29]: import sh
In [20]: com = sh.ls()
In [30]: print(com)
basic_types.rst exceptions.rst oop.rst standard_library.rst
control_flow.rst first_steps.rst python_language.rst
demo2.py functions.rst python-logo.png
demo.py io.rst reusing_code.rst
In [31]: print(com.exit_code)
0
In [23]: type(com)
Out[31]: sh.RunningCommand

Walking a directory

os.path.walk generates a list of filenames in a directory tree.

In [32]: for dirpath, dirnames, filenames in os.walk(os.curdir):
....: for fp in filenames:
....: print(os.path.abspath(fp))
....:
....:
/tmp/dotnet-diagnostic-1641-10535-socket
/tmp/dotnet-diagnostic-1625-10287-socket
/tmp/clr-debug-pipe-1641-10535-out
/tmp/dotnet-diagnostic-594-862-socket
/tmp/clr-debug-pipe-1625-10287-out
/tmp/clr-debug-pipe-1641-10535-in
/tmp/clr-debug-pipe-1625-10287-in
/tmp/clr-debug-pipe-594-862-out
/tmp/junk.txt
/tmp/clr-debug-pipe-594-862-in
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/dhist
/tmp/bookmarks
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README
/tmp/README

Environment variables:

In [33]: import os
In [34]: os.environ.keys()
Out[34]:
['_',
'FSLDIR',
'TERM_PROGRAM_VERSION',
'FSLREMOTECALL',
'USER',
'HOME',
'PATH',
'PS1',
'SHELL',
'EDITOR',
'WORKON_HOME',
'PYTHONPATH',
...
In [35]: os.environ['PYTHONPATH']
Out[35]: '.:/Users/cburns/src/utils:/Users/cburns/src/nitools:
/Users/cburns/local/lib/python2.5/site-packages/:
/usr/local/lib/python2.5/site-packages/:
/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5'
In [36]: os.getenv('PYTHONPATH')
Out[36]: '.:/Users/cburns/src/utils:/Users/cburns/src/nitools:
/Users/cburns/local/lib/python2.5/site-packages/:
/usr/local/lib/python2.5/site-packages/:
/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5'

1.2.7.2. shutil: high-level file operations

The shutil provides useful file operations:

  • shutil.rmtree: Recursively delete a directory tree.

  • shutil.move: Recursively move a file or directory to another location.

  • shutil.copy: Copy files or directories.

1.2.7.3. glob: Pattern matching on files

The glob module provides convenient file pattern matching.

Find all files ending in .txt:

In [37]: import glob
In [38]: glob.glob('*.txt')
Out[38]: ['junk.txt']

1.2.7.4. sys module: system-specific information

System-specific information related to the Python interpreter.

  • Which version of python are you running and where is it installed:

    In [39]: sys.platform
    
    Out[39]: 'linux'
    In [40]: sys.version
    Out[40]: '3.11.8 (main, Feb 7 2024, 04:02:05) [GCC 11.4.0]'
    In [41]: sys.prefix
    Out[41]: '/opt/hostedtoolcache/Python/3.11.8/x64'
  • List of command line arguments passed to a Python script:

    In [42]: sys.argv
    
    Out[42]:
    ['/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages/sphinx/__main__.py',
    '-b',
    'latex',
    '-d',
    'build/doctrees',
    '.',
    'build/latex']

sys.path is a list of strings that specifies the search path for modules. Initialized from PYTHONPATH:

In [43]: sys.path
Out[43]:
['/home/runner/work/scientific-python-lectures/scientific-python-lectures',
'/opt/hostedtoolcache/Python/3.11.8/x64/lib/python311.zip',
'/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11',
'/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/lib-dynload',
'/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages']

1.2.7.5. pickle: easy persistence

Useful to store arbitrary objects to a file. Not safe or fast!

In [44]: import pickle
In [45]: l = [1, None, 'Stan']
In [46]: with open('test.pkl', 'wb') as file:
....: pickle.dump(l, file)
....:
In [47]: with open('test.pkl', 'rb') as file:
....: out = pickle.load(file)
....:
In [48]: out
Out[48]: [1, None, 'Stan']

The PYTHONPATH Search Solution