1.2.7. Standard Library

Note

Reference document for this section:

1.2.7.1. os module: operating system functionality

“A portable way of using operating system dependent functionality.”

Directory and file manipulation

Current directory:

In [1]: os.getcwd()

Out[1]: '/tmp'

List a directory:

In [2]: os.listdir(os.curdir)

Out[2]: 

['profile_n6e2yvpb',

 'profile_i5w52r0g',

 'profile_dd2cps06',

 'systemd-private-6cb39a8f56ff4f35b098575cabcaef43-systemd-resolved.service-OOg9Vb',

 'profile_cnk77g07',

 'profile_1_1h313i',

 'dotnet-diagnostic-574-952-socket',

 'profile_keaew7vg',

 'clr-debug-pipe-574-952-out',

 'profile_sxp0ssjs',

 '.XIM-unix',

 'profile_nunb9eqn',

 'profile_mfh6bqd1',

 'profile_8z0a4gjk',

 'profile_sd_xd2gf',

 'profile_wzw8cz42',

 'clr-debug-pipe-1641-7927-in',

 'profile_zobinlgv',

 'profile_nhp9ycf4',

 'profile_2kw88hpm',

 'profile_e9rx1vda',

 'profile_ut0bijef',

 'profile_vlfxssxi',

 'profile_cbog82o_',

 'profile_u24byxq2',

 'profile_skxk13h0',

 'systemd-private-6cb39a8f56ff4f35b098575cabcaef43-chrony.service-8CePWJ',

 'profile_75y9dmrh',

 'dotnet-diagnostic-1641-7927-socket',

 'profile_05nf6zk2',

 'profile_vsugpmbe',

 'profile_adgo7sy4',

 'profile_4_b6k5py',

 'profile_jv5q83al',

 'profile_qf6bqm_r',

 'profile_eqrow769',

 'profile_qfr_ahv0',

 'profile_8d1wpoyb',

 'clr-debug-pipe-1626-7612-out',

 'profile_8ol6sefb',

 'profile_rdyp73p6',

 'profile_ptsq640r',

 'profile_8gh5hfgm',

 'profile_lx3_rbc8',

 '.ICE-unix',

 'profile_fv31fgl5',

 'profile_yfes5_sn',

 'profile_m1pc4zsw',

 'profile_8kven2t3',

 'profile_5ks776pb',

 '.X11-unix',

 'profile_c48tgqaw',

 'profile_wqxhnig9',

 'profile__naev4ar',

 'profile_bzm6iz0f',

 'systemd-private-6cb39a8f56ff4f35b098575cabcaef43-systemd-logind.service-oHLKJR',

 '.font-unix',

 'profile_v8mzbl3v',

 'dotnet-diagnostic-1626-7612-socket',

 'clr-debug-pipe-1626-7612-in',

 'profile_caa3dj8j',

 'profile_877e0m08',

 'profile_9rycip6w',

 'profile_6mb63fbw',

 'www-data-temp-aspnet-0',

 'profile_9frinjm5',

 'profile_75q6u26s',

 'profile_852bnxfo',

 'snap-private-tmp',

 'profile_i6hn4juk',

 'profile_3d0jo8go',

 'clr-debug-pipe-574-952-in',

 'profile_k0_scl8n',

 'profile_dq9t9ubp',

 '.Test-unix',

 'profile_h_wp9c2r',

 'profile_e7p4alz0',

 'profile_g9u03yvd',

 'profile_4y3fe0xi',

 'profile_srl7ornc',

 'clr-debug-pipe-1641-7927-out',

 'profile_l5n_8wr5',

 'profile_yufdnoc8',

 'systemd-private-6cb39a8f56ff4f35b098575cabcaef43-haveged.service-8gtdBz',

 'profile_4j5ui1hh',

 'profile_2liw_03i',

 'profile_4rbw_z_y',

 'profile_ji6nofdj',

 'profile_97ixwl8h',

 'profile_895grgus',

 'profile_lmxijk2x',

 'profile_4pdwudu_']

Make a directory:

In [3]: os.mkdir('junkdir')



In [4]: 'junkdir' in os.listdir(os.curdir)

Out[4]: True

Rename the directory:

In [5]: os.rename('junkdir', 'foodir')



In [6]: 'junkdir' in os.listdir(os.curdir)

Out[6]: False



In [7]: 'foodir' in os.listdir(os.curdir)

Out[7]: True



In [8]: os.rmdir('foodir')



In [9]: 'foodir' in os.listdir(os.curdir)

Out[9]: False

Delete a file:

In [10]: fp = open('junk.txt', 'w')



In [11]: fp.close()



In [12]: 'junk.txt' in os.listdir(os.curdir)

Out[12]: True



In [13]: os.remove('junk.txt')



In [14]: 'junk.txt' in os.listdir(os.curdir)

Out[14]: False

os.path: path manipulations

os.path provides common operations on pathnames.

In [15]: fp = open('junk.txt', 'w')



In [16]: fp.close()



In [17]: a = os.path.abspath('junk.txt')



In [18]: a

Out[18]: '/tmp/junk.txt'



In [19]: os.path.split(a)

Out[19]: ('/tmp', 'junk.txt')



In [20]: os.path.dirname(a)

Out[20]: '/tmp'



In [21]: os.path.basename(a)

Out[21]: 'junk.txt'



In [22]: os.path.splitext(os.path.basename(a))

Out[22]: ('junk', '.txt')



In [23]: os.path.exists('junk.txt')

Out[23]: True



In [24]: os.path.isfile('junk.txt')

Out[24]: True



In [25]: os.path.isdir('junk.txt')

Out[25]: False



In [26]: os.path.expanduser('~/local')

Out[26]: '/home/runner/local'



In [27]: os.path.join(os.path.expanduser('~'), 'local', 'bin')

Out[27]: '/home/runner/local/bin'

Running an external command

In [28]: os.system('ls')

Out[28]: 0

Note

Alternative to os.system

A noteworthy alternative to os.system is the sh module. Which provides much more convenient ways to obtain the output, error stream and exit code of the external command.

In [29]: import sh

In [20]: com = sh.ls()



In [30]: print(com)

basic_types.rst   exceptions.rst   oop.rst              standard_library.rst

control_flow.rst  first_steps.rst  python_language.rst

demo2.py          functions.rst    python-logo.png

demo.py           io.rst           reusing_code.rst



In [31]: print(com.exit_code)

0

In [23]: type(com)

Out[31]: sh.RunningCommand

Walking a directory

os.path.walk generates a list of filenames in a directory tree.

In [32]: for dirpath, dirnames, filenames in os.walk(os.curdir):

   ....:     for fp in filenames:

   ....:         print(os.path.abspath(fp))

   ....: 

   ....: 

/tmp/dotnet-diagnostic-574-952-socket

/tmp/clr-debug-pipe-574-952-out

/tmp/clr-debug-pipe-1641-7927-in

/tmp/dotnet-diagnostic-1641-7927-socket

/tmp/clr-debug-pipe-1626-7612-out

/tmp/junk.txt

/tmp/dotnet-diagnostic-1626-7612-socket

/tmp/clr-debug-pipe-1626-7612-in

/tmp/clr-debug-pipe-574-952-in

/tmp/clr-debug-pipe-1641-7927-out

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/dhist

/tmp/bookmarks

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

/tmp/README

Environment variables:

In [33]: import os



In [34]: os.environ.keys()

Out[34]: 

['_',

 'FSLDIR',

 'TERM_PROGRAM_VERSION',

 'FSLREMOTECALL',

 'USER',

 'HOME',

 'PATH',

 'PS1',

 'SHELL',

 'EDITOR',

 'WORKON_HOME',

 'PYTHONPATH',

 ...



In [35]: os.environ['PYTHONPATH']

Out[35]: '.:/Users/cburns/src/utils:/Users/cburns/src/nitools:

/Users/cburns/local/lib/python2.5/site-packages/:

/usr/local/lib/python2.5/site-packages/:

/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5'



In [36]: os.getenv('PYTHONPATH')

Out[36]: '.:/Users/cburns/src/utils:/Users/cburns/src/nitools:

/Users/cburns/local/lib/python2.5/site-packages/:

/usr/local/lib/python2.5/site-packages/:

/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5'

1.2.7.2. shutil: high-level file operations

The shutil provides useful file operations:

  • shutil.rmtree: Recursively delete a directory tree.

  • shutil.move: Recursively move a file or directory to another location.

  • shutil.copy: Copy files or directories.

1.2.7.3. glob: Pattern matching on files

The glob module provides convenient file pattern matching.

Find all files ending in .txt:

In [37]: import glob



In [38]: glob.glob('*.txt')

Out[38]: ['junk.txt']

1.2.7.4. sys module: system-specific information

System-specific information related to the Python interpreter.

  • Which version of python are you running and where is it installed:

    In [39]: sys.platform

    Out[39]: 'linux'

    

    In [40]: sys.version

    Out[40]: '3.11.5 (main, Aug 28 2023, 12:39:56) [GCC 11.4.0]'

    

    In [41]: sys.prefix

    Out[41]: '/opt/hostedtoolcache/Python/3.11.5/x64'

  • List of command line arguments passed to a Python script:

    In [42]: sys.argv

    Out[42]: 

    ['/opt/hostedtoolcache/Python/3.11.5/x64/lib/python3.11/site-packages/sphinx/__main__.py',

     '-b',

     'latex',

     '-d',

     'build/doctrees',

     '.',

     'build/latex']

sys.path is a list of strings that specifies the search path for modules. Initialized from PYTHONPATH:

In [43]: sys.path

Out[43]: 

['/home/runner/work/scientific-python-lectures/scientific-python-lectures',

 '/opt/hostedtoolcache/Python/3.11.5/x64/lib/python311.zip',

 '/opt/hostedtoolcache/Python/3.11.5/x64/lib/python3.11',

 '/opt/hostedtoolcache/Python/3.11.5/x64/lib/python3.11/lib-dynload',

 '/opt/hostedtoolcache/Python/3.11.5/x64/lib/python3.11/site-packages']

1.2.7.5. pickle: easy persistence

Useful to store arbitrary objects to a file. Not safe or fast!

In [44]: import pickle



In [45]: l = [1, None, 'Stan']



In [46]: with open('test.pkl', 'wb') as file:

   ....:     pickle.dump(l, file)

   ....: 



In [47]: with open('test.pkl', 'rb') as file:

   ....:     out = pickle.load(file)

   ....: 



In [48]: out

Out[48]: [1, None, 'Stan']

The PYTHONPATH Search Solution