How to use glob() to find files recursively?

pathlib.Path.rglob

Use pathlib.Path.rglob from the the pathlib module, which was introduced in Python 3.5.

from pathlib import Path

for path in Path('src').rglob('*.c'):
    print(path.name)

If you don't want to use pathlib, use can use glob.glob('**/*.c'), but don't forget to pass in the recursive keyword parameter and it will use inordinate amount of time on large directories.

For cases where matching files beginning with a dot (.); like files in the current directory or hidden files on Unix based system, use the os.walk solution below.

os.walk

For older Python versions, use os.walk to recursively walk a directory and fnmatch.filter to match against a simple expression:

import fnmatch
import os

matches = []
for root, dirnames, filenames in os.walk('src'):
    for filename in fnmatch.filter(filenames, '*.c'):
        matches.append(os.path.join(root, filename))

For python >= 3.5 you can use **, recursive=True :

import glob
for f in glob.glob('/path/**/*.c', recursive=True):
    print(f)

If recursive is True, the pattern ** will match any files and zero or more directories and subdirectories. If the pattern is followed by an os.sep, only directories and subdirectories match.

Python 3.6 Demo

Similar to other solutions, but using fnmatch.fnmatch instead of glob, since os.walk already listed the filenames:

import os, fnmatch


def find_files(directory, pattern):
    for root, dirs, files in os.walk(directory):
        for basename in files:
            if fnmatch.fnmatch(basename, pattern):
                filename = os.path.join(root, basename)
                yield filename


for filename in find_files('src', '*.c'):
    print 'Found C source:', filename

Also, using a generator alows you to process each file as it is found, instead of finding all the files and then processing them.

import os
import fnmatch


def recursive_glob(treeroot, pattern):
    results = []
    for base, dirs, files in os.walk(treeroot):
        goodfiles = fnmatch.filter(files, pattern)
        results.extend(os.path.join(base, f) for f in goodfiles)
    return results

fnmatch gives you exactly the same patterns as glob, so this is really an excellent replacement for glob.glob with very close semantics. An iterative version (e.g. a generator), IOW a replacement for glob.iglob, is a trivial adaptation (just yield the intermediate results as you go, instead of extending a single results list to return at the end).

You'll want to use os.walk to collect filenames that match your criteria. For example:

import os
cfiles = []
for root, dirs, files in os.walk('src'):
  for file in files:
    if file.endswith('.c'):
      cfiles.append(os.path.join(root, file))

Consider pathlib.rglob().

This is like calling Path.glob() with "**/" added in front of the given relative pattern:

import pathlib


for p in pathlib.Path("src").rglob("*.c"):
    print(p)

See also @taleinat's related post here and a similar post elsewhere.

In case this may interest anyone, I've profiled the top three proposed methods. I have about ~500K files in the globbed folder (in total), and 2K files that match the desired pattern.

here's the (very basic) code

import glob
import json
import fnmatch
import os
from pathlib import Path
from time import time


def find_files_iglob():
    return glob.iglob("./data/**/data.json", recursive=True)


def find_files_oswalk():
    for root, dirnames, filenames in os.walk('data'):
        for filename in fnmatch.filter(filenames, 'data.json'):
            yield os.path.join(root, filename)

def find_files_rglob():
    return Path('data').rglob('data.json')

t0 = time()
for f in find_files_oswalk(): pass    
t1 = time()
for f in find_files_rglob(): pass
t2 = time()
for f in find_files_iglob(): pass 
t3 = time()
print(t1-t0, t2-t1, t3-t2)

And the results I got were:
os_walk: ~3.6sec
rglob ~14.5sec
iglob: ~16.9sec

The platform: Ubuntu 16.04, x86_64 (core i7),

import os, glob

for each in glob.glob('path/**/*.c', recursive=True):
    print(f'Name with path: {each} \nName without path: {os.path.basename(each)}')

• glob.glob('*.c') :matches all files ending in .c in current directory
• glob.glob('*/*.c') :same as 1
• glob.glob('**/*.c') :matches all files ending in .c in the immediate subdirectories only, but not in the current directory
• glob.glob('*.c',recursive=True) :same as 1
• glob.glob('*/*.c',recursive=True) :same as 3
• glob.glob('**/*.c',recursive=True) :matches all files ending in .c in the current directory and in all subdirectories

For python 3.5 and later

import glob

#file_names_array = glob.glob('path/*.c', recursive=True)
#above works for files directly at path/ as guided by NeStack

#updated version
file_names_array = glob.glob('path/**/*.c', recursive=True)

further you might need

for full_path_in_src in  file_names_array:
    print (full_path_in_src ) # be like 'abc/xyz.c'
    #Full system path of this would be like => 'path till src/abc/xyz.c'

If the files are on a remote file system or inside an archive, you can use an implementation of the fsspec AbstractFileSystem class. For example, to list all the files in a zipfile:

from fsspec.implementations.zip import ZipFileSystem
fs = ZipFileSystem("/tmp/test.zip")
fs.glob("/**")  # equivalent: fs.find("/")

or to list all the files in a publicly available S3 bucket:

from s3fs import S3FileSystem
fs_s3 = S3FileSystem(anon=True)
fs_s3.glob("noaa-goes16/ABI-L1b-RadF/2020/045/**")  # or use fs_s3.find

you can also use it for a local filesystem, which may be interesting if your implementation should be filesystem-agnostic:

from fsspec.implementations.local import LocalFileSystem
fs = LocalFileSystem()
fs.glob("/tmp/test/**")

Other implementations include Google Cloud, Github, SFTP/SSH, Dropbox, and Azure. For details, see the fsspec API documentation.

This is a working code on Python 2.7. As part of my devops work, I was required to write a script which would move the config files marked with live-appName.properties to appName.properties. There could be other extension files as well like live-appName.xml.

Below is a working code for this, which finds the files in the given directories (nested level) and then renames (moves) it to the required filename

def flipProperties(searchDir):
   print "Flipping properties to point to live DB"
   for root, dirnames, filenames in os.walk(searchDir):
      for filename in fnmatch.filter(filenames, 'live-*.*'):
        targetFileName = os.path.join(root, filename.split("live-")[1])
        print "File "+ os.path.join(root, filename) + "will be moved to " + targetFileName
        shutil.move(os.path.join(root, filename), targetFileName)

This function is called from a main script

flipProperties(searchDir)

Hope this helps someone struggling with similar issues.