I've got to get a directory listing that contains about 2 million files, but when I do an ls command on it nothing comes back. I've waited 3 hours. I've tried ls | tee directory.txt, but that seems to hang forever.
I assume the server is doing a lot of inode sorting. Is there any way to speed up the ls command to just get a directory listing of filenames? I don't care about size, dates, permission or the like at this time.
ls -U
will do the ls without sorting.
Another source of slowness is --color. On some linux machines, there is a convenience alias which adds --color=auto' to the ls call, making it look up file attributes for each file found (slow), to color the display. This can be avoided by ls -U --color=never or \ls -U.
Try using:
find . -type f -maxdepth 1
This will only list the files in the directory, leave out the -type f argument if you want to list files and directories.
You can redirect output and run the ls process in the background.
ls > myls.txt &
This would allow you to go on about your business while its running. It wouldn't lock up your shell.
Not sure about what options are for running ls and getting less data back. You could always run man ls to check.
I'm assuming you are using GNU ls? try
\ls
It will unalias the usual ls (ls --color=auto).
If a process "doesn't come back", I recommend strace to analyze how a process is interacting with the operating system.
In case of ls:
$strace ls
you would have seen that it reads all directory entries (getdents(2)) before it actually outputs anything. (sorting… as it was already mentioned here)
How about find ./ -type f (which will find all files in the currently directory)? Take off the -type f to find everything.
Things to try:
Check ls isn't aliased?
alias ls
Perhaps try find instead?
find . \( -type d -name . -prune \) -o \( -type f -print \)
Hope this helps.
What partition type are you using?
Having millions of small files in one directory it might be a good idea to use JFS or ReiserFS which have better performance with many small sized files.
You should provide information about what operating system and the type of filesystem you are using. On certain flavours of UNIX and certain filesystems you might be able to use the commands ff and ncheck as alternatives.
I had a directory with timestamps in the file names. I wanted to check the date of the latest file and found find . -type f -maxdepth 1 | sort | tail -n 1 to be about twice as fast as ls -alh.
You can also make use of xargs. Just pipe the output of ls through xargs.
ls | xargs
If that doesn't work and the find examples above aren't working, try piping them to xargs as it can help the memory usage that might be causing your problems.