read a file with bash

I'm going to try to post a few howto articles for some of the simpler tasks. Many of these will be things that I use daily.

It is often necessary to read a file with bash, and act upon the entire line. There are many different ways to do this, but I'll outline two of the simpler methods, both suitable for stacking on a single command line.

For this exercise I'll assume the file is a list of files that we need to execute a command on.

# cat file.lst |while read line; do echo "${line}"; done /tmp/file1.txt /tmp/file with space.txt #

All we've managed to do here is cat the file.

Alternately we could move the cat to the end of the while loop as follows:

# while read line; do echo "${line}"; done < <(cat file.lst) /tmp/file1.txt /tmp/file with space.txt #

So we have a list of txt files. If we need to rename them all to .text we could use the following command.

# cat file.lst |while read line; do newname=$(echo ${line}|sed 's/txt$/text/'); mv -v "${line}" "${newname}"; done `/tmp/file1.txt' -> `/tmp/file1.text' `/tmp/file with space.txt' -> `/tmp/file with space.text' #

I prefer to use a -v when using a loop to perform tasks such as this. Adding the -i argument (inquire) will also ask before overwriting files. Another tip when doing something potentially destructive is to put an echo before the mv so you echo the commands instead of executing them.

You also aren't limited to getting the list of files from a file. ls, find, and many other commands can be used to obtain the list. For example:

# find /tmp -name '*.txt' | while read line; do echo "${line}"; done /tmp/file1.txt /tmp/file with space.txt #

That's it for tonight. If anyone has questions, requests, or suggestions, please post comments, or send me an email. My address is anton at this domain.

69 thoughts on “read a file with bash

  1. Pingback: Notes From The President » Blog Archive » how to manipulate a txt file one line at a time

  2. Thanks, this pointed me in the right direction for what I was trying to do.
    The problem with using cat though, is any variables assigned within the while loop are local to the subprocess that bash will execute cat in, meaning that:

    VALUE=””
    cat /path/to/file | while read line; do
    [ “${line%%:*}” == “Keyword” ] && VALUE=”${line#*:}”
    done
    echo $VALUE

    doesn’t work as expected (assuming there is a line ‘Keyword: something […]’ in /path/to/file), as VALUE is assigned local to the while loop, as bash spawned a subprocess for cat.

    A solution (there’s probably others):

    VALUE=””
    exec 9

  3. Sorry about the multiple posts.. ‘Submit Comment’ doesn’t substitute “<” for “&lt;”..

    A solution (there’s probably others):

    VALUE=””
    exec 9< /path/to/file # assign file descriptor 9 to file
    while read -u 9 line; do # read from file descriptor 9
    [ “${line%%:*}” == “Keyword” ] && VALUE=”${line#*:}”
    done
    exec 9<&- # free file descriptor 9
    echo $VALUE

    VALUE will be assigned local to the script, and will still be set to the value of ${line#*:} after the while loop has exited. Plus, the file is read using bash builtins only ๐Ÿ˜‰

  4. Switch the loop around like this:

    while read line ; do
    do stuff
    VAR=”123″
    done < <(cat file) echo $VAR

    Now the while loop happens in the current process and VAR is available when the loop finishes.

  5. Redirecting with a < is fine if you have a file. In most cases I am using commands to generate a list. while read line; do echo $line done < <(find . name '*mp3') Or more commonly: find . -name '*mp3'|while read line; do echo "$line" ; done

  6. A lot of the simple line processing jobs can more easily be done by use of xargs.

    For example:

    cat file.lst | xargs echo

    Instead of:

    cat file.lst |while read line; do echo “${line}”; done

    Yeah, I know this page was probably just an intro on how to read line by line from a shell script, but I wanted to point out that in many cases that is not necessary.

  7. The purpose is to show how to read a file line by line with bash and I would epxect you change the echo to a number of useful commands.

    As for using xargs:

    # cat file.lst
    file with spaces.txt
    file2.txt

    #cat file.lst | xargs echo
    file with spaces.txt file2.txt

    #cat file.lst | while read line ; do echo $line ; done
    file with spaces.txt
    file2.txt

    #cat file.lst | xargs rm -v
    rm: cannot remove `file’: No such file or directory
    rm: cannot remove `with’: No such file or directory
    rm: cannot remove `spaces.txt’: No such file or directory
    removed `file2.txt’

    #cat file.lst|while read line ; do rm -v “$line” ; done
    removed `file with spaces.txt’
    removed `file2.txt’

    There are differences and reasons not to use xargs, even for simple things like echo and rm.

  8. Hi ,

    Please see this example.

    The input file contents are as follows:
    Lets name it as test3

    ABC ABC ABC ??? + —- ##
    ABC ABC ABC ??? + —- ##
    ABC ABC ABC ??? + —- ##

    Now I am doing a while loop to read this ,like below

    cat test3|while read line
    do
    echo $line
    done

    The expected output is just like the above , however i am getting the below output

    ABC ABC ABC bin dev etc lib mnt net opt svm tmp usr var vol xfn + —- ##ABC ABC ABC bin dev etc lib mnt net opt svm tmp usr var vol xfn + —- ##
    ABC ABC ABC bin dev etc lib mnt net opt svm tmp usr var vol xfn + —- ##

    Could you please let me know why is this so ?

    Few things about this problem
    1. This problem happens only when I execute this script as a cron job , Manual execution gives the proper output
    2. This works fine If I remove the “???” characters from the file test3. that is ., I am getting this problem only when the input file has the character “???”

    Thanks ,
    Dilip

  9. BASH is expanding the ??? to all 3 letter files (and directories) in the current directory. Crontab runs from / so that’s what it’s matching. When I run it in my home dir I see bin, log, and src dirs.

    This is the same mechanism that allows rm ??? to remove all 3 letter files.

    Wrap the $line in “s to prevent bash from doing the replacement.

    cat tmp.txt | while read line ; do echo “$line” ; done

  10. Hi,

    Here all examples for “while read” are mentioned with a variable like “line”. My question is what variable should we check for the same while loop which has no variable. Example:

    while read
    do
    here a progress bar is generated
    done

    I need to log the output in text file too.

    your help will be appreciated greatly.
    Amit

  11. Amit,

    read, without a variable, uses the default of $REPLY.

    echo “Enter your name”
    read

    echo $REPLY

    I would recommend setting the variable though just to prevent confusion when you attempt to nest a while read inside another.

  12. Thank you Anton and everybody for messages,

    i want make bash file.
    everything is allright but i want get variable text other file. I can’t make this.
    how can make this?

  13. Erdem,

    I’m not entirely sure I understand your question. I’m assuming that you want to place the contents of a file into a variable. You could do this by:
    MYVAR=$(cat file.txt)

  14. Anton,

    — THIS IS VARIABLE FILE NAME download.txt —
    prg_1_name=”Anton’s Blog”
    prg_1_file=”http://……./anton.tar.gz”
    prg_1_fname=”anton.tar.gz”
    prg_1_dname=”anton”

    etc.
    .
    .
    i want get this file contents when bash file is execute..

    how can make it?

  15. Hey

    I needed a refresher on how to read line by line from a file using a loop. I just wanted you to know that your post was extremely helpful, thank you!

  16. Erdem,

    If you have a script that’s got commands in it (like setting variables) then you can run it by using ‘.’.

    Something like:

    #!/bin/bash
    . ./config.sh

    echo $prg_1_name

  17. Hi Anton:

    I’m usually always on windows, but today, I got a shell account from a friend, and although I knew bits and pieces about *nix and shell scripting because of being a web dev, you really helped point me in the right direction here.

    Thanks,
    Curtis

  18. hey anton,

    hope all is well, i have a question that i am sure you can answer in a heartbeat… i have a command that is returning two lines, essentially, i want to take each of those lines and make a variable out of them, how do i do that with bash? ๐Ÿ˜‰

  19. I’m accessing ‘http://www.random.org/cgi-bin/checkbuf’
    It returns the single-line file ‘checkbuf’ with these contents”
    4%
    In this case I wouldn’t want to request a number of random bytes.
    How do I check the number (note it’s 4%, not 004%) and do the Right Thing?

    Ted

  20. I want to read from ls -R1p for use with curl. how can i ‘filter’ the output, so that directories are ignored? i also need to prefix files in subdirectories with directory names, after stripping off the colon at the end.

    I basically need to know how to look for specific characters, and how to strip them off then end.

  21. I’d use find to get all the files.

    find . -type f

    This will give you a list of all files, including path.

    If you need to strip off the path, use basename to do so.

    > FILE=./subdir/file.txt
    > FILENAME=$(basename “$FILE”)
    > echo $FILENAME
    file.txt

  22. Anton,

    I have read through all of the examples on reading a file using bash, but none appear to do what I’d like. I have a text file with 10 lines of data. Each line has 1 field basically, such as 0, 10, 12, 13, 14, with the comma prepresenting a n. I can get the bash read command to read a line with numerous fields in it, and assign those to variables, but how do I get the read command to read a field per line, assign that to a variable name, then read the next line, etc until EOL?

    Thanks!

  23. You can call read inside the loop

    # cat tmp
    1
    2
    3
    4
    5
    6

    # cat tmp | while read x1 ; do read x2 ; read x3 ; read x4 ; echo $x1 $x2 $x3 $x4 ; done

    The output would be:
    1 2 3 4
    5 6

    If you run out of file in the middle of the last loop the remaining vars will be blank.

  24. Again, delete the previous posts… I hope is the right tag for posting those code. WordPress seems to not convert strings like <…

    # while read line; do echo “${line}”; done

    This is Useless Use Of Cat and very bad style. Instead, remove the cat and subshell completely and just use:

    # while read line; do echo “${line}”; done

    I’d also suggest not using ${line} here, because it is completely unneccessary. Quoting $line is OK. We’ll get:

    # while read line; do echo “$line”; done

    which will work just fine.

    Uuh… same here:

    # find /tmp -name ‘*.txt’ | while read line; do echo “${line}”; done
    /tmp/file1.txt
    /tmp/file with space.txt
    #

    to

    # find /tmp -name ‘*.txt’ | while read line; do echo “$line”; done
    /tmp/file1.txt
    /tmp/file with space.txt
    #

    Also, this example is pure nonsense. find uses -print automagically, if you don’t specifiy and -fprint*, -print* or -exec switch. The while is completely senseless. If you want to echo the things with echo anyway and not with print, you should use -exec anyway.

    # find /tmp -name ‘*.txt’ -exec echo {} ;
    /tmp/file1.txt
    /tmp/file with space.txt
    #

    Although I do not see any point in it, since…

    # find /tmp -name ‘*.txt’
    /tmp/file1.txt
    /tmp/file with space.txt
    #

    is working just fine.

    Okay, that’s all.

    -Ionic

  25. Anton,

    Thank you for the response. Your example worked just great. Thank you very much! Now I need to pass the file generated by redirecting into a file, i.e., the echo $x1 $x2 $x3 $x4 ; done > ids2.txt and passing that as an argument into another bash script. Thanks again

  26. Ionic,

    The point isn’t to replace cat, or get find to output the files. That is just a simple example.

    The intent is to demonstrate reading something line by line and acting upon it. echo can be replaced with any other command, or a series of commands and maybe some logic.

    # find . -name ‘*jpg’ | while read pic ; do
    convert ${pic} ${pic}.png
    mv ${pic}.png /some/new/dir
    done

    Or you could pull pic apart and, replace the jpg with png properly, and maybe resize if it’s over X bytes, etc.

    I like to escape my shell vars so I don’t get into problems with concatination. It’s a personal preference and while other methods work, this is what I like. That way I don’t accidently try something like “$PATHnewdir” which isn’t the same as “${PATH}newdir”

  27. @Anton’s response to Geir:

    You can work around the spaces problem very easily in xargs, don’t write it off so soon.

    $ cat files.lst
    File with spaces.txt
    file2.txt

    $ cat files.lst |xargs -i rm -v “{}”
    removed `File with spaces.txt’
    removed `file2.txt’

    $cat files.lst |xargs -i echo {}
    File with spaces.txt
    file2.txt

    Using the find command from findutils, you can also specify -print0 to print NUL characters for spaces instead of whitespace. Xargs can them interpret the whitespace via the -0 (that’s a zero) flag. find -print0 and xargs -0 should never fail because UNIX paths won’t contain NULs.

  28. Anton,

    I have the following code to read in a file an process accordingly :

    #/tmp/test.txt looks like this :
    # 0-1|0-2|0-3|0-4
    # 1-1||1-3|1-4
    # |2-2|2-3|2-4
    # 3-1|3-2||3-4

    FILENAME=$1; #input filename
    OIFS=$IFS;
    IFS=’|’;

    while read col1 col2 col3 col4
    do
    if [ -n “$col1” ]; then
    fred=”col1 = ‘$col1’, “;
    else
    fred=””;
    fi
    if [ -n “$col2” ]; then
    fred=$fred”col2 = ‘$col2’, “;
    else
    fred=$fred;
    fi
    if [ -n “$col3” ]; then
    fred=$fred”col3 = ‘$col3’, “;
    else
    fred=$fred;
    fi
    if [ -n “$col4″ ]; then
    fred=$fred”col4 = ‘$col4′”;
    else
    fred=$fred;
    fi
    echo $fred;
    done <$FILENAME

    #the output looks like this
    col1 = ‘0-1’, col2 = ‘0-2’, col3 = ‘0-3’, col4 = ‘0-4’
    col1 = ‘1-1’, col3 = ‘1-3’, col4 = ‘1-4’
    col2 = ‘2-2’, col3 = ‘2-3’, col4 = ‘2-4’
    col1 = ‘3-1’, col2 = ‘3-2’, col4 = ‘3-4’

    I want to know – can I substitute ‘col1 – col4’ for a variable, since there is about 40 fields per line on the “real” file and to do more or less the same validation repetitively seems pointless.

    any suggestions would be appreciated.

    thanks

  29. In all cases above you can remove the else clause, it isn’t doing anything for you. Put ‘fred=””‘ at the top of the loop, and just append to it in each “if”.

    You can use variable variables inside the loop so you don’t have to repeat each if block.

    To see how they work, try this:
    VAR1=abc
    abc=123
    echo ${!VAR1}

    Inside your while read loop you could do something like this:

    fred=””
    for VAR in col1 col2 col3 col4; do
    if [ -n “${!VAR} ] ; then
    fred=$fred”${!VAR}”
    fi
    done

  30. Wow! Comment with file descriptor 9 (“read -u 9”) really saved my day! Big thanks. I’ve got “bash: fork: Resource temporarily unavailable” and no command could be run except bash builtins. This is very helpful and elegant way. You guy is a real bash guru ๐Ÿ™‚ So I have run this to find out the pid of java process and kill it to free resources:

    for f in /proc/[0-9]* ; do
    VALUE=””
    exec 9< $f/cmdline # assign file descriptor 9 to file
    read -u 9 line
    [[ $line =~ java ]] && echo "|$f|: |$line|"
    exec 9<&- # free file descriptor 9
    echo $VALUE
    done

  31. Ever here of the useless uses of cat awards?

    Commands like:

    cat file.txt|while read line; do echo “${line}”; done
    while read line; do echo “${line}”; done < <(cat file.lst)

    are wrong because they use cat.

    Here is a better form WITHOUT cat:

    while read line; do echo "${line}"; done cat file.txt | while read line; do echo $line; myvar=”i am set”; done
    This is the file contents
    myuser@myhost ~> echo $myvar

    myuser@myhost ~> while read line; do echo $line; myvar=”i am set”; done echo $myvar
    i am set

  32. David,

    I’m trying to parse what you wrote and it isn’t working well. Your first example “without cat” is just my example above, and the second one does nothing to read a file, it will just stall waiting for stdin.

  33. Anton: the reason xargs did not work properly with rm is not because it is not valid. The reason is the spaces in the file name. In general xargs is far safer if you are not 100% sure the number of arguments is too many (or indeed other things like newlines which is dangerous indeed).

    This (newlines, spaces, etc) is why there exists the option -0 to xargs. Actually I see now that someone already pointed this out.

    And David is right: cat file | while …. is wrong. I get the impression the reason you couldn’t get it working is html characters (as in less than and greater than as part of html). I’ll try to correct it:

    while read line; do
    echo $line
    done; < < filename

    In fact your example:
    # cat file.lst|while read line ; do rm -v โ€œ$lineโ€ ; done

    is a useless use of cat too.

    Something else is I fail to see the use of your example:
    # find /tmp -name '*.txt' | while read line; do echo "${line}"; done

    If you're looking for files ending with .txt in /tmp (which is what the find invocation would be doing) then what is the point of the piping it to the loop?
    find itself shows files it finds (-print is assumed if you don't give any expression though as the man page rightly will point out you should use -print0 where you can anyway).

    And if you want to do something to each file it finds then you have much more flexible solutions: -exec and family options in find. If its to delete files you can (though there are some potential pitfalls) use the -delete option (cannot recall if this is a gnu extension or not). I would recommend you look over the man page for find as there is a lot to it that would really empower you by a lot (there's security considerations which I mention some more later).

    And sysadmin: If you're getting that error you should look at your limits (specifically how many processes the user you are logged in as can have at one time) because that is what it is saying. fork is used to create a process.

    As for find -exec and other functions I highly recommend reading the Security Considerations chapter in 'info findutils'. One example (and is the reason the -0 option is preferred) is a file with a newline in it (Yes, files can have a newline character in it which without intervention would be interpreted as more than one line. Consider something like you wanting to remove any file matching some pattern and the pattern happens to match a file like this: patternn/etc/shadow )

  34. And as for the trying to correct the example from David.

    Change the two less than signs to one less than sign and then it should work. (well okay and you’ll need a semi-colon before done – accidentally did not type it).

  35. First off, I wrote this about 8 years ago, so a lot has changed since then including my level of knowledge.

    Second, lay off the useless use of cat complaints. It’s simply a method of getting data for an example. In reality I would have used something like find, or recursive grep to get the data I needed to act upon.

    Third, this is just an example of how to loop through a file or pipe. It isn’t a definitive list of things you can do inside that loop. I chose echo for simplicity, but in practice it would be whatever command you wished to perform.

    As for the example:
    find /tmp -name ‘*.txt’ | while read line; do echo “${line}”; done

    That’s a simple way to ensure that your loop is correct before replacing the echo with some useful command and potentially destroying data.

Leave a Reply

Your email address will not be published. Required fields are marked *