Unix Tools

The Find Tool

An interesting tool to pattern match files and their content, and execute actions for each

find /etc/ -maxdepth 2 -name nginx

ls -la $(find /etc/ -maxdepth 2 -name nginx)

# with xargs
find /etc/ -maxdepth 2 -name nginx | xargs ls -la

# find files and remove them with with xargs and rm
find /tmp -name core -type f -print0 | xargs -0 /bin/rm -f

find /etc/ -maxdepth 2 -name nginx | xargs ls -la

The grep tool

Check existence of text / regex in file, and take action upon the result

target_line='/dev/disk/by-id/scsi-0Linode_Volume_zvol /nfs/zvol ext4 defaults,noatime,nofail 0 2'
target_file='/etc/fstab'

line_found=$(grep -rnw "$target_file" -e "$target_line")

if [[ ! "$line_found" ]] ; then
    echo "appending the line";
    echo "$target_line" | sudo tee -a "$target_file" ;
else
    echo "line exists" ;
fi

target_line="mk ALL=(ALL) NOPASSWD:ALL"
target_file='/etc/sudoers'

line_found=$(grep -rnw "$target_file" -e "$target_line")

if [[ ! "$line_found" ]] ; then
    echo "appending the line";
    echo "$target_line" | sudo tee -a "$target_file" ;
else
    echo "line exists" ;
fi

Text Manipulation

The sed tool

https://www.gnu.org/software/sed/manual/sed.html

sed - stream editor for filtering and transforming text.

String - Regex Substitution

sed -i 's/original/new/g' file.txt

  • -i inplace

  • s Substitute

  • original regex

  • new text

  • g global means all occurrences

String replacement examples

# replace white space with newlines
hostname -I | sed 's/ /\n/g'

# replace colon with a space
echo $PATH | sed 's/:/ /g'

# replace colo with a newline
echo $PATH | sed 's/:/\n/g'

# replace a pattern within a given lines range (address selection)
seq 10 | sed '3,6s/^.*/number/g'

# negation of address selection
seq 10 | sed '3,6!s/^.*/number/g'

# allow currently installed packages to accept pip upgrades
pip freeze | sed 's/==/>=/g' > requirements.txt
pip install -r requirements.txt --upgrade

Text Filtering - Selection

Text selection can be done with this pattern

sed -n '/regex/p'

  • option -n removes all stream lines by default

  • the p command is to print specific lines that match the regex

# all user profiles who's default shell is bash
sed -n '/bash$/p' /etc/passwd

# Match everything that starts with a b, third letter is a d
printf "%s\n" abode bad bed bit bid byte body | sed -n '/^b.d/p'

# Match all 3 letter words that starts with a b, third letter is a d
printf "%s\n" abode bad bed bit bid byte body | sed -n '/^b.d$/p'

printf "%s\n" abode bad bed bit bid byte body | sed '/^b.d$/d'

Text filtering can be done with this command

sed ‘/regex/d’

# print users whose default shell is not bash
sed '/bash$/d' /etc/passwd

the tr tool

Translate or delete a single character from a text stream

This tool is limited to replacing a single character, which is very limited in comparison to sed

# replace spaces with a newline
hostname -I | tr ' ' '\n'

# replace spaces with a comma
hostname -I | tr ' ' ','

the awk Tool

https://www.gnu.org/software/gawk/manual/gawk.html

# text stream passthrough
cat /etc/passwd |  awk '{print}'

# or printing the zeroth field of each line, which is all the fields
cat /etc/passwd |  awk '{print $0}'

# specify a field separator (FS) and print the first field of each line
cat /etc/passwd |  awk 'BEGIN {FS = ":"} {print $1}'

# find usernames that use the bash shell with a regex /bash/, and print the username with the user id
cat /etc/passwd |  awk 'BEGIN {FS = ":"} /bash/ {print $1,$3}'

# specify the input and output output field separator
cat /etc/passwd |  awk 'BEGIN {FS = ":" ; OFS = ":" } /bash/ {print $1,$3}'

the cut Tool

Select specific fields in a stream

# select the first and fifth field in the passwd file
cut -d: -f1,5 /etc/passwd