Linux/task_1&2.rst

Practical 1 : Bash Scripting
=================================

 .. hint::
    This practical shall give a starting point into bash scripting. It it impossible to really cover all aspects and possibilties in the given amount of time. If you really want to
dive deeper into specific aspects which are not covered here, we highly encourage self-study, as there are a lot of really good tutorials out there on the internet for nearly everything
related to bash scripting.


Part 1: Bash basics
-------------------

Task 1.1: Hello World
---------------------

Create a file **hello_world.sh** with an editor of your choice (e.g. *nano*)

.. code-block:: bash

   $ nano hello_world.sh

(Save: CTRL+O, then ENTER ; Exit: CTRL+X)

Write a little bash script that loops forever and prints out "Hello Word" every second.
Do not forget to make your file executable before running it:


.. code-block:: bash

   $ chmod +x hello_world.sh
   $ ./hello_world.sh


Solution:

.. code-block:: bash

	#!/bin/bash
	...
..	while true; do echo Hello World; sleep 1; done



Task 1.2: Variables
-------------------

A variable is a so called container that stores data (a number or a string) for later usage. The contents of a variable can change continously -> it is variable. Create an executable file
**variables.sh** :

.. code-block:: bash

   $ touch variables.sh
   $ chmod +x variables.sh

Now edit this file with an editor of your choice and assign some random values to four variables. These four variables shall then be printed out to the console. To access the content of a variable
the "$" operator is used.

example:

.. code-block:: bash

	#!/bin/bash

	Var1=SomeContent
	Var2="SomeContent"
	Var3=3
	Var4="3"

	echo $Var1
	echo $Var2
	echo $Var3
	echo $Var4

Save your file and run it:

.. code-block:: bash

	$ ./variables.sh
	SomeContent
	SomeContent
	3
	3

At first look it does not seem to make a difference wether to use quotation marks or not. Quotation marks need to be used though in case some elements contain empty spaces, e.g. "element 1".
Variables can also be used with curly brackets in order to separate them from the rest of the code. Change your script as following and re-run it to see the results:

.. code-block:: bash

	echo ${Var1}
	echo ${Var2}
	echo ${Var3}
	echo ${Var4}


.. code-block:: bash

	$ ./variables.sh
	SomeContent
	SomeContent
	3
	3

Also here at first glance it does not seem to make a difference wether to use curly brackets or not. This gets quite important though when it comes to merging variables or concatenate them.
Add the following lines to your script and re-run it. What is the difference?


.. code-block:: bash

	echo $Var1Extended
	echo ${Var1}Extended

Now add two more lines and re-run it:


.. code-block:: bash

	echo $Var1$Var3
	echo ${Var1}${Var3}

Is there any difference?

Now try to embedd your variables into some random text surrounding it and let it print to your console twice using single quotation marks and double quotation marks.

e.g.:

.. code-block:: bash

	echo "The content of Var1 is: ${Var1}"
	echo 'The content of Var1 is: ${Var1}'

What is the difference?


Task 1.3: Arrays
----------------

An array is a variable that can store more than one piece of data. An array can be declared explicitely by *declare -a arrayname*, but this is not necessary.
It is sufficient to just assign values to an array. Create an executable file **arrays.sh** and create an array with four elemets. All elements of the array need to be
surrounded by **round** brackets. Elements containing empty spaces further more require quotation marks.


.. code-block:: bash

	#!/bin/bash

	MyArray=("Element 1" "Element 2" 42 1337)

To access elements of an array the "$" operator is used and also the index of the element is needed in squared brackets, where the indices start at 0. When using arrays
the usage of curly brackets becomes important, as you can see what happens if you do not use them. Add the following hard-coded lines:


.. code-block:: bash

	echo $MyArray[0]
	echo $MyArray[1]
	echo $MyArray[2]
	echo $MyArray[3]

The variable *$MyArray"* was interpreted as the first element and the index was just appended as text. Now try curly brackets:

.. code-block:: bash

	echo ${MyArray[0]}
	echo ${MyArray[1]}
	echo ${MyArray[2]}
	echo ${MyArray[3]}

Now assign a new value to an element of your array. Further more append another new elememt to the array by using the next new free index (4):

.. code-block:: bash

	MyArray[3]="UpdatedContent"
	MyArray[4]="AppendedContent"

	echo ${MyArray[0]}
	echo ${MyArray[1]}
	echo ${MyArray[2]}
	echo ${MyArray[3]}
	echo ${MyArray[4]}

Let your script print out the length / size of each element by using the "#"-operator. Further more, the "@"-operator can be used to print out the number of elements in the array:

.. code-block:: bash

	echo ${#MyArray[0]}
	echo ${#MyArray[1]}
	echo ${#MyArray[2]}
	echo ${#MyArray[3]}
	echo ${#MyArray[4]}

	echo ${#MyArray[@]}


Task 1.4: Loops & Conditions
----------------------------

There are for- and while-loops in Bash. Loops end with **done**. The basic sytax is:

.. code-block:: bash

	while condition; do
		commands
	done

	for variable in array; do
		commands
	done


Conditions are expressed as following:

.. code-block:: bash

	[ expression ]

where "expression" can be a lot... here is just a short overview of what we are going to cover here. You could check:

* wether a file exists or not
* a value or the content of a variable is equal, unequal, lesser or greater to another value or variable
* a string has the lengt 0 or greater than 0
...

.. hint::
   See *man test* for the complete documentation.

It is very important that [ expression ] explicitely needs empty spaces between its arguments:

.. code-block:: bash

	#wrong
	if ["$#" -ne 1]; then
		echo "Illegal number of parameters"
	fi

	#correct
	if [ "$#" -ne 1 ]; then
		echo "Illegal number of parameters"
	fi


To be able to compare numbers it is necessary to put them into arithmetic context by using **round** brackets. e.g.:

.. code-block:: bash

	numberOfArgs=0

	if ((numberOfArgs < 1)); then
		echo "Not enough arguments provided, see --help for options"
	fi

To test this, run the following loops. Both print out the numbers from 1...10:

.. code-block:: bash

	echo "While:"
	i=1
	while [ "$i" -lt "11" ]; do   # if i less than 11
		echo "$i"
		let "i+=1"  # counter up, i=$i+1 can also be shortened to i+=1
	done

	echo "For:"
	for j in {1..10}; do
		echo $j
	done

Now use loops to print out the elements your array in **arrays.sh**. Use the "@"-operator as a placeholder for "all elememts":


.. code-block:: bash

	echo "print array using loop:"
	for j in ${MyArray[@]}; do
		echo $j
	done


Task 1.5: Flow control
----------------------

Bash offers if/then/else statements for flow control. A control structure is terminated with a **fi**. If the condition is not met commands for another condition or a default command can be defined as well.

.. code-block:: bash

	#if the condition is true, "commands" are executed
	if condition; then
		commands
	fi

	#commands that are executed exactly once are followed by the keyword "then"
	if condition1; then
		commands
	elif condition2; then
		commands
	else
		commands
	fi

Change the output of your array in a way, that only elements are printed out if they meet a specific requirement, such as minimum or maxmimum string length.



Task 1.6: Arguments
-------------------

Arguments can be passed to a script separated by spaces and they can be accessed via variables *$0 ... $n*, where *$n* is the total number of passed arguments. *$0* is reserved for the script's name itself.

Create a file **calc.sh** and let it print out all passed arguments. Here is an example of a script that takes three arguments:

.. code-block:: bash

	#!/bin/bash

	echo script name: $0
	echo arguments: $1 - $2 - $3

To make a variable number of arguments possible the placeholder "$#" can be used for the number of passed arguments, respectively "$@" for their content. This **does not** include $0:


.. code-block:: bash

	echo script name: $0
	echo number of arguments: $#

	for argument in "$@"
	do
		echo "$argument"
	done

Test your script if it behaves as planned if you pass a variable number of arguments. A possible test run:

.. code-block:: bash

	$ ./calc.sh 1 2 3 'test string' "hallo welt"
	script name: ./calc.sh
	number of arguments: 5
	1
	2
	3
	test string
	hallo welt


Task 1.7: User input
--------------------

The basic syntax for reading user input in Bash is as following:

.. code-block:: bash

	read -p "prompt" variable1 variable2 variableN

where:

* -p "prompt" : prints the content of "prompt" to the console without a line break
* -variable1 : the first word of the input is assigned to variable1
* -variable2 : the second word of the input is assigned to variable2
* -variableN : the n-th word of the input is assigned to variableN

Example:

.. code-block:: bash

	#!/bin/bash
	read -p "Enter your name : " name
	echo "Hi, $name. Let us be friends!"

	# read three numbers and assigned them to 3 vars
	read -p "Enter number one : " n1
	read -p "Enter number two : " n2
	read -p "Enter number three : " n3
	echo "Number1 - $n1"
	echo "Number2 - $n2"
	echo "Number3 - $n3"


Task 1.8: Calculator (for Integer numbers)
------------------------------------------

Use all of what you have learned so far to write a calculator program **calc.sh**. For simplicity it should only be able to handle integer numbers. The programm should have the following features:

* there must be 1, 2 or 3 arguments passed to the script but at least 1.
* if there is no argument passed or if arguments are invalid the program shall terminate and give a hint to use the "--help" option
* the first argument is the desired operation, which can be:

	* "add"  (addition)
	* "sub"  (subtraction)
	* "mult" (multiplication)
	* "div"  (division)
	* "cross" (cross sum)
	* "--help" (show available options)

* addition, subtraction, multiplication and division take 1 or 3 arguments. If there is only 1 argument, the user is asked to input two integer numbers. If there are 3 arguments, the program take argument 2 and 3 to calculate the result.
* cross sum takes 1 or 2 arguments. If there is 1 argument, the user is asked to input an integer number. If there are 2 arguments, the cross sum for argument 2 is calculated.


Part 2: Example for automation with Bash
----------------------------------------

Imaginge yourself beeing the administrator of a git server. Amongst other things your job is to handle all the indidivual ssh keys of all the users of the server
and their access rights to all of the different repositories. All keys are stored in a directory **keydir** and have the actual username as their filename (e.g. *mstuettgen.pub*).
Each time a new key for a new user is added or an existing key changes or gets deleted, the configuration file of the server needs to be adjusted accordingly by hand. Your
task shall be to automate this process with a bash script. The first stage of your script should return all existing usernames in the directory **keydir** by reading the contents and
cutting off the **.pub** file ending.


Task 2.1: Listing all files and directories of a given path
-----------------------------------------------------------

In order to prepare this task first create a folder **keydir**:

.. code-block:: bash

   $ mkdir keydir

Now create 10 pseudo-users inside that directory. The files do not need any content as we only want to read their filenames later.

.. code-block:: bash

   $ cd keydir
   $ touch mmustermann.pub
   $ touch mmusterfrau.pub
   $ touch rcallmund.pub
   $ touch plombardi.pub
   $ touch dbohlen.pub
   etc...

Now write a bash script **get_usernames.sh** which processes all **.pub** files and returns e.g. a concatenated string of all usernames which then gets printed to the console.
Keep in mind that this script has to ignore itself and other existing directories or files that might exist.


Solution:

.. code-block:: bash

	#!/bin/bash
	...
..	for filename in *; do
..		user="${filename%%.*}"
..		if [ "$user" != "get_usernames" ]; then
..			all_users="$all_users $user"
..		fi
..	done

..	printf "$all_users"



Task 2.2: Automatic replacement of text (in files) with SED
-----------------------------------------------------------

**S**\tream **ED**\itor is a non-interactive text editor. **SED** edits data based on defined rules and can be called like this:

.. code-block:: bash

	$ sed <options> <file>

**SED** is not limited to be used on files only, you can also pipe input from **STDIN** directly into **SED**. Try the following command:

.. code-block:: bash

	$ echo "Hello World" | sed 's/World/<InsertYourNameHere>'

The paramters *s* replaces the first textpattern (*World*) with the second (*YourName*).

Example:

.. code-block:: bash

	$ echo "Hello World" | sed 's/World/Marcel Stuettgen/'
	Hello Marcel Stuettgen


Since we are already capable of reading all existing usernames from the directory **keydir** we can now use **SED** to automate the update of the central configuration file.
To simulate this process please create a file **example.conf** with the following content:


.. code-block:: bash

	@demo_project_users = wputin dtrump

	repo    demo_project
	RW+   = @demo_project_users

And try invoking the following command:

.. code-block:: bash

 $ sed 's/dtrump/bobama/g' example.conf

The option *-g* replaces **all** occurences of the first text pattern, not only the first one. Check your results:


.. code-block:: bash

	$ cat example.conf

As you can see the replaced text was written to the console but the original file was untouched. In order to allow **SED** to actually replace text inside a file the option *--in-place* / *-i*
needs to be added:


.. code-block:: bash

	$ sed -i 's/dtrump/bobama/g' example.conf

Now the text should actually be replaced inside the original file. Check your results again:

.. code-block:: bash

	$ cat example.conf

The pure replacement of text is unfortunately not sufficient to solve our task. The usergroup *@demo_project_users* shall always contain all users that have a key inside **keydir**. If
a key gets deleted, the access right of that user shall be revoked automatically. This means that we have to rewrite the whole line *@demo_project_users = ...* each time. To do this
**SED** offers the usage of regular expressions. The following example removes **all** users from the group *@demo_project_users*:

.. code-block:: bash

	$ sed -i "s/@demo_project_users = .*/@demo_project_users = /g" example.conf


The placeholder  *.** makes sure that everything followed after *@demo_project_users = ...* gets removed. To be a little more precise, the whole line gets replaced by just the string "@demo_project_users = ".

Check your results:

.. code-block:: bash

	$ cat example.conf

Now copy over your existing script **get_usernames.sh** into a new file **update_users.sh**

.. code-block:: bash

	$ cp get_usernames.sh update_users.sh

(Do not forget to make that file executable).

Now change the script so that it does not just print out the concatenated string of users, but saves it into a variable. At the end of the script add an **SED** command that replaces all users
of the usergroup *@demo_project_users* in the file **example.conf** with the concatenated string of users.

Solution:

.. code-block:: bash

	#!/bin/bash
	...
..	for filename in *; do
..		user="${filename%%.*}"
..		if [ "$user" != "get_usernames" ] | [ "$user" != "update_users" ]; then
..			all_users="$all_users $user"
..		fi
..	done

..	printf "$all_users"

..	sed -i "s/@demo_project_users = .*/@demo_project_users = $all_users /g" example.conf


Check your results:

 .. code-block:: bash

	$ cat example.conf


Now delete a random user from the folder **keydir** and add a new pseudo-user. Invoke your automatic update script afterwards and check your results. e.g.:


 .. code-block:: bash

	$ rm mmustermann.pub
	$ touch djbobo.pub
	$ ./update_users
	$ cat example.conf

The user that was removed should no longer appear in *@demo_project_users*, but the new user should.





Bonus Tasks (optional)
----------------------

Bonus-Task 1: FH-Web-Grabber
----------------------------

Write a bash script that downloads all pictures from the FH Aachen Website (subfolder "user_upload") and convert the downloaded images automatically from *.jpg to *.png.

.. hint::
   First grab "index.html" from the starting page and extract the required URLs for the pictures, which then can be downloaded easily.



Bonus Task 2 : lstree in Bash
-----------------------------

Create a file **lstree.sh** with an editor of your choice (e.g. *emacs*)

.. code-block:: bash

   $ emacs lstree.sh

(Save: CTRL+X CTRL+S; Exit: CTRL+X CTRL+C)

.. hint::
	imagine "CTRL+X" beeing the click on "File" dialogue in emacs GUI

Write a bash script that prints out a graphical representation of all files and (sub)directories in
form of a tree, beginning with the directory it was called in. Do not forget to make your file executable!

.. code-block:: bash

   $ chmod +x lstree.sh
   $ ./lstree.sh


.. hint::
   - the output shall begin with the root, represented by "."
   - beneath the root (and all subentries) all entries shall be marked with a vertical line (the *pipe* symbol "|")
   - each new (sub)level shall be at least indented by one
   - directories are marked with "+"
   - files are marked with "-"

   An example output:

   .. code-block:: bash

     $ ./lstree.sh
     .
     |+ subdir01
     ||- file0101
     ||- file0102
     |+ subdir02
     ||- file0201
     ||- file0202
     ||- file0203
     |+ subdir03
     ||+ subsubdir01
     |||- file030101
     |||- file030102
     |||- file030103
     ||+ subsubdir02
     |||- file030201
     ||+ subsubdir03
     |+ subdir04
     |- file01
     |- file02
     |- file03
     |- file04


Solution:

.. code-block:: bash

   #!/bin/bash
   ...