Introduction to the Linux Terminal

If your technological experience is limited to graphical user interfaces like those provided by Windows and Android, using Linux may require a change in mindset. While there are graphical tools to perform many of the same tasks, the Unix philosophy (which also applies to Linux) is to have many tools that focus on doing one thing, and many of those tools are terminal-based, meaning they do not come with a graphical user interface (GUI).

One of the major advantages of this is that you can run a server with no GUI, which allows you to allocate your resources where they are needed - your database, web or WSGI server, etc. and not have a GUI use up memory needlessly. It also decreases the attack vector; services that contain bugs but are not active are less likely to be exploited by nefarious actors.

Also, mastering the terminal will make you a far more efficient system administrator, and allow you to easily configure remote servers.

What is a terminal?

What we call a terminal is actually a terminal emulator, a computer program that emulates the hardware terminals of years past. The terminal emulator can be running directly on a computer with no GUI, or it can be used in a desktop environment inside a graphical window. It can also be running on a remote machine to which you may connect using the SSH protocol. Finally, there are also Quake-style drop-down terminals, which run in desktop environments and can be toggled on/off with the press of a button on the keyboard, which are very handy when you often switch between a terminal and graphical applications.

A terminal emulator runs a Unix shell, a command-line interpreter. There are many different shells to choose from. The most popular one is probably the bash shell, which is the default shell on most Linux systems. A few other popular ones include zsh, which is now the default on Mac OS, and fish. Different shells have different characteristics, such as scripting language support, but any one should work for most needs.

Personally, I am a big fan of the fish shell. It has plenty of nice features like intelligent history and tab completion, and the fish language feels elegant to me.

Accessing the terminal

There's a few different ways to access the terminal, depending on your OS, installed packages, and environment. If you have a graphical install with a desktop environment, it will often have a default terminal application.

If you do not have a graphical install, chances are logging in will drop you directly into a shell.

KDE Plasma

The default terminal application for KDE Plasma is konsole. It can be accessed by opening the main menu, navigating to System, and selecting Konsole. Alternatively, you can press Alt-F2 to open the launcher, and type konsole.

You should also be able to access konsole by pressing ctrl-alt-t. (But why, when you could just use Yakuake?)

Konsole can be installed on Arch Linux with the command:

$ sudo pacman -S konsole

It is also a dependency of the KDE applications meta-package, so it will be automatically installed if you install the kde-applications-meta package.

It can be installed on Ubuntu with the command:

$ sudo apt install konsole

As it is a dependency of the KDE base applications, it will be automatically installed if you install the kde-baseapps package.

Yakuake

If you would like to have a drop-down terminal that is always accessible, I would recommend Yakuake. It is based on konsole, and integrates perfectly into the KDE Plasma environment. To ensure that it is always available, you can add it to the Autostart list by opening Plasma System Settings, navigating to Startup and Shutdown > Autostart, clicking the Add button, and adding the program yakuake.

To toggle the terminal on or off, simply press the F12 key.

You can see this document being edited in Neovim, in the Yakuake drop-down terminal in the image below.

Yakuake can be installed on Arch Linux with the command:

$ sudo pacman -S yakuake

It can also be installed on Ubuntu with the command:

$ sudo apt install yakuake

Pro-tips

If you enable a tiny bit of transparency in yakuake, you can watch music videos in the background. Note that you may find this distracting, but personally, I like it.

Gnome

The default terminal application for Gnome is gnome-terminal. You should be able to access it by pressing ctrl-alt-t, or clicking on the Show Apps icon and finding it in the application list. If you do not see it, you can also perform a search for terminal and it should be displayed, as shown on the screenshot below. Finally, you can press Alt-F2 and type gnome-terminal.

Using the terminal

As covered in the Files and Linux chapter, the terminal keeps track of your current working directory. The default working directory is the current user's home, as defined in the user database (which we have also covered in the Users and Permissions chapter). The terminal prompt waits for user commands and executes them. Note that these do not necessarily need to be executable programs. Shells can execute a built-in command, an alias, a script, or executable program.

Also, note that commands are case-sensitive. cd and CD are not equivalent.

Auto-completion

Depending on your shell and configuration, you may be able to press the tab key to trigger auto-completion. When there are multiple possible completions, hitting tab twice may list the various possibilities.

Running a command

To run a command, simply enter the name of the command, along with any arguments. Some arguments may be necessary, while some may be optional flags. Certain flags may also take their own arguments.

A positional argument is an argument that does not start with a dash, such as /etc/passwd in ls /etc/passwd or ls -lh /etc/passwd. Programs may take any number of positional arguments, including zero.

Flags come in short and long forms. Generally, short flags use a single dash (-) followed by the flag, while long flags use two dashes (--) followed by the flag name. For example, the commands ls -a and ls --all are equivalent. Flags may alter the execution of the commands. For example, ls -a lists all files, including hidden files, which are normally omitted. Note that not all flags come in both short and long forms. For example, ls -l has no long form equivalent.

Short flags can often be grouped together with a single dash. For example, ls -al is equivalent to ls -a -l.

When a flag takes an argument, the argument must be placed immediately after the flag. For example, the ls command takes a width argument to specify the maximum line length for its output. Its short form is -w COLS and long form is --width COLS. The long form can also have an equal sign (=) instead of space, as in --width=COLS. COLS must be replaced with an integer for the command to be valid. For example, these three commands are equivalent:

$ ls -w 80
$ ls --width=80
$ ls --width 80

Note that this command would be invalid, as the flag argument does not immediately follow the flag: ls -wal 80, but ls -w 80 -al and ls -alw 80 work.

Determining which command is run

Typically, the shell will look at built-in commands, aliases, and then look for the command in all directories defined in the $PATH environment variable, which we will cover in a later chapter, and run the first command with that name that it encounters (or print an error if it cannot be found).

Deviants

Note that while the previous descriptions apply to most Linux programs, they are not universal. For example, the dd command does not use traditional flags, but uses arguments with no dashes, as in dd if=/dev/zero of=/tmp/zerofile bs=1M count=10. Other programs, such as ffmpeg, may prefix long flags with a single dash, rather than two (though it still accepts --help with two dashes, go figure). When in doubt, refer to the man page.

Python and command arguments

While it is beyond the scope of this book, know that you can use the argparse module to write complex command-line python programs.

Getting help

To get help for a command, you can typically use the man command. Much GNU software is also covered by the info command, which may be more comprehensive than the man page.

A man page refers to an entry in the manual. For example, you can see the man man page by typing man man, as shown below.

As you can see from the man man page, the manual is divided into nine sections. Certain page names may repeat across different sections, and refer to different parts of the OS. For instance, chmod is documented as an executable in section 1, a kernel system call in section 2, and a POSIX library call in section 3. You can specify which section of the manual you would like to view by passing an extra argument as follows:

$ man chmod # opens section 1
$ man 2 chmod # opens section 2
$ man chmod.3 # opens section 3

As you can see in the following screenshot, this opens different manual pages.

Note: it has historically been fairly common to point out that a user did not do proper research before asking for help by responding RTFM, which stands for "read the fine manual" (or sometimes a different kind of f-word). Please do not respond that way to your fellow djangonauts, as it is considered rude.

Installing help programs and files on Arch Linux

To access the info command, you will need to install the texinfo package. To access man pages, you will need to install the man-db package. You may also install the man-pages command to install extra pages from the POSIX and Linux manual.

$ sudo pacman -S texinfo man-db man-pages

Help flags

Many programs also have a flag like --help, -h, or -? that can be used to display help.

Redirection

You can redirect the output from one command to another command with the | operator, commonly known as a pipe (or piping the output). This works very well with the UNIX philosophy of having commands do only one thing, and chaining commands together. For example, we can use the grep command to search a file, and pipe its output to the cut command to only the first field, as delimited by the colon (:) character. Let's get a list of users with nologin as their shell:

$ grep '/usr/bin/nologin' /etc/passwd | cut -d : -f 1
bin
daemon
mail
ftp
...

Note: the output has been cut (heh) for brevity.

There is no (unreasonable) limit to the number of commands you can pipe together. Let's repeat our previous command, but only get the first three users sorted alphabetically by username, by using the grep, cut, sort, and head commands:

$ grep '/usr/bin/nologin' /etc/passwd | cut -d : -f 1 | sort | head -n 3
avahi
bin
clamav

You can also redirect input from, and output to a file using the > and < operators. For example, you can you can save a list of all files in the current directory as follows:

$ ls > /tmp/files

By default, this only redirects the standard output stream, or stdout. Note that output to the standard error stream will still be displayed on the terminal. For example, this redirects the print() call, but not the exception:

$ python -c "print('hello, world'); raise Exception('oops')" > /tmp/pyerr
Traceback (most recent call last):
  File "<string>", line 1, in <module>
Exception: oops
$ cat /tmp/pyerr
hello, world

The special streams as numbered 0 for stdin, 1 for stdout, and 2 for stderr. You can redirect only one stream. For example, to suppress errors, you can redirect stream 2 to /dev/null:

$ python -c "print('hello, world'); raise Exception('oops')" 2> /dev/null
hello, world

As you can see, the error is now hidden. Note: hiding errors can lead to missing errors, meaning that you may not know what went wrong.

You can also redirect one stream to another. For example, to redirect the stderr stream to stdout, you can use 2>&1. This redirects stream 2 to stream 1. In order to redirect both to a file, you can first redirect stdout to the file, then redirect stderr to stdout. Note that this may change the output order.

$ python -c "print('hello, world'); raise Exception('oops')" > /tmp/pyerr 2>&1
$ cat /tmp/pyerr
Traceback (most recent call last):
  File "<string>", line 1, in <module>
Exception: oops
hello, world

Depending on your shell, this may be equivalent to &>, as in:

$ python -c "print('hello, world'); raise Exception('oops')" &> /tmp/pyerr

You can also open the destination file in append mode with >>, which will prevent it from being overwritten.

Finally, you can redirect a file to the input stream, which may be useful in cases such as loading an SQL script into the mysql command:

$ mysql < /path/to/script.sql

Note: redirecting output to a file does not open the output file with root privileges if you run a command as sudo. If you need to write to a file with root privileges, you can pipe the output to the tee command.

Useful commands

You can find a list of some of the most common commands below. I will not be going into too many details into all of them, but will try to warn you when they may be destructive so you know when to take extra care.

Read the man pages to learn more about them, and generally, do try to explore how they work.

Note: some of these commands may need to be installed separately before you can use them.

Printing text

You can use the echo built-in command to print text to the terminal. This will come in handy later in this book.

$ echo "hello, world"
hello, world

Files, directories, and file systems

Knowing where you are

It's always a good idea to know where you are. If you are not sure, run the pwd command, which will show you the current directory. You can think of it as print working directory.

$ pwd
/home/alice

Changing the working directory

To change the current working directory, you can use the cd command. You can think of it as Change Directory. The cd command is a shell built-in command, and may work differently on different shells, but it generally takes one argument; the destination of the new working directory. As we've previously discussed, the destination can be provided using an absolute path, or relative path. This is common to most commands that take paths as arguments. You may find some examples below, assuming that the user is named alice and their home directory is /home/alice/.

$ cd /home/alice
$ cd ~/Documents # /home/alice/Documents
$ cd .. # back to ~/
$ cd /root/ # should result in a permissions error
$ cd /tmp/
$ cd # special case

Note: when you run cd with no arguments, it normally changes to the user's home directory, or back to /home/alice/ in our example.

Listing files and directories

You can list files and directories with the ls command. With no arguments, it will list the current directory. If you specify a positional argument, it will list that file or directory.

The most common arguments are -a to list all files, -l to list in long format, and -h to list in human-readable format (which may be useless without the -l argument).

Creating files and directories

You can create a new directory with the mkdir command. Note that by default, it will refuse to create a directory if all its parent directories do not yet exist. You can use the -p flag to create parent directories as needed.

$ mkdir /tmp/foo/bar/baz/very/deeply/nested/
mkdir: cannot create directory ‘/tmp/foo/bar/baz/very/deeply/nested/’: No such file or directory
$ mkdir -p /tmp/foo/bar/baz/very/deeply/nested/

You can use the touch command to update the last modification time of a file. Honestly, that is probably utterly useless, but thankfully, the touch command also creates a new file if the destination does not yet exist.

$ touch /tmp/newfile # creates the file if needed

Moving and copying files.

Danger Zone: by default, copying files will overwrite existing files. You can use the interactive flag (-i) to ask for confirmation.

You can move a file or directory with the mv command. You can also copy a file or directory with the cp command. Note: by default, the copy command does not copy directories. You need the recursive (-r) flag to copy a directory.

$ cp /tmp/foo/bar/baz/very/deeply/nested/ /tmp/foo/bar/baz/very/deeply/nested2/
cp: -r not specified; omitting directory '/tmp/foo/bar/baz/very/deeply/nested/'
$ cp -r /tmp/foo/bar/baz/very/deeply/nested/ /tmp/foo/bar/baz/very/deeply/nested2/

Besides the previously mentioned interactive flag, both mv and cp can also use the verbose flag, -v.

Deleting files and directories

Danger Zone: for obvious reasons, deleting files is a destructive operation. Please ensure that you want the files gone before running this command.

You can delete a file with the rm command. Just like cp, it will not remove directories by default, unless you also pass the recursive flag -r. It can also take the interactive and verbose flags.

An alternative to rm is the shred command which can be used to securely delete files or even entire partitions. By default, it overwrites the file three times with random data. Danger Zone: this obviously makes files extremely difficult, if not impossible, to recover. It should also not be used with solid-state disks, as those use wear-leveling, meaning that it may not write to the same block, which will not actually physically alter the state of the drive where the file was originally located, but only wear your drive out faster.

Creating links

Both hard and symbolic links can be created using the ln command. Use the -s flag to create a symbolic link.

Reading text documents

You can use the cat command to print a text document to the terminal.

Some alternatives include bat, which has nice syntax highlighting and a built-in pager, and glow, which is a markdown viewer. You can compare the output of cat (at the top) with bat (bottom left), and glow (bottom right) on the image below:

You can also use the less pager command to read a file. This will allow you to scroll up and down the document.

Note: you can also redirect output from another command to the pager, which may make long outputs easier to navigate. For example, we can recursively list a large directory with ls -lhR /usr/lib/ | less as shown below:

You can navigate up and down the with respective arrows or PgUp and PgDn keys, use / to perform a forward search, ? to perform a backward search, and q to quit the pager.

Figuring out what a file is

You can use the file command to figure out what type of file an argument is, or the type shell built-in to determine what type a command is:

$ file docs/img/terminal-cats.png
docs/img/terminal-cats.png: PNG image data, 1920 x 998, 8-bit/color RGBA, non-interlaced
$ type cd
cd is a shell builtin
$ type ls
ls is aliased to 'ls --color=auto'
$ type which
which is /usr/bin/which
$ file /usr/bin/which
/usr/bin/which: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked

Searching

You can use the find command to search for files and directories, and optionally perform operations on them. Danger Zone: some of these commands are destructive, such as search and delete.

You can use grep to search for text in a file. This is particularly useful to search through a code base.

$ grep -r User ~/projects/foo/django # find references to User

Partitions

You can use the mount command to list mounted partition or to mount new partitions. You can use umount to unmount a partition.

You can also use the df command to list disk usage. Think of it as disk free.

$ df -h # list disk usage for all mounted partitions in human-readable format
$ df / # list disk usage for the root partition
$ df -h /tmp/ # list tmpfs usage

Measuring directory sizes

You can use the du command to measure directory sizes.

Calculating file hashes

You can use commands like md5sum, sha256sum, and others to calculate hashes of files. This can be used to check that two files are identical, for example if you copied one from one location to another or want to check if they are duplicates. You can also get the hash from text that is not a file.

$ sha512sum /usr/bin/sha512sum
f357f02252c5165138e2b2bd73f3c355960c5042b781a330f0d2ac48cfc71f20ceb0d0d19bfd22468afb0e41ae9f425e01adab2d9b68326d3abd6b4dade61619  /usr/bin/sha512sum
$ md5sum /usr/bin/md5sum
228089993f60bb763e439cdeb995c15c  /usr/bin/md5sum
$ echo -n "hello" | sha1sum
aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d -

Note: the MD5 algorithm is considered broken as there are ways to modify files without modifying the hash. Algorithms with longer hashes are generally considered more secure.

Text operations

You can use sort to sort text, uniq to remove duplicate lines (though the text needs to be sorted first), and tr and cut to manipulate text.

Users and permissions

You can use the chown and chmod commands to change ownership and permissions.

You can use the su command to temporarily switch to the root user, or another user. You can use the sudo command to run a command as the root user or another user.

You can use the useradd command to add new users, and usermod to modify existing users. For example, the user alice can set her shell to fish with sudo usermod -s /usr/bin/fish alice.

BusyBox

The previous commands are mostly based around the GNU core utilities. Note that certain distributions, such as Alpine Linux, may use BusyBox instead. BusyBox contains many utilities in a single executable that is much smaller than the many utilities provided by the GNU core utilities, but their implementation may contain less features, and work differently.

Links

• Computer terminal on Wiki
• Terminal emulator on Wiki
• Core utilities on the Arch Wiki - this lists alternatives to certain commands.
• Is shred bad for erasing SSDs? on Stack exchange
• MD5 collision demo
• Secure Hash Algorithms on Wikipedia
• BusyBox on Wikipedia