• Basics of programming in the shell command shell. The Evolution of Linux Shells

    What is a shell and why is it needed?

    The command shell in any unix-like systems, which includes GNU/Linux, is a regular program launched both in a text console (which is used less and less) and in a graphical environment - in the window of a terminal emulator, available on any Linux system .

    Its task is simple and obvious: accept a line (or lines) of input, produce them parsing and based on the results of this analysis, react accordingly - execute a command, run a program, display a diagnostic message, etc.

    Almost all Linux distributions the default shell for users is bash (Bourne Again SHell is another Bourne shell; Steve Bourne is the author of the first shell on Unix – sh). In fact, it has become an unofficial standard, and improvements to it functionality continues continuously. There are other command shells - tcsh (version of C-shell), ksh (Korn Shell), zsh, etc. – each has its own advantages and disadvantages, as well as its own fan groups. However, bash is more familiar to the general public with different levels preparation, that’s why I chose it. It is also worth noting that no matter what capabilities the various shells have, they are all compatible with their ideological progenitor - the Bourn Shell (sh). In other words, a script written for sh will work correctly in any modern shell (the reverse is generally not true).

    Benefits of the Command Line

    The question may arise: why bother with the command line if there are convenient and beautiful graphical interfaces? There are many reasons for this. Firstly, not all operations are more convenient or faster to perform using GUI. Secondly, each program follows the fundamental principle of Unix systems: do it clearly certain work and do it well. In other words, you always understand what happens when you run a particular utility (if something is not entirely clear, you should refer to the man manual). Thirdly, by mastering commands, trying their combinations and combinations of their parameters, the user studies the system, gaining valuable practical experience. You get access to these effective tools, like pipelines that allow you to organize a chain of commands for data processing, means of redirecting input/output, and in addition, you can program directly in the command shell. Perhaps it’s worth dwelling on programming in more detail, especially since many system scripts in Linux (for example, scripts for launching system services) are written for the shell.

    Command shell as a programming language

    So, the command shell can be considered as a programming language and as software environment execution simultaneously. Of course, this language is not compiled, but interpreted. It allows the use of variables: system or own. The sequence of execution of program commands is changed using condition checking constructs and selecting the appropriate option: if-then-else and case. While, until, and for loops allow you to automate repetitive actions. It is possible to combine groups of commands into logical blocks. You can even write real functions that pass parameters to them. Thus, all the signs and characteristics of a full-fledged programming language are available. Let's try to get double benefit from this - along with learning the basics of programming, we will automate our daily work.

    Hello, World! Simple backup system

    About the need for regular backup Everyone knows the data, but users never have enough time for this boring operation. The solution is simple - organize automatic creation backup copies. This will be our first shell programming task.

    #!/bin/bash # # Backup directories and files from the home directory # This batch script can be run automatically using cron # cd $HOME if [ ! -d archives ] then mkdir archives fi cur_date=`date +%Y%m%d%H%M` if [ $# -eq 0 ] ; then tar czf archive$(cur_date).tar.gz projects bin else tar czf archive$(cur_date).tar.gz $* fi if [ $? = 0 ] ; then mv archive$(cur_date).tar.gz $HOME/archives echo "$cur_date – Backup completed successfully." else echo "$cur_date - ERROR during backup." fi

    Any command script (script is a script, as command shell programs are called) begins with an identifier line, in which the command interpreter is explicitly specified, indicating the full path to it. The full path is a sequential listing of all directories, starting from the root, that must be entered to get to the target file, and, of course, the name of this file. Recording the full path is extremely important to uniquely identify each file in the file system hierarchy.

    Four lines of comments follow. As soon as the shell encounters the "#" character, it considers all subsequent characters to be comments and completely ignores them until the end current line. Therefore, you can start a comment not from the very beginning of the line, but accompany it with some command.

    After the comments there is an empty line. It means nothing to the command shell, and no action is taken. In scripts, blank lines are usually inserted to make the code easier to read.

    We finally got to the first “real” team. It allows you to change the directory (Change Directory), i.e. move from the current directory to another one passed to the command as an argument. In most cases, the target directory is specified explicitly, for example, cd /tmp or cd projects, but in our case the predefined system variable HOME is used - it contains the full path to the home directory of the current user under whose name the command script is being executed. This eliminates the need to make code changes every time we change users, because the command returns everyone to their personal directory. The dollar sign "$" in front of a variable name means that you need to extract the value contained in that variable and substitute it for its name on the command line. It should be especially noted that in command language letter case shells are important, i.e. HOME, Home and home are three different variables. By convention, uppercase letters denote the names of system variables: HOME, PATH, EDITOR, etc. This convention does not prevent users from creating their own variables with names from capital letters, but why complicate your life by violating generally accepted norms and rules? It is also not recommended to change the values ​​of system variables unless absolutely necessary. In general, we follow a simple rule: we use system variables for read-only purposes, and if we need our own, we write its name in lowercase letters.

    Our first command could be written more briefly:

    cd ~

    Here the "~" symbol also means the current user's home directory. Command line veterans put it even more succinctly:

    CD

    The idea is that when the cd command is given no argument, it changes to the home directory.

    Next up is the classic software design for checking conditions and making the appropriate decision. General scheme is this:

    if<условие>then<одна или несколько команд>fi

    The last word of the construct (if in reverse order) acts as a closing parenthesis, i.e. boundaries of the list of commands executed when the condition is true. The presence of fi is mandatory, even if there is only one team on the list.

    To check a condition, as a rule, the test command or its alternative form of notation in square brackets is used. In other words, records

    if [! -d archives ] if test ! -d archives

    absolutely equivalent. I prefer square brackets, since they more clearly define the boundaries of the condition being tested. Both the right and left parentheses must be separated from the condition by spaces.

    The criteria for checking the condition are determined by various flags. The test command recognizes a very large list of them. In our example, the -d flag is used to check whether the name specified after the flag corresponds to a real-life directory. The following flags are most often used when working with files:

    F – whether a regular file with the given name exists;

    R – whether the specified file has the right to read from it;

    W – whether the specified file has the right to write to it;

    X – whether the specified file has the right to execute it;

    S – whether the specified file has a non-zero size.

    In our case, the condition is preceded by exclamation mark, denoting the operation of logical negation, so the meaning of the condition being tested becomes completely opposite. Let's try to write down the meaning of these commands in ordinary Russian:

    if [! -d archives ] If the archives directory (in the current directory) does not exist, then start executing the command block: mkdir archives create the archives directory (in the current directory) fi end executing the command block.

    As you can see, everything turned out to be not so complicated. With a little practice, you can easily read and create similar designs yourself. The directory creation command is so obvious that no further explanation is required.

    On the next line we create our own local variable, cur_date. In the vast majority of cases, variables are created by simply assigning a specific value, for example:

    ten=10 string="This is a line of text"

    But in our example, a little trick is used. Please note that after the equals sign - the assignment symbol - the command is written in back quotes. This form of notation allows you to assign to a variable not the string itself, but the result of its execution. Here is the output of the date command, which returns the current date and time in a format specified by a list of parameters:

    %Y – current year in full form, i.e. of four digits (for example, 2009);

    %m – number current month(for example, 09 – for September);

    %d – current day number;

    %H – current hour in 24-hour format;

    %M – current minute.

    Thus, if you run the command

    cur_date=`date +%Y%m%d%H%M`

    on the tenth of September 2009 at 22:45, the variable cur_date will be assigned the string value "200909102245". The purpose of this trick is to create a unique, non-repeating name for the archive file. If you intend to run several instances of the program within one minute, you can improve the uniqueness of the names by adding the current seconds. How? Study the date utility manual (man date) - there is nothing complicated about it.

    Before we start creating an archive file, we need to determine which directories we will save in it. For greater flexibility, we can specify a set of directories to archive by default, but provide the ability to replace this set with a list of directories passed as an argument to our command script. For this purpose, special command shell variables are used: $# – the number of parameters passed to the script and $* – all parameters passed, written in one line format.

    if [ $# -eq 0 ] ; then

    Checking the condition “if the number of passed parameters is zero”, then execute the following command. Note that keyword then can be written on the condition line, separating it from the conditional expression with a semicolon.

    tar czf archive$(cur_date).tar.gz projects bin

    The command to create an archive file and compress this file. The tar utility itself does not perform compression, but only collects all specified files and directories into a single tar file. The first flag is intended for this - c (create). Compression is performed by an external program - here it is gzip, called by the second flag - z. If your system has more than effective program bzip2 compression, you can take advantage of it by changing the command as follows:

    tar cjf archive$(cur_date).tar.bz2 projects bin

    The third flag f indicates that what follows is the name of the archive file, so it is always the last one in the list of flags. Note that when substituting, the variable name is enclosed in curly braces. This is done to explicitly highlight the variable on the line surrounding it, thereby eliminating many potential problems. Extensions archive file are not assigned automatically; you fill in everything you need yourself. I've specified projects and bin as the default directories to archive, but you can write down the names of your most valuable directories here.

    The else keyword opens an alternative branch of execution. The commands of this block begin to work if the condition check returns the result “false” (in our example: “the number of parameters passed is non-zero,” i.e. the user specified directory names). In this case the command will look like this:

    tar czf archive$(cur_date).tar.gz $*

    Here the default directories are replaced by a directory name string accepted externally. It is possible to accept and process each external parameter separately, but it’s more convenient for us to pass the entire string.

    At the end of the program, another check is performed. In Unix environments, all commands return a completion status code. If the command was successful, it returns code 0, otherwise the exit code will be non-zero. To check the success of the previous archiving command, we will use another special variable $?, which always contains the value of the completion code of the most recent command. If in the variable $? contains 0, i.e. The backup file was successfully created, then we move it to the archive directory:

    mv archive$(cur_date).tar.gz $HOME/archives

    and display the corresponding message:

    echo "$cur_date – Backup completed successfully."

    If the check shows that the completion code of the archiving operation is not zero, then an error message is displayed:

    echo "$cur_date - ERROR during backup."

    This completes our command script.

    To test the operation of our program, we need to save the source code described above in a file, for example, named bckp, and then, for convenience, make it executable:

    chmod 750 bckp

    and run:

    ./bckp

    to create a backup of the default directories, and

    ./bckp docs progs works

    to create a backup copy of the listed directories (specify the names of the directories that actually exist on your system, otherwise you will receive an error message).

    You can place the bckp file in one of the directories specified in the system PATH variable. The most preferred locations are /usr/local/bin or $HOME/bin if you have them. After this you can run bckp as a system command.

    How to automate scheduled backup operations

    A few words about backup automation. For this purpose, the system serves cron scheduler, which reads work instructions from a special crontab file. To define such instructions, you need to create and edit your crontab file using the command:

    crontab -e

    Instructions are written in a strictly defined format (fields are separated by spaces):

    minutes hours day_of_month month day_of_week command

    One option for scheduling backup operations might look like this:

    30 23 10,20,30 * * /usr/local/bin/bckp

    This means that the backup script (you must provide the full path to this file) will run at 23:30 on the 10th, 20th and 30th of each month, regardless of the day of the week. (Asterisks indicate the entire permissible range of values, in this case: every month - in the 4th field, any day of the week - in the 5th field)

    If you prefer to summarize your results by week, and your system runs 24/7, then it makes sense to schedule backups during off-peak hours:

    0 5 * * 3.5 /usr/local/bin/bckp

    Here backups will be created at 5:00 on Wednesdays and Fridays in each month (asterisk in the 4th field), regardless of the date (asterisk in the 3rd field).

    You can read about all the intricacies of scheduling in the man 5 crontab manual.

    Results and conclusions

    The backup script discussed in this article has modest functional properties. But this was not his main task, but to ensure that the reader understood what can be done in command line, and not only copied and executed the proposed batch file, and became interested in expanding its functions, began researching the immense possibilities provided by command shells. And if someone, after reading this article, tries to improve the code given here, or writes their own version, or implements their own independent idea, then I will consider that the main goal has been achieved.

    Resources for download

    static.content.url=http://www.site/developerworks/js/artrating/

    ArticleID=458335

    ArticleTitle=Shell Programming Basics

    • Tutorial

    Why and for whom is the article?

    Initially, this was a reminder for students who are starting to work with Unix-like systems. In other words, the article is intended for those who have no previous experience working with the Unix command line, but for one reason or another want or need to learn how to interact effectively with it.

    There will be no retelling of mana (documentation), and the article does not in any way cancel or replace reading them. Instead, I will talk about the main things (commands, techniques and principles) that you need to understand from the very beginning of working in the unix shell in order for the work to be effective and enjoyable.

    The article concerns full-fledged unix-like environments, with a fully functional shell (preferably zsh or bash) and a fairly wide range of standard programs.

    What is shell

    Shell (shell, aka “command line”, aka CLI, aka “console”, aka “terminal”, aka “black window with white letters”) is a text interface for communicating with the operating system (well, strictly I mean, this is program, which provides such an interface, but now this difference is insignificant).

    In general, work through a shell looks like this: the user (i.e. you) enters a command from the keyboard, presses Enter, the system executes the command, writes the result of the execution to the screen, and again waits for the next command to be entered.

    Typical shell type:

    The shell is the primary way to interact with all Unix-like server systems.

    Where are command line systems found?

    Where a unix shell might be waiting for you, popular options:
    • MacOS (bash);
    • remote access to the server for work or for a personal web project;
    • home file server with remote access;
    • Ubuntu, PC-BSD on laptop/desktop - unix-like systems today are easy to install and use.

    What problems are reasonable to solve with a shell?

    Natural tasks for which the shell is suitable, useful and indispensable:
    • interactive work in the terminal:
      • compiling, running jobs via make;
      • comparison of text files;
      • fast ad-hoc data analysis (number of unique IPs in the log, distribution of records by hours/minutes, etc.);
      • one-time mass actions (kill many processes; if you work with a version control system, reverse or resolve a bunch of files);
      • diagnostics of what is happening in the system (semaphores, locks, processes, descriptors, disk space, etc.);
    • scripting:
      • installation scripts, for which you cannot rely on the presence of other interpreters - this is not for beginners;
      • functions for customizing the interactive shell (affecting the invitation, changing the directory, setting environment variables) are also not exactly for beginners;
      • one-time scripts such as mass file recoding;
      • makefiles.

    Absolutely first steps

    Let's get started: sign in and sign out

    Make sure you know exactly how to start the shell and how to exit it.

    If you are working on a machine with Ubuntu installed, you need to launch the Terminal program. When finished, you can simply close the window.

    On MacOS - also launch Terminal.

    To access a remote server, use ssh (if you have MacOS, Ubuntu or another unix-like system locally) or putty (if you have Windows).

    Who am I, where am I?

    Run the following commands:
    • hostname - displays the name of the machine (server) you are currently on;
    • whoami - displays your login (your name in the system);
    • tree -d / |less - pseudo-graphic representation of the directory tree on the machine; exit from scrolling - q ;
    • pwd - displays the directory you are currently in; on the command line you cannot be “just like that”, you must be in some directory (=current directory, working directory). The current working directory is probably displayed in your prompt.
    • ls - list of files in the current directory; ls /home - list of files in the specified directory;

    Command history (history)

    An important property of a full-fledged command line is the command history.

    Run several commands: hostname, ls, pwd, whoami. Now press the up key. The previous command appears in the input line. You can use the up and down keys to move forward and backward through the history. When you get to hostname, press Enter - the command will be executed again.

    Commands from history can not only be executed repeatedly, but also edited. Scroll through the history to the ls command, add the -l switch to it (it turns out ls -l , there is a space before the minus, but not after). Press Enter - the modified command will be executed.

    Scrolling through history, editing and re-executing commands are the most common actions when working on the command line, so get used to it.

    Copy-paste

    The command line is very text-centric: commands are text, input data for most standard programs is text, and the result of the work is most often also text.

    The great thing about text is that it can be copied and pasted, and this is true on the command line as well.

    Try the command date +"%y-%m-%d, %A"
    Did you enter it entirely by hand or copied it from the article? Make sure you can copy it, paste it into a terminal and execute it.

    Once you've learned how to use man, make sure you can copy and run example commands from the help. To check, look up the EXAMPLES section of the date program help, copy and run the first example given (just in case: the dollar sign is not part of the command , this is a symbolic image of an input prompt).

    How exactly to copy text from the terminal and paste it into the terminal depends on your system and its settings, so give universal instructions, unfortunately, it won't work. On Ubuntu, try this: copy - just select with the mouse, paste - the middle mouse button. If it doesn’t work, or if you have a different system, look on the Internet or ask more experienced friends.

    Keys and options

    As you've explored the command history, you've already encountered that the ls command has at least two options. If you call it just like that, it outputs a simple list:

    Akira@latitude-e7240: ~/shell-survival-quide> ls Makefile shell-first-steps.md shell-first-steps.pdf shell-survival-quide.md shell-survival-quide.pdf
    If you add the -l switch, detailed information is displayed for each file:

    Akira@latitude-e7240: ~/shell-survival-quide> ls -l total 332 -rw-rw-r-- 1 akira akira 198 Feb 13 11:48 Makefile -rw-rw-r-- 1 akira akira 15107 Feb 14 22:26 shell-first-steps.md -rw-rw-r-- 1 akira akira 146226 Feb 13 11:49 shell-first-steps.pdf -rw-rw-r-- 1 akira akira 16626 Feb 13 11 :45 shell-survival-quide.md -rw-rw-r-- 1 akira akira 146203 Feb 13 11:35 shell-survival-quide.pdf
    This is a very typical situation: if you add special modifiers (keys, options, parameters) to a command call, the behavior of the command changes. Compare: tree / and tree -d / , hostname and hostname -f .

    In addition, commands can take file names, directory names, or simply text strings as parameters. Try:

    Ls -ld /home ls -l /home grep root /etc/passwd

    man

    man - Help with the commands and programs available on your machine, as well as system calls and the standard C library.

    Try: man grep, man atoi, man chdir, man man.

    Scrolling forward and backward is done with the “up”, “down”, “PageUp”, “PageDown” buttons, exiting the help view is done with the q button. Search for specific text in a help article: press / (forward slash), enter text to search, press Enter. Move to next occurrences - key n.

    All help articles are divided into categories. The most important:

    It is necessary to indicate from which category the certificate should be shown in cases of coincidence of names. For example, man 3 printf describes a function from the C standard library, and man 1 printf describes a console program with the same name.

    You can view a list of all help articles available on your machine using the man -k command. (the dot is also part of the komada).

    less

    When you need to view very long text in a small terminal window (the contents of a file, a long man, etc.), special “pager” programs are used (from the word page, that is, page flippers). The most popular scroller is less, and it is the one that provides you with the scrolling when you read man pages.

    Try and compare the behavior:

    Cat /etc/bash.bashrc cat /etc/bash.bashrc |less

    You can transfer the file to the pager directly in the parameters:

    Less /etc/bash.bashrc

    Scrolling up and down - buttons "up", "down", "PageUp", "PageDown", exit - button q. Search for specific text: press / (forward slash), enter the text to search, press Enter. Move to next occurrences - key n. (Do you recognize the instructions about man ? No wonder, less is also used to display help.)

    Rights

    Any file or directory is associated with a set of “rights”: the right to read the file, the right to write to the file, the right to execute the file. All users are divided into three categories: file owner, file owner group, all other users.

    You can view file permissions using ls -l . For example:

    > ls -l Makefile -rw-r--r-- 1 akira students 198 Feb 13 11:48 Makefile
    This output means that the owner (akira) can read and write the file, the group (students) can only read, and all other users can also only read.

    If you receive a permission denied message while working, this means that you do not have sufficient permissions for the object you wanted to work with.

    Read more in man chmod.

    STDIN, STDOUT, conveyors (pipes)

    There are 3 standard data streams associated with each executing program: input data stream STDIN, output data stream STDOUT, error output stream STDERR.

    Run the wc program, enter the text Good day today, press Enter, enter the text good day, press Enter, press Ctrl+d. The wc program will show statistics on the number of letters, words and lines in your text and end:

    > wc good day today good day 2 5 24
    In this case, you supplied a two-line text to the program's STDIN, and received three numbers in STDOUT.

    Now run the command head -n3 /etc/passwd , it should look something like this:

    > head -n3 /etc/passwd root:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin bin:x: 2:2:bin:/bin:/usr/sbin/nologin
    In this case, the head program did not read anything from STDIN, but wrote three lines to STDOUT.

    You can imagine it this way: the program is a pipe into which STDIN flows and STDOUT flows out.

    The most important property of the Unix command line is that “pipe” programs can be connected to each other: the output (STDOUT) of one program can be passed as input data (STDIN) to another program.

    This construction of connected programs is called a pipe in English, or a conveyor or pipe in Russian.

    Combining programs into a pipeline is done with the symbol | (vertical bar)

    Run the command head -n3 /etc/passwd |wc , it will look something like this:

    > head -n3 /etc/passwd |wc 3 3 117
    Here's what happened: the head program output three lines of text in STDOUT, which immediately went to the input of the wc program, which in turn counted the number of characters, words and lines in the resulting text.

    You can combine as many programs as you like into a pipeline. For example, you can add another wc program to the previous pipeline, which will count how many words and letters were in the output of the first wc:

    > head -n3 /etc/passwd |wc |wc 1 3 24

    Creating pipelines (pipes) is a very common task when working on the command line. For an example of how this is done in practice, read the section “Creating a one-liner pipeline.”

    I/O redirection

    The output (STDOUT) of a program can not only be transferred to another program via a pipeline, but also simply written to a file. This redirection is done using > (greater than sign):

    Date > /tmp/today.txt
    As a result of executing this command, the file /tmp/today.txt will appear on the disk. View its contents using cat /tmp/today.txt

    If a file with the same name already existed, its old contents will be destroyed. If the file did not exist, it will be created. The directory in which the file is created must exist before the command is executed.

    If you don't want to overwrite a file, but rather add output to the end of it, use >> :

    Date >> /tmp/today.txt
    Check what is now written in the file.

    In addition, you can pass any file to the program instead of STDIN. Try:

    Wc

    What to do when something is unclear

    If you encounter system behavior that you don’t understand, or want to achieve a certain result, but don’t know how, I advise you to proceed in the following order (by the way, this applies not only to shells):
    • As clearly as possible, formulate the question or task - there is nothing more difficult than solving “something I don’t know what”;
    • remember if you have already encountered the same or similar problem - in this case, it is worth trying the solution that worked last time;
    • read the appropriate man pages (if you understand which man pages are suitable in your case) - perhaps you will find suitable examples of using commands, the necessary options or links to other commands;
    • think: is it possible to change the task a little? - perhaps, by slightly changing the conditions, you will get a problem that you already know how to solve;
    • ask your clearly formulated question in a search engine - perhaps the answer can be found on Stack Overflow or other sites;
    If none of the above helps, seek advice from a teacher, an experienced colleague or friend. And don’t be afraid to ask “stupid” questions - it’s not a shame not to know, it’s a shame not to ask.

    If you solve a difficult problem (on your own, with the help of the Internet or other people), write down your solution in case the same problem arises again for you or your friends. You can record it in a simple text file, in Evernote, or publish it on social networks.

    Working methods

    Copy-and-paste- from man pages, from articles on StackOverflow, etc. The command line consists of text, take advantage of this: copy and use example commands, write down successful findings as a keepsake, publish them on Twitter and blogs.

    Pull the previous command from the history, add another command to the pipeline, run, repeat.Cm. See also the section “Creating a one-liner pipeline.”

    Basic commands

    • change to another directory: cd ;
    • viewing the contents of files: cat, less, head, tail;
    • file manipulation: cp, mv, rm;
    • viewing directory contents: ls , ls -l , ls -lS ;
    • directory structure: tree , tree -d (directory can be passed as a parameter);
    • search for files: find . -name ... ;

    Analytics

    • wc, wc -l;
    • sort -k - sort by the specified field;
    • sort -n - numeric sorting;
    • diff - file comparison;
    • grep , grep -v , grep -w , grep "\ " , grep -E - search for text;
    • uniq , uniq -c - string uniqueization;
    • awk - in the awk "(print $1)" option, to leave only the first field from each line, $1 can be changed to $2, $3, etc.;

    System diagnostics

    • ps axuww - information about processes (running programs) running on the machine;
    • top - interactive viewing of the most resource-intensive processes;
    • df - used and free disk space;
    • du - total size of files in the directory (recursively with subdirectories);
    • strace , ktrace - what system calls the process makes;
    • lsof - what files the process uses;
    • netstat -na, netstat -nap - which ports and sockets are open in the system.

    You may not have some programs; they need to be installed additionally. In addition, some options of these programs are available only to privileged users (root).

    Bulk and semi-automatic execution

    At first, skip this section; you will need these commands and constructs when you get to simple shell scripting.
    • test - checking conditions;
    • while read - loop line by line STDIN ;
    • xargs - substitution of strings from STDIN into parameters of the specified program;
    • seq - generation of sequences of natural numbers;
    • () - combine the output of several commands;
    • ; - do one thing after another;
    • && - execute if the first command completes successfully;
    • || - execute if the first command fails;
    • tee - duplicate the program output to STDOUT and to a file on disk.

    Miscellaneous

    • date - current date;
    • curl - downloads a document from the specified url and writes the result to STDOUT;
    • touch - update file modification date;
    • kill - send a signal to the process;
    • true - does nothing, returns true, useful for organizing eternal loops;
    • sudo - execute the command as root "a.

    Creating a one-liner pipeline

    Let's look at an example of a real task: we need to kill all task-6-server processes running as the current user.

    Step 1.
    Understand which program produces approximately the necessary data, even if not in its pure form. For our task, it is worth getting a list of all processes in the system: ps axuww. Launch.

    Step 2.
    Look at the data received with your eyes, come up with a filter that will throw out some of the unnecessary data. This is often grep or grep -v . Use the “Up” key to pull out the previous command from the history, assign an invented filter to it, and run it.

    Ps axuww |grep `whoami`
    - only processes of the current user.

    Step 3.
    Repeat step 2 until you get the clean data you need.

    "
    - all processes with the required name (plus, perhaps, extra ones like vim task-6-server.c, etc.),

    Ps axuww |grep `whoami` | grep "\ " | grep -v vim ps axuww |grep `whoami` | grep "\ " | grep -v vim |grep -v less
    - only processes with the required name

    Ps axuww |grep `whoami` | grep "\ " | grep -v vim |grep -v less |awk "(print $2)"

    Pids of the required processes, step 3 completed

    Step 4.
    Apply a suitable final handler. Using the “Up” key, we pull out the previous command from the history and add processing that will complete the solution to the problem:

    • |wc -l to count the number of processes;
    • >pids to write pids to a file;
    • |xargs kill -9 kill processes.

    Training tasks

    Want to practice new skills? Try the following tasks:
    • get a list of all files and directories in your home directory;
    • get a list of all man articles from category 2 (system calls);
    • count how many times the word grep appears in the grep program man page;
    • count how many processes are currently running as root;
    • find which command appears in the maximum number of help categories (man);
    • count how many times the word var appears on the page ya.ru.
    Hint: you will need find , grep -o , awk "(print $1)" , regular expressions in grep , curl -s .

    What to study next?

    If you start to like the command line, don't stop, keep improving your skills.

    Here are some programs that will definitely come in handy if you live on the command line:

    • find with complex options
    • apropos
    • locate
    • telnet
    • netcat
    • tcpdump
    • rsync
    • screen
    • zgrep, zless
    • visudo
    • crontab -e
    • sendmail
    In addition, over time it is worth mastering a scripting language, such as perl or python, or even both.

    Who needs this?

    Is it even worth learning the command line and shell scripting today? Definitely worth it. I’ll give just a few examples of Facebook’s requirements for candidates who want to get a job at FB.

    Command interpreter

    To provide a command line interface, the OS often uses command interpreters, which can be independent programming languages ​​with their own syntax and distinctive functionality.

    The Windows 9x operating system includes the command interpreter command.com, and Windows NT includes the UNIX command shells, popular csh, ksh and others.

    As a rule, when low-level tuning the OS, the user has the opportunity to change the default shell.

    Functions

    The command interpreter executes commands in its language, given on the command line or coming from standard input or a specified .

    Calls to system or application utilities, as well as control structures, are interpreted as commands. In addition, the shell is responsible for expanding filename patterns and for redirecting and binding utility I/O.

    Together with a set of utilities, the shell is an operating environment, a full-fledged programming language and a powerful tool for solving both system and some application problems, especially automating frequently executed sequences of commands.

    Standard command interpreter

    Implementations and Availability

    Alternatives

    Along with the standard ones, open operating systems also use alternative tcsh shells, which differ in the syntax of control constructs and the behavior of variables.

    Some alternative operating systems come with interpreters of their own command file languages ​​(such as Microsoft Windows NT command file language, OS/2 language, etc.)

    Some people prefer to use new interpreted languages, such as Python, to automate frequently executed command sequences.

    Graphic shell

    Graphic shells for OS Windows

    The latest versions of Windows OS use the Windows Explorer integrated environment as their shell. Windows Explorer is a visual management environment that includes the Desktop, Start Menu, Taskbar, and file management functions. Early versions of Windows OS 3.xx included a program manager as a graphical shell.

    Many third-party developers offer alternative environments that can be used instead of the Explorer shell that Microsoft includes by default in Windows.

    • Aston shell
    • BB4Win
    • BBlean
    • Cairo (Under Development)
    • Chroma
    • Emerge Desktop
    • Geoshell
    • Packard Bell Navigator
    • Program Manager
    • Secure Desktop
    • SharpE
    • Talisman Desktop
    • WinStep
    • Microsoft Bob

    See also

    Wikimedia Foundation. 2010.

    See what “Command Shell” is in other dictionaries:

      - (English Unix shell, often just “shell” or “sh”) a command interpreter used in operating systems of the UNIX family, in which the user ... Wikipedia

      This term has other meanings, see Wish. wish (Windowing Shell) is a simple scriptable or interactive UNIX command shell for the X Window System and Mac OS X. It provides users with control over components... ... Wikipedia

      - ... Wikipedia

      The Command Line Interpreter request is redirected here. A separate article is needed on the topic “Command Line Interpreter”. Appearance of the shell (English Co ... Wikipedia

      Operating system shell (from the English shell shell) is an interpreter of operating system (OS) commands, providing an interface for user interaction with system functions. In general, there are shells with two types of interface for ... Wikipedia

      UNIX command shell (English Unix shell, often just “shell” or “sh”) is a command interpreter used in operating systems of the POSIX family of compatible shells, dating back to the Bourne shell, which appeared in Unix Version 7. Contents 1 ... ... Wikipedia

      UNIX command shell (English Unix shell, often just “shell” or “sh”) is a command interpreter used in operating systems of the POSIX family of compatible shells, dating back to the Bourne shell, which appeared in Unix Version 7. Contents 1 ... ... Wikipedia

      UNIX command shell (English Unix shell, often just “shell” or “sh”) is a command interpreter used in operating systems of the POSIX family of compatible shells, dating back to the Bourne shell, which appeared in Unix Version 7. Contents 1 ... ... Wikipedia

    Books

    • Shell scripts. Linux, OS X and Unix. Guide, Taylor Dave. Shell scripts have helped system administrators and programmers automate routine tasks since the dawn of computers. Since the release of the first...

    As mentioned above, to build arbitrary algorithms it is necessary to have condition checking operators. Shell bash supports selection statements ifthenelse and case, as well as looping operators for,while, until, making it a powerful programming language.

    5.8.1 Operators if And test(or )

    The construction of the conditional operator in a slightly simplified form looks like this:

    if list1 then list2 else list3 fi

    Where list1, list2 and list3 are sequences of commands separated by commas and ending with a semicolon or newline character. Additionally, these sequences can be enclosed in curly braces: (list).

    Operator if checks the value returned by commands from list1. If there are several commands in this list, then the value returned by the last command in the list is checked. If this value is 0, then commands from list2; if this value is not zero, the commands from list3. The value returned by such a compound operator if, is the same as the value produced by the last command in the sequence being executed.

    Full command format if has the form:

    if list then list [ elif list then list ] ... [ else list ] fi

    (here the square brackets only mean that what is contained in them is not necessarily present in the operator).

    As an expression that comes immediately after if or elif, a frequently used command test, which can also be denoted by square brackets. Team test evaluates some expression and returns 0 if the expression is true and 1 otherwise. The expression is passed to the program test as an argument. Instead of writing

    test expression,

    You can enclose the expression in square brackets:

    [expression].

    Please note that test and [ are two names of the same program, not some magical conversion performed by the shell bash(only the [ syntax requires a closing parenthesis to be included). Note also that instead of test in design if any program can be used.

    In conclusion, we give an example of using the operator if:

    if [ -e textmode2.htm ] ; then

    ls textmode*

    else

    pwd

    About the operator test(or […]) we need to have a special conversation.

    5.8.2 Operator test and conditionals

    Conditional expressions used in the statement test, are built on the basis of checking file attributes, string comparisons and ordinary arithmetic comparisons. Complex expressions are built from the following unary or binary operations (“elementary building blocks”):

      A file

    True if a file named file exists.

      B file

    True if file exists and is a special block device file.

      C file

    True if file exists and is a special character device file.

      D file

    True if file exists and is a directory.

      E file

    True if the file named file exists.

      F file

    True if the file named file exists and is a regular file.

      G file

    True if the file named file exists and its group change bit is set.

      H file or -L file

    True if the file named file exists and is a symbolic link.

      K file

    True if the file named file exists and its "sticky" bit is set.

      P file

    True if the file named file exists and is named pipe(FIFO).

      R file

    True if the file named file exists and has read permission set

      S file

    True if the file named file exists and its size is greater than zero.

      Tfd

    True if the file descriptor fd is open and points to the terminal.

      U file

    True if the file named file exists and its change user bit is set.

      W file

    True if the file named file exists and has write permission set.

      X file

    True if the file named file exists and is executable.

      O file

    True if the file named file exists and is owned by the user pointed to by the effective user ID.

      G file

    True if the file named file exists and belongs to the group identified by the effective group ID.

      S file

    True if the file named file exists and is a socket.

      N file

    True if the file named file exists and has changed since it was last read.

      file1 -nt file2

    True if the file file1 has a later modification time than file2.

      file1 -ot file2

    True if the file file1 older than file2.

      file1 -ef file2

    True if the files file1 And file2have the same device and inode numbers(inode).

      O optname

    True if shell option is enabled optname. For an explanation, see the bash man page.

      Z string

    True if the length of the string is zero.

      N string

    True if the length of the string is not zero.

      string1 == string2

    True if the strings match. Instead of == can be used = .

      string1 !== string2

    True if the strings do not match.

      string1< string2

    True if the line string1 lexicographically precedes the string string2(for the current locale).

      string1 > string2

    True if the line string1 lexicographically comes after the line string2(for the current locale).

      arg1 OP arg2

    Here OP- uh then one of the arithmetic comparison operations: -eq(equals), -ne(not equal) -lt(less than) -le(less than or equal to) -gt(more), -ge(greater than or equal to). Positive or negative integers can be used as arguments.

    From these elementary conditional expressions you can build as complex as you like using the usual logical operations of NEGATION, AND and OR:

      !(expression)

    Boolean negation operator.

      expression1 -a expression2

    Boolean operator AND(AND). True if both expressions are true.

      expression1 -o expression2

    Boolean operator OR(OR). True if either of the two expressions is true.

    The same conditional expressions are used in operators while And until, which we will look at below.

    5.8.3 Operator case

    Operator format case is:

    case word in [ [(] pattern [ | pattern ] ...) list ;; ]...esac

    Team case first produces word expansion word, and tries to match the result with each of the samples pattern one by one. After the first match is found, no further checks are performed; the list of commands following the pattern with which the match was found is executed. The value returned by the operator is 0 if no pattern matches were found. Otherwise, the value produced by the last command in the corresponding list is returned.

    The following example of using the case statement is taken from the system script /etc/rc.d/rc.sysinit.

    case "$UTC" in

    yes|true)

    CLOCKFLAGS="$CLOCKFLAGS -u";

    CLOCKDEF="$CLOCKDEF (utc)";

    no|false)

    CLOCKFLAGS="$CLOCKFLAGS --localtime";

    CLOCKDEF="$CLOCKDEF (localtime)";

    esac

    If the variable evaluates to yes or true, then the first pair of commands will be executed, and if its value is no or false, the second pair will be executed.

    5.8.4 Operator select

    Operator select allows you to organize interactive interaction with the user. It has the following format:

    select name [in word; ] do list ; done

    First from the template word a list of words matching the pattern is generated. This set of words is output to standard error, with each word followed by serial number. If the pattern word omitted, positional parameters are derived in the same way. The standard PS3 prompt is then issued and the shell waits for a line to be entered on standard input. If the entered string contains a number corresponding to one of the displayed words, then the variable name is assigned a value equal to that word. If an empty line is entered, the numbers and corresponding words are displayed again. If any other value is entered, the variable name is assigned a value of zero. The string entered by the user is stored in a variable REPLY. List of commands list executed with the selected variable value name.

    Here's a small script:

    #!/bin/sh

    echo "Which OS do you prefer?"

    select var in "Linux" "Gnu Hurd" "Free BSD" "Other"; do

    break

    done

    echo "You would choose $var"

    Which OS do you prefer?
    1) Linux
    2) Gnu Hurd
    3) Free BSD
    4)Other
    #?

    Press any of the 4 suggested numbers (1,2,3,4). If you enter 1, for example, you will see the message:

    “Would you choose Linux”

    5.8.5 Operator for

    Operator for works a little differently than in regular programming languages. Instead of causing the value of some variable to increase or decrease by one each time it passes through the loop, it assigns the next value from a given list of words to the variable each time it passes through the loop. In general, the design looks something like this:

    for name in words do list done.

    Rules for constructing command lists ( list) are the same as in the operator if.

    Example. The following script creates the files foo_1, foo_2 and foo_3:

    for a in 1 2 3 ; do

    touch foo_$a

    done

    In general, the for statement has the format:

    for name [in word; ] do list ; done

    First, the word is revealed word in accordance with the expression disclosure rules given above. Then the variable name the resulting values ​​are assigned one by one, and a list of commands is executed each time ist. If " in word" is missing, then the list of commands list is executed once for each positional parameter that is specified.

    Linux has a program seq, which takes two numbers as arguments and produces a sequence of all numbers located between the given ones. With this command you can force for V bash work exactly the same way as a similar operator works in conventional programming languages. To do this, just write the cycle for as follows:

    for a in $(seq 1 10) ; do

    cat file_$a

    done

    This command displays the contents of 10 files: " file_1", ..., "file_10".

    5.8.6 Operators while And until

    Operator while works like if, only executing operators from the list list2 loop continues as long as the condition is true and aborts if the condition is not true. The design looks like this:

    while list1 do list2 done.

    while [ -d mydirectory ] ; do

    ls -l mydirectory >> logfile

    echo -- SEPARATOR -- >> logfile

    sleep 60

    done

    Such a program will log the contents of the directory "mydirectory" every minute as long as the directory exists.

    Operator until similar to operator while:

    until list1 do list2 done.

    The difference is that the result returned when executing a list of statements list1, taken with negation: list2 executed if the last command in the list list1 returns a non-zero exit status.

    5.8.7 Functions

    Syntax

    Shell bash allows the user to create their own functions. Functions behave and are used exactly like regular shell commands, meaning we can create new commands ourselves. The functions are constructed as follows:

    function name() (list)

    And the word function not necessarily name defines the name of the function by which it can be accessed, and the body of the function consists of a list of commands list, located between ( and ). This list of commands is executed every time the name name specified as the name of the command to invoke. Note that functions can be defined recursively, so it is permissible to call the function we define within itself.

    Functions are executed in the context of the current shell: a new process is not started to interpret the function (unlike executing shell scripts).

    Arguments

    When a function is called for execution, the function's arguments become positional parameters(positional parameters) for the duration of the function. They are referred to as $n, Where n— the number of the argument we want to access. Argument numbering starts at 1, so $1 - this is the first argument. We can also get all the arguments at once with $* , and the number of arguments using $# . Positional parameter 0 does not change.

    If a built-in command occurs in the function body return, the execution of the function is interrupted and control is transferred to the command after the function call. When the function completes, the positional parameters and the special parameter # the values ​​they had before the function started are returned.

    Local variables (local)

    If we want to create local parameter, you can use the keyword local. The syntax for specifying it is exactly the same as for regular parameters, only the definition is preceded by a keyword local: local name=value.

    Here is an example of specifying a function that implements the command mentioned above seq:

    seq()

    local I=$1;

    while [ $2 != $I ]; do

    echo -n "$I ";

    I=$(($I + 1))

    done;

    echo $2

    Please note the option -n operator echo, it cancels the transition to new line. Although this is not essential for the purposes we have in mind here, it may be useful for using the function for other purposes.

    Factorial calculation function fact

    Another example:

    fact()

    if [ $1 = 0 ]; then

    echo 1;

    else

    echo $(($1 * $(fact $(($1 - 1)))))

    This is the factorial function, an example of a recursive function. Note arithmetic expansion and command substitution.

    V. Kostromin (kos at rus-linux dot net) - 5.8. Shell as a programming language

    The main environment for interacting with UNIX is command line. Its essence is that each line transmitted by the user to the system is a command that it must execute. Until a key is pressed Enter, the line can be edited, then it is sent to the system.

    The commands are interpreted and executed by a special program - command shell(or "shell" in English). User processes are managed through the command shell - interprocess communication tools are used for this.

    The command shell is directly connected to the terminal, through which control sequences and text are transmitted. Figure (3.1) shows a general diagram of user interaction with the system when using the command line.

    Figure 3.1. Command Line Interface

    Simultaneous access to the system

    Each computer running UNIX allows multiple users to log in and access the system simultaneously. Even if all users have only one monitor and one system keyboard, this feature is useful: having multiple users logged in at the same time allows you to work in turn without having to complete all started tasks and then resume them each time. It is also possible to register in the system several times under the same login name. This way, you can access the same resources and organize parallel work on several tasks.

    A characteristic way for modern versions of UNIX to organize parallel work of users is virtual consoles. Virtual consoles are several programs executed in parallel by the operating system that provide the user with the ability to log into the system in text mode and gain access to the command line (see Figure 3.2, “Virtual and graphical consoles”).

    IN operating system Linux switching between virtual consoles in text mode is done using a keyboard shortcut Alt-F1,Alt-F2 etc. In this case, each virtual console is designated by a special name: “tty1”, “tty2”, etc. Each combination is associated with a corresponding virtual console number. Each of these consoles is associated with its own terminal, which is characterized by a device file with a corresponding name (for example, /dev/tty1).

    Modern versions of UNIX provide users with graphical user interfaces that also provide command line functionality. This feature is achieved using graphic terminals– special programs that emulate a text terminal in a graphical window.

    It is also possible to launch several graphic subsystems, then switching between them will be carried out similarly to switching between virtual consoles - using a key combination Ctrl-Alt-F1.

    Figure 3.2. Virtual and graphical consoles

    Each terminal device has its own possibilities on input and output of information. Examples of such capabilities are: the number of display colors, the ability to move the cursor and change the screen size, a set of control characters, etc. Terminals are divided by types: a set of capabilities regulated in a special configuration file. Examples of types are: tty (teletypewriter) or xterm (graphics terminal). The terminal type is explicitly specified in the terminal device name (for example, /dev/tty1). All terminal type settings are located in the /etc/termcap directory.