When Stephen Bourne released the his shell in 1977, shell scripting was born and he turned every UNIX user into a developer. The UNIX shell can be used interactively, but the real power is when the shell is used as a programming language. Modern shells like bash can be seen as a general-purpose language, but the syntax and semantics can be awkward at times. Moreover, UNIX includes a broad range of small utilities (cut, sed, tr, etc.). These utilities together, with the programming constructs of the shell, enable the user to automate many tasks and thereby become a more productive user. In this talk at Øredev Conference 2015, Kenneth Geisshirt will show how to use the command-line/terminal/shell efficiently, and go over general patterns and pitfall in shell scripting. To feel the real power of OS X, writing shell scripts is essential.

---

Become a Console Cowboy (0:00)

I had been a Unix (Venix) user for many years, but then I ended up with a Mac. I opened the terminal. All the applications have Emacs bindings (I am an old Emacs user). And I realized: Mac is made not for end users but for developers. That is the motivation behind my talk.

You should not use the mouse very much. Use the shell, use bash especially as a productivity tool. Writing scripts, there is a way to boost your performance, and your productivity at work. At the end of the talk, I hope that you are a console cowboy.

Agenda (three quarters of the talk about Unix; a quarter is Mac specific):

1. the terminal, and the shell
2. the basic usage of bash, and how to live on the command line
3. scripting.

The Shell (2:58)

Terminal and shell (3:04)

When you work on a shell you have to use a terminal emulator. Apple provides one (do not use that); the iTerm2 is a better alternative: keeps your secure shell connections alive.

You should change some of the keyboard layout (system preferences), the bindings. For example, easier to change tab: CTRL left + right (Mac); alt left + right for desktop preferences. If you have multiple terminals and shells running at the same time, it is easier to change between them.

I have colleagues that claim that fish, or bash (since 10.3 of Mac OS, the default shell) is the best shell. The current shell installed on a Mac is a fairly recent one. It secures your bug, it has to be fixed very quickly. Home brew has many great bash related packages.

Basic usage of bash: the philosophy of Unix is that a tool should only do one thing, but do it well. Text is universal, basic format. The output of one utility should be the input for the next utility.

Redirection (7:25)

Bash implements redirections - you can stdout output (using > or character). You have small ones that do have outputs and useful input. You can also take the stdout arrow and put it into stdout input, one file descriptor to another. You can get all messages from some utility (e.g. clang is a small utility used by developers). You can output all the outputs, both stdout input and output, to a file. That is a way you can save; even though the Unix utility cannot save files, you can always use redirection on your shells. If you write your own utility (read from stdout input, and you can write to stdout output), the shell can take care of the rest: you do not have to take care of reading from a file.

Pipes (9:40)

One utility writes to an output; the next one has to read that output as input: you stick them together using pipes. For example, you have a text terminal, keyboard input output for program one. Program one might output something, which program two can then read. You can have this pipeline of data flowing from one utility to another.

Configuration (11:55)

Bash can be configured: you normally have a system wide, but you can also have your own. Typically you have your own home directory. Bash profile, $HOME/.bash_profile, is very often used. alias, useful for often used options. You can also set up the prompt for the command line (PS1) and search path (PATH). You can reload configuration using the source command.

Bash (.bash_profile) can remember the commands in your history, and which ones not to remember (see video for an example).

Keyboard shortcuts (14:53)

Bash uses Emacs bindings by default (it is also possible to set up VI bindings). The CTRL-a and -e goes to either the beginning / end of the line (also true for Mac OS 10 applications). If you are in pages, CTRL-e will go to the end of the line. To remap, CTRL left and right that go one word on the command line. If you are using the default settings in your desktop, then it will change to the other workspace (that is why you have to change it).

You can also do Cut-n-paste on your command line. CTRL-k erases the rest of the line. ESC BACK space to delete one word. Arrow up gives you the previous command. CTRL left to hop to the place where you want to modify. Maybe you press CTRL ESC BACK space to delete that parameter. Then return, and runs command. If it is a long command, you go back by one, arrow left, and if you still end up getting off the line.

History (18:05)

Bash stores your command history. You can rerun the latest command using ! !. Typing history, you can get all the commands; ! number, will run that command numberagain. If you type CTRL-r, it will search back and find the command that matches. And you might have to be typing the same thing, and it is not the first one it finds, but the next one you want.; CTRL-r is search reverse in Emacs (you can search backwards). Normally cd and ls is something you type much, and nothing worth remembering - you can ignore them in your history (HISTIGNORE="pwd:ls:ls -l:cd").

Completion (19:18)

You start typing, type TAP Bash will try to guess what you are trying. There is this programmable completion in bash (in the newer versions). If you are using home brew, there is ways to set that up; if you are a Linux user, there is packages for bash completion.

Living on the Command Line (20:34)

If you type cd -, and some directory, or ghost, you change to that directory. If you type cd -, it will go back to the previous one. Also, file which tries to guess the content of the file (uses magic numbers). There is lsof, which lists open files (very often you have an application you cannot close, because it has open files; or because you have no clue what files opened). There is ps for listing all the processes. And a small utility called watch (you do not have that on a Mac, you can do you own small watch by writing a one line script; see video).

OS X–specific commands (21:57)

• Open. If you are on a command line, type Open, as in some file, it will try to… For example, If you type open, and then file, it will open it in your browser. You can type Open, and then a number file, for it will open numbers. Or if you type open on C file, it will open it in Xcode, and so on.
• Say. Read aloud in Mac OS 10, very useful for speech synthesis.
• pbcopy. It will take all the output and input, and put it into the paste part. ls pipe PBcopy, and then in another program type command v, it will paste it in there. You can do command paste directly from the shell.
• pbpaste: if you copy something from any Mac application, you can do a pbpaste, and it will paste it into your shell directly.
• dns-sd -B _ssh._tcp show Bonjour enabled SSH hosts.

Useful utilities (24:53)

Other utilities (stdoud Unix): find; grep, and cut (to cut fields out of, if you have a comma separated file); Wordcount for wc; transform (transform any character, tr, to another); sorting; look into last/first lines of a file (see video).

sed - the stream editor (25:52)

If you are an old Linux user, you are used to the canoe. -i option is tricky. It can take a file, or a stream of characters and do changes to that. Option -E gives an editing expression. s/FISH/HORSE/g, for example, to substitute; or /FISH/d to delete lines.

awk - a processing tool (27:12)

awk is a programming language by itself. It matches line, finds line and then processes it. This one is very much a spreadsheet (data in columns). For instance, we want to sum column one (command line spreadsheet function). I often use AWK for counting and doing calculations on comma separated files.

Scripting (28:17)

Bash for programmers (28:20)

Bash is also for programmers; a complete programming language by itself. Maintaining shell scripts of more than 1000 lines are not that fun. It is also an interpreted language: very slow.

Basic syntax (29:09)

The basic syntax: white spaces (space and tab). Comments begin with a #. Statements end-of-line, have a ;. Variables begin with a letters, digits or underscore. Hash mark, exclamation mark to identify a bash script.

This is a small program that does estimate pi using Monte Carlo simulations (see video). You can do simulations in bash; it is not recommended - it is slow.

Variables (30:01)

Build-in variables: $ , the first argument of the script. $? the exit code from the command you just executed (check if it failed or not). You can also do math directly. But you cannot type in an expression: you have to put it in $ (( )) (see video for examples).

Branches (30:57)

if, then, else fi construct for simple branching. Switch is interesting because it is a case in esac. The if ends, which is also if in reverse (you know this is where it stops). There is string operators, and integer comparisons (see video).

For example (see video), this case construct is your switches. Notice this semicolon, semicolon, which ends a case.

Loops (32:12)

Loops (list of things), such as for or while. This one is particularly useful: you can read, and it reads from the stand up input. You will often take the outputs of some utility, while read line. You read the output from that utility one by one, and do something about it. (see video for an example).

Functions (33:38)

There is also functions; a function can return integers only (it cannot return a string). If you need to return a string, you have to use a global variable. (see video for multiplication function example). Bash is very good for processing text files. It is not very good for calculations.

Tips and tricks (34:22)

When you are writing scripts, you can use the set -e: if any of the utilities in your script fails, the script will fail immediately. You do not have to do any check. There is also a problem if you have pipes, and the first programming a pipe fails, and the second one does not fail: you will not get that error code. If you want an early warning, you have to set -O pipefail. tee command is taking stdoud input, and it can write it to a file, and to the stdoud output. Very often you want to see the stdoud output, but you also want to keep a log file of what is happening; you use if for creating log files. Then you can set -x, and that will write all the commands it is doing and what the variables are assigned to.

When you have a variable, put quotes around it (see video for examples: There is one way I do not quote the string variable, and the next one I do. The first one will output two lines, and the second one would output one line only. Different, because there is space in the string. When you have spaces in strings, you have to be careful. You might have spaces in directory names, and file names Then you have spaces in file names, you have to remember your strings).

Using -p for the mkdir - the command will fail if it cannot make that directory. You can also create temp. files with mktemp /tmp/$$.XXXXX. There is also this one about subshells (start a subshell): (cd foo && rm -f bar). I will not have to remember which directory to go back to because it is a subshell, and will disappear.

Tools for Developers (38:52)

Homebrew (38:53)

Homebrew is for the console. It is only the open source small utilities that (especially the packages managers in the Linux distribution) comes with. They also have one for binary distribution: brew cask (used local).

Tools for developers (39:40)

When you install Xcode, you get all the command line utilities there as well. otool -L, which can display which shared libraries are needed. You can also use libtool to create libraries, lipo to fat/universal binaries.

git (40:27)

git tells me which branch I am in all the time. For example, I was committing to the wrong branch, I decided to do something about it (see video for full command).

Xcode (41:34)

You can build Xcode projects the command-line. Very useful if you have built system, where you have to do builds automatically. In bash, or in Xcode, you can also add into your built faces any bash script you want. When it runs, it will also run bash scripts within Xcode. Then you can implement the features that are missing in Xcode yourself using bash, or any other language (ruby and python).

xcpretty and xctool (42:08)

If you try to run Xcode built on the command line, you will see that the output is horrible and very hard to read. xcpretty makes it a bit easier to read. You pass the output from Xcode built into xcpretty and it formats it - you can see it is running (much easier to see and read). xctool is a similar one. You can use that as well. It depends on which one you like. Yes, that is about it.

Further Information (43:19)

• “Classical Shell Scripting” by Arnolds, to learn anything about shell.
• The sed FAQ
• Advance Bash-Scripting guide. Everything about bash, and how to write scripts.