summaryrefslogtreecommitdiffstats
path: root/software/command_line_primer.page
blob: b3f0809ecc894fae29a26fe9603bbff44ac76bea (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
---
format: markdown
title: UNIX Shell/Command-Line Primer
...


I'll start with un-intuitive things that took me the longest to learn. It 
can be hard to figure out what are command line programs, and what 
commands are built-in to bash... sometimes there is both a program version 
and a shell version! Eg, `test`. It's very UNIX-y to have even the most 
basic things be programs, like `ls`. Some things must be built-in, like 
`fg`/`bg` (foreground and background jobs), `for`, etc.

You get help with programs using `--help` or `man`. You get help with bash 
built-ins using `help` (eg, `help for`). I'm pretty sure built-ins 
override programs if they have the same name. You can check if a program 
exists with `whereis`. `help` on it's own gives a listing of commands. 
Some examples of commands that are built-in and programs:

    echo
    true/false
    [      (sic!)
    kill

There are basically three categories of UNIX shell:

- simple standards-compliant (POSIX) shell: /bin/sh
- bash-compatible: /usr/bin/bash, zsh
- non-bash compatible: ksh, csh, fish, xonsh, etc

For things to run on any UNIX machine (including macOS), stick to POSIX 
shell... but in reality bash is almost always installed (is it default on 
macOS? I don't know. It isn't on FreeBSD). The bash/zsh/csh thing is like 
emacs/vi. You can see which you are running with `echo $0`.

To make a script on UNIX, you make the file executable (`chmod +x 
thefile.sh`), and add a "shebang" line (the `#!` is called "shebang"):

    #!/usr/bin/bash

You can put any program on shebang line, and the body of the file will be 
passed to that program:

    #!/usr/bin/python3

or

    #!/usr/bin/parallel

For portability, and to avoid issues like "python3 is installed under 
/opt, not /usr/bin", you can do this:

    #!/usr/bin/env python3

The "env" will look up the program by name on the $PATH.

To do a for loop in one line you do:

    for l in `ls /somedir`; do echo $l; done

In a script, for better legibility:

    for l in `ls /somedir`; do
        echo $l
    done

If statements:

    if [ -e /some/file ]; do echo 'file exists'; fi

    if [ -e /some/file ]; do
        echo 'file exists'
    fi

You'll see those brackets (single [ or double [[), and it took me forever 
to learn about them... `help [` and `help if` are not helpful, you need 
`help test`.

### Specific Commands

There are a bajillion command line commands. Here are some helpful 
categories...

For doing csv/tsv/column data munging:

    cut - select columns
    grep/rg - select rows (by pattern), use '-v' to invert
    join - what it says
    sort - note the -n flag for numerical. lexical/case etc are a mess
    uniq
    sed - search/replace using regex
    rg - more complex search/replace using the -o or -e options
    awk - simple string substitutions, re-order columns with 'print'
    tr - character-level search/replace or filtering (eg, lowercase)
    xsv - a whole lot!

Here's an example of the above; I run things like this all day:

    cut -f1 ~/.bash_history -d' ' | sort | uniq -c | sort -n

For doing batch operations:

    xargs - superceded by parallel
    parallel - like 'for' but in parallel. can also work on piped stdin
        I basically never use 'for' loops after learning this command
    find - helpful for finding files, but can also run a command for each
        file found

Other every-day tools:

    jq - the JSON swiss army knife. A whole scripting language
    http (httpie) - better than curl/wget for debugging (though maybe not
        for big downloads)
    xsv - can do conversions between csv types
    rg/ripgrep - faster + better than grep
    pv - show progress
    screen/tmux - a whole world of it's own