bash tutorial and notes

3 files changed, 151 insertions, 24 deletions

diff --git a/software/bash.page b/software/bash.page
index 0a3f73f..088d456 100644
--- a/software/bash.page
+++ b/software/bash.page
@@ -1,43 +1,48 @@
 ---
-format: rst
+format: md
 toc: no
 ...
 
-==================
 BASH
 ==================
 
-Job Control
-    The syntax for "job number" or "JOBSPEC" (when using ``kill`` or similar) is ``%4`` or ``%5``.
 
-Startup
--------
-``bash`` by default takes a very long time to initialize because the
+# Tricks and Specifics
+
+## Job Control
+
+The syntax for "job number" or "JOBSPEC" (when using `kill` or similar) is
+`%4` or `%5`.
+
+
+## Startup
+
+`bash` by default takes a very long time to initialize because the
 auto-completion scripts are loaded multiple times; disable this in
-``~/.bashrc``?
+`~/.bashrc`?
+
+
+## Piping
 
-Piping
----------
-You can pipe both ``stdout`` and ``stderr`` together either to a file or two another command::
+You can pipe both `stdout` and `stderr` together either to a file or two another command::
 
     grep --asdf >& /some/file
     grep --asdf |& less
 
-Network Access
-----------------
-You can directly access network sockets as if they were files from bash using
-the virtual devices ``/dev/tcp/HOSTNAME/PORT`` and ``/dev/udp/HOSTNAME/PORT``.
 
-printf
-------
-The ``printf`` command is much more powerful than "echo".
+## printf
+
+The `printf` command is much more powerful than `echo`.
+
 
-Prelude
--------
+## Prelude
 
-`set`:
+I frequently add a one-line version of the following to shell scripts:
 
-    -e          fail on error
-    -u          fail if variable not set in substitution
-    -o pipefail fail if part of a '|' command fails
+    set -e              # fail on error
+    set -u              # fail if variable not set in substitution
+    set -o pipefail     # fail if part of a '|' command fails
 
+Note that `join`, `grep`, and others sometimes exit non-zero return codes on
+purpose (eg, pipe input closed or found no matches, as expected), which makes
+life difficult. Sometimes `|| true` is enough to get around this.
diff --git a/software/command_line_primer.page b/software/command_line_primer.page
new file mode 100644
index 0000000..72aafb6
--- /dev/null
+++ b/software/command_line_primer.page
@@ -0,0 +1,117 @@
+---
+format: md
+title: UNIX Shell/Command-Line Primer
+...
+
+
+I'll start with un-intuitive things that took me the longest to learn. It 
+can be hard to figure out what are command line programs, and what 
+commands are built-in to bash... sometimes there is both a program version 
+and a shell version! Eg, `test`. It's very UNIX-y to have even the most 
+basic things be programs, like `ls`. Some things must be built-in, like 
+`fg`/`bg` (foreground and background jobs), `for`, etc.
+
+You get help with programs using `--help` or `man`. You get help with bash 
+built-ins using `help` (eg, `help for`). I'm pretty sure built-ins 
+override programs if they have the same name. You can check if a program 
+exists with `whereis`. `help` on it's own gives a listing of commands. 
+Some examples of commands that are built-in and programs:
+
+    echo
+    true/false
+    [      (sic!)
+    kill
+
+There are basically three categories of UNIX shell:
+
+- simple standards-compliant (POSIX) shell: /bin/sh
+- bash-compatible: /usr/bin/bash, zsh
+- non-bash compatible: ksh, csh, fish, xonsh, etc
+
+For things to run on any UNIX machine (including macOS), stick to POSIX 
+shell... but in reality bash is almost always installed (is it default on 
+macOS? I don't know. It isn't on FreeBSD). The bash/zsh/csh thing is like 
+emacs/vi. You can see which you are running with `echo $0`.
+
+To make a script on UNIX, you make the file executable (`chmod +x 
+thefile.sh`), and add a "shebang" line (the `#!` is called "shebang"):
+
+    #!/usr/bin/bash
+
+You can put any program on shebang line, and the body of the file will be 
+passed to that program:
+
+    #!/usr/bin/python3
+
+or
+
+    #!/usr/bin/parallel
+
+For portability, and to avoid issues like "python3 is installed under 
+/opt, not /usr/bin", you can do this:
+
+    #!/usr/bin/env python3
+
+The "env" will look up the program by name on the $PATH.
+
+To do a for loop in one line you do:
+
+    for l in `ls /somedir`; do echo $l; done
+
+In a script, for better legibility:
+
+    for l in `ls /somedir`; do
+        echo $l
+    done
+
+If statements:
+
+    if [ -e /some/file ]; do echo 'file exists'; fi
+
+    if [ -e /some/file ]; do
+        echo 'file exists'
+    fi
+
+You'll see those brackets (single [ or double [[), and it took me forever 
+to learn about them... `help [` and `help if` are not helpful, you need 
+`help test`.
+
+### Specific Commands
+
+There are a bajillion command line commands. Here are some helpful 
+categories...
+
+For doing csv/tsv/column data munging:
+
+    cut - select columns
+    grep/rg - select rows (by pattern), use '-v' to invert
+    join - what it says
+    sort - note the -n flag for numerical. lexical/case etc are a mess
+    uniq
+    sed - search/replace using regex
+    rg - more complex search/replace using the -o or -e options
+    awk - simple string substitutions, re-order columns with 'print'
+    tr - character-level search/replace or filtering (eg, lowercase)
+    xsv - a whole lot!
+
+Here's an example of the above; I run things like this all day:
+
+    cut -f1 ~/.bash_history -d' ' | sort | uniq -c | sort -n
+
+For doing batch operations:
+
+    xargs - superceded by parallel
+    parallel - like 'for' but in parallel. can also work on piped stdin
+        I basically never use 'for' loops after learning this command
+    find - helpful for finding files, but can also run a command for each
+        file found
+
+Other every-day tools:
+
+    jq - the JSON swiss army knife. A whole scripting language
+    http (httpie) - better than curl/wget for debugging (though maybe not
+        for big downloads)
+    xsv - can do conversions between csv types
+    rg/ripgrep - faster + better than grep
+    pv - show progress
+    screen/tmux - a whole world of it's own
diff --git a/software/unix-tricks.page b/software/unix-tricks.page
index 26d6877..d6c363e 100644
--- a/software/unix-tricks.page
+++ b/software/unix-tricks.page
@@ -89,3 +89,8 @@ Using sed:
 
     sed -i 's/OldString/NewString/g' *.txt
 
+File-based Network Access
+---------------------------------------------
+
+You can directly access network sockets as if they were files from bash using
+the virtual devices ``/dev/tcp/HOSTNAME/PORT`` and ``/dev/udp/HOSTNAME/PORT``.