|author||bnewbold <email@example.com>||2017-09-28 22:45:53 -0700|
|committer||bnewbold <firstname.lastname@example.org>||2017-09-28 22:45:53 -0700|
bash tutorial and notes
Diffstat (limited to 'software')
3 files changed, 151 insertions, 24 deletions
diff --git a/software/bash.page b/software/bash.page
index 0a3f73f..088d456 100644
@@ -1,43 +1,48 @@
- The syntax for "job number" or "JOBSPEC" (when using ``kill`` or similar) is ``%4`` or ``%5``.
-``bash`` by default takes a very long time to initialize because the
+# Tricks and Specifics
+## Job Control
+The syntax for "job number" or "JOBSPEC" (when using `kill` or similar) is
+`%4` or `%5`.
+`bash` by default takes a very long time to initialize because the
auto-completion scripts are loaded multiple times; disable this in
-You can pipe both ``stdout`` and ``stderr`` together either to a file or two another command::
+You can pipe both `stdout` and `stderr` together either to a file or two another command::
grep --asdf >& /some/file
grep --asdf |& less
-You can directly access network sockets as if they were files from bash using
-the virtual devices ``/dev/tcp/HOSTNAME/PORT`` and ``/dev/udp/HOSTNAME/PORT``.
-The ``printf`` command is much more powerful than "echo".
+The `printf` command is much more powerful than `echo`.
+I frequently add a one-line version of the following to shell scripts:
- -e fail on error
- -u fail if variable not set in substitution
- -o pipefail fail if part of a '|' command fails
+ set -e # fail on error
+ set -u # fail if variable not set in substitution
+ set -o pipefail # fail if part of a '|' command fails
+Note that `join`, `grep`, and others sometimes exit non-zero return codes on
+purpose (eg, pipe input closed or found no matches, as expected), which makes
+life difficult. Sometimes `|| true` is enough to get around this.
diff --git a/software/command_line_primer.page b/software/command_line_primer.page
new file mode 100644
@@ -0,0 +1,117 @@
+title: UNIX Shell/Command-Line Primer
+I'll start with un-intuitive things that took me the longest to learn. It
+can be hard to figure out what are command line programs, and what
+commands are built-in to bash... sometimes there is both a program version
+and a shell version! Eg, `test`. It's very UNIX-y to have even the most
+basic things be programs, like `ls`. Some things must be built-in, like
+`fg`/`bg` (foreground and background jobs), `for`, etc.
+You get help with programs using `--help` or `man`. You get help with bash
+built-ins using `help` (eg, `help for`). I'm pretty sure built-ins
+override programs if they have the same name. You can check if a program
+exists with `whereis`. `help` on it's own gives a listing of commands.
+Some examples of commands that are built-in and programs:
+ [ (sic!)
+There are basically three categories of UNIX shell:
+- simple standards-compliant (POSIX) shell: /bin/sh
+- bash-compatible: /usr/bin/bash, zsh
+- non-bash compatible: ksh, csh, fish, xonsh, etc
+For things to run on any UNIX machine (including macOS), stick to POSIX
+shell... but in reality bash is almost always installed (is it default on
+macOS? I don't know. It isn't on FreeBSD). The bash/zsh/csh thing is like
+emacs/vi. You can see which you are running with `echo $0`.
+To make a script on UNIX, you make the file executable (`chmod +x
+thefile.sh`), and add a "shebang" line (the `#!` is called "shebang"):
+You can put any program on shebang line, and the body of the file will be
+passed to that program:
+For portability, and to avoid issues like "python3 is installed under
+/opt, not /usr/bin", you can do this:
+ #!/usr/bin/env python3
+The "env" will look up the program by name on the $PATH.
+To do a for loop in one line you do:
+ for l in `ls /somedir`; do echo $l; done
+In a script, for better legibility:
+ for l in `ls /somedir`; do
+ echo $l
+ if [ -e /some/file ]; do echo 'file exists'; fi
+ if [ -e /some/file ]; do
+ echo 'file exists'
+You'll see those brackets (single [ or double [[), and it took me forever
+to learn about them... `help [` and `help if` are not helpful, you need
+### Specific Commands
+There are a bajillion command line commands. Here are some helpful
+For doing csv/tsv/column data munging:
+ cut - select columns
+ grep/rg - select rows (by pattern), use '-v' to invert
+ join - what it says
+ sort - note the -n flag for numerical. lexical/case etc are a mess
+ sed - search/replace using regex
+ rg - more complex search/replace using the -o or -e options
+ awk - simple string substitutions, re-order columns with 'print'
+ tr - character-level search/replace or filtering (eg, lowercase)
+ xsv - a whole lot!
+Here's an example of the above; I run things like this all day:
+ cut -f1 ~/.bash_history -d' ' | sort | uniq -c | sort -n
+For doing batch operations:
+ xargs - superceded by parallel
+ parallel - like 'for' but in parallel. can also work on piped stdin
+ I basically never use 'for' loops after learning this command
+ find - helpful for finding files, but can also run a command for each
+ file found
+Other every-day tools:
+ jq - the JSON swiss army knife. A whole scripting language
+ http (httpie) - better than curl/wget for debugging (though maybe not
+ for big downloads)
+ xsv - can do conversions between csv types
+ rg/ripgrep - faster + better than grep
+ pv - show progress
+ screen/tmux - a whole world of it's own
diff --git a/software/unix-tricks.page b/software/unix-tricks.page
index 26d6877..d6c363e 100644
@@ -89,3 +89,8 @@ Using sed:
sed -i 's/OldString/NewString/g' *.txt
+File-based Network Access
+You can directly access network sockets as if they were files from bash using
+the virtual devices ``/dev/tcp/HOSTNAME/PORT`` and ``/dev/udp/HOSTNAME/PORT``.