diff options
Diffstat (limited to 'software')
| -rw-r--r-- | software/bash.page | 53 | ||||
| -rw-r--r-- | software/command_line_primer.page | 117 | ||||
| -rw-r--r-- | software/unix-tricks.page | 5 |
3 files changed, 151 insertions, 24 deletions
diff --git a/software/bash.page b/software/bash.page index 0a3f73f..088d456 100644 --- a/software/bash.page +++ b/software/bash.page @@ -1,43 +1,48 @@ --- -format: rst +format: md toc: no ... -================== BASH ================== -Job Control - The syntax for "job number" or "JOBSPEC" (when using ``kill`` or similar) is ``%4`` or ``%5``. -Startup -------- -``bash`` by default takes a very long time to initialize because the +# Tricks and Specifics + +## Job Control + +The syntax for "job number" or "JOBSPEC" (when using `kill` or similar) is +`%4` or `%5`. + + +## Startup + +`bash` by default takes a very long time to initialize because the auto-completion scripts are loaded multiple times; disable this in -``~/.bashrc``? +`~/.bashrc`? + + +## Piping -Piping ---------- -You can pipe both ``stdout`` and ``stderr`` together either to a file or two another command:: +You can pipe both `stdout` and `stderr` together either to a file or two another command:: grep --asdf >& /some/file grep --asdf |& less -Network Access ----------------- -You can directly access network sockets as if they were files from bash using -the virtual devices ``/dev/tcp/HOSTNAME/PORT`` and ``/dev/udp/HOSTNAME/PORT``. -printf ------- -The ``printf`` command is much more powerful than "echo". +## printf + +The `printf` command is much more powerful than `echo`. + -Prelude -------- +## Prelude -`set`: +I frequently add a one-line version of the following to shell scripts: - -e fail on error - -u fail if variable not set in substitution - -o pipefail fail if part of a '|' command fails + set -e # fail on error + set -u # fail if variable not set in substitution + set -o pipefail # fail if part of a '|' command fails +Note that `join`, `grep`, and others sometimes exit non-zero return codes on +purpose (eg, pipe input closed or found no matches, as expected), which makes +life difficult. Sometimes `|| true` is enough to get around this. diff --git a/software/command_line_primer.page b/software/command_line_primer.page new file mode 100644 index 0000000..72aafb6 --- /dev/null +++ b/software/command_line_primer.page @@ -0,0 +1,117 @@ +--- +format: md +title: UNIX Shell/Command-Line Primer +... + + +I'll start with un-intuitive things that took me the longest to learn. It +can be hard to figure out what are command line programs, and what +commands are built-in to bash... sometimes there is both a program version +and a shell version! Eg, `test`. It's very UNIX-y to have even the most +basic things be programs, like `ls`. Some things must be built-in, like +`fg`/`bg` (foreground and background jobs), `for`, etc. + +You get help with programs using `--help` or `man`. You get help with bash +built-ins using `help` (eg, `help for`). I'm pretty sure built-ins +override programs if they have the same name. You can check if a program +exists with `whereis`. `help` on it's own gives a listing of commands. +Some examples of commands that are built-in and programs: + + echo + true/false + [ (sic!) + kill + +There are basically three categories of UNIX shell: + +- simple standards-compliant (POSIX) shell: /bin/sh +- bash-compatible: /usr/bin/bash, zsh +- non-bash compatible: ksh, csh, fish, xonsh, etc + +For things to run on any UNIX machine (including macOS), stick to POSIX +shell... but in reality bash is almost always installed (is it default on +macOS? I don't know. It isn't on FreeBSD). The bash/zsh/csh thing is like +emacs/vi. You can see which you are running with `echo $0`. + +To make a script on UNIX, you make the file executable (`chmod +x +thefile.sh`), and add a "shebang" line (the `#!` is called "shebang"): + + #!/usr/bin/bash + +You can put any program on shebang line, and the body of the file will be +passed to that program: + + #!/usr/bin/python3 + +or + + #!/usr/bin/parallel + +For portability, and to avoid issues like "python3 is installed under +/opt, not /usr/bin", you can do this: + + #!/usr/bin/env python3 + +The "env" will look up the program by name on the $PATH. + +To do a for loop in one line you do: + + for l in `ls /somedir`; do echo $l; done + +In a script, for better legibility: + + for l in `ls /somedir`; do + echo $l + done + +If statements: + + if [ -e /some/file ]; do echo 'file exists'; fi + + if [ -e /some/file ]; do + echo 'file exists' + fi + +You'll see those brackets (single [ or double [[), and it took me forever +to learn about them... `help [` and `help if` are not helpful, you need +`help test`. + +### Specific Commands + +There are a bajillion command line commands. Here are some helpful +categories... + +For doing csv/tsv/column data munging: + + cut - select columns + grep/rg - select rows (by pattern), use '-v' to invert + join - what it says + sort - note the -n flag for numerical. lexical/case etc are a mess + uniq + sed - search/replace using regex + rg - more complex search/replace using the -o or -e options + awk - simple string substitutions, re-order columns with 'print' + tr - character-level search/replace or filtering (eg, lowercase) + xsv - a whole lot! + +Here's an example of the above; I run things like this all day: + + cut -f1 ~/.bash_history -d' ' | sort | uniq -c | sort -n + +For doing batch operations: + + xargs - superceded by parallel + parallel - like 'for' but in parallel. can also work on piped stdin + I basically never use 'for' loops after learning this command + find - helpful for finding files, but can also run a command for each + file found + +Other every-day tools: + + jq - the JSON swiss army knife. A whole scripting language + http (httpie) - better than curl/wget for debugging (though maybe not + for big downloads) + xsv - can do conversions between csv types + rg/ripgrep - faster + better than grep + pv - show progress + screen/tmux - a whole world of it's own diff --git a/software/unix-tricks.page b/software/unix-tricks.page index 26d6877..d6c363e 100644 --- a/software/unix-tricks.page +++ b/software/unix-tricks.page @@ -89,3 +89,8 @@ Using sed: sed -i 's/OldString/NewString/g' *.txt +File-based Network Access +--------------------------------------------- + +You can directly access network sockets as if they were files from bash using +the virtual devices ``/dev/tcp/HOSTNAME/PORT`` and ``/dev/udp/HOSTNAME/PORT``. |