summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorbnewbold <bnewbold@robocracy.org>2017-09-28 22:45:53 -0700
committerbnewbold <bnewbold@robocracy.org>2017-09-28 22:45:53 -0700
commita2f4b6ff17a51c190ea4f35bdaedc44a7f50c35e (patch)
tree88be4850c3f52ef8d8ccd715ec2336eb6865fed7
parente6edc12f06231becdaa47b6f10884d0ebf62e587 (diff)
downloadknowledge-a2f4b6ff17a51c190ea4f35bdaedc44a7f50c35e.tar.gz
knowledge-a2f4b6ff17a51c190ea4f35bdaedc44a7f50c35e.zip
bash tutorial and notes
-rw-r--r--software/bash.page53
-rw-r--r--software/command_line_primer.page117
-rw-r--r--software/unix-tricks.page5
3 files changed, 151 insertions, 24 deletions
diff --git a/software/bash.page b/software/bash.page
index 0a3f73f..088d456 100644
--- a/software/bash.page
+++ b/software/bash.page
@@ -1,43 +1,48 @@
---
-format: rst
+format: md
toc: no
...
-==================
BASH
==================
-Job Control
- The syntax for "job number" or "JOBSPEC" (when using ``kill`` or similar) is ``%4`` or ``%5``.
-Startup
--------
-``bash`` by default takes a very long time to initialize because the
+# Tricks and Specifics
+
+## Job Control
+
+The syntax for "job number" or "JOBSPEC" (when using `kill` or similar) is
+`%4` or `%5`.
+
+
+## Startup
+
+`bash` by default takes a very long time to initialize because the
auto-completion scripts are loaded multiple times; disable this in
-``~/.bashrc``?
+`~/.bashrc`?
+
+
+## Piping
-Piping
----------
-You can pipe both ``stdout`` and ``stderr`` together either to a file or two another command::
+You can pipe both `stdout` and `stderr` together either to a file or two another command::
grep --asdf >& /some/file
grep --asdf |& less
-Network Access
-----------------
-You can directly access network sockets as if they were files from bash using
-the virtual devices ``/dev/tcp/HOSTNAME/PORT`` and ``/dev/udp/HOSTNAME/PORT``.
-printf
-------
-The ``printf`` command is much more powerful than "echo".
+## printf
+
+The `printf` command is much more powerful than `echo`.
+
-Prelude
--------
+## Prelude
-`set`:
+I frequently add a one-line version of the following to shell scripts:
- -e fail on error
- -u fail if variable not set in substitution
- -o pipefail fail if part of a '|' command fails
+ set -e # fail on error
+ set -u # fail if variable not set in substitution
+ set -o pipefail # fail if part of a '|' command fails
+Note that `join`, `grep`, and others sometimes exit non-zero return codes on
+purpose (eg, pipe input closed or found no matches, as expected), which makes
+life difficult. Sometimes `|| true` is enough to get around this.
diff --git a/software/command_line_primer.page b/software/command_line_primer.page
new file mode 100644
index 0000000..72aafb6
--- /dev/null
+++ b/software/command_line_primer.page
@@ -0,0 +1,117 @@
+---
+format: md
+title: UNIX Shell/Command-Line Primer
+...
+
+
+I'll start with un-intuitive things that took me the longest to learn. It
+can be hard to figure out what are command line programs, and what
+commands are built-in to bash... sometimes there is both a program version
+and a shell version! Eg, `test`. It's very UNIX-y to have even the most
+basic things be programs, like `ls`. Some things must be built-in, like
+`fg`/`bg` (foreground and background jobs), `for`, etc.
+
+You get help with programs using `--help` or `man`. You get help with bash
+built-ins using `help` (eg, `help for`). I'm pretty sure built-ins
+override programs if they have the same name. You can check if a program
+exists with `whereis`. `help` on it's own gives a listing of commands.
+Some examples of commands that are built-in and programs:
+
+ echo
+ true/false
+ [ (sic!)
+ kill
+
+There are basically three categories of UNIX shell:
+
+- simple standards-compliant (POSIX) shell: /bin/sh
+- bash-compatible: /usr/bin/bash, zsh
+- non-bash compatible: ksh, csh, fish, xonsh, etc
+
+For things to run on any UNIX machine (including macOS), stick to POSIX
+shell... but in reality bash is almost always installed (is it default on
+macOS? I don't know. It isn't on FreeBSD). The bash/zsh/csh thing is like
+emacs/vi. You can see which you are running with `echo $0`.
+
+To make a script on UNIX, you make the file executable (`chmod +x
+thefile.sh`), and add a "shebang" line (the `#!` is called "shebang"):
+
+ #!/usr/bin/bash
+
+You can put any program on shebang line, and the body of the file will be
+passed to that program:
+
+ #!/usr/bin/python3
+
+or
+
+ #!/usr/bin/parallel
+
+For portability, and to avoid issues like "python3 is installed under
+/opt, not /usr/bin", you can do this:
+
+ #!/usr/bin/env python3
+
+The "env" will look up the program by name on the $PATH.
+
+To do a for loop in one line you do:
+
+ for l in `ls /somedir`; do echo $l; done
+
+In a script, for better legibility:
+
+ for l in `ls /somedir`; do
+ echo $l
+ done
+
+If statements:
+
+ if [ -e /some/file ]; do echo 'file exists'; fi
+
+ if [ -e /some/file ]; do
+ echo 'file exists'
+ fi
+
+You'll see those brackets (single [ or double [[), and it took me forever
+to learn about them... `help [` and `help if` are not helpful, you need
+`help test`.
+
+### Specific Commands
+
+There are a bajillion command line commands. Here are some helpful
+categories...
+
+For doing csv/tsv/column data munging:
+
+ cut - select columns
+ grep/rg - select rows (by pattern), use '-v' to invert
+ join - what it says
+ sort - note the -n flag for numerical. lexical/case etc are a mess
+ uniq
+ sed - search/replace using regex
+ rg - more complex search/replace using the -o or -e options
+ awk - simple string substitutions, re-order columns with 'print'
+ tr - character-level search/replace or filtering (eg, lowercase)
+ xsv - a whole lot!
+
+Here's an example of the above; I run things like this all day:
+
+ cut -f1 ~/.bash_history -d' ' | sort | uniq -c | sort -n
+
+For doing batch operations:
+
+ xargs - superceded by parallel
+ parallel - like 'for' but in parallel. can also work on piped stdin
+ I basically never use 'for' loops after learning this command
+ find - helpful for finding files, but can also run a command for each
+ file found
+
+Other every-day tools:
+
+ jq - the JSON swiss army knife. A whole scripting language
+ http (httpie) - better than curl/wget for debugging (though maybe not
+ for big downloads)
+ xsv - can do conversions between csv types
+ rg/ripgrep - faster + better than grep
+ pv - show progress
+ screen/tmux - a whole world of it's own
diff --git a/software/unix-tricks.page b/software/unix-tricks.page
index 26d6877..d6c363e 100644
--- a/software/unix-tricks.page
+++ b/software/unix-tricks.page
@@ -89,3 +89,8 @@ Using sed:
sed -i 's/OldString/NewString/g' *.txt
+File-based Network Access
+---------------------------------------------
+
+You can directly access network sockets as if they were files from bash using
+the virtual devices ``/dev/tcp/HOSTNAME/PORT`` and ``/dev/udp/HOSTNAME/PORT``.