diff options
Diffstat (limited to 'software')
| -rw-r--r-- | software/bash.page | 2 | ||||
| -rw-r--r-- | software/postgres.page | 17 | ||||
| -rw-r--r-- | software/python.page | 21 | ||||
| -rw-r--r-- | software/rsyslog.page | 20 | ||||
| -rw-r--r-- | software/rust.page | 117 |
5 files changed, 174 insertions, 3 deletions
diff --git a/software/bash.page b/software/bash.page index 0d5325b..5d82fdc 100644 --- a/software/bash.page +++ b/software/bash.page @@ -37,6 +37,8 @@ Note that `join`, `grep`, and others sometimes exit non-zero return codes on purpose (eg, pipe input closed or found no matches, as expected), which makes life difficult. Sometimes `|| true` is enough to get around this. +More on this: <http://redsymbol.net/articles/unofficial-bash-strict-mode/> + ## General Style Google has a style guide: https://google.github.io/styleguide/shell.xml diff --git a/software/postgres.page b/software/postgres.page index bdb01d8..7d05074 100644 --- a/software/postgres.page +++ b/software/postgres.page @@ -15,3 +15,20 @@ Basic status from the psql command line: \l+ (list databases) \dt+ (describe table) + +## Temporary Databases in Tests + +Try `pg_tmp`, though it may not work with high concurrency (due to internal +postgres locking), and might need special configuration in CI environments. + +Debian also ships with `pg_virtualenv` for temporary databases. + +--------- + +More refs/links: + +- https://stackoverflow.com/questions/9604723/alternate-output-format-for-psql +- https://brandur.org/postgres-connections +- https://www.citusdata.com/blog/2017/07/16/customizing-my-postgres-shell-using-psqlrc/ +- `bind "^R" em-inc-search-prev` + diff --git a/software/python.page b/software/python.page index 0fd2531..0fc3c8d 100644 --- a/software/python.page +++ b/software/python.page @@ -205,3 +205,24 @@ and create a `pytest.ini` like: norecursedirs = .svn _build tmp* Need to mock? <https://blog.fugue.co/2016-02-11-python-mocking-101.html> + +Debugging Memory Usage +------------------------ + +Most helpful tools I found were `psutil` and `pympler` (both need to be +installed). + + import os, psutil + process = psutil.Process(os.getpid()) + print(process.memory_info().rss) + # ... do some stuff ... + print(process.memory_info().rss) + +and + + from pympler import tracker + tr = tracker.SummaryTracker() + tr.print_diff() + + # ... do some stuff ... + tr.print_diff() diff --git a/software/rsyslog.page b/software/rsyslog.page new file mode 100644 index 0000000..b55c127 --- /dev/null +++ b/software/rsyslog.page @@ -0,0 +1,20 @@ + +Put these in files like `/etc/rsyslog.d/40-grobid-skip.conf`. + +Filter out lines from a given program: + + :programname, isequal, "GROBID" ~ + +Or matching a pattern: + + :msg, startswith, "inserting line to HBase: sha1:" + :msg, contains, " INFO GET http" + +For list of operations ("isequal", "regex", "contains"): + + https://www.rsyslog.com/doc/v8-stable/configuration/filters.html + +For list of properties (":msg", ":programname"): + + https://www.rsyslog.com/doc/v8-stable/configuration/properties.html + diff --git a/software/rust.page b/software/rust.page index df99de4..dc900c8 100644 --- a/software/rust.page +++ b/software/rust.page @@ -3,9 +3,10 @@ Rust ## Resources -- [http://xion.io/post/code/rust-iter-patterns.html]() -- [https://deterministic.space/rust-cli-tips.html]() -- [https://manishearth.github.io/blog/2018/01/10/whats-tokio-and-async-io-all-about/]() +- <http://xion.io/post/code/rust-iter-patterns.html> +- <https://deterministic.space/rust-cli-tips.htm> +- <https://manishearth.github.io/blog/2018/01/10/whats-tokio-and-async-io-all-about/> +- <https://saghm.github.io/five-rust-things/> Optimization: use `RUSTFLAGS="-C target-cpu=native"` to take advantage of CPU special features. @@ -22,3 +23,113 @@ Run tests with stdout output: To run tests with logging enabled (eg, with `env_logger`), make sure you add `env_logger::init()` to the test function itself. + + +## map() and Result Ergonomics + +`.collect()` has some magical features! In addition to turning an iterator of +`Item` into `Vec<Item>`, it will turn an iterator of `Result<Item>` into +`Result<Vec<Item>>`. This makes it really useful for the end of functions. + +This is particularly useful for resolving some categories of "error handling in +map closures": you can use `?` in the map closure as long as you wrap the happy +path with `Ok()` and call collect on the outside. Eg: + + let list: Vec<Item> = junk + .iter() + .map(|thing| Ok(Item { + a: thing.a, + b: fixup(thing.widget)?, + })) + .collect::Result<Vec<Item>>()?; + +What about when `map` over an `Option`? Eg: + + let toy = Shiny { + a: 123, + b: component.map(|v| paint(v).expect("paint to succeed"), + }; + +Should use match in this case: + + let toy = Shiny { + a: 123, + b: match component { + None => None, + Some(v) => Some(paint(v)?), + }, + }; + +## 2020-05-17 Reading + +While working on fatcat-cli tool, checked the The Rust Programming Language +book to read about trait objects and the `dyn` keyword, which I had ignored +previously. + +They seem like they could be used in a few places in fatcat-server rust code. +We don't particularly care about per-function-call performance there, and most +entities are already allocated on the heap. + +Other small syntax and thing learned: + +Can copy a struct while only updating specific fields with ".." syntax. Might +use this in fatcat-cli for update mutation. + +This is the cleanest example of using ErrorKind that I have seen: + + let f = match f { + Ok(file) => file, + Err(error) => match error.kind() { + ErrorKind::NotFound => match File::create("hello.txt") { + Ok(fc) => fc, + Err(e) => panic!("Problem creating the file: {:?}", e), + }, + other_error => { + panic!("Problem opening the file: {:?}", other_error) + } + }, + }; + +I didn't realize that test code may get compiled into non-test binaries unless +annotated with `#[cfg(test)]`. You are supposed to create a sub-module within +each `src/` file with unittests, like: + + #[cfg(test)] + mod tests { + use super::*; + + #[test] + fn it_works() { + assert_eq!(2 + 2, 4); + } + } +This doesn't apply to `tests/` directory, which is for integration tests. + +The common pattern for binary crates (vs. library crates) is to have `main.rs` +and `lib.rs`, with any code that needs to be tested in `lib.rs` (aka, all the +actual logic). + +I think I knew `eprintln!()` (for stderr) vs. `println!()` (for stdout), but +good to remember. + +There is a description of how to avoid memory leaks with reference counting +using "weak" `Rc` references. Probably worth reading the [entire chapter on smart pointers](https://doc.rust-lang.org/book/ch15-06-reference-cycles.html#preventing-reference-cycles-turning-an-rct-into-a-weakt) +(including Box, Rc, RefCell) again. + +For the `Sized` trait, and `Sized` trait alone, can specify an ambiguous trait +constraint with `?` to indicate "may or may not be Sized", which doesn't really +mean anything but does explicitly allow generic functions over non-sized traits +like: + + fn my_generic_func<T: ?Sized>(t: &T) { + // --snip-- + } + +A trait can depend on another trait. For example, a PrettyPrint trait could +rely on Display (and impl functions could call functions from Display). This is +done on the trait definition line. Such a trait is called a "supertrait". + +Implementing Deref on a wrapper type allows transparent access to all the trait +methods on the interior object. + +Also, a new longer post on error handling: <https://nick.groenen.me/posts/rust-error-handling/> |