From 534a42f619b8a7e3226ad7e67601bbaef6bd2f31 Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Thu, 20 Sep 2018 21:35:02 -0700 Subject: add libsodium-dev as a dep --- rust/README.md | 1 + 1 file changed, 1 insertion(+) (limited to 'rust/README.md') diff --git a/rust/README.md b/rust/README.md index 2bf86d3c..ddde9b80 100644 --- a/rust/README.md +++ b/rust/README.md @@ -7,6 +7,7 @@ Rust implementation of fatcat API server (`fatcatd`). - diesel (`cargo install diesel_cli`) - postgres (9.6+; targetting 11.1 for production) - postgres libs (debian: `sudo apt install libsqlite3-dev libpq-dev`) +- libsodium library and development headers (debian: `libsodium-dev`) Create a new postgres superuser. A regular postgres user and an existing database should also work (with up/down migrations), but it's easier to just -- cgit v1.2.3 From b7da0669b1568bd17192bfbe86f8a248279a870a Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Mon, 31 Dec 2018 17:15:30 -0800 Subject: bit of auth docs --- notes/auth.md | 98 +++++++++++++++++++++++++++++++++++++++++++++++++ notes/auth_thoughts.txt | 98 ------------------------------------------------- rust/HACKING.md | 4 ++ rust/README.md | 3 ++ 4 files changed, 105 insertions(+), 98 deletions(-) create mode 100644 notes/auth.md delete mode 100644 notes/auth_thoughts.txt (limited to 'rust/README.md') diff --git a/notes/auth.md b/notes/auth.md new file mode 100644 index 00000000..c82b204e --- /dev/null +++ b/notes/auth.md @@ -0,0 +1,98 @@ + +For users: use openid connect (oauth2) to sign up and login to web app. From +web app, can create (and disable?) API tokens + +For impl: fatcat-web has private key to create tokens. tokens used both in +cookies and as API keys. tokens are macaroons (?). fatcatd only verifies +tokens. optionally, some redis or other fast shared store to verify that tokens +haven't been revoked. + +Could use portier with openid connect as an email-based option. Otherwise, +orcid, github, google. + +--------- + +Use macaroons! + +editor/user table has a "auth_epoch" timestamp; only macaroons generated +after this timestamp are valid. revocation is done by incrementing this +timestamp ("touch"). + +Rust CLI tool for managing users: +- create editor + +Special users/editor that can create editor accounts via API; eg, one for +fatcat-web. + +Associate one oauth2 id per domain per editor/user. + +Users come to fatcat-web and do oauth2 to login or create an account. All +oauth2 internal to fatcat-web. If successful, fatcat-web does an +(authenticated) lookup to API for that identifier. If found, requests a +new macaroon to use as a cookie for auth. All future requests pass this +cookie through as bearer auth. fatcat-web remains stateless! macaroon +contains username (for display); no lookup-per page. Need to logout/login for +this to update? + +Later, can do a "add additional account" feature. + +Backend: +- oauth2 account table, foreign key to editor table + => this is the only private table +- auth_epoch timestamp column on editor table +- lock editor by setting auth_epoch to deep future + +Deploy process: +- auto-create root (admin), import-bootstrap (admin,bot), and demo-user + editors, with fixed editor_id and "early" auth_epoch, as part of SQL. save + tokens in env files, on laptop and QA instance. +- on live QA instance, revoke all keys when live (?) + +TODO: privacy policy + +fatcat API doesn't *require* auth, but if auth is provided, it will check +macaroon, and validate against editor table's timestamp. + +support oauth2 against: +- orcid +- git.archive.org +- github +? google + +Macaroon details: +- worth looking at "bakery" projects (python and golang) for example of how to + actually implement macaroon authentication/authorization +- location is fatcat.wiki (and/or qa.fatcat.wiki, or test or localhost or test.fatcat.wiki?) +- identifier is a UUID in upper-case string format +- will need some on-disk key storage thing? + => how to generate new keys? which one should be used, most recent? + conception of revoking keys? simple JSON/TOML, or LMDB? +- call them "authentication tokens"? +- params/constraints + - editor_id: always, fcid format + - created: always, some date format (seconds/iso) + - expires: optional, same date format + +It's a huge simplification to have webface generate macaroons as well, using a +root key. webface doesn't need multiple keys because it only creates, doesn't +verify. + +Code structure: +- auth service/struct is generated at startup; reads environment and on-disk keys +- verify helper does the thing +- some sort of auth/edit context + +Roles? +- public: unauthenticated +- editor: any authenticated, active account +- bot +- admin + +Caveats: +- general model is that macaroon is omnipotent and passes all verification, + unless caveats are added. eg, adding verification checks doesn't constrain + auth, only the caveats constrain auth; verification check *allow* additional + auth. each caveat only needs to be allowed by one verifiation. +- can (and should?) add as many caveat checkers/constrants in code as possible + +http://evancordell.com/2015/09/27/macaroons-101-contextual-confinement.html diff --git a/notes/auth_thoughts.txt b/notes/auth_thoughts.txt deleted file mode 100644 index c82b204e..00000000 --- a/notes/auth_thoughts.txt +++ /dev/null @@ -1,98 +0,0 @@ - -For users: use openid connect (oauth2) to sign up and login to web app. From -web app, can create (and disable?) API tokens - -For impl: fatcat-web has private key to create tokens. tokens used both in -cookies and as API keys. tokens are macaroons (?). fatcatd only verifies -tokens. optionally, some redis or other fast shared store to verify that tokens -haven't been revoked. - -Could use portier with openid connect as an email-based option. Otherwise, -orcid, github, google. - ---------- - -Use macaroons! - -editor/user table has a "auth_epoch" timestamp; only macaroons generated -after this timestamp are valid. revocation is done by incrementing this -timestamp ("touch"). - -Rust CLI tool for managing users: -- create editor - -Special users/editor that can create editor accounts via API; eg, one for -fatcat-web. - -Associate one oauth2 id per domain per editor/user. - -Users come to fatcat-web and do oauth2 to login or create an account. All -oauth2 internal to fatcat-web. If successful, fatcat-web does an -(authenticated) lookup to API for that identifier. If found, requests a -new macaroon to use as a cookie for auth. All future requests pass this -cookie through as bearer auth. fatcat-web remains stateless! macaroon -contains username (for display); no lookup-per page. Need to logout/login for -this to update? - -Later, can do a "add additional account" feature. - -Backend: -- oauth2 account table, foreign key to editor table - => this is the only private table -- auth_epoch timestamp column on editor table -- lock editor by setting auth_epoch to deep future - -Deploy process: -- auto-create root (admin), import-bootstrap (admin,bot), and demo-user - editors, with fixed editor_id and "early" auth_epoch, as part of SQL. save - tokens in env files, on laptop and QA instance. -- on live QA instance, revoke all keys when live (?) - -TODO: privacy policy - -fatcat API doesn't *require* auth, but if auth is provided, it will check -macaroon, and validate against editor table's timestamp. - -support oauth2 against: -- orcid -- git.archive.org -- github -? google - -Macaroon details: -- worth looking at "bakery" projects (python and golang) for example of how to - actually implement macaroon authentication/authorization -- location is fatcat.wiki (and/or qa.fatcat.wiki, or test or localhost or test.fatcat.wiki?) -- identifier is a UUID in upper-case string format -- will need some on-disk key storage thing? - => how to generate new keys? which one should be used, most recent? - conception of revoking keys? simple JSON/TOML, or LMDB? -- call them "authentication tokens"? -- params/constraints - - editor_id: always, fcid format - - created: always, some date format (seconds/iso) - - expires: optional, same date format - -It's a huge simplification to have webface generate macaroons as well, using a -root key. webface doesn't need multiple keys because it only creates, doesn't -verify. - -Code structure: -- auth service/struct is generated at startup; reads environment and on-disk keys -- verify helper does the thing -- some sort of auth/edit context - -Roles? -- public: unauthenticated -- editor: any authenticated, active account -- bot -- admin - -Caveats: -- general model is that macaroon is omnipotent and passes all verification, - unless caveats are added. eg, adding verification checks doesn't constrain - auth, only the caveats constrain auth; verification check *allow* additional - auth. each caveat only needs to be allowed by one verifiation. -- can (and should?) add as many caveat checkers/constrants in code as possible - -http://evancordell.com/2015/09/27/macaroons-101-contextual-confinement.html diff --git a/rust/HACKING.md b/rust/HACKING.md index 57642b2d..9d161b87 100644 --- a/rust/HACKING.md +++ b/rust/HACKING.md @@ -57,3 +57,7 @@ Debug SQL schema errors (if diesel commands fail): Creating entities via API: http --json post localhost:9411/v0/container name=asdf issn=1234-5678 + +## Authentication + +Uses macaroons. See `notes/auth.md` and maybe look in the guide. diff --git a/rust/README.md b/rust/README.md index ddde9b80..ecbfba2d 100644 --- a/rust/README.md +++ b/rust/README.md @@ -17,6 +17,9 @@ Create a `.env` file with configuration: DATABASE_URL=postgres://fatcat:tactaf@localhost/fatcat_rs TEST_DATABASE_URL=postgres://fatcat:tactaf@localhost/fatcat_rs_test + AUTH_LOCATION=dev.fatcat.wiki + AUTH_KEY_IDENT=2018-12-31-dev + AUTH_SECRET_KEY=VQe8kdn8laZ3MArKAzOeWWNUQgM6IjduG2jwKnSWehQ= Re-create database from scratch: -- cgit v1.2.3 From 67d56af9447a69700a5a2ad816efcace679145d2 Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Fri, 4 Jan 2019 13:19:45 -0800 Subject: document rust fatcatd config --- rust/.gitignore | 1 + rust/README.md | 38 +++++++++++++++++++++++++++++--------- rust/env.example | 6 ++++++ 3 files changed, 36 insertions(+), 9 deletions(-) create mode 100644 rust/env.example (limited to 'rust/README.md') diff --git a/rust/.gitignore b/rust/.gitignore index 18b70ae0..a2068a8a 100644 --- a/rust/.gitignore +++ b/rust/.gitignore @@ -1,3 +1,4 @@ +.env target/ !.cargo diff --git a/rust/README.md b/rust/README.md index ecbfba2d..a6bb34b6 100644 --- a/rust/README.md +++ b/rust/README.md @@ -3,25 +3,24 @@ Rust implementation of fatcat API server (`fatcatd`). ## Development +You need the following dependencies installed locally to build, run tests, and +do development work: + - rust stable, 1.29+ (eg, via "rustup", includes cargo tool) - diesel (`cargo install diesel_cli`) - postgres (9.6+; targetting 11.1 for production) - postgres libs (debian: `sudo apt install libsqlite3-dev libpq-dev`) - libsodium library and development headers (debian: `libsodium-dev`) +Copying commands out of `../.gitlab-ci.yml` file may be the fastest way to get +started. + Create a new postgres superuser. A regular postgres user and an existing database should also work (with up/down migrations), but it's easier to just blow the entire database away. -Create a `.env` file with configuration: - - DATABASE_URL=postgres://fatcat:tactaf@localhost/fatcat_rs - TEST_DATABASE_URL=postgres://fatcat:tactaf@localhost/fatcat_rs_test - AUTH_LOCATION=dev.fatcat.wiki - AUTH_KEY_IDENT=2018-12-31-dev - AUTH_SECRET_KEY=VQe8kdn8laZ3MArKAzOeWWNUQgM6IjduG2jwKnSWehQ= - -Re-create database from scratch: +Copy `env.example` to `.env`, update if needed, then re-create database from +scratch: diesel database reset @@ -34,3 +33,24 @@ Tests: cargo test -- --test-threads 1 See `HACKING` for some more advanced tips and commands. + +## Configuration + +All configuration goes through environment variables, the notable ones being: + +- `DATABASE_URL`: postgres connection details (username, password, host, and database) +- `TEST_DATABASE_URL`: used when running `cargo test` +- `AUTH_LOCATION`: the domain authentication tokens should be valid over +- `AUTH_KEY_IDENT`: a unique name for the primary auth signing key (used to + find the correct key after key rotation has occured) +- `AUTH_SECRET_KEY`: base64-encoded secret key used to both sign and verify + authentication tokens (symmetric encryption) +- `AUTH_ALT_KEYS`: additional ident/key pairs that can be used to verify tokens + (to enable key rotation). Syntax is like `:,:key2,...`. + +To setup authentication with a new secret authentication key, run: + + cargo run --bin fatcat-auth create-key + +then copy the last line as `AUTH_SECRET_KEY` in `.env`, and update +`AUTH_KEY_IDENT` with a unique name for this new key (eg, including the date). diff --git a/rust/env.example b/rust/env.example new file mode 100644 index 00000000..0ecf58a5 --- /dev/null +++ b/rust/env.example @@ -0,0 +1,6 @@ +DATABASE_URL="postgres://fatcat:tactaf@localhost/fatcat" +TEST_DATABASE_URL="postgres://fatcat:tactaf@localhost/fatcat_test +AUTH_LOCATION="dev.fatcat.wiki" +AUTH_KEY_IDENT="20190101-dev-dummy-key" +AUTH_SECRET_KEY="5555555555555555555555555555555555555555555=" +AUTH_ALT_KEYS="20181220-dev:6666666666666666666666666666666666666666666=,20181210-dev:7777777777777777777777777777777777777777777=" -- cgit v1.2.3 From 3aee89745355b0ced5223c601a03f77e46d997dc Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Fri, 4 Jan 2019 13:26:18 -0800 Subject: update rust README --- rust/README.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) (limited to 'rust/README.md') diff --git a/rust/README.md b/rust/README.md index a6bb34b6..decfc74d 100644 --- a/rust/README.md +++ b/rust/README.md @@ -1,5 +1,17 @@ -Rust implementation of fatcat API server (`fatcatd`). +Rust implementation of fatcat API server. Commands include: + +- `fatcatd`: the API server itself +- `fatcat-auth`: privileged command to manage authentication keys, tokens, and + accounts. Useful to generate admin accounts, new signing keys, etc. +- `fatcat-export`: high-speed JSON export tool, which talks directly to the + database (instead of going through the API). See `README.export.md`. + +The `fatcat-api-spec` crate is generated from the openapi/swagger spec and +contains Rust models, response types, and endpoint definitions (but not +implementations). + +The SQL database schema (and migrations) are under `./migrations/`. ## Development @@ -32,6 +44,9 @@ Tests: cargo test -- --test-threads 1 +Note that most "integration" level tests are written in python and run by +`pytest`; see `../python/README.md`. + See `HACKING` for some more advanced tips and commands. ## Configuration -- cgit v1.2.3