fatcat/extra/elasticsearch, branch v0.3.3 [no description] https://git.bnewbold.net/fatcat/atom?h=v0.3.3 2020-07-01T23:34:13+00:00 commit example of an elasticsearch SQL query 2020-07-01T23:34:13+00:00 Bryan Newbold bnewbold@robocracy.org 2020-07-01T23:34:13+00:00 urn:sha1:274da5d3994e9f1a4ddabf2d3ddba06c5db1aa73 ES schema: add best_url to file schema 2020-06-04T21:12:30+00:00 Bryan Newbold bnewbold@robocracy.org 2020-06-04T21:01:34+00:00 urn:sha1:a42d5f0d00e76bf8474647fae4e1d9d61693a7d9 This will increase index size (URLs are often long in our corpus, and we have many file entities), but seems worth it. Initially added `ia_url` as a second field, guaranteed to always be an *.archive.org URL, but `best_url` defaults to that anyways so didn't seem worthwhile. ES README: really need to limit to 1k esbulk batches 2020-02-27T07:11:11+00:00 Bryan Newbold bnewbold@robocracy.org 2020-02-27T07:11:11+00:00 urn:sha1:b4ab8501636b2891976ff867a064e02a478de065 update ES transform README 2020-02-26T20:27:30+00:00 Bryan Newbold bnewbold@robocracy.org 2020-02-26T20:27:28+00:00 urn:sha1:0ab3f66664fd4cc63cf9040e351d725c6a5c22b9 - smaller batch sizes to prevent esbulk errors - file transform/index ES container last tweaks 2020-02-26T19:29:30+00:00 Bryan Newbold bnewbold@robocracy.org 2020-02-26T19:28:05+00:00 urn:sha1:4e6bc246d01183f4c7ffad7d0d474e683f04c07f ES release: last minor tweaks 2020-02-26T19:22:30+00:00 Bryan Newbold bnewbold@robocracy.org 2020-02-26T19:22:30+00:00 urn:sha1:0450f22006c9b991cdc4695458fc3b3e3e97bfbb release schema: do doc_value on DOIs 2020-02-13T22:23:01+00:00 Bryan Newbold bnewbold@robocracy.org 2020-02-13T22:22:59+00:00 urn:sha1:2f8788152ff740d049d11e2e263cac978d526e2a Because DOIs are pseudo-structured (prefix, and often structure within the publisher-controlled area), I suspect we will in fact be wanting to do analytics over these strings. ES release: actually do want doc_values for work_id 2020-02-05T23:42:45+00:00 Bryan Newbold bnewbold@robocracy.org 2020-02-05T23:42:43+00:00 urn:sha1:3655bbe6c539fdeccfbfaa19b6fc93a4859e0ca7 Eg, for fast "unique count" fix axiv/arxiv typo in release schema 2020-02-04T23:10:26+00:00 Bryan Newbold bnewbold@robocracy.org 2020-02-04T23:10:26+00:00 urn:sha1:8007cdfc4e06753a9bbba56d1fa7f9686775e5e8 ES release schema: fix typo 2020-01-31T21:33:38+00:00 Bryan Newbold bnewbold@robocracy.org 2020-01-31T21:33:38+00:00 urn:sha1:fbd79c7315cad4789eb0e92c136c59da8f38c4f3