fatcat/python/tests/files, branch v0.3.2 [no description] https://git.bnewbold.net/fatcat/atom?h=v0.3.2 2020-03-20T20:00:52+00:00 pubmed: handle multiple ReferenceList 2020-03-20T20:00:52+00:00 Bryan Newbold bnewbold@robocracy.org 2020-03-20T20:00:50+00:00 urn:sha1:a6f74183dd1cf1eaa44f7edeb98dbc5dc737dabb This resolves a situation noticed in prod where we were only importing/updating a single reference per article. Includes a regression test. Merge branch 'martin-kafka-bs4-import' into 'master' 2020-03-10T15:33:17+00:00 Martin Czygan martin@archive.org 2020-03-10T15:33:17+00:00 urn:sha1:336630e1d445fb9d233447f9af4bac94473a12bf pubmed and arxiv harvest preparations See merge request webgroup/fatcat!28 Merge branch 'bnewbold-elastic-v03b' 2020-02-27T06:05:43+00:00 Bryan Newbold bnewbold@robocracy.org 2020-02-27T06:05:43+00:00 urn:sha1:ae50ee2274031ddc178fa4a10b59280e8440a24c more pubmed adjustments 2020-02-22T16:44:38+00:00 Martin Czygan martin.czygan@gmail.com 2020-02-19T01:28:11+00:00 urn:sha1:376053a479a8d683fc5e099d0b0b3cb76c026d16 * regenerate map in continuous mode * add tests shadow import: more filtering of file_meta fields 2020-02-14T06:24:20+00:00 Bryan Newbold bnewbold@robocracy.org 2020-01-30T20:15:09+00:00 urn:sha1:87029cb13d244381f915fe66e40760477edb5675 basic shadow importer 2020-02-14T06:24:20+00:00 Bryan Newbold bnewbold@robocracy.org 2019-12-24T01:59:10+00:00 urn:sha1:e59d1b617d4abd5f002d9e59b6bbaebc9ff30993 datacite: add exception for https://www.micropublication.org/ 2020-01-31T00:44:46+00:00 Martin Czygan martin.czygan@gmail.com 2020-01-31T00:44:46+00:00 urn:sha1:a42206d2603e28f1311ac3873dc168c78eabffee datacite: improve date handling and minor tweak 2020-01-30T12:36:01+00:00 Martin Czygan martin.czygan@gmail.com 2020-01-30T12:36:01+00:00 urn:sha1:7dec2d1560ebf5ca6d0d337eb246fe345f6ec0bb Records from https://www.micropublication.org/ did not have a date in FC, although raw data contained date strings - they were not using the finer-grained "attributes.date" but "attributes.published" and/or "attributes.publicationYear". Support for those fields has been added, including a test case. During this test (#30) a processing gap for names became clear (author may have "given_name" and "surname", but no "name"). This bug has been fixed, too. fix some transform bugs, add some tests 2020-01-30T05:59:05+00:00 Bryan Newbold bnewbold@robocracy.org 2020-01-30T05:52:33+00:00 urn:sha1:5d458a3df7e58e6551d8ec72979e376c62fdd2f7 do not normalize "en dash" in DOI 2020-01-17T13:03:00+00:00 Martin Czygan martin.czygan@gmail.com 2020-01-17T13:03:00+00:00 urn:sha1:53756811572bab0679cb8cee1b9de95e7b29b96a Technically, [...] DOI names may incorporate any printable characters from the Universal Character Set (UCS-2), of ISO/IEC 10646, which is the character set defined by Unicode (https://www.doi.org/doi_handbook/2_Numbering.html#2.5.1). For mostly QA reasons, we currently treat a DOI with an "en dash" as invalid.