Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | datacite: resolve formatting issues in tests | Martin Czygan | 2020-07-10 | 16 | -72/+28 |
|\ | |||||
| * | datacite: fix attribute error | Martin Czygan | 2020-07-07 | 1 | -1/+1 |
| | | | | | | | | refs: #44035 | ||||
| * | lint (flake8) tool python files | Bryan Newbold | 2020-07-01 | 16 | -72/+27 |
| | | |||||
* | | datacite: there should be no index gaps | Martin Czygan | 2020-07-10 | 1 | -2/+8 |
| | | |||||
* | | datacite: document contributor types | Martin Czygan | 2020-07-10 | 1 | -0/+25 |
| | | |||||
* | | wip: contrib, GH59 | Martin Czygan | 2020-07-10 | 1 | -16/+22 |
| | | |||||
* | | datacite: address duplicated contributor issue | Martin Czygan | 2020-07-07 | 1 | -0/+16 |
|/ | | | | | | | Use string comparison. * https://fatcat.wiki/release/spjysmrnsrgyzgq6ise5o44rlu/contribs * https://api.datacite.org/dois/10.25940/roper-31098406 | ||||
* | add new license mappings | Bryan Newbold | 2020-06-30 | 2 | -0/+27 |
| | |||||
* | datacite: improve license mapping | Martin Czygan | 2020-06-30 | 1 | -9/+15 |
| | | | | via "missed potential license", refs #58 | ||||
* | datacite: hard cast possible date value to string | Martin Czygan | 2020-06-29 | 1 | -1/+1 |
| | |||||
* | ingest importer: check that stage is consistent with release | Bryan Newbold | 2020-05-26 | 1 | -0/+5 |
| | |||||
* | Merge remote-tracking branch 'github/master' | Bryan Newbold | 2020-05-22 | 1 | -2/+2 |
|\ | |||||
| * | Indentity is not the same this as equality in Python | Christian Clauss | 2020-05-14 | 1 | -2/+2 |
| | | |||||
* | | importers: clarify handling of ApiException | Bryan Newbold | 2020-05-22 | 3 | -4/+10 |
| | | | | | | | | | | | | | | | | One of these (in ingest importer pipeline) is an actual bug, the others are just changing the syntax to be more explicit/conservative. The ingest importer bug seems to have resulted in some bad file match imports; scale of impact is unknown. | ||||
* | | ingest importer: don't use glutton matches | Bryan Newbold | 2020-05-22 | 1 | -3/+3 |
| | | | | | | | | | | | | | | Until reviewing I didn't realize we were even doing this currently. Hopefluly has not impacted too many imports, as almost all ingests use an external identifer, so only those with identifers not in fatcat for whatever reason. | ||||
* | | datacite: fix type error | Martin Czygan | 2020-04-22 | 1 | -1/+3 |
| | | | | | | | | | | | | | | Up to now, we expected the description to be a string or list. Add handling for int as well. First appeared: Apr 22 19:58:39. | ||||
* | | datacite: fix a raw name constraint violation | Martin Czygan | 2020-04-20 | 1 | -0/+8 |
|/ | | | | | | | It was possible that contribs got added which had no raw name. One example would be a name consisting of whitespace only. This fix adds a final check for this case. | ||||
* | consistently use raw string prefix for regex | Bryan Newbold | 2020-04-17 | 1 | -1/+1 |
| | |||||
* | pubmed: use untranslated title if translated not available | Bryan Newbold | 2020-04-01 | 1 | -0/+6 |
| | | | | | | | The primary motivation for this change is that fatcat *requires* a non-empty title for each release entity. Pubmed/Medline occasionally indexes just a VenacularTitle with no ArticleTitle for foreign publications, and currently those records don't end up in fatcat at all. | ||||
* | importers: replace newlines in get_text() strings | Bryan Newbold | 2020-04-01 | 4 | -23/+25 |
| | |||||
* | importers: more string/get_text swaps | Bryan Newbold | 2020-03-28 | 3 | -27/+27 |
| | | | | See previous pubmed commit for details. | ||||
* | pubmed: bunch of .get_text() instead of .string | Bryan Newbold | 2020-03-28 | 1 | -12/+12 |
| | | | | | | | | | | | Yikes! Apparently when a tag has child tags, .string will return None instead of all the strings. .get_text() returns all of it: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#get-text https://www.crummy.com/software/BeautifulSoup/bs4/doc/#string I've things like identifiers as .string, when we expect only a single string inside. | ||||
* | Merge pull request #53 from EdwardBetts/spelling | bnewbold | 2020-03-27 | 3 | -7/+7 |
|\ | | | | | Correct spelling mistakes | ||||
| * | Correct spelling mistakes | Edward Betts | 2020-03-27 | 3 | -7/+7 |
| | | |||||
* | | datacite: nameIdentifier corner case | Bryan Newbold | 2020-03-26 | 1 | -1/+2 |
| | | | | | | | | | | | | | | | | | | Works around a bug in production: AttributeError: 'NoneType' object has no attribute 'replace' (datacite.py:724) NOTE: there are no tests for this code path | ||||
* | | jalc: avoid meaningless pages values | Bryan Newbold | 2020-03-23 | 1 | -4/+8 |
| | | |||||
* | | datacite: add year sanity restrictions | bnewbold | 2020-03-23 | 1 | -0/+7 |
| | | | | | | | | | | | | | | | | | | Example of entities with bogus years: https://fatcat.wiki/release/search?q=doi_registrar%3Adatacite+year%3A%3E2100 We can do a clean-up task, but first need to prevent creation of new bad metadata. | ||||
* | | pubmed: handle multiple ReferenceList | Bryan Newbold | 2020-03-20 | 1 | -1/+4 |
| | | | | | | | | | | | | | | This resolves a situation noticed in prod where we were only importing/updating a single reference per article. Includes a regression test. | ||||
* | | pubmed: update many more metadata fields | Bryan Newbold | 2020-03-19 | 1 | -0/+22 |
| | | | | | | | | | | | | | | In particular, with daily updates in most cases the DOI will be registered first, then the entity updated with PMID when that is available. Often the pubmed metadata will be more complete, with abstracts etc, and we'll want those improvements. | ||||
* | | crossref: skip stub OUP title | Bryan Newbold | 2020-03-19 | 1 | -0/+8 |
| | | | | | | | | | | | | It seems like OUP pre-registers DOIs with this place-holder title, then updates the Crossref metdata when the paper is actually published. We should wait until the real title is available before creating an entity. | ||||
* | | Merge branch 'martin-kafka-bs4-import' into 'master' | Martin Czygan | 2020-03-10 | 2 | -1/+66 |
|\ \ | |/ |/| | | | | | pubmed and arxiv harvest preparations See merge request webgroup/fatcat!28 | ||||
| * | common: use smaller batch size since XML parsing may be slow | Martin Czygan | 2020-03-10 | 1 | -1/+1 |
| | | | | | | | | | | | | | | | | Address kafka tradeoff between long and short time-outs. Shorter time-outs would facilitate > consumer group re-balances and other consumer group state changes [...] in a reasonable human time-frame. | ||||
| * | pubmed ftp harvest and KafkaBs4XmlPusher | Martin Czygan | 2020-02-19 | 2 | -1/+66 |
| | | | | | | | | | | | | | | * add PubmedFTPWorker * utils are currently stored alongside pubmed (e.g. ftpretr, xmlstream) but may live elsewhere, as they are more generic * add KafkaBs4XmlPusher | ||||
* | | add some more domain/rel URL mappings | Bryan Newbold | 2020-02-22 | 1 | -0/+9 |
| | | |||||
* | | Merge branch 'bnewbold-shadow-import' | Bryan Newbold | 2020-02-19 | 2 | -0/+196 |
|\ \ | |||||
| * | | remove arabesque short wayback URL hack | Bryan Newbold | 2020-02-14 | 1 | -6/+0 |
| | | | |||||
| * | | improve shadow import file url cleanup path | Bryan Newbold | 2020-02-13 | 1 | -2/+12 |
| | | | | | | | | | | | | Should probably be refactored out in to shared cleanup code. | ||||
| * | | shadow import fixes from QA testing | Bryan Newbold | 2020-02-13 | 1 | -0/+6 |
| | | | |||||
| * | | shadow import: more filtering of file_meta fields | Bryan Newbold | 2020-02-13 | 1 | -0/+10 |
| | | | |||||
| * | | basic shadow importer | Bryan Newbold | 2020-02-13 | 2 | -0/+176 |
| |/ | |||||
* | | ingest import: fix edit_extra path | Bryan Newbold | 2020-02-18 | 1 | -1/+1 |
| | | |||||
* | | ingest importer: edit_extra is a top-level key | Bryan Newbold | 2020-02-18 | 1 | -1/+1 |
| | | |||||
* | | ingest import: allow short version of corpus names | Bryan Newbold | 2020-02-18 | 1 | -0/+3 |
| | | |||||
* | | ingest importer: pass through link rel | Bryan Newbold | 2020-02-18 | 1 | -1/+6 |
|/ | |||||
* | check ingest_request_source existance for SPN as well as ingest | Bryan Newbold | 2020-02-06 | 1 | -0/+3 |
| | |||||
* | additional trusted link sources | Bryan Newbold | 2020-02-06 | 1 | -0/+3 |
| | |||||
* | add mag and s2 as trusted link sources | Bryan Newbold | 2020-02-06 | 1 | -1/+1 |
| | |||||
* | ingest worker: handle missing ingest_request_source | Bryan Newbold | 2020-02-06 | 1 | -0/+3 |
| | | | | | Seeing a bunch of these due to re-ingests not including this field because of an earlier persist bug. | ||||
* | datacite: prevent none | Martin Czygan | 2020-01-31 | 1 | -1/+1 |
| | |||||
* | datacite: name shall not be None | Martin Czygan | 2020-01-31 | 1 | -1/+1 |
| |