summaryrefslogtreecommitdiffstats
path: root/python/fatcat_tools/importers
Commit message (Collapse)AuthorAgeFilesLines
* ingest importer: check that stage is consistent with releaseBryan Newbold2020-05-261-0/+5
|
* Merge remote-tracking branch 'github/master'Bryan Newbold2020-05-221-2/+2
|\
| * Indentity is not the same this as equality in PythonChristian Clauss2020-05-141-2/+2
| |
* | importers: clarify handling of ApiExceptionBryan Newbold2020-05-223-4/+10
| | | | | | | | | | | | | | | | One of these (in ingest importer pipeline) is an actual bug, the others are just changing the syntax to be more explicit/conservative. The ingest importer bug seems to have resulted in some bad file match imports; scale of impact is unknown.
* | ingest importer: don't use glutton matchesBryan Newbold2020-05-221-3/+3
| | | | | | | | | | | | | | Until reviewing I didn't realize we were even doing this currently. Hopefluly has not impacted too many imports, as almost all ingests use an external identifer, so only those with identifers not in fatcat for whatever reason.
* | datacite: fix type errorMartin Czygan2020-04-221-1/+3
| | | | | | | | | | | | | | Up to now, we expected the description to be a string or list. Add handling for int as well. First appeared: Apr 22 19:58:39.
* | datacite: fix a raw name constraint violationMartin Czygan2020-04-201-0/+8
|/ | | | | | | It was possible that contribs got added which had no raw name. One example would be a name consisting of whitespace only. This fix adds a final check for this case.
* consistently use raw string prefix for regexBryan Newbold2020-04-171-1/+1
|
* pubmed: use untranslated title if translated not availableBryan Newbold2020-04-011-0/+6
| | | | | | | The primary motivation for this change is that fatcat *requires* a non-empty title for each release entity. Pubmed/Medline occasionally indexes just a VenacularTitle with no ArticleTitle for foreign publications, and currently those records don't end up in fatcat at all.
* importers: replace newlines in get_text() stringsBryan Newbold2020-04-014-23/+25
|
* importers: more string/get_text swapsBryan Newbold2020-03-283-27/+27
| | | | See previous pubmed commit for details.
* pubmed: bunch of .get_text() instead of .stringBryan Newbold2020-03-281-12/+12
| | | | | | | | | | | Yikes! Apparently when a tag has child tags, .string will return None instead of all the strings. .get_text() returns all of it: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#get-text https://www.crummy.com/software/BeautifulSoup/bs4/doc/#string I've things like identifiers as .string, when we expect only a single string inside.
* Merge pull request #53 from EdwardBetts/spellingbnewbold2020-03-273-7/+7
|\ | | | | Correct spelling mistakes
| * Correct spelling mistakesEdward Betts2020-03-273-7/+7
| |
* | datacite: nameIdentifier corner caseBryan Newbold2020-03-261-1/+2
| | | | | | | | | | | | | | | | | | Works around a bug in production: AttributeError: 'NoneType' object has no attribute 'replace' (datacite.py:724) NOTE: there are no tests for this code path
* | jalc: avoid meaningless pages valuesBryan Newbold2020-03-231-4/+8
| |
* | datacite: add year sanity restrictionsbnewbold2020-03-231-0/+7
| | | | | | | | | | | | | | | | | | Example of entities with bogus years: https://fatcat.wiki/release/search?q=doi_registrar%3Adatacite+year%3A%3E2100 We can do a clean-up task, but first need to prevent creation of new bad metadata.
* | pubmed: handle multiple ReferenceListBryan Newbold2020-03-201-1/+4
| | | | | | | | | | | | | | This resolves a situation noticed in prod where we were only importing/updating a single reference per article. Includes a regression test.
* | pubmed: update many more metadata fieldsBryan Newbold2020-03-191-0/+22
| | | | | | | | | | | | | | In particular, with daily updates in most cases the DOI will be registered first, then the entity updated with PMID when that is available. Often the pubmed metadata will be more complete, with abstracts etc, and we'll want those improvements.
* | crossref: skip stub OUP titleBryan Newbold2020-03-191-0/+8
| | | | | | | | | | | | It seems like OUP pre-registers DOIs with this place-holder title, then updates the Crossref metdata when the paper is actually published. We should wait until the real title is available before creating an entity.
* | Merge branch 'martin-kafka-bs4-import' into 'master'Martin Czygan2020-03-102-1/+66
|\ \ | |/ |/| | | | | pubmed and arxiv harvest preparations See merge request webgroup/fatcat!28
| * common: use smaller batch size since XML parsing may be slowMartin Czygan2020-03-101-1/+1
| | | | | | | | | | | | | | | | Address kafka tradeoff between long and short time-outs. Shorter time-outs would facilitate > consumer group re-balances and other consumer group state changes [...] in a reasonable human time-frame.
| * pubmed ftp harvest and KafkaBs4XmlPusherMartin Czygan2020-02-192-1/+66
| | | | | | | | | | | | | | * add PubmedFTPWorker * utils are currently stored alongside pubmed (e.g. ftpretr, xmlstream) but may live elsewhere, as they are more generic * add KafkaBs4XmlPusher
* | add some more domain/rel URL mappingsBryan Newbold2020-02-221-0/+9
| |
* | Merge branch 'bnewbold-shadow-import'Bryan Newbold2020-02-192-0/+196
|\ \
| * | remove arabesque short wayback URL hackBryan Newbold2020-02-141-6/+0
| | |
| * | improve shadow import file url cleanup pathBryan Newbold2020-02-131-2/+12
| | | | | | | | | | | | Should probably be refactored out in to shared cleanup code.
| * | shadow import fixes from QA testingBryan Newbold2020-02-131-0/+6
| | |
| * | shadow import: more filtering of file_meta fieldsBryan Newbold2020-02-131-0/+10
| | |
| * | basic shadow importerBryan Newbold2020-02-132-0/+176
| |/
* | ingest import: fix edit_extra pathBryan Newbold2020-02-181-1/+1
| |
* | ingest importer: edit_extra is a top-level keyBryan Newbold2020-02-181-1/+1
| |
* | ingest import: allow short version of corpus namesBryan Newbold2020-02-181-0/+3
| |
* | ingest importer: pass through link relBryan Newbold2020-02-181-1/+6
|/
* check ingest_request_source existance for SPN as well as ingestBryan Newbold2020-02-061-0/+3
|
* additional trusted link sourcesBryan Newbold2020-02-061-0/+3
|
* add mag and s2 as trusted link sourcesBryan Newbold2020-02-061-1/+1
|
* ingest worker: handle missing ingest_request_sourceBryan Newbold2020-02-061-0/+3
| | | | | Seeing a bunch of these due to re-ingests not including this field because of an earlier persist bug.
* datacite: prevent noneMartin Czygan2020-01-311-1/+1
|
* datacite: name shall not be NoneMartin Czygan2020-01-311-1/+1
|
* datacite: add exception for https://www.micropublication.org/Martin Czygan2020-01-311-0/+5
|
* datacite: do not skip records w/o dateMartin Czygan2020-01-311-2/+1
|
* datacite: improve docstringMartin Czygan2020-01-311-4/+4
|
* datacite: improve date handling and minor tweakMartin Czygan2020-01-301-19/+42
| | | | | | | | | | | | | Records from https://www.micropublication.org/ did not have a date in FC, although raw data contained date strings - they were not using the finer-grained "attributes.date" but "attributes.published" and/or "attributes.publicationYear". Support for those fields has been added, including a test case. During this test (#30) a processing gap for names became clear (author may have "given_name" and "surname", but no "name"). This bug has been fixed, too.
* fix KafkaError worker reporting for partition errorsBryan Newbold2020-01-291-1/+1
|
* fix trivial typo in file importerBryan Newbold2020-01-201-1/+1
|
* ingest: improve tests, support old ingest resultsBryan Newbold2020-01-151-3/+12
|
* update ingest worker for schema tweaksBryan Newbold2020-01-151-8/+15
| | | | | | Should be backwards compatible with old ingest results. Fixed a bug with glutton ident detection.
* ingest: allow more sources to auto-importBryan Newbold2020-01-151-1/+2
|
* datacite: skip records without a doiMartin Czygan2020-01-131-0/+4
|
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-24 17:53:36 +0000