summaryrefslogtreecommitdiffstats
path: root/python/fatcat_tools/importers
Commit message (Collapse)AuthorAgeFilesLines
* Merge branch 'martin-kafka-bs4-import' into 'master'Martin Czygan2020-03-102-1/+66
|\ | | | | | | | | pubmed and arxiv harvest preparations See merge request webgroup/fatcat!28
| * common: use smaller batch size since XML parsing may be slowMartin Czygan2020-03-101-1/+1
| | | | | | | | | | | | | | | | Address kafka tradeoff between long and short time-outs. Shorter time-outs would facilitate > consumer group re-balances and other consumer group state changes [...] in a reasonable human time-frame.
| * pubmed ftp harvest and KafkaBs4XmlPusherMartin Czygan2020-02-192-1/+66
| | | | | | | | | | | | | | * add PubmedFTPWorker * utils are currently stored alongside pubmed (e.g. ftpretr, xmlstream) but may live elsewhere, as they are more generic * add KafkaBs4XmlPusher
* | add some more domain/rel URL mappingsBryan Newbold2020-02-221-0/+9
| |
* | Merge branch 'bnewbold-shadow-import'Bryan Newbold2020-02-192-0/+196
|\ \
| * | remove arabesque short wayback URL hackBryan Newbold2020-02-141-6/+0
| | |
| * | improve shadow import file url cleanup pathBryan Newbold2020-02-131-2/+12
| | | | | | | | | | | | Should probably be refactored out in to shared cleanup code.
| * | shadow import fixes from QA testingBryan Newbold2020-02-131-0/+6
| | |
| * | shadow import: more filtering of file_meta fieldsBryan Newbold2020-02-131-0/+10
| | |
| * | basic shadow importerBryan Newbold2020-02-132-0/+176
| |/
* | ingest import: fix edit_extra pathBryan Newbold2020-02-181-1/+1
| |
* | ingest importer: edit_extra is a top-level keyBryan Newbold2020-02-181-1/+1
| |
* | ingest import: allow short version of corpus namesBryan Newbold2020-02-181-0/+3
| |
* | ingest importer: pass through link relBryan Newbold2020-02-181-1/+6
|/
* check ingest_request_source existance for SPN as well as ingestBryan Newbold2020-02-061-0/+3
|
* additional trusted link sourcesBryan Newbold2020-02-061-0/+3
|
* add mag and s2 as trusted link sourcesBryan Newbold2020-02-061-1/+1
|
* ingest worker: handle missing ingest_request_sourceBryan Newbold2020-02-061-0/+3
| | | | | Seeing a bunch of these due to re-ingests not including this field because of an earlier persist bug.
* datacite: prevent noneMartin Czygan2020-01-311-1/+1
|
* datacite: name shall not be NoneMartin Czygan2020-01-311-1/+1
|
* datacite: add exception for https://www.micropublication.org/Martin Czygan2020-01-311-0/+5
|
* datacite: do not skip records w/o dateMartin Czygan2020-01-311-2/+1
|
* datacite: improve docstringMartin Czygan2020-01-311-4/+4
|
* datacite: improve date handling and minor tweakMartin Czygan2020-01-301-19/+42
| | | | | | | | | | | | | Records from https://www.micropublication.org/ did not have a date in FC, although raw data contained date strings - they were not using the finer-grained "attributes.date" but "attributes.published" and/or "attributes.publicationYear". Support for those fields has been added, including a test case. During this test (#30) a processing gap for names became clear (author may have "given_name" and "surname", but no "name"). This bug has been fixed, too.
* fix KafkaError worker reporting for partition errorsBryan Newbold2020-01-291-1/+1
|
* fix trivial typo in file importerBryan Newbold2020-01-201-1/+1
|
* ingest: improve tests, support old ingest resultsBryan Newbold2020-01-151-3/+12
|
* update ingest worker for schema tweaksBryan Newbold2020-01-151-8/+15
| | | | | | Should be backwards compatible with old ingest results. Fixed a bug with glutton ident detection.
* ingest: allow more sources to auto-importBryan Newbold2020-01-151-1/+2
|
* datacite: skip records without a doiMartin Czygan2020-01-131-0/+4
|
* datacite: add entry to license slug mapMartin Czygan2020-01-091-0/+1
|
* datacite: ignore known unknown values in resourceType*Martin Czygan2020-01-091-2/+2
|
* datacite: abstracts may be strings or list of stringsMartin Czygan2020-01-091-2/+15
|
* datacite: improve license_slug handlingMartin Czygan2020-01-091-60/+101
|
* datacite: add 'Unknown' to blacklistMartin Czygan2020-01-091-1/+5
|
* datacite: get rid of schemaVersionMartin Czygan2020-01-091-3/+0
|
* Merge branch 'martin-datacite-import'Martin Czygan2020-01-082-0/+1024
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Pipfile.lock is broken. * martin-datacite-import: (68 commits) datacite: pass in doi into factored out method datacite: reformat test cases and use jq . --sort-keys datacite: factor out contributor handling datacite: catch type mismatch in language detection datacite: adjust tests for release_month datacite: name extra.month, extra.release_month datacite: mark additional files as stub datacite: CCDC are entries, mostly datacite: use more specific release_type, if possible datacite: ignore certain names datacite: over 3% records have the same title: stub datacite: fill a few more release_type gaps datacite: adding datacite-specific extra metadata datacite: apply pylint suggestions datacite: fix typos datacite: set release_stage to published by default datacite: month field should be top-level datacite: include month in extra datacite: indicate mismatched file in test datacite: clean abstracts, use unknown value tokens ...
| * datacite: pass in doi into factored out methodMartin Czygan2020-01-081-2/+3
| |
| * datacite: factor out contributor handlingMartin Czygan2020-01-081-80/+103
| | | | | | | | | | | | | | Use values from: * attributes.creators[] * attributes.contributors[]
| * datacite: catch type mismatch in language detectionMartin Czygan2020-01-081-3/+2
| |
| * datacite: name extra.month, extra.release_monthMartin Czygan2020-01-081-1/+3
| |
| * datacite: mark additional files as stubMartin Czygan2020-01-081-0/+4
| |
| * datacite: CCDC are entries, mostlyMartin Czygan2020-01-081-0/+4
| |
| * datacite: use more specific release_type, if possibleMartin Czygan2020-01-081-0/+6
| |
| * datacite: ignore certain namesMartin Czygan2020-01-081-0/+6
| |
| * datacite: over 3% records have the same title: stubMartin Czygan2020-01-081-0/+7
| | | | | | | | | | | | | | | | | | | | The GBIF (https://www.gbif.org/) deposits most records under the titles: * 599243 GBIF Occurrence Download * 41176 Occurrence Download Mark them as "stub" for the moment (https://guide.fatcat.wiki/entity_release.html#release_type-vocabulary).
| * datacite: fill a few more release_type gapsMartin Czygan2020-01-081-17/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * citeproc: http://docs.citationstyles.org/en/stable/specification.html#appendix-iii-types * resourceTypeGeneral: https://schema.datacite.org/meta/kernel-4.0/doc/DataCite-MetadataKernel_v4.0.pdf#page=32 * resourceType: uncontrolled, over 170000 distinct values, frequent: null, Dataset, JournalArticle, PGRFA Material, Journal Article, Dataset/UNITE Species Hypothesis, ... General frequency: * "attributes.types": 18210075, * "attributes.types.ris": 18058890, * "attributes.types.bibtex": 18058888, * "attributes.types.citeproc": 18058890, * "attributes.types.schemaOrg": 18058929, * "attributes.types.resourceType": 12737988, * "attributes.types.resourceTypeGeneral": 16576139,
| * datacite: adding datacite-specific extra metadataMartin Czygan2020-01-071-0/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | * attributes.metadataVersion * attributes.schemaVersion * attributes.version (source dependent values, follows suggestions in https://schema.datacite.org/meta/kernel-4.3/doc/DataCite-MetadataKernel_v4.3.pdf#page=26, but values vary) Furthermore: * attributes.types.resourceTypeGeneral * attributes.types.resourceType
| * datacite: apply pylint suggestionsMartin Czygan2020-01-071-8/+10
| |
| * datacite: fix typosMartin Czygan2020-01-071-1/+1
| |
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-24 18:23:53 +0000