Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | Merge branch 'bnewbold-more-ingest' into 'master' | bnewbold | 2020-02-25 | 1 | -1/+37 |
|\ | | | | | | | | | entity worker: ingest more Datacite releases; filter some out See merge request webgroup/fatcat!29 | ||||
| * | entity worker: ingest more releases | Bryan Newbold | 2020-02-22 | 1 | -1/+37 |
| | | | | | | | | | | | | | | | | If release is a dataset or image, don't do a pdf ingest request. If release is a datacite DOI, and release_type is a "document", crawl regardless of is_oa detection. This is mostly to crawl repositories (institutional or subject). | ||||
* | | add some more domain/rel URL mappings | Bryan Newbold | 2020-02-22 | 1 | -0/+9 |
|/ | |||||
* | Merge branch 'bnewbold-shadow-import' | Bryan Newbold | 2020-02-19 | 2 | -0/+196 |
|\ | |||||
| * | remove arabesque short wayback URL hack | Bryan Newbold | 2020-02-14 | 1 | -6/+0 |
| | | |||||
| * | improve shadow import file url cleanup path | Bryan Newbold | 2020-02-13 | 1 | -2/+12 |
| | | | | | | | | Should probably be refactored out in to shared cleanup code. | ||||
| * | shadow import fixes from QA testing | Bryan Newbold | 2020-02-13 | 1 | -0/+6 |
| | | |||||
| * | shadow import: more filtering of file_meta fields | Bryan Newbold | 2020-02-13 | 1 | -0/+10 |
| | | |||||
| * | basic shadow importer | Bryan Newbold | 2020-02-13 | 2 | -0/+176 |
| | | |||||
* | | always crawl researchgate DOIs | Bryan Newbold | 2020-02-18 | 1 | -0/+2 |
| | | | | | | | | Now that ingest is fixed | ||||
* | | ingest import: fix edit_extra path | Bryan Newbold | 2020-02-18 | 1 | -1/+1 |
| | | |||||
* | | ingest importer: edit_extra is a top-level key | Bryan Newbold | 2020-02-18 | 1 | -1/+1 |
| | | |||||
* | | ingest import: allow short version of corpus names | Bryan Newbold | 2020-02-18 | 1 | -0/+3 |
| | | |||||
* | | ingest importer: pass through link rel | Bryan Newbold | 2020-02-18 | 1 | -1/+6 |
| | | |||||
* | | harvest: log state on startup and use stderr for diagnostics | Martin Czygan | 2020-02-14 | 3 | -17/+22 |
|/ | |||||
* | add acceptlist override for biorxiv/medrxiv | Bryan Newbold | 2020-02-10 | 1 | -2/+12 |
| | |||||
* | check ingest_request_source existance for SPN as well as ingest | Bryan Newbold | 2020-02-06 | 1 | -0/+3 |
| | |||||
* | additional trusted link sources | Bryan Newbold | 2020-02-06 | 1 | -0/+3 |
| | |||||
* | add mag and s2 as trusted link sources | Bryan Newbold | 2020-02-06 | 1 | -1/+1 |
| | |||||
* | ingest worker: handle missing ingest_request_source | Bryan Newbold | 2020-02-06 | 1 | -0/+3 |
| | | | | | Seeing a bunch of these due to re-ingests not including this field because of an earlier persist bug. | ||||
* | default to PMC ingest URLs over DOI | Bryan Newbold | 2020-02-04 | 1 | -4/+4 |
| | | | | | | | For cases where there might be both PMC and DOI urls, do the europmc.org PMC ones over DOI option. May want to turn this into a config or command-line option in the future. | ||||
* | datacite: prevent none | Martin Czygan | 2020-01-31 | 1 | -1/+1 |
| | |||||
* | datacite: name shall not be None | Martin Czygan | 2020-01-31 | 1 | -1/+1 |
| | |||||
* | datacite: add exception for https://www.micropublication.org/ | Martin Czygan | 2020-01-31 | 1 | -0/+5 |
| | |||||
* | datacite: do not skip records w/o date | Martin Czygan | 2020-01-31 | 1 | -2/+1 |
| | |||||
* | datacite: improve docstring | Martin Czygan | 2020-01-31 | 1 | -4/+4 |
| | |||||
* | datacite: improve date handling and minor tweak | Martin Czygan | 2020-01-30 | 1 | -19/+42 |
| | | | | | | | | | | | | | Records from https://www.micropublication.org/ did not have a date in FC, although raw data contained date strings - they were not using the finer-grained "attributes.date" but "attributes.published" and/or "attributes.publicationYear". Support for those fields has been added, including a test case. During this test (#30) a processing gap for names became clear (author may have "given_name" and "surname", but no "name"). This bug has been fixed, too. | ||||
* | fix KafkaError worker reporting for partition errors | Bryan Newbold | 2020-01-29 | 3 | -3/+3 |
| | |||||
* | additional DOI prefix filters | Bryan Newbold | 2020-01-28 | 1 | -0/+8 |
| | | | | From martin, thanks. | ||||
* | apply ingest request filtering in entity worker | Bryan Newbold | 2020-01-28 | 1 | -3/+34 |
| | | | | | | | `ingest_oa_only` behavior, and other filters, now handled in the entity update worker, instead of in the transform function. Also add a DOI prefix blocklist feature. | ||||
* | remove 'oa_only' feature from ingest transform | Bryan Newbold | 2020-01-28 | 1 | -14/+1 |
| | | | | Refactoring to move this filter elsewhere | ||||
* | fix trivial typo in file importer | Bryan Newbold | 2020-01-20 | 1 | -1/+1 |
| | |||||
* | normal: DOI corner-case from pubmed import | Bryan Newbold | 2020-01-19 | 1 | -0/+9 |
| | |||||
* | do not normalize "en dash" in DOI | Martin Czygan | 2020-01-17 | 1 | -2/+5 |
| | | | | | | | | | Technically, [...] DOI names may incorporate any printable characters from the Universal Character Set (UCS-2), of ISO/IEC 10646, which is the character set defined by Unicode (https://www.doi.org/doi_handbook/2_Numbering.html#2.5.1). For mostly QA reasons, we currently treat a DOI with an "en dash" as invalid. | ||||
* | ingest: improve tests, support old ingest results | Bryan Newbold | 2020-01-15 | 1 | -3/+12 |
| | |||||
* | update ingest worker for schema tweaks | Bryan Newbold | 2020-01-15 | 1 | -8/+15 |
| | | | | | | Should be backwards compatible with old ingest results. Fixed a bug with glutton ident detection. | ||||
* | ingest: allow more sources to auto-import | Bryan Newbold | 2020-01-15 | 1 | -1/+2 |
| | |||||
* | datacite: skip records without a doi | Martin Czygan | 2020-01-13 | 1 | -0/+4 |
| | |||||
* | datacite: add entry to license slug map | Martin Czygan | 2020-01-09 | 1 | -0/+1 |
| | |||||
* | datacite: ignore known unknown values in resourceType* | Martin Czygan | 2020-01-09 | 1 | -2/+2 |
| | |||||
* | datacite: abstracts may be strings or list of strings | Martin Czygan | 2020-01-09 | 1 | -2/+15 |
| | |||||
* | datacite: improve license_slug handling | Martin Czygan | 2020-01-09 | 1 | -60/+101 |
| | |||||
* | datacite: add 'Unknown' to blacklist | Martin Czygan | 2020-01-09 | 1 | -1/+5 |
| | |||||
* | datacite: get rid of schemaVersion | Martin Czygan | 2020-01-09 | 1 | -3/+0 |
| | |||||
* | Merge branch 'martin-datacite-import' | Martin Czygan | 2020-01-08 | 2 | -0/+1024 |
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Pipfile.lock is broken. * martin-datacite-import: (68 commits) datacite: pass in doi into factored out method datacite: reformat test cases and use jq . --sort-keys datacite: factor out contributor handling datacite: catch type mismatch in language detection datacite: adjust tests for release_month datacite: name extra.month, extra.release_month datacite: mark additional files as stub datacite: CCDC are entries, mostly datacite: use more specific release_type, if possible datacite: ignore certain names datacite: over 3% records have the same title: stub datacite: fill a few more release_type gaps datacite: adding datacite-specific extra metadata datacite: apply pylint suggestions datacite: fix typos datacite: set release_stage to published by default datacite: month field should be top-level datacite: include month in extra datacite: indicate mismatched file in test datacite: clean abstracts, use unknown value tokens ... | ||||
| * | datacite: pass in doi into factored out method | Martin Czygan | 2020-01-08 | 1 | -2/+3 |
| | | |||||
| * | datacite: factor out contributor handling | Martin Czygan | 2020-01-08 | 1 | -80/+103 |
| | | | | | | | | | | | | | | Use values from: * attributes.creators[] * attributes.contributors[] | ||||
| * | datacite: catch type mismatch in language detection | Martin Czygan | 2020-01-08 | 1 | -3/+2 |
| | | |||||
| * | datacite: name extra.month, extra.release_month | Martin Czygan | 2020-01-08 | 1 | -1/+3 |
| | | |||||
| * | datacite: mark additional files as stub | Martin Czygan | 2020-01-08 | 1 | -0/+4 |
| | |