Commit message (Collapse) | Author | Age | Files | Lines | ||
---|---|---|---|---|---|---|
... | ||||||
| * | | fix some transform bugs, add some tests | Bryan Newbold | 2020-01-29 | 6 | -13/+48 | |
| | | | ||||||
| * | | ES release schema updates | Bryan Newbold | 2020-01-29 | 1 | -5/+76 | |
| | | | ||||||
| * | | container ES schema changes | Bryan Newbold | 2020-01-29 | 1 | -16/+18 | |
| | | | ||||||
| * | | first implementation of ES file schema | Bryan Newbold | 2020-01-29 | 3 | -3/+69 | |
| | | | | | | | | | | | | | | | Includes a trivial test and transform, but not any workers or doc updates. | |||||
* | | | Merge branch 'bnewbold-more-ingest' into 'master' | bnewbold | 2020-02-25 | 1 | -1/+37 | |
|\ \ \ | | | | | | | | | | | | | | | | | entity worker: ingest more Datacite releases; filter some out See merge request webgroup/fatcat!29 | |||||
| * | | | entity worker: ingest more releases | Bryan Newbold | 2020-02-22 | 1 | -1/+37 | |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If release is a dataset or image, don't do a pdf ingest request. If release is a datacite DOI, and release_type is a "document", crawl regardless of is_oa detection. This is mostly to crawl repositories (institutional or subject). | |||||
* | | | | add some more domain/rel URL mappings | Bryan Newbold | 2020-02-22 | 1 | -0/+9 | |
|/ / / | ||||||
* | | | Merge branch 'bnewbold-shadow-import' | Bryan Newbold | 2020-02-19 | 7 | -2/+289 | |
|\ \ \ | ||||||
| * | | | remove arabesque short wayback URL hack | Bryan Newbold | 2020-02-14 | 1 | -6/+0 | |
| | | | | ||||||
| * | | | clarify shadow filter code | Bryan Newbold | 2020-02-13 | 1 | -1/+1 | |
| | | | | ||||||
| * | | | filter out shadows-only files in webface | Bryan Newbold | 2020-02-13 | 2 | -2/+7 | |
| | | | | ||||||
| * | | | improve shadow import file url cleanup path | Bryan Newbold | 2020-02-13 | 1 | -2/+12 | |
| | | | | | | | | | | | | | | | | Should probably be refactored out in to shared cleanup code. | |||||
| * | | | shadow import fixes from QA testing | Bryan Newbold | 2020-02-13 | 2 | -1/+7 | |
| | | | | ||||||
| * | | | shadow import: more filtering of file_meta fields | Bryan Newbold | 2020-02-13 | 3 | -18/+28 | |
| | | | | ||||||
| * | | | basic shadow importer | Bryan Newbold | 2020-02-13 | 5 | -0/+262 | |
| | | | | ||||||
* | | | | always crawl researchgate DOIs | Bryan Newbold | 2020-02-18 | 1 | -0/+2 | |
| | | | | | | | | | | | | | | | | Now that ingest is fixed | |||||
* | | | | ingest import: fix edit_extra path | Bryan Newbold | 2020-02-18 | 1 | -1/+1 | |
| | | | | ||||||
* | | | | ingest importer: edit_extra is a top-level key | Bryan Newbold | 2020-02-18 | 1 | -1/+1 | |
| | | | | ||||||
* | | | | ingest import: allow short version of corpus names | Bryan Newbold | 2020-02-18 | 1 | -0/+3 | |
| | | | | ||||||
* | | | | ingest importer: pass through link rel | Bryan Newbold | 2020-02-18 | 1 | -1/+6 | |
| |_|/ |/| | | ||||||
* | | | fatcat_ingest: as 'fatcat-ingest', not 'fatcat-ingest-container' | Bryan Newbold | 2020-02-14 | 1 | -1/+1 | |
| | | | | | | | | | | | | This tool is more generic now. | |||||
* | | | harvest: log state on startup and use stderr for diagnostics | Martin Czygan | 2020-02-14 | 3 | -17/+22 | |
|/ / | ||||||
* | | add acceptlist override for biorxiv/medrxiv | Bryan Newbold | 2020-02-10 | 1 | -2/+12 | |
| | | ||||||
* | | check ingest_request_source existance for SPN as well as ingest | Bryan Newbold | 2020-02-06 | 1 | -0/+3 | |
| | | ||||||
* | | additional trusted link sources | Bryan Newbold | 2020-02-06 | 1 | -0/+3 | |
| | | ||||||
* | | add mag and s2 as trusted link sources | Bryan Newbold | 2020-02-06 | 1 | -1/+1 | |
| | | ||||||
* | | ingest worker: handle missing ingest_request_source | Bryan Newbold | 2020-02-06 | 1 | -0/+3 | |
| | | | | | | | | | | Seeing a bunch of these due to re-ingests not including this field because of an earlier persist bug. | |||||
* | | switch '!= None' to 'is not None' | Bryan Newbold | 2020-02-04 | 1 | -3/+3 | |
| | | | | | | | | As reminded in code review, thanks Martin. | |||||
* | | allow-non-oa is a top-level flag, not sub-command | Bryan Newbold | 2020-02-04 | 1 | -3/+0 | |
| | | ||||||
* | | default to PMC ingest URLs over DOI | Bryan Newbold | 2020-02-04 | 1 | -4/+4 | |
| | | | | | | | | | | | | | | For cases where there might be both PMC and DOI urls, do the europmc.org PMC ones over DOI option. May want to turn this into a config or command-line option in the future. | |||||
* | | ingest: add 'extid' and 'query' modes; filters; refactor | Bryan Newbold | 2020-02-04 | 1 | -38/+147 | |
| | | | | | | | | | | | | This is a large refactor of the ingest script. It adds a number of filtering options (for all modes), and new modes for free-form queries or limiting to specific external identifiers. | |||||
* | | datacite: prevent none | Martin Czygan | 2020-01-31 | 1 | -1/+1 | |
| | | ||||||
* | | datacite: name shall not be None | Martin Czygan | 2020-01-31 | 1 | -1/+1 | |
| | | ||||||
* | | datacite: add exception for https://www.micropublication.org/ | Martin Czygan | 2020-01-31 | 2 | -1/+7 | |
| | | ||||||
* | | datacite: do not skip records w/o date | Martin Czygan | 2020-01-31 | 1 | -2/+1 | |
| | | ||||||
* | | datacite: improve docstring | Martin Czygan | 2020-01-31 | 1 | -4/+4 | |
| | | ||||||
* | | datacite: improve date handling and minor tweak | Martin Czygan | 2020-01-30 | 4 | -21/+153 | |
|/ | | | | | | | | | | | | | Records from https://www.micropublication.org/ did not have a date in FC, although raw data contained date strings - they were not using the finer-grained "attributes.date" but "attributes.published" and/or "attributes.publicationYear". Support for those fields has been added, including a test case. During this test (#30) a processing gap for names became clear (author may have "given_name" and "surname", but no "name"). This bug has been fixed, too. | |||||
* | fix KafkaError worker reporting for partition errors | Bryan Newbold | 2020-01-29 | 3 | -3/+3 | |
| | ||||||
* | additional DOI prefix filters | Bryan Newbold | 2020-01-28 | 1 | -0/+8 | |
| | | | | From martin, thanks. | |||||
* | increase kafka-pixy timeout to 25 seconds | Bryan Newbold | 2020-01-28 | 1 | -1/+1 | |
| | ||||||
* | apply ingest request filtering in entity worker | Bryan Newbold | 2020-01-28 | 1 | -3/+34 | |
| | | | | | | | `ingest_oa_only` behavior, and other filters, now handled in the entity update worker, instead of in the transform function. Also add a DOI prefix blocklist feature. | |||||
* | remove 'oa_only' feature from ingest transform | Bryan Newbold | 2020-01-28 | 2 | -15/+1 | |
| | | | | Refactoring to move this filter elsewhere | |||||
* | refactor fatcat_import kafka group names | Bryan Newbold | 2020-01-21 | 1 | -13/+54 | |
| | | | | | | | | | | | | | My current understanding is that consumer group names should be one-to-one with topic names. I previously though offsets were stored on a {topic, group} key, but they seem to be mixed and having too many workers in the same group is bad. In particular, we don't want cross-talk or load between QA and prod. All these topics are caught up in prod, so deploying this change and restarting workers should be safe. This commit does not update the elasticsearch or entity updates workers. | |||||
* | fix trivial typo in file importer | Bryan Newbold | 2020-01-20 | 1 | -1/+1 | |
| | ||||||
* | normal: DOI corner-case from pubmed import | Bryan Newbold | 2020-01-19 | 1 | -0/+9 | |
| | ||||||
* | Merge branch 'martin-fix-container-search' into 'master' | Martin Czygan | 2020-01-18 | 2 | -6/+6 | |
|\ | | | | | | | | | pagination: use correct endpoint in navigation See merge request webgroup/fatcat!19 | |||||
| * | pagination: use correct endpoint in navigation | Martin Czygan | 2020-01-18 | 2 | -6/+6 | |
| | | ||||||
* | | fix trivial one-character typo in fatcat_import.py | Bryan Newbold | 2020-01-17 | 1 | -1/+1 | |
| | | | | | | | | Should have run tests before pushing! | |||||
* | | actually control pubmed updates with a flag | Bryan Newbold | 2020-01-17 | 1 | -0/+4 | |
| | | ||||||
* | | do not normalize "en dash" in DOI | Martin Czygan | 2020-01-17 | 2 | -3/+6 | |
|/ | | | | | | | | | Technically, [...] DOI names may incorporate any printable characters from the Universal Character Set (UCS-2), of ISO/IEC 10646, which is the character set defined by Unicode (https://www.doi.org/doi_handbook/2_Numbering.html#2.5.1). For mostly QA reasons, we currently treat a DOI with an "en dash" as invalid. |