Commit message (Collapse) | Author | Age | Files | Lines | ||
---|---|---|---|---|---|---|
... | ||||||
| * | | ES files: don't remove archive.org domains/hosts | Bryan Newbold | 2020-02-07 | 1 | -5/+0 | |
| | | | ||||||
| * | | ES releases: host/domain fixes | Bryan Newbold | 2020-01-31 | 2 | -2/+5 | |
| | | | ||||||
| * | | pipenv: lock zipp version to work around python3.6 requirement | Bryan Newbold | 2020-01-30 | 2 | -7/+20 | |
| | | | ||||||
| * | | fix release es transform missing 'issue' | Bryan Newbold | 2020-01-30 | 1 | -0/+1 | |
| | | | ||||||
| * | | add upper-case work-around from kibana map join | Bryan Newbold | 2020-01-30 | 1 | -0/+1 | |
| | | | ||||||
| * | | tweak file ES archive.org domain tracking | Bryan Newbold | 2020-01-30 | 1 | -0/+6 | |
| | | | ||||||
| * | | implement host+domain parsing for file ES transform | Bryan Newbold | 2020-01-30 | 2 | -13/+8 | |
| | | | ||||||
| * | | pipenv: add tldextract (url parser) and update deps | Bryan Newbold | 2020-01-30 | 2 | -136/+159 | |
| | | | ||||||
| * | | fix ES file schema plural field names | Bryan Newbold | 2020-01-29 | 2 | -5/+4 | |
| | | | ||||||
| * | | new biblio-only general search | Bryan Newbold | 2020-01-29 | 1 | -2/+2 | |
| | | | | | | | | | | | | The other fields are now "copy_to" the merged biblio field. | |||||
| * | | elastic schema fixes | Bryan Newbold | 2020-01-29 | 1 | -0/+5 | |
| | | | ||||||
| * | | add country to v03b release schema | Bryan Newbold | 2020-01-29 | 1 | -0/+2 | |
| | | | ||||||
| * | | actually implement changelog transform | Bryan Newbold | 2020-01-29 | 2 | -18/+68 | |
| | | | ||||||
| * | | fix some transform bugs, add some tests | Bryan Newbold | 2020-01-29 | 6 | -13/+48 | |
| | | | ||||||
| * | | ES release schema updates | Bryan Newbold | 2020-01-29 | 1 | -5/+76 | |
| | | | ||||||
| * | | container ES schema changes | Bryan Newbold | 2020-01-29 | 1 | -16/+18 | |
| | | | ||||||
| * | | first implementation of ES file schema | Bryan Newbold | 2020-01-29 | 3 | -3/+69 | |
| | | | | | | | | | | | | | | | Includes a trivial test and transform, but not any workers or doc updates. | |||||
* | | | Merge branch 'bnewbold-more-ingest' into 'master' | bnewbold | 2020-02-25 | 1 | -1/+37 | |
|\ \ \ | | | | | | | | | | | | | | | | | entity worker: ingest more Datacite releases; filter some out See merge request webgroup/fatcat!29 | |||||
| * | | | entity worker: ingest more releases | Bryan Newbold | 2020-02-22 | 1 | -1/+37 | |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If release is a dataset or image, don't do a pdf ingest request. If release is a datacite DOI, and release_type is a "document", crawl regardless of is_oa detection. This is mostly to crawl repositories (institutional or subject). | |||||
* | | | | add some more domain/rel URL mappings | Bryan Newbold | 2020-02-22 | 1 | -0/+9 | |
|/ / / | ||||||
* | | | Merge branch 'bnewbold-shadow-import' | Bryan Newbold | 2020-02-19 | 7 | -2/+289 | |
|\ \ \ | ||||||
| * | | | remove arabesque short wayback URL hack | Bryan Newbold | 2020-02-14 | 1 | -6/+0 | |
| | | | | ||||||
| * | | | clarify shadow filter code | Bryan Newbold | 2020-02-13 | 1 | -1/+1 | |
| | | | | ||||||
| * | | | filter out shadows-only files in webface | Bryan Newbold | 2020-02-13 | 2 | -2/+7 | |
| | | | | ||||||
| * | | | improve shadow import file url cleanup path | Bryan Newbold | 2020-02-13 | 1 | -2/+12 | |
| | | | | | | | | | | | | | | | | Should probably be refactored out in to shared cleanup code. | |||||
| * | | | shadow import fixes from QA testing | Bryan Newbold | 2020-02-13 | 2 | -1/+7 | |
| | | | | ||||||
| * | | | shadow import: more filtering of file_meta fields | Bryan Newbold | 2020-02-13 | 3 | -18/+28 | |
| | | | | ||||||
| * | | | basic shadow importer | Bryan Newbold | 2020-02-13 | 5 | -0/+262 | |
| | | | | ||||||
* | | | | always crawl researchgate DOIs | Bryan Newbold | 2020-02-18 | 1 | -0/+2 | |
| | | | | | | | | | | | | | | | | Now that ingest is fixed | |||||
* | | | | ingest import: fix edit_extra path | Bryan Newbold | 2020-02-18 | 1 | -1/+1 | |
| | | | | ||||||
* | | | | ingest importer: edit_extra is a top-level key | Bryan Newbold | 2020-02-18 | 1 | -1/+1 | |
| | | | | ||||||
* | | | | ingest import: allow short version of corpus names | Bryan Newbold | 2020-02-18 | 1 | -0/+3 | |
| | | | | ||||||
* | | | | ingest importer: pass through link rel | Bryan Newbold | 2020-02-18 | 1 | -1/+6 | |
| |_|/ |/| | | ||||||
* | | | fatcat_ingest: as 'fatcat-ingest', not 'fatcat-ingest-container' | Bryan Newbold | 2020-02-14 | 1 | -1/+1 | |
| | | | | | | | | | | | | This tool is more generic now. | |||||
* | | | harvest: log state on startup and use stderr for diagnostics | Martin Czygan | 2020-02-14 | 3 | -17/+22 | |
|/ / | ||||||
* | | add acceptlist override for biorxiv/medrxiv | Bryan Newbold | 2020-02-10 | 1 | -2/+12 | |
| | | ||||||
* | | check ingest_request_source existance for SPN as well as ingest | Bryan Newbold | 2020-02-06 | 1 | -0/+3 | |
| | | ||||||
* | | additional trusted link sources | Bryan Newbold | 2020-02-06 | 1 | -0/+3 | |
| | | ||||||
* | | add mag and s2 as trusted link sources | Bryan Newbold | 2020-02-06 | 1 | -1/+1 | |
| | | ||||||
* | | ingest worker: handle missing ingest_request_source | Bryan Newbold | 2020-02-06 | 1 | -0/+3 | |
| | | | | | | | | | | Seeing a bunch of these due to re-ingests not including this field because of an earlier persist bug. | |||||
* | | switch '!= None' to 'is not None' | Bryan Newbold | 2020-02-04 | 1 | -3/+3 | |
| | | | | | | | | As reminded in code review, thanks Martin. | |||||
* | | allow-non-oa is a top-level flag, not sub-command | Bryan Newbold | 2020-02-04 | 1 | -3/+0 | |
| | | ||||||
* | | default to PMC ingest URLs over DOI | Bryan Newbold | 2020-02-04 | 1 | -4/+4 | |
| | | | | | | | | | | | | | | For cases where there might be both PMC and DOI urls, do the europmc.org PMC ones over DOI option. May want to turn this into a config or command-line option in the future. | |||||
* | | ingest: add 'extid' and 'query' modes; filters; refactor | Bryan Newbold | 2020-02-04 | 1 | -38/+147 | |
| | | | | | | | | | | | | This is a large refactor of the ingest script. It adds a number of filtering options (for all modes), and new modes for free-form queries or limiting to specific external identifiers. | |||||
* | | datacite: prevent none | Martin Czygan | 2020-01-31 | 1 | -1/+1 | |
| | | ||||||
* | | datacite: name shall not be None | Martin Czygan | 2020-01-31 | 1 | -1/+1 | |
| | | ||||||
* | | datacite: add exception for https://www.micropublication.org/ | Martin Czygan | 2020-01-31 | 2 | -1/+7 | |
| | | ||||||
* | | datacite: do not skip records w/o date | Martin Czygan | 2020-01-31 | 1 | -2/+1 | |
| | | ||||||
* | | datacite: improve docstring | Martin Czygan | 2020-01-31 | 1 | -4/+4 | |
| | | ||||||
* | | datacite: improve date handling and minor tweak | Martin Czygan | 2020-01-30 | 4 | -21/+153 | |
|/ | | | | | | | | | | | | | Records from https://www.micropublication.org/ did not have a date in FC, although raw data contained date strings - they were not using the finer-grained "attributes.date" but "attributes.published" and/or "attributes.publicationYear". Support for those fields has been added, including a test case. During this test (#30) a processing gap for names became clear (author may have "given_name" and "surname", but no "name"). This bug has been fixed, too. |