aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
...
| * | update ES docs and proposalBryan Newbold2020-01-292-4/+6
| | |
| * | actually implement changelog transformBryan Newbold2020-01-293-19/+78
| | |
| * | fix some transform bugs, add some testsBryan Newbold2020-01-296-13/+48
| | |
| * | ES release schema updatesBryan Newbold2020-01-292-28/+122
| | |
| * | container ES schema changesBryan Newbold2020-01-292-29/+38
| | |
| * | first implementation of ES file schemaBryan Newbold2020-01-294-3/+115
| | | | | | | | | | | | | | | Includes a trivial test and transform, but not any workers or doc updates.
* | | Merge branch 'bnewbold-more-ingest' into 'master'bnewbold2020-02-251-1/+37
|\ \ \ | | | | | | | | | | | | | | | | entity worker: ingest more Datacite releases; filter some out See merge request webgroup/fatcat!29
| * | | entity worker: ingest more releasesBryan Newbold2020-02-221-1/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If release is a dataset or image, don't do a pdf ingest request. If release is a datacite DOI, and release_type is a "document", crawl regardless of is_oa detection. This is mostly to crawl repositories (institutional or subject).
* | | | add some more domain/rel URL mappingsBryan Newbold2020-02-221-0/+9
|/ / /
* | | table size snapshotsBryan Newbold2020-02-192-0/+47
| | |
* | | Merge branch 'bnewbold-shadow-import'Bryan Newbold2020-02-197-2/+289
|\ \ \
| * | | remove arabesque short wayback URL hackBryan Newbold2020-02-141-6/+0
| | | |
| * | | clarify shadow filter codeBryan Newbold2020-02-131-1/+1
| | | |
| * | | filter out shadows-only files in webfaceBryan Newbold2020-02-132-2/+7
| | | |
| * | | improve shadow import file url cleanup pathBryan Newbold2020-02-131-2/+12
| | | | | | | | | | | | | | | | Should probably be refactored out in to shared cleanup code.
| * | | shadow import fixes from QA testingBryan Newbold2020-02-132-1/+7
| | | |
| * | | shadow import: more filtering of file_meta fieldsBryan Newbold2020-02-133-18/+28
| | | |
| * | | basic shadow importerBryan Newbold2020-02-135-0/+262
| | | |
* | | | more SQL db size reduction notesBryan Newbold2020-02-191-0/+16
| | | |
* | | | always crawl researchgate DOIsBryan Newbold2020-02-181-0/+2
| | | | | | | | | | | | | | | | Now that ingest is fixed
* | | | ingest import: fix edit_extra pathBryan Newbold2020-02-181-1/+1
| | | |
* | | | ingest importer: edit_extra is a top-level keyBryan Newbold2020-02-181-1/+1
| | | |
* | | | ingest import: allow short version of corpus namesBryan Newbold2020-02-181-0/+3
| | | |
* | | | ingest importer: pass through link relBryan Newbold2020-02-181-1/+6
| |_|/ |/| |
* | | fatcat_ingest: as 'fatcat-ingest', not 'fatcat-ingest-container'Bryan Newbold2020-02-141-1/+1
| | | | | | | | | | | | This tool is more generic now.
* | | Merge branch 'martin-harvest-print' into 'master'bnewbold2020-02-143-17/+22
|\ \ \ | |/ / |/| | | | | | | | harvest: log state on startup and use stderr for diagnostics See merge request webgroup/fatcat!27
| * | harvest: log state on startup and use stderr for diagnosticsMartin Czygan2020-02-143-17/+22
|/ /
* | add acceptlist override for biorxiv/medrxivBryan Newbold2020-02-101-2/+12
| |
* | check ingest_request_source existance for SPN as well as ingestBryan Newbold2020-02-061-0/+3
| |
* | additional trusted link sourcesBryan Newbold2020-02-061-0/+3
| |
* | add mag and s2 as trusted link sourcesBryan Newbold2020-02-061-1/+1
| |
* | ingest worker: handle missing ingest_request_sourceBryan Newbold2020-02-061-0/+3
| | | | | | | | | | Seeing a bunch of these due to re-ingests not including this field because of an earlier persist bug.
* | switch '!= None' to 'is not None'Bryan Newbold2020-02-041-3/+3
| | | | | | | | As reminded in code review, thanks Martin.
* | allow-non-oa is a top-level flag, not sub-commandBryan Newbold2020-02-041-3/+0
| |
* | default to PMC ingest URLs over DOIBryan Newbold2020-02-041-4/+4
| | | | | | | | | | | | | | For cases where there might be both PMC and DOI urls, do the europmc.org PMC ones over DOI option. May want to turn this into a config or command-line option in the future.
* | ingest: add 'extid' and 'query' modes; filters; refactorBryan Newbold2020-02-041-38/+147
| | | | | | | | | | | | This is a large refactor of the ingest script. It adds a number of filtering options (for all modes), and new modes for free-form queries or limiting to specific external identifiers.
* | Merge branch 'martin-datacite-strptime-none' into 'master'Martin Czygan2020-01-311-1/+1
|\ \ | | | | | | | | | | | | datacite: prevent none See merge request webgroup/fatcat!25
| * | datacite: prevent noneMartin Czygan2020-01-311-1/+1
|/ /
* | Merge branch 'martin-datacite-micropub' into 'master'Martin Czygan2020-01-314-25/+162
|\ \ | |/ |/| | | | | datacite: improve date handling and minor tweak See merge request webgroup/fatcat!24
| * datacite: name shall not be NoneMartin Czygan2020-01-311-1/+1
| |
| * datacite: add exception for https://www.micropublication.org/Martin Czygan2020-01-312-1/+7
| |
| * datacite: do not skip records w/o dateMartin Czygan2020-01-311-2/+1
| |
| * datacite: improve docstringMartin Czygan2020-01-311-4/+4
| |
| * datacite: improve date handling and minor tweakMartin Czygan2020-01-304-21/+153
|/ | | | | | | | | | | | | Records from https://www.micropublication.org/ did not have a date in FC, although raw data contained date strings - they were not using the finer-grained "attributes.date" but "attributes.published" and/or "attributes.publicationYear". Support for those fields has been added, including a test case. During this test (#30) a processing gap for names became clear (author may have "given_name" and "surname", but no "name"). This bug has been fixed, too.
* fix KafkaError worker reporting for partition errorsBryan Newbold2020-01-293-3/+3
|
* additional DOI prefix filtersBryan Newbold2020-01-281-0/+8
| | | | From martin, thanks.
* increase kafka-pixy timeout to 25 secondsBryan Newbold2020-01-281-1/+1
|
* apply ingest request filtering in entity workerBryan Newbold2020-01-281-3/+34
| | | | | | | `ingest_oa_only` behavior, and other filters, now handled in the entity update worker, instead of in the transform function. Also add a DOI prefix blocklist feature.
* remove 'oa_only' feature from ingest transformBryan Newbold2020-01-282-15/+1
| | | | Refactoring to move this filter elsewhere
* more TODO/proposal cleanupBryan Newbold2020-01-224-10/+34
|
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-29 16:48:51 +0000