aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* common: use smaller batch size since XML parsing may be slowMartin Czygan2020-03-101-1/+1
| | | | | | | | Address kafka tradeoff between long and short time-outs. Shorter time-outs would facilitate > consumer group re-balances and other consumer group state changes [...] in a reasonable human time-frame.
* pubmed: log to stderrMartin Czygan2020-03-101-1/+1
|
* pubmed: move mapping generation out of fetch_dateMartin Czygan2020-03-102-7/+10
| | | | | * fetch_date will fail on missing mapping * adjust tests (test will require access to pubmed ftp)
* harvest: fix imports from HarvestPubmedWorker cleanupMartin Czygan2020-03-102-4/+4
|
* pubmed: citations is a bit more preciseMartin Czygan2020-03-091-1/+1
| | | | | > Each day, NLM produces update files that include new, revised and deleted citations. -- ftp://ftp.ncbi.nlm.nih.gov/pubmed/updatefiles/README.txt
* pubmed: we sync from FTPMartin Czygan2020-03-091-1/+1
|
* oaipmh: HarvestPubmedWorker obsoleted by PubmedFTPWorkerMartin Czygan2020-03-091-34/+0
|
* fatcat_import: address potential hanging, if stdin is emptyMartin Czygan2020-03-091-0/+2
|
* more pubmed adjustmentsMartin Czygan2020-02-226-71/+197
| | | | | * regenerate map in continuous mode * add tests
* pubmed ftp: fix urlMartin Czygan2020-02-191-4/+6
|
* pubmed ftp harvest and KafkaBs4XmlPusherMartin Czygan2020-02-196-21/+307
| | | | | | | * add PubmedFTPWorker * utils are currently stored alongside pubmed (e.g. ftpretr, xmlstream) but may live elsewhere, as they are more generic * add KafkaBs4XmlPusher
* fatcat_ingest: as 'fatcat-ingest', not 'fatcat-ingest-container'Bryan Newbold2020-02-141-1/+1
| | | | This tool is more generic now.
* Merge branch 'martin-harvest-print' into 'master'bnewbold2020-02-143-17/+22
|\ | | | | | | | | harvest: log state on startup and use stderr for diagnostics See merge request webgroup/fatcat!27
| * harvest: log state on startup and use stderr for diagnosticsMartin Czygan2020-02-143-17/+22
|/
* add acceptlist override for biorxiv/medrxivBryan Newbold2020-02-101-2/+12
|
* check ingest_request_source existance for SPN as well as ingestBryan Newbold2020-02-061-0/+3
|
* additional trusted link sourcesBryan Newbold2020-02-061-0/+3
|
* add mag and s2 as trusted link sourcesBryan Newbold2020-02-061-1/+1
|
* ingest worker: handle missing ingest_request_sourceBryan Newbold2020-02-061-0/+3
| | | | | Seeing a bunch of these due to re-ingests not including this field because of an earlier persist bug.
* switch '!= None' to 'is not None'Bryan Newbold2020-02-041-3/+3
| | | | As reminded in code review, thanks Martin.
* allow-non-oa is a top-level flag, not sub-commandBryan Newbold2020-02-041-3/+0
|
* default to PMC ingest URLs over DOIBryan Newbold2020-02-041-4/+4
| | | | | | | For cases where there might be both PMC and DOI urls, do the europmc.org PMC ones over DOI option. May want to turn this into a config or command-line option in the future.
* ingest: add 'extid' and 'query' modes; filters; refactorBryan Newbold2020-02-041-38/+147
| | | | | | This is a large refactor of the ingest script. It adds a number of filtering options (for all modes), and new modes for free-form queries or limiting to specific external identifiers.
* Merge branch 'martin-datacite-strptime-none' into 'master'Martin Czygan2020-01-311-1/+1
|\ | | | | | | | | datacite: prevent none See merge request webgroup/fatcat!25
| * datacite: prevent noneMartin Czygan2020-01-311-1/+1
|/
* Merge branch 'martin-datacite-micropub' into 'master'Martin Czygan2020-01-314-25/+162
|\ | | | | | | | | datacite: improve date handling and minor tweak See merge request webgroup/fatcat!24
| * datacite: name shall not be NoneMartin Czygan2020-01-311-1/+1
| |
| * datacite: add exception for https://www.micropublication.org/Martin Czygan2020-01-312-1/+7
| |
| * datacite: do not skip records w/o dateMartin Czygan2020-01-311-2/+1
| |
| * datacite: improve docstringMartin Czygan2020-01-311-4/+4
| |
| * datacite: improve date handling and minor tweakMartin Czygan2020-01-304-21/+153
|/ | | | | | | | | | | | | Records from https://www.micropublication.org/ did not have a date in FC, although raw data contained date strings - they were not using the finer-grained "attributes.date" but "attributes.published" and/or "attributes.publicationYear". Support for those fields has been added, including a test case. During this test (#30) a processing gap for names became clear (author may have "given_name" and "surname", but no "name"). This bug has been fixed, too.
* fix KafkaError worker reporting for partition errorsBryan Newbold2020-01-293-3/+3
|
* additional DOI prefix filtersBryan Newbold2020-01-281-0/+8
| | | | From martin, thanks.
* increase kafka-pixy timeout to 25 secondsBryan Newbold2020-01-281-1/+1
|
* apply ingest request filtering in entity workerBryan Newbold2020-01-281-3/+34
| | | | | | | `ingest_oa_only` behavior, and other filters, now handled in the entity update worker, instead of in the transform function. Also add a DOI prefix blocklist feature.
* remove 'oa_only' feature from ingest transformBryan Newbold2020-01-282-15/+1
| | | | Refactoring to move this filter elsewhere
* more TODO/proposal cleanupBryan Newbold2020-01-224-10/+34
|
* more details on potential _edit table disk savingsBryan Newbold2020-01-221-3/+23
|
* proposal of ideas for reducing database sizeBryan Newbold2020-01-211-0/+154
|
* cleanup some of old TODO list into proposalsBryan Newbold2020-01-214-44/+269
|
* refactor fatcat_import kafka group namesBryan Newbold2020-01-211-13/+54
| | | | | | | | | | | | | My current understanding is that consumer group names should be one-to-one with topic names. I previously though offsets were stored on a {topic, group} key, but they seem to be mixed and having too many workers in the same group is bad. In particular, we don't want cross-talk or load between QA and prod. All these topics are caught up in prod, so deploying this change and restarting workers should be safe. This commit does not update the elasticsearch or entity updates workers.
* fix trivial typo in file importerBryan Newbold2020-01-201-1/+1
|
* stats: remove internal PG table sizes from old dumpsBryan Newbold2020-01-192-292/+0
| | | | For ease of reading and comparison
* update stats and table sizesBryan Newbold2020-01-194-0/+96
|
* Merge branch 'martin-openapi-typo-exmaple' into 'master'bnewbold2020-01-191-1/+1
|\ | | | | | | | | fix a typo in openapi definition See merge request webgroup/fatcat!20
| * fix a typo in openapi definitionMartin Czygan2020-01-181-1/+1
| |
* | Merge branch 'martin-guide-typos-sentance' into 'master'bnewbold2020-01-191-2/+2
|\ \ | | | | | | | | | | | | fix two typos in editing guide See merge request webgroup/fatcat!21
| * | fix two typos in editing guideMartin Czygan2020-01-181-2/+2
| |/
* | basic notes in bulk edit changelogBryan Newbold2020-01-191-0/+7
| |
* | bulk edit notes for datacite (QA)Bryan Newbold2020-01-191-0/+152
| |
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-26 02:30:07 +0000