aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* Merge branch 'martin-kafka-bs4-import' into 'master'Martin Czygan2020-03-1010-43/+428
|\ | | | | | | | | pubmed and arxiv harvest preparations See merge request webgroup/fatcat!28
| * common: use smaller batch size since XML parsing may be slowMartin Czygan2020-03-101-1/+1
| | | | | | | | | | | | | | | | Address kafka tradeoff between long and short time-outs. Shorter time-outs would facilitate > consumer group re-balances and other consumer group state changes [...] in a reasonable human time-frame.
| * pubmed: log to stderrMartin Czygan2020-03-101-1/+1
| |
| * pubmed: move mapping generation out of fetch_dateMartin Czygan2020-03-102-7/+10
| | | | | | | | | | * fetch_date will fail on missing mapping * adjust tests (test will require access to pubmed ftp)
| * harvest: fix imports from HarvestPubmedWorker cleanupMartin Czygan2020-03-102-4/+4
| |
| * pubmed: citations is a bit more preciseMartin Czygan2020-03-091-1/+1
| | | | | | | | | | > Each day, NLM produces update files that include new, revised and deleted citations. -- ftp://ftp.ncbi.nlm.nih.gov/pubmed/updatefiles/README.txt
| * pubmed: we sync from FTPMartin Czygan2020-03-091-1/+1
| |
| * oaipmh: HarvestPubmedWorker obsoleted by PubmedFTPWorkerMartin Czygan2020-03-091-34/+0
| |
| * fatcat_import: address potential hanging, if stdin is emptyMartin Czygan2020-03-091-0/+2
| |
| * more pubmed adjustmentsMartin Czygan2020-02-226-71/+197
| | | | | | | | | | * regenerate map in continuous mode * add tests
| * pubmed ftp: fix urlMartin Czygan2020-02-191-4/+6
| |
| * pubmed ftp harvest and KafkaBs4XmlPusherMartin Czygan2020-02-196-21/+307
| | | | | | | | | | | | | | * add PubmedFTPWorker * utils are currently stored alongside pubmed (e.g. ftpretr, xmlstream) but may live elsewhere, as they are more generic * add KafkaBs4XmlPusher
* | add --force-crawl flag to ingest toolBryan Newbold2020-03-021-0/+5
| |
* | pipenv: lock authlib to less than v0.13; rebuild lock fileBryan Newbold2020-02-282-112/+109
| |
* | ES README: really need to limit to 1k esbulk batchesBryan Newbold2020-02-261-3/+3
| |
* | Merge branch 'bnewbold-elastic-v03b'Bryan Newbold2020-02-2616-257/+674
|\ \
| * | improve is_oa flag accuracyBryan Newbold2020-02-262-10/+6
| | | | | | | | | | | | | | | | | | Particularly, the ezb=green match seems mostly incorrect. Note that pmcid being assigned could still be in an embargo window?
| * | update ES transform READMEBryan Newbold2020-02-261-2/+3
| | | | | | | | | | | | | | | - smaller batch sizes to prevent esbulk errors - file transform/index
| * | fix fatcat_transform state filtersBryan Newbold2020-02-261-4/+4
| | |
| * | bulk ES transform: skip non-active entitiesBryan Newbold2020-02-261-0/+8
| | |
| * | ES container last tweaksBryan Newbold2020-02-262-3/+7
| | |
| * | ES release: last minor tweaksBryan Newbold2020-02-262-5/+7
| | |
| * | ES updates: fix tests to accept archive.org in host/domainBryan Newbold2020-02-141-2/+3
| | |
| * | release schema: do doc_value on DOIsBryan Newbold2020-02-131-1/+1
| | | | | | | | | | | | | | | | | | Because DOIs are pseudo-structured (prefix, and often structure within the publisher-controlled area), I suspect we will in fact be wanting to do analytics over these strings.
| * | ES files: don't remove archive.org domains/hostsBryan Newbold2020-02-071-5/+0
| | |
| * | ES release: actually do want doc_values for work_idBryan Newbold2020-02-051-1/+1
| | | | | | | | | | | | Eg, for fast "unique count"
| * | fix axiv/arxiv typo in release schemaBryan Newbold2020-02-041-1/+1
| | |
| * | ES release schema: fix typoBryan Newbold2020-01-311-1/+1
| | |
| * | ES releases: host/domain fixesBryan Newbold2020-01-312-2/+5
| | |
| * | pipenv: lock zipp version to work around python3.6 requirementBryan Newbold2020-01-302-7/+20
| | |
| * | fix release es transform missing 'issue'Bryan Newbold2020-01-301-0/+1
| | |
| * | fix json typos in changelog schemaBryan Newbold2020-01-301-2/+2
| | |
| * | add upper-case work-around from kibana map joinBryan Newbold2020-01-302-0/+2
| | |
| * | JSON typo in release mappingBryan Newbold2020-01-301-1/+0
| | |
| * | ES schemas: make keywords case-insensitive by defaultBryan Newbold2020-01-304-66/+115
| | | | | | | | | | | | But not applying asciifolding; don't see any need to do so?
| * | tweak file ES archive.org domain trackingBryan Newbold2020-01-302-0/+7
| | |
| * | implement host+domain parsing for file ES transformBryan Newbold2020-01-302-13/+8
| | |
| * | pipenv: add tldextract (url parser) and update depsBryan Newbold2020-01-302-136/+159
| | |
| * | fix ES file schema plural field namesBryan Newbold2020-01-292-5/+4
| | |
| * | new biblio-only general searchBryan Newbold2020-01-291-2/+2
| | | | | | | | | | | | The other fields are now "copy_to" the merged biblio field.
| * | elastic schema fixesBryan Newbold2020-01-293-7/+12
| | |
| * | add country to v03b release schemaBryan Newbold2020-01-292-0/+3
| | |
| * | update ES docs and proposalBryan Newbold2020-01-292-4/+6
| | |
| * | actually implement changelog transformBryan Newbold2020-01-293-19/+78
| | |
| * | fix some transform bugs, add some testsBryan Newbold2020-01-296-13/+48
| | |
| * | ES release schema updatesBryan Newbold2020-01-292-28/+122
| | |
| * | container ES schema changesBryan Newbold2020-01-292-29/+38
| | |
| * | first implementation of ES file schemaBryan Newbold2020-01-294-3/+115
| | | | | | | | | | | | | | | Includes a trivial test and transform, but not any workers or doc updates.
* | | Merge branch 'bnewbold-more-ingest' into 'master'bnewbold2020-02-251-1/+37
|\ \ \ | | | | | | | | | | | | | | | | entity worker: ingest more Datacite releases; filter some out See merge request webgroup/fatcat!29
| * | | entity worker: ingest more releasesBryan Newbold2020-02-221-1/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If release is a dataset or image, don't do a pdf ingest request. If release is a datacite DOI, and release_type is a "document", crawl regardless of is_oa detection. This is mostly to crawl repositories (institutional or subject).
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-25 02:03:34 +0000