aboutsummaryrefslogtreecommitdiffstats
path: root/python/fatcat_tools
Commit message (Expand)AuthorAgeFilesLines
* Merge branch 'martin-kafka-bs4-import' into 'master'Martin Czygan2020-03-105-22/+318
|\
| * common: use smaller batch size since XML parsing may be slowMartin Czygan2020-03-101-1/+1
| * pubmed: log to stderrMartin Czygan2020-03-101-1/+1
| * pubmed: move mapping generation out of fetch_dateMartin Czygan2020-03-101-7/+8
| * harvest: fix imports from HarvestPubmedWorker cleanupMartin Czygan2020-03-101-2/+2
| * pubmed: citations is a bit more preciseMartin Czygan2020-03-091-1/+1
| * pubmed: we sync from FTPMartin Czygan2020-03-091-1/+1
| * oaipmh: HarvestPubmedWorker obsoleted by PubmedFTPWorkerMartin Czygan2020-03-091-34/+0
| * more pubmed adjustmentsMartin Czygan2020-02-222-70/+118
| * pubmed ftp: fix urlMartin Czygan2020-02-191-4/+6
| * pubmed ftp harvest and KafkaBs4XmlPusherMartin Czygan2020-02-194-1/+280
* | Merge branch 'bnewbold-elastic-v03b'Bryan Newbold2020-02-262-46/+198
|\ \
| * | improve is_oa flag accuracyBryan Newbold2020-02-261-8/+4
| * | ES container last tweaksBryan Newbold2020-02-261-0/+3
| * | ES release: last minor tweaksBryan Newbold2020-02-261-2/+2
| * | ES files: don't remove archive.org domains/hostsBryan Newbold2020-02-071-5/+0
| * | ES releases: host/domain fixesBryan Newbold2020-01-311-2/+2
| * | fix release es transform missing 'issue'Bryan Newbold2020-01-301-0/+1
| * | add upper-case work-around from kibana map joinBryan Newbold2020-01-301-0/+1
| * | tweak file ES archive.org domain trackingBryan Newbold2020-01-301-0/+6
| * | implement host+domain parsing for file ES transformBryan Newbold2020-01-301-9/+5
| * | fix ES file schema plural field namesBryan Newbold2020-01-291-4/+3
| * | elastic schema fixesBryan Newbold2020-01-291-0/+5
| * | add country to v03b release schemaBryan Newbold2020-01-291-0/+2
| * | actually implement changelog transformBryan Newbold2020-01-291-17/+45
| * | fix some transform bugs, add some testsBryan Newbold2020-01-291-6/+8
| * | ES release schema updatesBryan Newbold2020-01-291-5/+76
| * | container ES schema changesBryan Newbold2020-01-291-16/+18
| * | first implementation of ES file schemaBryan Newbold2020-01-292-1/+46
* | | Merge branch 'bnewbold-more-ingest' into 'master'bnewbold2020-02-251-1/+37
|\ \ \
| * | | entity worker: ingest more releasesBryan Newbold2020-02-221-1/+37
* | | | add some more domain/rel URL mappingsBryan Newbold2020-02-221-0/+9
|/ / /
* | | Merge branch 'bnewbold-shadow-import'Bryan Newbold2020-02-192-0/+196
|\ \ \
| * | | remove arabesque short wayback URL hackBryan Newbold2020-02-141-6/+0
| * | | improve shadow import file url cleanup pathBryan Newbold2020-02-131-2/+12
| * | | shadow import fixes from QA testingBryan Newbold2020-02-131-0/+6
| * | | shadow import: more filtering of file_meta fieldsBryan Newbold2020-02-131-0/+10
| * | | basic shadow importerBryan Newbold2020-02-132-0/+176
* | | | always crawl researchgate DOIsBryan Newbold2020-02-181-0/+2
* | | | ingest import: fix edit_extra pathBryan Newbold2020-02-181-1/+1
* | | | ingest importer: edit_extra is a top-level keyBryan Newbold2020-02-181-1/+1
* | | | ingest import: allow short version of corpus namesBryan Newbold2020-02-181-0/+3
* | | | ingest importer: pass through link relBryan Newbold2020-02-181-1/+6
| |_|/ |/| |
* | | harvest: log state on startup and use stderr for diagnosticsMartin Czygan2020-02-143-17/+22
|/ /
* | add acceptlist override for biorxiv/medrxivBryan Newbold2020-02-101-2/+12
* | check ingest_request_source existance for SPN as well as ingestBryan Newbold2020-02-061-0/+3
* | additional trusted link sourcesBryan Newbold2020-02-061-0/+3
* | add mag and s2 as trusted link sourcesBryan Newbold2020-02-061-1/+1
* | ingest worker: handle missing ingest_request_sourceBryan Newbold2020-02-061-0/+3
* | default to PMC ingest URLs over DOIBryan Newbold2020-02-041-4/+4
generated by cgit v1.2.3 (git 2.39.1) at 2024-05-28 18:05:32 +0000