aboutsummaryrefslogtreecommitdiffstats
path: root/python
Commit message (Collapse)AuthorAgeFilesLines
* add --force-crawl flag to ingest toolBryan Newbold2020-03-021-0/+5
|
* pipenv: lock authlib to less than v0.13; rebuild lock fileBryan Newbold2020-02-282-112/+109
|
* Merge branch 'bnewbold-elastic-v03b'Bryan Newbold2020-02-2610-190/+465
|\
| * improve is_oa flag accuracyBryan Newbold2020-02-261-8/+4
| | | | | | | | | | | | Particularly, the ezb=green match seems mostly incorrect. Note that pmcid being assigned could still be in an embargo window?
| * fix fatcat_transform state filtersBryan Newbold2020-02-261-4/+4
| |
| * bulk ES transform: skip non-active entitiesBryan Newbold2020-02-261-0/+8
| |
| * ES container last tweaksBryan Newbold2020-02-261-0/+3
| |
| * ES release: last minor tweaksBryan Newbold2020-02-261-2/+2
| |
| * ES updates: fix tests to accept archive.org in host/domainBryan Newbold2020-02-141-2/+3
| |
| * ES files: don't remove archive.org domains/hostsBryan Newbold2020-02-071-5/+0
| |
| * ES releases: host/domain fixesBryan Newbold2020-01-312-2/+5
| |
| * pipenv: lock zipp version to work around python3.6 requirementBryan Newbold2020-01-302-7/+20
| |
| * fix release es transform missing 'issue'Bryan Newbold2020-01-301-0/+1
| |
| * add upper-case work-around from kibana map joinBryan Newbold2020-01-301-0/+1
| |
| * tweak file ES archive.org domain trackingBryan Newbold2020-01-301-0/+6
| |
| * implement host+domain parsing for file ES transformBryan Newbold2020-01-302-13/+8
| |
| * pipenv: add tldextract (url parser) and update depsBryan Newbold2020-01-302-136/+159
| |
| * fix ES file schema plural field namesBryan Newbold2020-01-292-5/+4
| |
| * new biblio-only general searchBryan Newbold2020-01-291-2/+2
| | | | | | | | The other fields are now "copy_to" the merged biblio field.
| * elastic schema fixesBryan Newbold2020-01-291-0/+5
| |
| * add country to v03b release schemaBryan Newbold2020-01-291-0/+2
| |
| * actually implement changelog transformBryan Newbold2020-01-292-18/+68
| |
| * fix some transform bugs, add some testsBryan Newbold2020-01-296-13/+48
| |
| * ES release schema updatesBryan Newbold2020-01-291-5/+76
| |
| * container ES schema changesBryan Newbold2020-01-291-16/+18
| |
| * first implementation of ES file schemaBryan Newbold2020-01-293-3/+69
| | | | | | | | | | Includes a trivial test and transform, but not any workers or doc updates.
* | Merge branch 'bnewbold-more-ingest' into 'master'bnewbold2020-02-251-1/+37
|\ \ | | | | | | | | | | | | entity worker: ingest more Datacite releases; filter some out See merge request webgroup/fatcat!29
| * | entity worker: ingest more releasesBryan Newbold2020-02-221-1/+37
| | | | | | | | | | | | | | | | | | | | | | | | If release is a dataset or image, don't do a pdf ingest request. If release is a datacite DOI, and release_type is a "document", crawl regardless of is_oa detection. This is mostly to crawl repositories (institutional or subject).
* | | add some more domain/rel URL mappingsBryan Newbold2020-02-221-0/+9
|/ /
* | Merge branch 'bnewbold-shadow-import'Bryan Newbold2020-02-197-2/+289
|\ \
| * | remove arabesque short wayback URL hackBryan Newbold2020-02-141-6/+0
| | |
| * | clarify shadow filter codeBryan Newbold2020-02-131-1/+1
| | |
| * | filter out shadows-only files in webfaceBryan Newbold2020-02-132-2/+7
| | |
| * | improve shadow import file url cleanup pathBryan Newbold2020-02-131-2/+12
| | | | | | | | | | | | Should probably be refactored out in to shared cleanup code.
| * | shadow import fixes from QA testingBryan Newbold2020-02-132-1/+7
| | |
| * | shadow import: more filtering of file_meta fieldsBryan Newbold2020-02-133-18/+28
| | |
| * | basic shadow importerBryan Newbold2020-02-135-0/+262
| | |
* | | always crawl researchgate DOIsBryan Newbold2020-02-181-0/+2
| | | | | | | | | | | | Now that ingest is fixed
* | | ingest import: fix edit_extra pathBryan Newbold2020-02-181-1/+1
| | |
* | | ingest importer: edit_extra is a top-level keyBryan Newbold2020-02-181-1/+1
| | |
* | | ingest import: allow short version of corpus namesBryan Newbold2020-02-181-0/+3
| | |
* | | ingest importer: pass through link relBryan Newbold2020-02-181-1/+6
| | |
* | | fatcat_ingest: as 'fatcat-ingest', not 'fatcat-ingest-container'Bryan Newbold2020-02-141-1/+1
| | | | | | | | | | | | This tool is more generic now.
* | | harvest: log state on startup and use stderr for diagnosticsMartin Czygan2020-02-143-17/+22
|/ /
* | add acceptlist override for biorxiv/medrxivBryan Newbold2020-02-101-2/+12
| |
* | check ingest_request_source existance for SPN as well as ingestBryan Newbold2020-02-061-0/+3
| |
* | additional trusted link sourcesBryan Newbold2020-02-061-0/+3
| |
* | add mag and s2 as trusted link sourcesBryan Newbold2020-02-061-1/+1
| |
* | ingest worker: handle missing ingest_request_sourceBryan Newbold2020-02-061-0/+3
| | | | | | | | | | Seeing a bunch of these due to re-ingests not including this field because of an earlier persist bug.
* | switch '!= None' to 'is not None'Bryan Newbold2020-02-041-3/+3
| | | | | | | | As reminded in code review, thanks Martin.
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-27 01:08:08 +0000