aboutsummaryrefslogtreecommitdiffstats
path: root/python
Commit message (Collapse)AuthorAgeFilesLines
...
| * | implement host+domain parsing for file ES transformBryan Newbold2020-01-302-13/+8
| | |
| * | pipenv: add tldextract (url parser) and update depsBryan Newbold2020-01-302-136/+159
| | |
| * | fix ES file schema plural field namesBryan Newbold2020-01-292-5/+4
| | |
| * | new biblio-only general searchBryan Newbold2020-01-291-2/+2
| | | | | | | | | | | | The other fields are now "copy_to" the merged biblio field.
| * | elastic schema fixesBryan Newbold2020-01-291-0/+5
| | |
| * | add country to v03b release schemaBryan Newbold2020-01-291-0/+2
| | |
| * | actually implement changelog transformBryan Newbold2020-01-292-18/+68
| | |
| * | fix some transform bugs, add some testsBryan Newbold2020-01-296-13/+48
| | |
| * | ES release schema updatesBryan Newbold2020-01-291-5/+76
| | |
| * | container ES schema changesBryan Newbold2020-01-291-16/+18
| | |
| * | first implementation of ES file schemaBryan Newbold2020-01-293-3/+69
| | | | | | | | | | | | | | | Includes a trivial test and transform, but not any workers or doc updates.
* | | Merge branch 'bnewbold-more-ingest' into 'master'bnewbold2020-02-251-1/+37
|\ \ \ | | | | | | | | | | | | | | | | entity worker: ingest more Datacite releases; filter some out See merge request webgroup/fatcat!29
| * | | entity worker: ingest more releasesBryan Newbold2020-02-221-1/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If release is a dataset or image, don't do a pdf ingest request. If release is a datacite DOI, and release_type is a "document", crawl regardless of is_oa detection. This is mostly to crawl repositories (institutional or subject).
* | | | add some more domain/rel URL mappingsBryan Newbold2020-02-221-0/+9
|/ / /
* | | Merge branch 'bnewbold-shadow-import'Bryan Newbold2020-02-197-2/+289
|\ \ \
| * | | remove arabesque short wayback URL hackBryan Newbold2020-02-141-6/+0
| | | |
| * | | clarify shadow filter codeBryan Newbold2020-02-131-1/+1
| | | |
| * | | filter out shadows-only files in webfaceBryan Newbold2020-02-132-2/+7
| | | |
| * | | improve shadow import file url cleanup pathBryan Newbold2020-02-131-2/+12
| | | | | | | | | | | | | | | | Should probably be refactored out in to shared cleanup code.
| * | | shadow import fixes from QA testingBryan Newbold2020-02-132-1/+7
| | | |
| * | | shadow import: more filtering of file_meta fieldsBryan Newbold2020-02-133-18/+28
| | | |
| * | | basic shadow importerBryan Newbold2020-02-135-0/+262
| | | |
* | | | always crawl researchgate DOIsBryan Newbold2020-02-181-0/+2
| | | | | | | | | | | | | | | | Now that ingest is fixed
* | | | ingest import: fix edit_extra pathBryan Newbold2020-02-181-1/+1
| | | |
* | | | ingest importer: edit_extra is a top-level keyBryan Newbold2020-02-181-1/+1
| | | |
* | | | ingest import: allow short version of corpus namesBryan Newbold2020-02-181-0/+3
| | | |
* | | | ingest importer: pass through link relBryan Newbold2020-02-181-1/+6
| |_|/ |/| |
* | | fatcat_ingest: as 'fatcat-ingest', not 'fatcat-ingest-container'Bryan Newbold2020-02-141-1/+1
| | | | | | | | | | | | This tool is more generic now.
* | | harvest: log state on startup and use stderr for diagnosticsMartin Czygan2020-02-143-17/+22
|/ /
* | add acceptlist override for biorxiv/medrxivBryan Newbold2020-02-101-2/+12
| |
* | check ingest_request_source existance for SPN as well as ingestBryan Newbold2020-02-061-0/+3
| |
* | additional trusted link sourcesBryan Newbold2020-02-061-0/+3
| |
* | add mag and s2 as trusted link sourcesBryan Newbold2020-02-061-1/+1
| |
* | ingest worker: handle missing ingest_request_sourceBryan Newbold2020-02-061-0/+3
| | | | | | | | | | Seeing a bunch of these due to re-ingests not including this field because of an earlier persist bug.
* | switch '!= None' to 'is not None'Bryan Newbold2020-02-041-3/+3
| | | | | | | | As reminded in code review, thanks Martin.
* | allow-non-oa is a top-level flag, not sub-commandBryan Newbold2020-02-041-3/+0
| |
* | default to PMC ingest URLs over DOIBryan Newbold2020-02-041-4/+4
| | | | | | | | | | | | | | For cases where there might be both PMC and DOI urls, do the europmc.org PMC ones over DOI option. May want to turn this into a config or command-line option in the future.
* | ingest: add 'extid' and 'query' modes; filters; refactorBryan Newbold2020-02-041-38/+147
| | | | | | | | | | | | This is a large refactor of the ingest script. It adds a number of filtering options (for all modes), and new modes for free-form queries or limiting to specific external identifiers.
* | datacite: prevent noneMartin Czygan2020-01-311-1/+1
| |
* | datacite: name shall not be NoneMartin Czygan2020-01-311-1/+1
| |
* | datacite: add exception for https://www.micropublication.org/Martin Czygan2020-01-312-1/+7
| |
* | datacite: do not skip records w/o dateMartin Czygan2020-01-311-2/+1
| |
* | datacite: improve docstringMartin Czygan2020-01-311-4/+4
| |
* | datacite: improve date handling and minor tweakMartin Czygan2020-01-304-21/+153
|/ | | | | | | | | | | | | Records from https://www.micropublication.org/ did not have a date in FC, although raw data contained date strings - they were not using the finer-grained "attributes.date" but "attributes.published" and/or "attributes.publicationYear". Support for those fields has been added, including a test case. During this test (#30) a processing gap for names became clear (author may have "given_name" and "surname", but no "name"). This bug has been fixed, too.
* fix KafkaError worker reporting for partition errorsBryan Newbold2020-01-293-3/+3
|
* additional DOI prefix filtersBryan Newbold2020-01-281-0/+8
| | | | From martin, thanks.
* increase kafka-pixy timeout to 25 secondsBryan Newbold2020-01-281-1/+1
|
* apply ingest request filtering in entity workerBryan Newbold2020-01-281-3/+34
| | | | | | | `ingest_oa_only` behavior, and other filters, now handled in the entity update worker, instead of in the transform function. Also add a DOI prefix blocklist feature.
* remove 'oa_only' feature from ingest transformBryan Newbold2020-01-282-15/+1
| | | | Refactoring to move this filter elsewhere
* refactor fatcat_import kafka group namesBryan Newbold2020-01-211-13/+54
| | | | | | | | | | | | | My current understanding is that consumer group names should be one-to-one with topic names. I previously though offsets were stored on a {topic, group} key, but they seem to be mixed and having too many workers in the same group is bad. In particular, we don't want cross-talk or load between QA and prod. All these topics are caught up in prod, so deploying this change and restarting workers should be safe. This commit does not update the elasticsearch or entity updates workers.
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-23 21:55:14 +0000