summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* Update EntityImporter docstring.Martin Czygan2019-12-131-13/+4
| | | | I believe the required method is `parse_record`, not `parse`.
* Merge branch 'bnewbold-ingest-oa-container' into 'master'bnewbold2019-12-126-3/+181
|\ | | | | | | | | container-ingest tool See merge request webgroup/fatcat!8
| * container_issnl, not issnl, for ES release queryBryan Newbold2019-12-121-1/+1
| | | | | | | | Caught by Martin in review; Thanks!
| * improve argparse usageBryan Newbold2019-12-111-6/+4
| | | | | | | | | | | | | | | | | | | | --fatcat-api-url is clearer than --host-url remove unimplemented --debug (copy/paste from webface argparse) use formater which will display 'default' parameters with --help Thanks to Martin for pointing out the later, which i've always wanted!
| * simplify ES scroll deletion using param()Bryan Newbold2019-12-111-29/+29
| | | | | | | | | | | | | | | | | | | | | | This gets rid of some mess error handling code by properly configuring the elasticsearch client to just not clean up scroll iterators when accessing the public (prod or qa) search interfaces. Leaving the scroll state around isn't ideal, so we still delete them if possible (eg, connecting directly to elasticsearch). Thanks to Martin for pointing out this solution in review.
| * add ingest-container command (new CLI tool)Bryan Newbold2019-12-101-0/+136
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The intent of this tool is to make it easy to enque ingest requests into kafka, to be processed by a worker pool and eventually end up inserted into fatcat (for ingest hits that pass various checks). As a specific example use-case, we have pretty good coverage of eLife (a prominent OA publisher), but have missed some publications in the past, and have a large gap for the year 2019: https://fatcat.wiki/container/en4qj5ijrbf5djxx7p5zzpjyoq/coverage This tool would make it trivial to enqueue all the missing releases to be crawled. Future variants on this tool could query for, eg, long-tail OA works.
| * factor out some basic kafka helpersBryan Newbold2019-12-102-0/+23
| |
| * add another ingest request source to whitelistBryan Newbold2019-12-101-2/+5
| |
| * pipenv: add elasticsearch and elasticsearch-dsl librariesBryan Newbold2019-12-102-1/+19
| | | | | | | | | | These are low-level and high-level (respectively) client wrappers for elasticsearch
* | improve argparse usageBryan Newbold2019-12-1110-78/+95
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Use --fatcat-api-url instead of (ambiguous) --host-url for commands that aren't deployed/running via systemd. TODO: update the other --host-url usage, and either roll-out change consistently or support the old arg as an alias during cut-over Use argparse.ArgumentDefaultsHelpFormatter (thanks Martin!) Add help messages for all sub-commands, both as documentation and as a way to get argparse to print available commands in a more readable format.
* | add kafka-pixy to docker-compose fileBryan Newbold2019-12-101-0/+8
| |
* | tweaks to docker-compose imageBryan Newbold2019-12-101-0/+5
|/ | | | | - don't start kafka image until zookeeper is running - set very liberal "watermarks" for elasticsearch disk monitoring
* another schema update idea (containers)Bryan Newbold2019-12-091-0/+1
|
* fix delete release history viewBryan Newbold2019-12-091-1/+1
| | | | | | This was causing 5xx errors in production and qa. Eg, at: https://qa.fatcat.wiki/release/aaaaaaaaaaaaarceaaaaaaaaai/history
* regression test for deleted entity history viewBryan Newbold2019-12-091-0/+25
|
* add missing underline in deleted entity web viewBryan Newbold2019-12-091-1/+1
|
* Merge branch 'bnewbold-crossref-harvest-test' into 'master'Martin Czygan2019-12-095-22/+82
|\ | | | | | | | | Basic mocked test for crossref harvester See merge request webgroup/fatcat!7
| * add basic test for crossref harvest API callBryan Newbold2019-12-062-0/+46
| |
| * refactor kafka producer in crossref harvesterBryan Newbold2019-12-061-21/+26
| | | | | | | | | | | | | | | | producer creation/configuration should be happening in __init__() time, not 'daily' call. This specific refactor motivated by mocking out the producer in unit tests.
| * add pytest-mock helper library to dev depsBryan Newbold2019-12-062-1/+10
| |
* | Merge branch 'martin-increase-docker-kafka-message-size' into 'master'bnewbold2019-12-061-0/+1
|\ \ | |/ |/| | | | | increase max.message.bytes in container See merge request webgroup/fatcat!5
| * increase max.message.bytes in containerMartin Czygan2019-12-051-0/+1
|/ | | | | While working on datacite, some message were larger than the default of 1000012 bytes.
* improve previous commit (JATS abstract hack)Bryan Newbold2019-12-031-4/+6
|
* hack: remove enclosing JATS XML tags around abstractsBryan Newbold2019-12-031-1/+7
| | | | | | The more complete fix is to actually render the JATS to HTML and display that. This is just to fix a nit with the most common case of XML tags in abstracts.
* tweaks to file ingest importerBryan Newbold2019-12-032-3/+10
| | | | | - allow overriding source filter whitelist (common case for CLI use) - fix editgroup description env variable pass-through
* crossref is_update isn't what I thoughtBryan Newbold2019-12-031-6/+2
| | | | | | | | I thought this would filter for metadata updates to an existing DOI, but actually "updates" are a type of DOI (eg, a retraction). TODO: handle 'updates' field. Should both do a lookup and set work_ident appropriately, and store in crossref-specific metadata.
* bump required rust to 1.36Bryan Newbold2019-12-032-2/+2
| | | | | | | | | | | | This isn't a fatcat rust requirement, but instead a diesel requirement, via rust-smallvec, which in v1.0 uses the alloc crate: https://github.com/servo/rust-smallvec/issues/73 I think the reason this came up now is that diesel-cli is an application and doesn't have a Cargo.lock file, and the build was updated. Using some binary mechanism to install these dependencies would be more robust, but feels like a yak shave right now.
* update gitlab-ci to rust 1.34Bryan Newbold2019-12-031-1/+1
| | | | | Apparently the rust:1.34-stretch image is gone from docker hub, and this was causing CI errors.
* make file edit form hash values case insensitiveBryan Newbold2019-12-021-0/+3
| | | | | | | Test in previous commit. This fixes a user-reported 500 error when creating a file with SHA1/SHA256/MD5 hashes in upper-case.
* add regression test for upper-case SHA-1 form submitBryan Newbold2019-12-021-0/+10
|
* re-order ingest want() for better statsBryan Newbold2019-11-151-7/+10
|
* project -> ingest_request_sourceBryan Newbold2019-11-153-9/+9
|
* have ingest-file-results importer operate as crawl-botBryan Newbold2019-11-151-1/+1
| | | | As opposed to sandcrawler-bot
* fix release.pmcid typoBryan Newbold2019-11-151-2/+2
|
* better ingest-file-results import nameBryan Newbold2019-11-151-1/+1
|
* ingest importer fixesBryan Newbold2019-11-151-3/+4
|
* more ingest importer comments and countsBryan Newbold2019-11-152-2/+29
|
* crude support for 'sandcrawler' kafka topicsBryan Newbold2019-11-151-2/+3
|
* ingest file result importerBryan Newbold2019-11-155-2/+228
|
* test for ingest transformBryan Newbold2019-11-151-0/+57
|
* add ingest request feature to entity_updates workerBryan Newbold2019-11-152-4/+22
| | | | | | | | | | | | | Initially was going to create a new worker to consume from the release update channel, but couldn't get the edit context ("is this a new release, or update to an existing") from that context. Currently there is a flag in source code to control whether we only do OA releases or all releases. Starting with OA only to start slow, but should probably default to all, and make this a config flag. Should probably also have a config flag to control this entire feature. Tested locally in dev.
* add ingest request transform (and test)Bryan Newbold2019-11-153-1/+68
|
* update next schema tweaks proposal docBryan Newbold2019-11-151-0/+1
|
* Merge branch 'martin-search-results-pagination' into 'master'Martin Czygan2019-11-156-20/+82
|\ | | | | | | | | Add basic pagination to search results See merge request webgroup/fatcat!4
| * address test issueMartin Czygan2019-11-151-2/+3
| |
| * adjust search test case for new wordingMartin Czygan2019-11-141-2/+2
| | | | | | | | > "Showing top " -> "Showing first "
| * gray out inactive navigation linksMartin Czygan2019-11-141-2/+2
| | | | | | | | | | | | | | | | As per [this issue](https://github.com/Semantic-Org/Semantic-UI/issues/1885#issuecomment-77619519), text colors are not supported in semantic ui. To not move text too much, gray out inactive links.
| * move pagination into macrosMartin Czygan2019-11-143-43/+51
| | | | | | | | | | | | | | | | | | Two new macros: * top_results(found) * bottom_results(found) wip: move pagination into macro
| * Add basic pagination to search resultsMartin Czygan2019-11-084-14/+67
| | | | | | | | | | | | | | | | | | | | | | | | The "deep paging problem" imposes some limit, which currently is a hardcoded default value, `deep_page_limit=2000` in `do_search`. Elasticsearch can be configured, too: > Note that from + size can not be more than the index.max_result_window index setting, which defaults to 10,000. -- https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-body.html#request-body-search-from-size
* | web: catch MacaroonInitExceptionBryan Newbold2019-11-121-0/+4
| | | | | | | | | | Caught one of these in sentry. Probably due to a crawler? Or typing gibberish in the token form.
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-27 21:08:44 +0000