summaryrefslogtreecommitdiffstats
path: root/python
Commit message (Collapse)AuthorAgeFilesLines
* complete parse_record docstringMartin Czygan2019-12-141-0/+6
|
* Update EntityImporter docstring.Martin Czygan2019-12-131-13/+4
| | | | I believe the required method is `parse_record`, not `parse`.
* Merge branch 'bnewbold-ingest-oa-container' into 'master'bnewbold2019-12-126-3/+181
|\ | | | | | | | | container-ingest tool See merge request webgroup/fatcat!8
| * container_issnl, not issnl, for ES release queryBryan Newbold2019-12-121-1/+1
| | | | | | | | Caught by Martin in review; Thanks!
| * improve argparse usageBryan Newbold2019-12-111-6/+4
| | | | | | | | | | | | | | | | | | | | --fatcat-api-url is clearer than --host-url remove unimplemented --debug (copy/paste from webface argparse) use formater which will display 'default' parameters with --help Thanks to Martin for pointing out the later, which i've always wanted!
| * simplify ES scroll deletion using param()Bryan Newbold2019-12-111-29/+29
| | | | | | | | | | | | | | | | | | | | | | This gets rid of some mess error handling code by properly configuring the elasticsearch client to just not clean up scroll iterators when accessing the public (prod or qa) search interfaces. Leaving the scroll state around isn't ideal, so we still delete them if possible (eg, connecting directly to elasticsearch). Thanks to Martin for pointing out this solution in review.
| * add ingest-container command (new CLI tool)Bryan Newbold2019-12-101-0/+136
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The intent of this tool is to make it easy to enque ingest requests into kafka, to be processed by a worker pool and eventually end up inserted into fatcat (for ingest hits that pass various checks). As a specific example use-case, we have pretty good coverage of eLife (a prominent OA publisher), but have missed some publications in the past, and have a large gap for the year 2019: https://fatcat.wiki/container/en4qj5ijrbf5djxx7p5zzpjyoq/coverage This tool would make it trivial to enqueue all the missing releases to be crawled. Future variants on this tool could query for, eg, long-tail OA works.
| * factor out some basic kafka helpersBryan Newbold2019-12-102-0/+23
| |
| * add another ingest request source to whitelistBryan Newbold2019-12-101-2/+5
| |
| * pipenv: add elasticsearch and elasticsearch-dsl librariesBryan Newbold2019-12-102-1/+19
| | | | | | | | | | These are low-level and high-level (respectively) client wrappers for elasticsearch
* | improve argparse usageBryan Newbold2019-12-1110-78/+95
|/ | | | | | | | | | | | | | Use --fatcat-api-url instead of (ambiguous) --host-url for commands that aren't deployed/running via systemd. TODO: update the other --host-url usage, and either roll-out change consistently or support the old arg as an alias during cut-over Use argparse.ArgumentDefaultsHelpFormatter (thanks Martin!) Add help messages for all sub-commands, both as documentation and as a way to get argparse to print available commands in a more readable format.
* fix delete release history viewBryan Newbold2019-12-091-1/+1
| | | | | | This was causing 5xx errors in production and qa. Eg, at: https://qa.fatcat.wiki/release/aaaaaaaaaaaaarceaaaaaaaaai/history
* regression test for deleted entity history viewBryan Newbold2019-12-091-0/+25
|
* add missing underline in deleted entity web viewBryan Newbold2019-12-091-1/+1
|
* add basic test for crossref harvest API callBryan Newbold2019-12-062-0/+46
|
* refactor kafka producer in crossref harvesterBryan Newbold2019-12-061-21/+26
| | | | | | | | producer creation/configuration should be happening in __init__() time, not 'daily' call. This specific refactor motivated by mocking out the producer in unit tests.
* add pytest-mock helper library to dev depsBryan Newbold2019-12-062-1/+10
|
* improve previous commit (JATS abstract hack)Bryan Newbold2019-12-031-4/+6
|
* hack: remove enclosing JATS XML tags around abstractsBryan Newbold2019-12-031-1/+7
| | | | | | The more complete fix is to actually render the JATS to HTML and display that. This is just to fix a nit with the most common case of XML tags in abstracts.
* tweaks to file ingest importerBryan Newbold2019-12-032-3/+10
| | | | | - allow overriding source filter whitelist (common case for CLI use) - fix editgroup description env variable pass-through
* crossref is_update isn't what I thoughtBryan Newbold2019-12-031-6/+2
| | | | | | | | I thought this would filter for metadata updates to an existing DOI, but actually "updates" are a type of DOI (eg, a retraction). TODO: handle 'updates' field. Should both do a lookup and set work_ident appropriately, and store in crossref-specific metadata.
* make file edit form hash values case insensitiveBryan Newbold2019-12-021-0/+3
| | | | | | | Test in previous commit. This fixes a user-reported 500 error when creating a file with SHA1/SHA256/MD5 hashes in upper-case.
* add regression test for upper-case SHA-1 form submitBryan Newbold2019-12-021-0/+10
|
* re-order ingest want() for better statsBryan Newbold2019-11-151-7/+10
|
* project -> ingest_request_sourceBryan Newbold2019-11-153-9/+9
|
* have ingest-file-results importer operate as crawl-botBryan Newbold2019-11-151-1/+1
| | | | As opposed to sandcrawler-bot
* fix release.pmcid typoBryan Newbold2019-11-151-2/+2
|
* better ingest-file-results import nameBryan Newbold2019-11-151-1/+1
|
* ingest importer fixesBryan Newbold2019-11-151-3/+4
|
* more ingest importer comments and countsBryan Newbold2019-11-152-2/+29
|
* crude support for 'sandcrawler' kafka topicsBryan Newbold2019-11-151-2/+3
|
* ingest file result importerBryan Newbold2019-11-155-2/+228
|
* test for ingest transformBryan Newbold2019-11-151-0/+57
|
* add ingest request feature to entity_updates workerBryan Newbold2019-11-152-4/+22
| | | | | | | | | | | | | Initially was going to create a new worker to consume from the release update channel, but couldn't get the edit context ("is this a new release, or update to an existing") from that context. Currently there is a flag in source code to control whether we only do OA releases or all releases. Starting with OA only to start slow, but should probably default to all, and make this a config flag. Should probably also have a config flag to control this entire feature. Tested locally in dev.
* add ingest request transform (and test)Bryan Newbold2019-11-153-1/+68
|
* Merge branch 'martin-search-results-pagination' into 'master'Martin Czygan2019-11-156-20/+82
|\ | | | | | | | | Add basic pagination to search results See merge request webgroup/fatcat!4
| * address test issueMartin Czygan2019-11-151-2/+3
| |
| * adjust search test case for new wordingMartin Czygan2019-11-141-2/+2
| | | | | | | | > "Showing top " -> "Showing first "
| * gray out inactive navigation linksMartin Czygan2019-11-141-2/+2
| | | | | | | | | | | | | | | | As per [this issue](https://github.com/Semantic-Org/Semantic-UI/issues/1885#issuecomment-77619519), text colors are not supported in semantic ui. To not move text too much, gray out inactive links.
| * move pagination into macrosMartin Czygan2019-11-143-43/+51
| | | | | | | | | | | | | | | | | | Two new macros: * top_results(found) * bottom_results(found) wip: move pagination into macro
| * Add basic pagination to search resultsMartin Czygan2019-11-084-14/+67
| | | | | | | | | | | | | | | | | | | | | | | | The "deep paging problem" imposes some limit, which currently is a hardcoded default value, `deep_page_limit=2000` in `do_search`. Elasticsearch can be configured, too: > Note that from + size can not be more than the index.max_result_window index setting, which defaults to 10,000. -- https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-body.html#request-body-search-from-size
* | web: catch MacaroonInitExceptionBryan Newbold2019-11-121-0/+4
| | | | | | | | | | Caught one of these in sentry. Probably due to a crawler? Or typing gibberish in the token form.
* | Merge branch 'martin-python-readme-es-note' into 'master'bnewbold2019-11-081-0/+5
|\ \ | | | | | | | | | | | | mention elasticsearch empty index setup See merge request webgroup/fatcat!3
| * | mention elasticsearch empty index setupMartin Czygan2019-11-081-0/+5
| |/ | | | | | | | | | | When setting up with the defaults, all works fine, except that the web search will try to access a local elasticsearch. Mention in README, how to create empty indices.
* | crossref: accurate blank title countsBryan Newbold2019-11-051-0/+1
| |
* | fix crossref component testBryan Newbold2019-11-041-1/+1
| |
* | crossref: component typeBryan Newbold2019-11-041-1/+3
| |
* | crossref: count why skip happenedBryan Newbold2019-11-041-1/+7
| | | | | | | | | | | | Might skip based on release type (eg container, not a paper/release), or missing title, or other reasons. Over 7 million DOIs are getting skipped, curious why.
* | crossref: don't skip on short/null subtitleBryan Newbold2019-11-041-1/+1
|/ | | | This was a bug. Should only set subtitle black, not skip the import.
* commit file cleaner testsBryan Newbold2019-10-081-0/+58
|
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-24 01:01:21 +0000