aboutsummaryrefslogtreecommitdiffstats
path: root/python
Commit message (Collapse)AuthorAgeFilesLines
* have ingest-file-results importer operate as crawl-botBryan Newbold2019-11-151-1/+1
| | | | As opposed to sandcrawler-bot
* fix release.pmcid typoBryan Newbold2019-11-151-2/+2
|
* better ingest-file-results import nameBryan Newbold2019-11-151-1/+1
|
* ingest importer fixesBryan Newbold2019-11-151-3/+4
|
* more ingest importer comments and countsBryan Newbold2019-11-152-2/+29
|
* crude support for 'sandcrawler' kafka topicsBryan Newbold2019-11-151-2/+3
|
* ingest file result importerBryan Newbold2019-11-155-2/+228
|
* test for ingest transformBryan Newbold2019-11-151-0/+57
|
* add ingest request feature to entity_updates workerBryan Newbold2019-11-152-4/+22
| | | | | | | | | | | | | Initially was going to create a new worker to consume from the release update channel, but couldn't get the edit context ("is this a new release, or update to an existing") from that context. Currently there is a flag in source code to control whether we only do OA releases or all releases. Starting with OA only to start slow, but should probably default to all, and make this a config flag. Should probably also have a config flag to control this entire feature. Tested locally in dev.
* add ingest request transform (and test)Bryan Newbold2019-11-153-1/+68
|
* Merge branch 'martin-search-results-pagination' into 'master'Martin Czygan2019-11-156-20/+82
|\ | | | | | | | | Add basic pagination to search results See merge request webgroup/fatcat!4
| * address test issueMartin Czygan2019-11-151-2/+3
| |
| * adjust search test case for new wordingMartin Czygan2019-11-141-2/+2
| | | | | | | | > "Showing top " -> "Showing first "
| * gray out inactive navigation linksMartin Czygan2019-11-141-2/+2
| | | | | | | | | | | | | | | | As per [this issue](https://github.com/Semantic-Org/Semantic-UI/issues/1885#issuecomment-77619519), text colors are not supported in semantic ui. To not move text too much, gray out inactive links.
| * move pagination into macrosMartin Czygan2019-11-143-43/+51
| | | | | | | | | | | | | | | | | | Two new macros: * top_results(found) * bottom_results(found) wip: move pagination into macro
| * Add basic pagination to search resultsMartin Czygan2019-11-084-14/+67
| | | | | | | | | | | | | | | | | | | | | | | | The "deep paging problem" imposes some limit, which currently is a hardcoded default value, `deep_page_limit=2000` in `do_search`. Elasticsearch can be configured, too: > Note that from + size can not be more than the index.max_result_window index setting, which defaults to 10,000. -- https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-body.html#request-body-search-from-size
* | web: catch MacaroonInitExceptionBryan Newbold2019-11-121-0/+4
| | | | | | | | | | Caught one of these in sentry. Probably due to a crawler? Or typing gibberish in the token form.
* | Merge branch 'martin-python-readme-es-note' into 'master'bnewbold2019-11-081-0/+5
|\ \ | | | | | | | | | | | | mention elasticsearch empty index setup See merge request webgroup/fatcat!3
| * | mention elasticsearch empty index setupMartin Czygan2019-11-081-0/+5
| |/ | | | | | | | | | | When setting up with the defaults, all works fine, except that the web search will try to access a local elasticsearch. Mention in README, how to create empty indices.
* | crossref: accurate blank title countsBryan Newbold2019-11-051-0/+1
| |
* | fix crossref component testBryan Newbold2019-11-041-1/+1
| |
* | crossref: component typeBryan Newbold2019-11-041-1/+3
| |
* | crossref: count why skip happenedBryan Newbold2019-11-041-1/+7
| | | | | | | | | | | | Might skip based on release type (eg container, not a paper/release), or missing title, or other reasons. Over 7 million DOIs are getting skipped, curious why.
* | crossref: don't skip on short/null subtitleBryan Newbold2019-11-041-1/+1
|/ | | | This was a bug. Should only set subtitle black, not skip the import.
* commit file cleaner testsBryan Newbold2019-10-081-0/+58
|
* file cleanup tweaks to actually runBryan Newbold2019-10-082-5/+4
|
* refactor duplicated b32_hex function in importersBryan Newbold2019-10-083-21/+11
|
* dict wrapper for entity_from_json()Bryan Newbold2019-10-082-3/+7
|
* new cleanup python tool/frameworkBryan Newbold2019-10-085-0/+300
|
* redirect direct entity underscore linksBryan Newbold2019-10-032-0/+30
|
* webface: extra <br> in container lookup linksBryan Newbold2019-09-211-1/+1
|
* remove duplicate style ref in container edit viewBryan Newbold2019-09-201-5/+0
|
* review/fix all confluent-kafka produce codeBryan Newbold2019-09-206-27/+75
|
* small fixes to confluent-kafka importers/workersBryan Newbold2019-09-208-26/+69
| | | | | | | | - decrease default changelog pipeline to 5.0sec - fix missing KafkaException harvester imports - more confluent-kafka tweaks - updates to kafka consumer configs - bump elastic updates consumergroup (again)
* update Pipfile.lock after confluent-kafka rebaseBryan Newbold2019-09-201-1/+33
|
* convert pipeline workers from pykafka to confluent-kafkaBryan Newbold2019-09-203-125/+230
|
* small kafka tweaks for robustnessBryan Newbold2019-09-202-0/+5
|
* convert importers to confluent-kafka libraryBryan Newbold2019-09-202-21/+74
|
* bump max message size to ~20 MBytesBryan Newbold2019-09-202-0/+2
|
* fixes to confluent-kafka harvestersBryan Newbold2019-09-203-20/+21
|
* first draft harvesters using confluent-kafkaBryan Newbold2019-09-203-48/+104
|
* make default kafka env 'dev', not 'qa'Bryan Newbold2019-09-202-4/+4
|
* add confluent-kafka library (to replace pykafka)Bryan Newbold2019-09-201-0/+1
|
* handle more external identifiers in pythonBryan Newbold2019-09-182-14/+101
| | | | | This makes it possible to, eg, past an arxiv identifier or SHA-1 hash in the general search box and do a quick lookup.
* webface: fix duration_seconds parsingBryan Newbold2019-09-181-1/+1
|
* add guide editing links to edit forms and signup messageBryan Newbold2019-09-185-5/+26
|
* python webface impl token generationBryan Newbold2019-09-184-1/+85
|
* slightly less annoying 'flash' message headerBryan Newbold2019-09-181-1/+1
|
* remove '@' from archive.org identBryan Newbold2019-09-171-1/+1
|
* IA auth: use itemname not screenname for usernameBryan Newbold2019-09-171-1/+1
| | | | | | | Have run in to several issues with IA screenname being invalid fatcat usernames (eg, containing whitespace). This probably won't catch all such issues, but hopefully most of them.
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-22 23:27:19 +0000