summaryrefslogtreecommitdiffstats
path: root/python
Commit message (Collapse)AuthorAgeFilesLines
* ES schema: add best_url to file schemaBryan Newbold2020-06-041-0/+12
| | | | | | | | | This will increase index size (URLs are often long in our corpus, and we have many file entities), but seems worth it. Initially added `ia_url` as a second field, guaranteed to always be an *.archive.org URL, but `best_url` defaults to that anyways so didn't seem worthwhile.
* re-use 'best pdf url' for release green buttonBryan Newbold2020-06-041-2/+2
| | | | | | | I thought this was the existing behavior, but it looks like we were just taking the first link from the first file. In the future may refactor this out even further.
* Merge remote-tracking branch 'origin/martin-harvest-fail-on-400'Bryan Newbold2020-05-291-4/+0
|\ | | | | | | | | | | Manually resolved conflicts: python/fatcat_tools/harvest/doi_registrars.py
| * harvest: fail on HTTP 400Martin Czygan2020-05-291-4/+0
| | | | | | | | | | | | | | | | | | In the past harvest of datacite resulted in occasional HTTP 400. Meanwhile, various API bugs have been fixed (most recently: https://github.com/datacite/lupo/pull/537, https://github.com/datacite/datacite/issues/1038). Downside of ignoring this error was that state lives in kafka, which has limited support for deletion of arbitrary messages from a topic.
* | Merge branch 'martin-datacite-harvest-log-output' into 'master'Martin Czygan2020-05-291-1/+1
|\ \ | | | | | | | | | | | | harvest: log the failed url See merge request webgroup/fatcat!55
| * | harvest: log the failed urlMartin Czygan2020-05-291-1/+1
| |/
* / datacite: fix test docsMartin Czygan2020-05-291-3/+3
|/
* Merge branch 'bnewbold-ingest-stage' into 'master'Martin Czygan2020-05-283-7/+46
|\ | | | | | | | | verify release_stage in ingest importer See merge request webgroup/fatcat!52
| * ingest importer: check that stage is consistent with releaseBryan Newbold2020-05-261-0/+5
| |
| * regression test for release_stage mismatch with ingest requestBryan Newbold2020-05-262-7/+41
| |
* | rename HarvestState.next() to HarvestState.next_span()Bryan Newbold2020-05-265-7/+7
|/ | | | | | | | | "span" short for "timespan" to harvest; there may be a better name to use. Motivation for this is to work around a pylint erorr that .next() was not callable. This might be a bug with pylint, but .next() is also a very generic name.
* add a work-in-progress python MakefileBryan Newbold2020-05-261-0/+24
|
* pylintrc: skip many spurious WTForm no-member errorsBryan Newbold2020-05-261-0/+2
|
* HACK: try to squelch pylint in CIBryan Newbold2020-05-261-2/+2
| | | | | | | | | | | | | | | | | Gitlab CI is showing lint errors like: =================================== FAILURES =================================== 6316 _______________________ [pylint] tests/harvest_state.py ________________________ 6317 E: 19,11: hs.next is not callable (not-callable) 6318 E: 33,11: hs.next is not callable (not-callable) 6319 E: 19,11: hs.next is not callable (not-callable) [...] this is confusing as we use pipenv with a lock, so I should see the exact same errors locally. This commit is a hack to try and fix this and unbreak builds until we can debug further.
* HACK: skip pylint errors on lines that seem to be fineBryan Newbold2020-05-223-3/+3
| | | | | It seems to be an inadvertantly ugraded version of pylint saying that these lines are not-callable.
* pipenv: add flake8Bryan Newbold2020-05-222-183/+213
|
* Merge remote-tracking branch 'github/master'Bryan Newbold2020-05-222-7/+7
|\
| * Indentity is not the same this as equality in PythonChristian Clauss2020-05-141-2/+2
| |
| * Indentity is not the same this as equality in PythonChristian Clauss2020-05-141-5/+5
| |
* | importers: clarify handling of ApiExceptionBryan Newbold2020-05-223-4/+10
| | | | | | | | | | | | | | | | One of these (in ingest importer pipeline) is an actual bug, the others are just changing the syntax to be more explicit/conservative. The ingest importer bug seems to have resulted in some bad file match imports; scale of impact is unknown.
* | ingest importer: don't use glutton matchesBryan Newbold2020-05-221-3/+3
| | | | | | | | | | | | | | Until reviewing I didn't realize we were even doing this currently. Hopefluly has not impacted too many imports, as almost all ingests use an external identifer, so only those with identifers not in fatcat for whatever reason.
* | Merge branch 'martin-fix-container-empty-search' into 'master'Martin Czygan2020-04-291-0/+4
|\ \ | | | | | | | | | | | | search: assume * when q is not set or empty See merge request webgroup/fatcat!51
| * | search: assume * when q is not set or emptyMartin Czygan2020-04-291-0/+4
| | | | | | | | | | | | An example would be a blank search from a container details page.
* | | Merge branch 'bnewbold-search-tweaks' into 'master'bnewbold2020-04-273-92/+132
|\ \ \ | |/ / |/| | | | | | | | tweaks to search result pages See merge request webgroup/fatcat!50
| * | web search: tweak release search result styleBryan Newbold2020-04-231-25/+51
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is also back-ported from covid19.fatcat.wiki, though with some more tweaks on top. The changes are: - show original title if available (usually non-English) - move release_type label to title line suffix, and only show if not a "paper" - show publication status and withdrawl as text after the journal title, not as a label
| * | web search: improve indentation, fix missing div tagsBryan Newbold2020-04-232-67/+81
| |/ | | | | | | These are back-ported fixes from covid19.fatcat.wiki
* | Merge branch 'bnewbold-non-ident-fix' into 'master'Martin Czygan2020-04-243-6/+10
|\ \ | | | | | | | | | | | | | | | | | | fix ident=None broken links Closes #3 See merge request webgroup/fatcat!49
| * | web: fix ident=None broken linksBryan Newbold2020-04-233-6/+10
| |/ | | | | | | | | | | On web interface views for revisions, we had a bunch of broken links because the ident is "None". This commit fixes these by removing the links.
* | datacite: fix type errorMartin Czygan2020-04-224-2/+80
| | | | | | | | | | | | | | Up to now, we expected the description to be a string or list. Add handling for int as well. First appeared: Apr 22 19:58:39.
* | Merge branch 'martin-datacite-fix-release-contrib-raw-name-check-violation' ↵bnewbold2020-04-204-1/+86
|\ \ | | | | | | | | | | | | | | | | | | into 'master' datacite: fix a raw name constraint violation See merge request webgroup/fatcat!47
| * | datacite: fix a raw name constraint violationMartin Czygan2020-04-204-1/+86
| |/ | | | | | | | | | | | | It was possible that contribs got added which had no raw name. One example would be a name consisting of whitespace only. This fix adds a final check for this case.
* | more changelog ES fixesBryan Newbold2020-04-171-4/+6
| |
* | ES changelog worker: fixes for ident; fetch update from API if neededBryan Newbold2020-04-171-2/+9
|/ | | | | The API fetch update may be needed for old changelog entries in the kafka feed.
* Merge branch 'bnewbold-py37-cleanups' into 'master'bnewbold2020-04-177-64/+39
|\ | | | | | | | | py37 cleanups See merge request webgroup/fatcat!44
| * example of starting to use format stringsBryan Newbold2020-04-171-12/+12
| |
| * pytest: ignore remaining deprecation warnings in 3rd party librariesBryan Newbold2020-04-171-0/+2
| |
| * consistently use raw string prefix for regexBryan Newbold2020-04-173-7/+7
| |
| * pipenv: update deps for python 3.7Bryan Newbold2020-04-172-45/+18
| | | | | | | | | | We had some pre-3.6 work arounds. Also seems like a reasonable time to update all depdencies to most recent versions.
* | Merge branch 'martin-changelog-to-es' into 'master'bnewbold2020-04-173-3/+41
|\ \ | | | | | | | | | | | | derive changelog worker from release worker See merge request webgroup/fatcat!43
| * | derive changelog worker from release workerMartin Czygan2020-04-173-3/+41
| |/ | | | | | | | | Early versions of changelog entries may not have all the fields required for the current transform.
* | changelog: limit typesMartin Czygan2020-04-161-5/+1
| | | | | | | | | | No partial docs (e.g. abstract), too generic components and entries, not HTML blogs.
* | changelog: extend release_types considered documentsMartin Czygan2020-04-161-10/+19
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | according to release_rev.release_type, we have 29 values: fatcat_prod=# select release_type, count(release_type) from release_rev group by release_type; release_type | count -------------------+----------- abstract | 2264 article | 6371076 article-journal | 101083841 article-newspaper | 17062 book | 1676941 chapter | 13914854 component | 58990 dataset | 6860325 editorial | 133573 entry | 1628487 graphic | 1809471 interview | 19898 legal_case | 3581 legislation | 1626 letter | 275119 paper-conference | 6074669 peer_review | 30581 post | 245807 post-weblog | 135 report | 1010699 retraction | 1292 review-book | 96219 software | 316 song | 24027 speech | 4263 standard | 312364 stub | 1036813 thesis | 414397 | 0 (29 rows)
* update README and coveragerc for python3.7Bryan Newbold2020-04-132-5/+5
|
* pipenv: switch from python3.5 to python3.7Bryan Newbold2020-04-132-210/+204
| | | | Also updates dependencies.
* ingest: configurable ES indexv0.3.2Bryan Newbold2020-04-081-1/+4
|
* Merge branch 'bnewbold-pubmed-get_text' into 'master'bnewbold2020-04-014-39/+47
|\ | | | | | | | | beautifulsoup XML parsing: .string vs. .get_text() See merge request webgroup/fatcat!40
| * pubmed: use untranslated title if translated not availableBryan Newbold2020-04-011-0/+6
| | | | | | | | | | | | | | The primary motivation for this change is that fatcat *requires* a non-empty title for each release entity. Pubmed/Medline occasionally indexes just a VenacularTitle with no ArticleTitle for foreign publications, and currently those records don't end up in fatcat at all.
| * importers: replace newlines in get_text() stringsBryan Newbold2020-04-014-23/+25
| |
| * importers: more string/get_text swapsBryan Newbold2020-03-283-27/+27
| | | | | | | | See previous pubmed commit for details.
| * pubmed: bunch of .get_text() instead of .stringBryan Newbold2020-03-281-12/+12
| | | | | | | | | | | | | | | | | | | | | | Yikes! Apparently when a tag has child tags, .string will return None instead of all the strings. .get_text() returns all of it: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#get-text https://www.crummy.com/software/BeautifulSoup/bs4/doc/#string I've things like identifiers as .string, when we expect only a single string inside.
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-16 18:51:47 +0000