summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* ES changelog worker: fixes for ident; fetch update from API if neededBryan Newbold2020-04-171-2/+9
| | | | | The API fetch update may be needed for old changelog entries in the kafka feed.
* Merge branch 'bnewbold-py37-cleanups' into 'master'bnewbold2020-04-177-64/+39
|\ | | | | | | | | py37 cleanups See merge request webgroup/fatcat!44
| * example of starting to use format stringsBryan Newbold2020-04-171-12/+12
| |
| * pytest: ignore remaining deprecation warnings in 3rd party librariesBryan Newbold2020-04-171-0/+2
| |
| * consistently use raw string prefix for regexBryan Newbold2020-04-173-7/+7
| |
| * pipenv: update deps for python 3.7Bryan Newbold2020-04-172-45/+18
| | | | | | | | | | We had some pre-3.6 work arounds. Also seems like a reasonable time to update all depdencies to most recent versions.
* | Merge branch 'martin-changelog-to-es' into 'master'bnewbold2020-04-173-3/+41
|\ \ | | | | | | | | | | | | derive changelog worker from release worker See merge request webgroup/fatcat!43
| * | derive changelog worker from release workerMartin Czygan2020-04-173-3/+41
| | | | | | | | | | | | | | | Early versions of changelog entries may not have all the fields required for the current transform.
* | | Merge branch 'martin-changelog-release-types' into 'master'bnewbold2020-04-171-12/+17
|\ \ \ | | | | | | | | | | | | | | | | changelog: extend release_types considered documents See merge request webgroup/fatcat!42
| * | | changelog: limit typesMartin Czygan2020-04-161-5/+1
| | | | | | | | | | | | | | | | | | | | No partial docs (e.g. abstract), too generic components and entries, not HTML blogs.
| * | | changelog: extend release_types considered documentsMartin Czygan2020-04-161-10/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | according to release_rev.release_type, we have 29 values: fatcat_prod=# select release_type, count(release_type) from release_rev group by release_type; release_type | count -------------------+----------- abstract | 2264 article | 6371076 article-journal | 101083841 article-newspaper | 17062 book | 1676941 chapter | 13914854 component | 58990 dataset | 6860325 editorial | 133573 entry | 1628487 graphic | 1809471 interview | 19898 legal_case | 3581 legislation | 1626 letter | 275119 paper-conference | 6074669 peer_review | 30581 post | 245807 post-weblog | 135 report | 1010699 retraction | 1292 review-book | 96219 software | 316 song | 24027 speech | 4263 standard | 312364 stub | 1036813 thesis | 414397 | 0 (29 rows)
* | | | update prod statsBryan Newbold2020-04-177-0/+149
| |_|/ |/| |
* | | CHANGELOG entry for python 3.7Bryan Newbold2020-04-171-0/+4
| | |
* | | retro-active v0.3.2 changelog updatesBryan Newbold2020-04-172-2/+41
| | |
* | | Add missing packages to Dockerfile and CI fileBryan Newbold2020-04-162-3/+3
| | |
* | | ci: don't re-build/install commands if existingBryan Newbold2020-04-161-2/+2
| | |
* | | ci: only build postgres feature for dieselBryan Newbold2020-04-161-1/+1
| | |
* | | test-base DockerfileBryan Newbold2020-04-162-0/+51
| | | | | | | | | | | | Used to create bnewbold/fatcat-test-base image
* | | ci: switch to fatcat-test-base Docker imageBryan Newbold2020-04-161-1/+1
| |/ |/| | | | | Goal is to speed up CI runs.
* | CI: add libpq-dev (for diesel build)Bryan Newbold2020-04-161-1/+1
|/ | | | Not sure why things build without this.
* get gitlab-ci working with python3.7Bryan Newbold2020-04-131-2/+2
| | | | | | Required updating to newer 'buster' Debian distro, and a newer rust release to work around a Docker/OCI containerization issue with older docker images.
* update README and coveragerc for python3.7Bryan Newbold2020-04-132-5/+5
|
* pipenv: switch from python3.5 to python3.7Bryan Newbold2020-04-132-210/+204
| | | | Also updates dependencies.
* ingest: configurable ES indexv0.3.2Bryan Newbold2020-04-081-1/+4
|
* update bulk export instructionsBryan Newbold2020-04-071-4/+2
| | | | | - don't do expanded and regular release dumps - default to sqldump_public for item name (as that is common-case)
* Merge branch 'bnewbold-pubmed-get_text' into 'master'bnewbold2020-04-014-39/+47
|\ | | | | | | | | beautifulsoup XML parsing: .string vs. .get_text() See merge request webgroup/fatcat!40
| * pubmed: use untranslated title if translated not availableBryan Newbold2020-04-011-0/+6
| | | | | | | | | | | | | | The primary motivation for this change is that fatcat *requires* a non-empty title for each release entity. Pubmed/Medline occasionally indexes just a VenacularTitle with no ArticleTitle for foreign publications, and currently those records don't end up in fatcat at all.
| * importers: replace newlines in get_text() stringsBryan Newbold2020-04-014-23/+25
| |
| * importers: more string/get_text swapsBryan Newbold2020-03-283-27/+27
| | | | | | | | See previous pubmed commit for details.
| * pubmed: bunch of .get_text() instead of .stringBryan Newbold2020-03-281-12/+12
| | | | | | | | | | | | | | | | | | | | | | Yikes! Apparently when a tag has child tags, .string will return None instead of all the strings. .get_text() returns all of it: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#get-text https://www.crummy.com/software/BeautifulSoup/bs4/doc/#string I've things like identifiers as .string, when we expect only a single string inside.
* | Merge branch 'bnewbold-match-proposal' into 'master'Martin Czygan2020-04-011-0/+430
|\ \ | | | | | | | | | | | | proposal: fuzzy matching See merge request webgroup/fatcat!39
| * | proposal: fuzzy matchingbnewbold2020-04-011-0/+430
|/ /
* | sql_dumps: stop doing redundant release dumpsBryan Newbold2020-04-011-1/+3
| |
* | Merge branch 'bnewbold-crossref-deposit' into 'master'bnewbold2020-04-012-4/+24
|\ \ | | | | | | | | | | | | change crossref harvest date field See merge request webgroup/fatcat!41
| * | crossref: switch from index-date to update-dateBryan Newbold2020-03-302-2/+2
| | | | | | | | | | | | | | | | | | This goes against what the API docs recommend, but we are currently far behind on updates and need to catch up. Other than what the docs say, this seems to be consistent with the behavior we want.
| * | crossref: longer comment about crossref API date fieldsBryan Newbold2020-03-301-2/+22
|/ /
* | web: small corrections to subtitle/original_nameBryan Newbold2020-03-282-2/+4
| |
* | Merge branch 'bnewbold-translation-view'Bryan Newbold2020-03-282-0/+7
|\ \ | |/ |/|
| * web: use more idiomatic type checkBryan Newbold2020-03-281-1/+1
| | | | | | | | Thanks to Martin for suggestion
| * web: display container original titlesBryan Newbold2020-03-261-0/+3
| |
| * web: copy extra.subtitle to subtitle for release viewBryan Newbold2020-03-261-0/+4
| | | | | | | | | | The release view will display subtitles, but it needs to be in the correct "location".
* | ingest: more DOI patterns to treat as OABryan Newbold2020-03-281-0/+26
| | | | | | | | | | | | | | These are journal/publisher patterns which we suspect to actually be OA based on the large quantity of papers that crawl successfully. The better long-term solution will be to flag containers in some way as OA (or "should crawl"), but this is a good short-term solution.
* | update CONTRIBUTORSBryan Newbold2020-03-271-2/+15
| | | | | | | | | | | | | | So far only updating "what was contributed" for past work, not recent or (potentially) ongoing contributions. Thank you everybody!
* | Merge pull request #53 from EdwardBetts/spellingbnewbold2020-03-2713-20/+20
|\ \ | | | | | | Correct spelling mistakes
| * | Correct spelling mistakesEdward Betts2020-03-2713-20/+20
| | |
* | | Merge branch 'bnewbold-400-bad-revisions' into 'master'Martin Czygan2020-03-262-2/+14
|\ \ \ | | | | | | | | | | | | | | | | catch ApiValueError in some generic API calls See merge request webgroup/fatcat!35
| * | | catch ApiValueError in some generic API callsBryan Newbold2020-03-252-2/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The motivation for this change is to handle bogus revision IDs in URLs, which were causing 500 errors not 400 errors. Eg: https://qa.fatcat.wiki/file/rev/5d5d5162-b676-4f0a-968f-e19dadeaf96e%2B2019-11-27%2B13:49:51%2B0%2B6 I have no idea where these URLs are actually coming from, but they should be 4xx not 5xx. Investigating made me realize there is a whole category of ApiValueError exceptions we were not catching and should have been.
* | | | Merge branch 'bnewbold-citeproc-fixes' into 'master'bnewbold2020-03-265-22/+72
|\ \ \ \ | |_|_|/ |/| | | | | | | | | | | improve citeproc/CSL web interface See merge request webgroup/fatcat!36
| * | | improve citeproc/CSL web interfaceBryan Newbold2020-03-255-22/+72
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This tries to show the citeproc (bibtext, MLA, CSL-JSON) options for more releases, and not show the links when they would break. The primary motivation here is to work around two exceptions being thrown in prod every day (according to sentry): KeyError: 'role' ValueError: CLS requries some surname (family name) I'm guessing these are mostly coming from crawlers following the citeproc links on release landing pages.
* | | | datacite: nameIdentifier corner caseBryan Newbold2020-03-261-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Works around a bug in production: AttributeError: 'NoneType' object has no attribute 'replace' (datacite.py:724) NOTE: there are no tests for this code path
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-27 22:30:47 +0000