| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| ... | |
| | |
|
| | |
|
| | |
|
| |\
| |
| |
| |
| | |
better download button links
See merge request webgroup/fatcat!57
|
| | |
| |
| |
| | |
Similar to recent change for release download pages.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
This will increase index size (URLs are often long in our corpus, and we
have many file entities), but seems worth it.
Initially added `ia_url` as a second field, guaranteed to always be an
*.archive.org URL, but `best_url` defaults to that anyways so didn't
seem worthwhile.
|
| | |
| |
| |
| |
| |
| |
| | |
I thought this was the existing behavior, but it looks like we were just
taking the first link from the first file.
In the future may refactor this out even further.
|
| |/ |
|
| |\
| |
| |
| |
| |
| | |
Manually resolved conflicts:
python/fatcat_tools/harvest/doi_registrars.py
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
In the past harvest of datacite resulted in occasional HTTP 400.
Meanwhile, various API bugs have been fixed (most recently:
https://github.com/datacite/lupo/pull/537,
https://github.com/datacite/datacite/issues/1038). Downside of ignoring
this error was that state lives in kafka, which has limited support for
deletion of arbitrary messages from a topic.
|
| |\ \
| | |
| | |
| | |
| | | |
harvest: log the failed url
See merge request webgroup/fatcat!55
|
| | |/ |
|
| |/ |
|
| |\
| |
| |
| |
| | |
verify release_stage in ingest importer
See merge request webgroup/fatcat!52
|
| | | |
|
| | | |
|
| |/
|
|
|
|
|
|
|
| |
"span" short for "timespan" to harvest; there may be a better name to
use.
Motivation for this is to work around a pylint erorr that .next() was
not callable. This might be a bug with pylint, but .next() is also a
very generic name.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Gitlab CI is showing lint errors like:
=================================== FAILURES ===================================
6316 _______________________ [pylint] tests/harvest_state.py ________________________
6317 E: 19,11: hs.next is not callable (not-callable)
6318 E: 33,11: hs.next is not callable (not-callable)
6319 E: 19,11: hs.next is not callable (not-callable)
[...]
this is confusing as we use pipenv with a lock, so I should see the
exact same errors locally.
This commit is a hack to try and fix this and unbreak builds until we
can debug further.
|
| |
|
|
|
| |
It seems to be an inadvertantly ugraded version of pylint saying that
these lines are not-callable.
|
| | |
|
| |\ |
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
One of these (in ingest importer pipeline) is an actual bug, the others
are just changing the syntax to be more explicit/conservative.
The ingest importer bug seems to have resulted in some bad file match
imports; scale of impact is unknown.
|
| | |
| |
| |
| |
| |
| |
| | |
Until reviewing I didn't realize we were even doing this currently.
Hopefluly has not impacted too many imports, as almost all ingests use
an external identifer, so only those with identifers not in fatcat for
whatever reason.
|
| |\ \
| | |
| | |
| | |
| | | |
search: assume * when q is not set or empty
See merge request webgroup/fatcat!51
|
| | | |
| | |
| | |
| | | |
An example would be a blank search from a container details page.
|
| |\ \ \
| |/ /
|/| |
| | |
| | | |
tweaks to search result pages
See merge request webgroup/fatcat!50
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
This is also back-ported from covid19.fatcat.wiki, though with some more
tweaks on top.
The changes are:
- show original title if available (usually non-English)
- move release_type label to title line suffix, and only show if not a
"paper"
- show publication status and withdrawl as text after the journal title,
not as a label
|
| | |/
| |
| |
| | |
These are back-ported fixes from covid19.fatcat.wiki
|
| |\ \
| | |
| | |
| | |
| | |
| | |
| | | |
fix ident=None broken links
Closes #3
See merge request webgroup/fatcat!49
|
| | |/
| |
| |
| |
| |
| | |
On web interface views for revisions, we had a bunch of broken links
because the ident is "None". This commit fixes these by removing the
links.
|
| | |
| |
| |
| |
| |
| |
| | |
Up to now, we expected the description to be a string or list. Add
handling for int as well.
First appeared: Apr 22 19:58:39.
|
| |\ \
| | |
| | |
| | |
| | |
| | |
| | | |
into 'master'
datacite: fix a raw name constraint violation
See merge request webgroup/fatcat!47
|
| | |/
| |
| |
| |
| |
| |
| | |
It was possible that contribs got added which had no raw name. One
example would be a name consisting of whitespace only.
This fix adds a final check for this case.
|
| | | |
|
| |/
|
|
|
| |
The API fetch update may be needed for old changelog entries in the
kafka feed.
|
| |\
| |
| |
| |
| | |
py37 cleanups
See merge request webgroup/fatcat!44
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| | |
We had some pre-3.6 work arounds. Also seems like a reasonable time to
update all depdencies to most recent versions.
|
| |\ \
| | |
| | |
| | |
| | | |
derive changelog worker from release worker
See merge request webgroup/fatcat!43
|
| | |/
| |
| |
| |
| | |
Early versions of changelog entries may not have all the fields
required for the current transform.
|
| | |
| |
| |
| |
| | |
No partial docs (e.g. abstract), too generic components and entries, not
HTML blogs.
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
according to release_rev.release_type, we have 29 values:
fatcat_prod=# select release_type, count(release_type) from release_rev group by release_type;
release_type | count
-------------------+-----------
abstract | 2264
article | 6371076
article-journal | 101083841
article-newspaper | 17062
book | 1676941
chapter | 13914854
component | 58990
dataset | 6860325
editorial | 133573
entry | 1628487
graphic | 1809471
interview | 19898
legal_case | 3581
legislation | 1626
letter | 275119
paper-conference | 6074669
peer_review | 30581
post | 245807
post-weblog | 135
report | 1010699
retraction | 1292
review-book | 96219
software | 316
song | 24027
speech | 4263
standard | 312364
stub | 1036813
thesis | 414397
| 0
(29 rows)
|
| | |
|
| |
|
|
| |
Also updates dependencies.
|
