fatcat/python/fatcat_tools/transforms, branch v0.3.3[no description]https://git.bnewbold.net/fatcat/atom?h=v0.3.32020-12-18T03:22:26+00:00bug fix: is_preserved should always be bool2020-12-18T03:22:26+00:00Bryan Newboldbnewbold@robocracy.org2020-12-18T03:22:26+00:00urn:sha1:28aa515a964421abba8cefd2f43ef2bf75fe47c5fix indentation2020-12-16T22:39:57+00:00Bryan Newboldbnewbold@robocracy.org2020-12-16T22:39:57+00:00urn:sha1:0174f7976e4cf5f288539e8a82ba07cc8f45f5c8have release elasticsearch transform count webcaptures and filesets towards preservation2020-12-16T22:34:28+00:00Bryan Newboldbnewbold@robocracy.org2020-12-16T22:34:26+00:00urn:sha1:486bbd7ea65fa50b3a839e5d371f04b8655a00c8
These are simple/partial changes to have webcaptures and filesets show
up in 'preservation', 'in_ia', and 'in_web' ES schema flags. A
longer-term TODO is to update the ES schema to have more granular
analytic flags.
Also includes a small generalization refactor for URL object parsing
into preservation status, shared across file+fileset+webcapture entity
types (all have similar URL objects with url+rel fields).
small release_to_elasticsearch refactors2020-12-16T19:31:22+00:00Bryan Newboldbnewbold@robocracy.org2020-12-16T19:29:45+00:00urn:sha1:532a25205f2cd2929c4258dee87bc6c53cd5cdc3
These should have almost no change in behavior, but improve code
quality.
The one behavior change is counting ftp URLs as "in_web"
refactor release_to_elasticsearch transform2020-12-16T19:24:55+00:00Bryan Newboldbnewbold@robocracy.org2020-12-16T19:24:53+00:00urn:sha1:d6ad61c28ddf5bd7dc57f9766ce57d5b48022d3e
This method was huge an monolithic. This commit splits out the content
and container specific sections into helper functions to make it more
managable. This involved refactoring to make many flags ("is_*" and
"in_*") part of the output dict through the entire code path, allowing
simple update() calls on the dict.
Noting that in the future should refactor to use a type-annotated class
for the elasticsearch output object. Perhaps something auto-generated
from the ES schema itself (JSON files).
if a release has DOAJ article id, count as OA2020-11-19T22:55:15+00:00Bryan Newboldbnewbold@robocracy.org2020-11-18T03:41:56+00:00urn:sha1:6b948b6b6d480940571de244c664efe420e05d50ingest tool: support for setting ingest type2020-11-07T03:16:31+00:00Bryan Newboldbnewbold@robocracy.org2020-11-07T03:16:31+00:00urn:sha1:b1b34d44ce1a416ee70be665b71b99ba9f98d9a3elastic transform: more preservation keepers2020-10-09T00:41:52+00:00Bryan Newboldbnewbold@robocracy.org2020-10-09T00:41:52+00:00urn:sha1:e3320e27999567ec9b687c25cc8040ff600496cdrelease ES transform tweaks2020-08-08T03:05:31+00:00Bryan Newboldbnewbold@robocracy.org2020-08-08T03:05:29+00:00urn:sha1:de0fb59f0e36d8079649feefb7592189d8f7c6ed
pass-through publisher_type from container extra metadata (ES field
already existed; this is from newer chocula metadata)
count arxiv and PMCID papers which haven't been crawled (by IA) as
"dark", not "bright"
basic toml transform helper2020-07-31T06:45:30+00:00Bryan Newboldbnewbold@robocracy.org2020-07-30T02:27:35+00:00urn:sha1:69e281deaf601b39e8ef51d603e3e5e16dc71777