aboutsummaryrefslogtreecommitdiffstats
path: root/skate
diff options
context:
space:
mode:
authorMartin Czygan <martin.czygan@gmail.com>2022-01-15 01:16:33 +0100
committerMartin Czygan <martin.czygan@gmail.com>2022-01-15 01:16:33 +0100
commit29c0e34c932f7d085152be6745c554786e1e786a (patch)
tree672da320f307b900a7a1e4b794b20d0486626b79 /skate
parent2f249d62905eb2c6a61703c6cddc317d071b124f (diff)
downloadrefcat-29c0e34c932f7d085152be6745c554786e1e786a.tar.gz
refcat-29c0e34c932f7d085152be6745c554786e1e786a.zip
skate: minor doc tweaks
Diffstat (limited to 'skate')
-rw-r--r--skate/README.md12
-rw-r--r--skate/map.go4
2 files changed, 8 insertions, 8 deletions
diff --git a/skate/README.md b/skate/README.md
index 40a863d..2a816f6 100644
--- a/skate/README.md
+++ b/skate/README.md
@@ -8,9 +8,11 @@ graph](https://en.wikipedia.org/wiki/Citation_graph).
## Background
-Python was a bit too slow, even when parallelized (with GNU parallel), e.g. for
-generating clusters of similar documents or to do verification. An option for
-the future would be to resort to [Cython](https://cython.org/). Parts of
+Python was a bit too slow, even when parallelized (with the great [GNU
+parallel](https://www.gnu.org/software/parallel/), e.g. for generating clusters
+of similar documents or to do verification. An option for the future would be
+to resort to [Cython](https://cython.org/) or
+[pyspark](https://spark.apache.org/docs/latest/api/python/index.html). Parts of
[fuzzycat](https://git.archive.org/webgroup/fuzzycat) has been ported into this
project for performance (and we saw a 25x speedup for certain tasks).
@@ -19,8 +21,8 @@ project for performance (and we saw a 25x speedup for certain tasks).
## Overview
We follow a map-reduce style approach (on a single machine): We extract
-specific keys from data. We group items (via sort) with the same *key* together
-and apply some computation on these groups.
+specific **keys** from data. We **group** items (via *sort*) with the same
+*key* together and **apply** a function over these groups.
Mapper is defined as function type, mapping a blob of data (e.g. a single JSON
object) to a number of fields (e.g. key, value).
diff --git a/skate/map.go b/skate/map.go
index 730726d..b9fedad 100644
--- a/skate/map.go
+++ b/skate/map.go
@@ -381,9 +381,7 @@ func MapperOpenLibraryReleaseNormalizedISBN(p []byte) (fields [][]byte, err erro
}
func MapperCdxSummary(p []byte) (fields [][]byte, err error) {
- var (
- cdx cdxSummary
- )
+ var cdx cdxSummary
if err := json.Unmarshal(p, &cdx); err != nil {
return nil, err
}