sandcrawler - [no description]

	Commit message (Collapse)	Author	Age	Files	Lines
*	Merge branch 'bnewbold-match-quality'	Bryan Newbold	2018-08-24	12	-27/+2125
\|\ \| \| \| \| \| \| \| \| \| \|	Manually resolved merge conflict in: please
\| *	start tracking benchmark results	Bryan Newbold	2018-08-24	1	-0/+17
\| \|
\| *	rework fetch_hadoop script	Bryan Newbold	2018-08-24	5	-27/+45
\| \| \| \| \| \| \| \| \| \|	Should work on macOS now, and fetches hadoop in addition to pig. Still requires wget (not installed by default on macOS).
\| *	BibjsonScorable: fix ScorableFeatures (after rebase)	Bryan Newbold	2018-08-21	1	-1/+1
\| \|
\| *	please support for match-benchmark	Bryan Newbold	2018-08-21	1	-0/+26
\| \|
\| *	fix bug with qa/prod detection	Bryan Newbold	2018-08-21	1	-0/+1
\| \|
\| *	local bibjson-to-bibjson matching job (no tests)	Bryan Newbold	2018-08-21	1	-0/+29
\| \|
\| *	bibjson scorable class (no tests)	Bryan Newbold	2018-08-21	1	-0/+50
\| \|
\| *	add matching bibjson files	Bryan Newbold	2018-08-21	3	-0/+1956
\| \|
* \|	add counters to ScoreJob	Bryan Newbold	2018-08-24	1	-13/+47
\| \|
* \|	remove duplicate imports	Bryan Newbold	2018-08-24	1	-2/+0
\| \|
* \|	add a content-type filter for crossref works	Bryan Newbold	2018-08-23	2	-1/+26
\| \|
* \|	require crossref works to have at least one author (for matching)	Bryan Newbold	2018-08-23	2	-1/+7
\| \|
* \|	author parsing (and year, for crossref)	Bryan Newbold	2018-08-23	5	-7/+57
\| \|
* \|	set a minimum slug size (8 chars)	Bryan Newbold	2018-08-23	4	-15/+31
\| \|
* \|	clean up indendation in ScoreJob.scala	Bryan Newbold	2018-08-23	1	-5/+5
\| \|
* \|	clean up commented-out code in ScoreJob.scala	Bryan Newbold	2018-08-23	1	-30/+0
\| \|
* \|	increase MaxTitleLength from 255 to 1023	Bryan Newbold	2018-08-23	1	-1/+1
\| \| \| \| \| \| \| \| \| \| \| \|	Motivated after finding some long titles with MathML mixed in. Until this issue can be investigated further, bumping this limit to pass the handful of matches found.
* \|	additions to slug blacklist	Bryan Newbold	2018-08-23	1	-383/+414
\| \|
* \|	Merge branch 'bnewbold-extraction-tweaks'	Bryan Newbold	2018-08-23	1	-4/+7
\|\ \
\| * \|	extraction: do want content, not text	Bryan Newbold	2018-08-21	1	-1/+1
\| \| \| \| \| \| \| \| \| \| \| \|	XML can have non-unicode characters? Who knew.
\| * \|	extraction: status reporting tweaks	Bryan Newbold	2018-08-21	1	-5/+8
\| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	Improvements to how the extraction function in the extraction script reports status (in output, hbase, and counters)
* \| \|	Merge branch 'ellen-length-filtering' into 'master'	bnewbold	2018-08-23	6	-24/+138
\|\ \ \ \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	Filtering titles by length See merge request webgroup/sandcrawler!21
\| * \| \|	Fixed style violations.	Ellen Spertus	2018-08-22	2	-4/+5
\| \| \| \|
\| * \| \|	Added ScoreJob test for title-length filtering.	Ellen Spertus	2018-08-22	1	-5/+13
\| \| \| \|
\| * \| \|	Merge branch 'master' into ellen-length-filtering	Ellen Spertus	2018-08-22	3	-2/+6
\| \|\ \ \ \| \|/ / / \|/\| \| \|
* \| \| \|	Merge branch 'copyright' into 'master'	bnewbold	2018-08-22	1	-0/+4
\|\ \ \ \ \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	Copyright See merge request webgroup/sandcrawler!20
\| * \| \| \|	Created CONTRIBUTORS.	Ellen Spertus	2018-08-22	1	-0/+4
\| \| \| \| \|
* \| \| \| \|	add more punctuation characters to slug filter	Bryan Newbold	2018-08-22	2	-2/+2
\| \| \| \| \|
\| \| * \| \|	Added title-length filtering to CrossrefScorable.	Ellen Spertus	2018-08-22	2	-15/+71
\| \| \| \| \|
\| \| * \| \|	Added more tests of GrobidScorable.keepRecord	Ellen Spertus	2018-08-22	1	-0/+5
\| \| \| \| \|
\| \| * \| \|	Added title length filtering to GrobidScorable	Ellen Spertus	2018-08-22	3	-2/+46
\| \|/ / / \|/\| \| \|
* \| \| \|	expand slug-blacklist with results from prod GROBID/crossref match	Bryan Newbold	2018-08-21	1	-0/+393
\| \| \| \|
* \| \| \|	remove slug-blacklist conservative test	Bryan Newbold	2018-08-21	1	-16/+0
\| \| \| \|
* \| \| \|	Merge branch 'bnewbold-match-scale'	Bryan Newbold	2018-08-21	3	-5/+15
\|\ \ \ \
\| * \| \| \|	explicit spill and compression settings for ScoreJob	Bryan Newbold	2018-08-20	1	-0/+5
\| \| \| \| \|
\| * \| \| \|	add a trap to ScoreJob	Bryan Newbold	2018-08-20	2	-5/+10
\| \|/ / /
* \| \| \|	update spyglass patch/version (as an experiment)	Bryan Newbold	2018-08-21	1	-1/+1
\| \| \| \|
* \| \| \|	scalastyle	Bryan Newbold	2018-08-21	1	-9/+8
\| \| \| \|
* \| \| \|	HDFS doesn't like colons	Bryan Newbold	2018-08-21	1	-1/+1
\| \| \| \|
* \| \| \|	fix bugs/typos in HBaseColCountJob and HBaseStatusCountJob	Bryan Newbold	2018-08-21	4	-18/+11
\| \| \| \|
* \| \| \|	please support for status-code-count	Bryan Newbold	2018-08-21	1	-0/+24
\| \| \| \|
* \| \| \|	make col counter generic	Bryan Newbold	2018-08-21	2	-6/+35
\| \| \| \|
* \| \| \|	add dedicated job for counting GrobidMetadata column	Bryan Newbold	2018-08-21	1	-0/+36
\| \| \| \|
* \| \| \|	use grobid0:metadata, not tei_json	Bryan Newbold	2018-08-21	2	-10/+10
\| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \| \|	This is for efficiency. I had forgotten that the extract script actually writes this path!
* \| \| \|	distinction between status_code and status counting	Bryan Newbold	2018-08-21	4	-10/+111
\| \| \| \|
* \| \| \|	fold all scorable code into sanity check; counters	Bryan Newbold	2018-08-21	1	-7/+52
\| \| \| \|
* \| \| \|	please support for grobid-scorable-dump	Bryan Newbold	2018-08-21	1	-0/+24
\| \| \| \|
* \| \| \|	add GrobidScorableDumpJob and basic test	Bryan Newbold	2018-08-21	2	-0/+142
\| \|_\|/ \|/\| \|
* \| \|	Merge branch 'strings'	Bryan Newbold	2018-08-21	4	-9/+65
\|\ \ \ \| \|_\|/ \|/\| \| \| \| \| \| \| \| \| \| \|	Resolved conflicts in: scalding/src/main/scala/sandcrawler/ScorableFeatures.scala