| Commit message (Collapse) | Author | Age | Files | Lines | |
|---|---|---|---|---|---|
| * | refactor and expand wall/block/cookie URL patterns | Bryan Newbold | 2021-09-03 | 1 | -0/+14 |
| | | |||||
| * | check for simple URL patterns that are usually paywalls or loginwalls | Bryan Newbold | 2020-08-11 | 1 | -0/+18 |
| | | |||||
| * | pdfextract support in ingest worker | Bryan Newbold | 2020-06-25 | 1 | -0/+7 |
| | | |||||
| * | ingest: add URL blocklist feature | Bryan Newbold | 2020-01-17 | 1 | -0/+17 |
| | | | | | And, temporarily, block zenodo and figshare. | ||||
| * | clarify ingest result schema and semantics | Bryan Newbold | 2020-01-15 | 1 | -3/+19 |
| | | |||||
| * | add postgrest checks to test mocks | Bryan Newbold | 2020-01-14 | 1 | -1/+9 |
| | | |||||
| * | tests: don't use localhost as a responses mock host | Bryan Newbold | 2020-01-14 | 1 | -2/+2 |
| | | |||||
| * | refactor ingest to a loop, allowing multiple hops | Bryan Newbold | 2020-01-09 | 1 | -2/+9 |
| | | |||||
| * | add ingest test file | Bryan Newbold | 2020-01-09 | 1 | -0/+120 |
| Forgot to commit earlier! | |||||
 |
index : sandcrawler | |
| [no description] |
| aboutsummaryrefslogtreecommitdiffstats |