kalou/LensRT
Watch
1
0
Fork
You've already forked LensRT
 0
Code Issues 2 Pull requests Projects Releases 4 Packages Wiki Activity Actions

TEM-129: OCR fallback for PDFs and images #11

Merged
kalou merged 0 commits from refs/pull/11/head into master 2018-12-02 03:56:22 +00:00
Conversation 11 Commits - Files changed -
kalou commented 2018-11-26 00:33:08 +00:00 (Migrated from git.kalou.net)
Copy link
  • Improved PDF indexing:
    • New PDF text extraction library
    • OCR fallback for PDFs with too little text
  • Image OCR fallback
  • Significant refactors
    • Mocks for classes dependent on tensorflow and ipfs
    • Dependency injection for some annoying dependencies
    • New Lens::Magnify() tests
    • Misc improvements
  • CI needs fixing
  • update integration tests
- [x] Improved PDF indexing: - [x] New PDF text extraction library - [x] OCR fallback for PDFs with too little text - [x] Image OCR fallback - [x] Significant refactors - [x] Mocks for classes dependent on tensorflow and ipfs - [x] Dependency injection for some annoying dependencies - [x] New `Lens::Magnify()` tests - [x] Misc improvements - [x] CI needs fixing - [x] update integration tests
kalou commented 2018-11-29 23:29:07 +00:00 (Migrated from git.kalou.net)
Copy link

Codecov Report

Merging #11 into master will increase coverage by 11.56%.
The diff coverage is 53.27%.

Impacted file tree graph

@@             Coverage Diff             @@
##           master      #11       +/-   ##
===========================================
+ Coverage   30.96%   42.53%   +11.56%     
===========================================
  Files           8       11        +3     
  Lines         578      710      +132     
===========================================
+ Hits          179      302      +123     
+ Misses        368      355       -13     
- Partials       31       53       +22
Impacted Files Coverage Δ
text/textrank.go 79.54% <ø> (ø)
server/server.go 0% <0%> (ø) ⬆️
analyzer/images/tensorflow.go 55.79% <100%> (+0.79%) ⬆️
logs/mock.go 100% <100%> (ø)
lens.go 41.04% <63.04%> (+41.04%) ⬆️
analyzer/ocr/ocr.go 63.75% <63.75%> (ø)
logs/logs.go 77.27% <77.27%> (ø)
search/search.go 30.72% <83.33%> (-1.32%) ⬇️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 983fe73...539e84c. Read the comment docs.

# [Codecov](https://codecov.io/gh/RTradeLtd/Lens/pull/11?src=pr&el=h1) Report > Merging [#11](https://codecov.io/gh/RTradeLtd/Lens/pull/11?src=pr&el=desc) into [master](https://codecov.io/gh/RTradeLtd/Lens/commit/983fe731173b11178d11cad652dbfe3a050bc76b?src=pr&el=desc) will **increase** coverage by `11.56%`. > The diff coverage is `53.27%`. [![Impacted file tree graph](https://codecov.io/gh/RTradeLtd/Lens/pull/11/graphs/tree.svg?width=650&token=nQgFgGoU0R&height=150&src=pr)](https://codecov.io/gh/RTradeLtd/Lens/pull/11?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master #11 +/- ## =========================================== + Coverage 30.96% 42.53% +11.56% =========================================== Files 8 11 +3 Lines 578 710 +132 =========================================== + Hits 179 302 +123 + Misses 368 355 -13 - Partials 31 53 +22 ``` | [Impacted Files](https://codecov.io/gh/RTradeLtd/Lens/pull/11?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [text/textrank.go](https://codecov.io/gh/RTradeLtd/Lens/pull/11/diff?src=pr&el=tree#diff-dGV4dC90ZXh0cmFuay5nbw==) | `79.54% <ø> (ø)` | | | [server/server.go](https://codecov.io/gh/RTradeLtd/Lens/pull/11/diff?src=pr&el=tree#diff-c2VydmVyL3NlcnZlci5nbw==) | `0% <0%> (ø)` | :arrow_up: | | [analyzer/images/tensorflow.go](https://codecov.io/gh/RTradeLtd/Lens/pull/11/diff?src=pr&el=tree#diff-YW5hbHl6ZXIvaW1hZ2VzL3RlbnNvcmZsb3cuZ28=) | `55.79% <100%> (+0.79%)` | :arrow_up: | | [logs/mock.go](https://codecov.io/gh/RTradeLtd/Lens/pull/11/diff?src=pr&el=tree#diff-bG9ncy9tb2NrLmdv) | `100% <100%> (ø)` | | | [lens.go](https://codecov.io/gh/RTradeLtd/Lens/pull/11/diff?src=pr&el=tree#diff-bGVucy5nbw==) | `41.04% <63.04%> (+41.04%)` | :arrow_up: | | [analyzer/ocr/ocr.go](https://codecov.io/gh/RTradeLtd/Lens/pull/11/diff?src=pr&el=tree#diff-YW5hbHl6ZXIvb2NyL29jci5nbw==) | `63.75% <63.75%> (ø)` | | | [logs/logs.go](https://codecov.io/gh/RTradeLtd/Lens/pull/11/diff?src=pr&el=tree#diff-bG9ncy9sb2dzLmdv) | `77.27% <77.27%> (ø)` | | | [search/search.go](https://codecov.io/gh/RTradeLtd/Lens/pull/11/diff?src=pr&el=tree#diff-c2VhcmNoL3NlYXJjaC5nbw==) | `30.72% <83.33%> (-1.32%)` | :arrow_down: | | ... and [2 more](https://codecov.io/gh/RTradeLtd/Lens/pull/11/diff?src=pr&el=tree-more) | | ------ [Continue to review full report at Codecov](https://codecov.io/gh/RTradeLtd/Lens/pull/11?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/RTradeLtd/Lens/pull/11?src=pr&el=footer). Last update [983fe73...539e84c](https://codecov.io/gh/RTradeLtd/Lens/pull/11?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
kalou (Migrated from git.kalou.net) reviewed 2018-11-29 23:53:39 +00:00
kalou (Migrated from git.kalou.net) reviewed 2018-11-29 23:55:05 +00:00
kalou commented 2018-11-30 01:13:09 +00:00 (Migrated from git.kalou.net)
Copy link

Getting an issue with the integration tests:

=== RUN   TestContentTypeDetect_Integration/pdf
=== RUN   TestContentTypeDetect_Integration/markdown
=== RUN   TestContentTypeDetect_Integration/jpg
--- FAIL: TestContentTypeDetect_Integration (0.16s)
    --- FAIL: TestContentTypeDetect_Integration/pdf (0.05s)
        lens_test.go:75: retrieving QmTbvUMmniE7wUP1ucbtC9s4ree7s8mSiQBt1c4odzKnY4
        lens_test.go:84: content type: application/pdf
        lens_test.go:86: wanted pdf, got application/pdf
    --- FAIL: TestContentTypeDetect_Integration/markdown (0.00s)
        lens_test.go:75: retrieving QmS5yadpmuu5hPz884XoRFnTTTKaTS4GmdJddd7maysznm
        lens_test.go:84: content type: text/html; charset=utf-8
        lens_test.go:86: wanted markdown, got text/html; charset=utf-8
    --- FAIL: TestContentTypeDetect_Integration/jpg (0.00s)
        lens_test.go:75: retrieving QmNWaM9vM4LUs8ZUHThAqC3hCHeQF8fYdJhLjJMwzJmzYS
        lens_test.go:84: content type: image/jpeg
        lens_test.go:86: wanted jpg, got image/jpeg
Getting an issue with the integration tests: ``` === RUN TestContentTypeDetect_Integration/pdf === RUN TestContentTypeDetect_Integration/markdown === RUN TestContentTypeDetect_Integration/jpg --- FAIL: TestContentTypeDetect_Integration (0.16s) --- FAIL: TestContentTypeDetect_Integration/pdf (0.05s) lens_test.go:75: retrieving QmTbvUMmniE7wUP1ucbtC9s4ree7s8mSiQBt1c4odzKnY4 lens_test.go:84: content type: application/pdf lens_test.go:86: wanted pdf, got application/pdf --- FAIL: TestContentTypeDetect_Integration/markdown (0.00s) lens_test.go:75: retrieving QmS5yadpmuu5hPz884XoRFnTTTKaTS4GmdJddd7maysznm lens_test.go:84: content type: text/html; charset=utf-8 lens_test.go:86: wanted markdown, got text/html; charset=utf-8 --- FAIL: TestContentTypeDetect_Integration/jpg (0.00s) lens_test.go:75: retrieving QmNWaM9vM4LUs8ZUHThAqC3hCHeQF8fYdJhLjJMwzJmzYS lens_test.go:84: content type: image/jpeg lens_test.go:86: wanted jpg, got image/jpeg ```
kalou (Migrated from git.kalou.net) reviewed 2018-11-30 19:09:52 +00:00
kalou commented 2018-11-30 19:28:44 +00:00 (Migrated from git.kalou.net)
Copy link

@postables i think i will remove those tests for now - they dont seem to really be doing anything other than check extraction and content-type check

plus the extraction doesnt work for me lol it just hangs forever :(

@postables i think i will remove those tests for now - they dont seem to really be doing anything other than check extraction and content-type check plus the extraction doesnt work for me lol it just hangs forever :(
kalou commented 2018-11-30 21:26:25 +00:00 (Migrated from git.kalou.net)
Copy link

@bobheadxi Sounds good.

@bobheadxi Sounds good.
kalou (Migrated from git.kalou.net) reviewed 2018-11-30 21:27:17 +00:00
kalou (Migrated from git.kalou.net) reviewed 2018-11-30 22:09:01 +00:00
kalou (Migrated from git.kalou.net) reviewed 2018-11-30 22:11:15 +00:00
kalou (Migrated from git.kalou.net) left a comment
Copy link
No description provided.
Well done :ok_hand:
kalou (Migrated from git.kalou.net) approved these changes 2018-12-02 03:50:52 +00:00
kalou (Migrated from git.kalou.net) left a comment
Copy link

AWESOME 🔥

AWESOME :fire:
Sign in to join this conversation.
Reviewers
No reviewers
kalou
Labels
Clear labels   
bug
Something isn't working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature or request

  
good first issue
Good for newcomers

  
help wanted
Extra attention is needed

  
invalid
This doesn't seem right

  
question
Further information is requested

  
status:review

   
status:wip

   
wontfix
This will not be worked on

No labels
bug
duplicate
enhancement
good first issue
help wanted
invalid
question
status:review
status:wip
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kalou/LensRT!11
No description provided.