TEM-131: Lens v2 - new datastore, new API #19

Merged

kalou merged 0 commits from refs/pull/19/head into master

2019-02-18 23:08:15 +00:00

kalou commented

2019-01-23 07:41:39 +00:00

(Migrated from git.kalou.net)

Opening this PR to allow continued input on the V2 prototype of Lens' internal search functionality.

TODO: write up PR description once closer to finalizing

Opening this PR to allow continued input on the V2 prototype of Lens' internal search functionality. TODO: write up PR description once closer to finalizing

kalou commented

2019-01-23 19:16:37 +00:00

(Migrated from git.kalou.net)

Glad we're switching over to a solid search engine. A note about badger since we're only using it for persistence, we can look into dumping the index onto disk, and adding that to IPFS to better handle replicating the dataset across our nodes?

kalou (Migrated from git.kalou.net) reviewed

2019-01-23 19:16:58 +00:00

kalou (Migrated from git.kalou.net) reviewed

2019-01-24 01:57:04 +00:00

kalou commented

2019-01-24 01:59:04 +00:00

(Migrated from git.kalou.net)

IPFS to better handle replicating the dataset across our nodes?

@postables I don't think we want to just replicate, since that would cause some redundancy and issues where nodes are out of sync. The riot repo advertises it as "distributed", so I think there's a way to do it properly - there's this issue, that someone replied to with a rather... unhelpful example. I'll keep looking into it

> IPFS to better handle replicating the dataset across our nodes? @postables I don't think we want to just replicate, since that would cause some redundancy and issues where nodes are out of sync. The `riot` repo advertises it as "distributed", so I think there's a way to do it properly - there's this [issue](https://github.com/go-ego/riot/issues/62), that someone replied to with a rather... unhelpful example. I'll keep looking into it

kalou (Migrated from git.kalou.net) reviewed

2019-02-01 03:42:09 +00:00

kalou commented

2019-02-01 03:45:14 +00:00

(Migrated from git.kalou.net)

@bobheadxi based on some quick look-through of the links you sent, I think it works by creating gprc workers on various machines, and anytime an operation comes it, it's synced across teh workers?

kalou (Migrated from git.kalou.net) reviewed

2019-02-01 21:39:03 +00:00

kalou commented

2019-02-18 23:04:31 +00:00

(Migrated from git.kalou.net)

Codecov Report

Merging #19 into master will increase coverage by 11.25%.
The diff coverage is 79.02%.

@@             Coverage Diff             @@
##           master      #19       +/-   ##
===========================================
+ Coverage   49.73%   60.99%   +11.25%     
===========================================
  Files          12       18        +6     
  Lines         752     1110      +358     
===========================================
+ Hits          374      677      +303     
- Misses        316      364       +48     
- Partials       62       69        +7

Impacted Files	Coverage Δ
server/run.go	`0% <0%> (ø)`
server/server.go	`23.46% <0%> (ø)`	⬆️
xtractor/planetary/service.go	`0% <0%> (ø)`	⬆️
engine/result.go	`100% <100%> (ø)`
engine/labels.go	`100% <100%> (ø)`
search/search.go	`31.66% <22.22%> (ø)`	⬆️
lens.go	`54.19% <54.54%> (+0.46%)`	⬆️
v2_util.go	`78.08% <78.08%> (ø)`
v2.go	`90.16% <90.16%> (ø)`
engine/engine.go	`96.74% <96.74%> (ø)`
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4fbb6d5...ca62194. Read the comment docs.

# [Codecov](https://codecov.io/gh/RTradeLtd/Lens/pull/19?src=pr&el=h1) Report > Merging [#19](https://codecov.io/gh/RTradeLtd/Lens/pull/19?src=pr&el=desc) into [master](https://codecov.io/gh/RTradeLtd/Lens/commit/4fbb6d5ec1c049dd1cedcac23ed6c28719b8cd3a?src=pr&el=desc) will **increase** coverage by `11.25%`. > The diff coverage is `79.02%`. [![Impacted file tree graph](https://codecov.io/gh/RTradeLtd/Lens/pull/19/graphs/tree.svg?width=650&token=nQgFgGoU0R&height=150&src=pr)](https://codecov.io/gh/RTradeLtd/Lens/pull/19?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master #19 +/- ## =========================================== + Coverage 49.73% 60.99% +11.25% =========================================== Files 12 18 +6 Lines 752 1110 +358 =========================================== + Hits 374 677 +303 - Misses 316 364 +48 - Partials 62 69 +7 ``` | [Impacted Files](https://codecov.io/gh/RTradeLtd/Lens/pull/19?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [server/run.go](https://codecov.io/gh/RTradeLtd/Lens/pull/19/diff?src=pr&el=tree#diff-c2VydmVyL3J1bi5nbw==) | `0% <0%> (ø)` | | | [server/server.go](https://codecov.io/gh/RTradeLtd/Lens/pull/19/diff?src=pr&el=tree#diff-c2VydmVyL3NlcnZlci5nbw==) | `23.46% <0%> (ø)` | :arrow_up: | | [xtractor/planetary/service.go](https://codecov.io/gh/RTradeLtd/Lens/pull/19/diff?src=pr&el=tree#diff-eHRyYWN0b3IvcGxhbmV0YXJ5L3NlcnZpY2UuZ28=) | `0% <0%> (ø)` | :arrow_up: | | [engine/result.go](https://codecov.io/gh/RTradeLtd/Lens/pull/19/diff?src=pr&el=tree#diff-ZW5naW5lL3Jlc3VsdC5nbw==) | `100% <100%> (ø)` | | | [engine/labels.go](https://codecov.io/gh/RTradeLtd/Lens/pull/19/diff?src=pr&el=tree#diff-ZW5naW5lL2xhYmVscy5nbw==) | `100% <100%> (ø)` | | | [search/search.go](https://codecov.io/gh/RTradeLtd/Lens/pull/19/diff?src=pr&el=tree#diff-c2VhcmNoL3NlYXJjaC5nbw==) | `31.66% <22.22%> (ø)` | :arrow_up: | | [lens.go](https://codecov.io/gh/RTradeLtd/Lens/pull/19/diff?src=pr&el=tree#diff-bGVucy5nbw==) | `54.19% <54.54%> (+0.46%)` | :arrow_up: | | [v2\_util.go](https://codecov.io/gh/RTradeLtd/Lens/pull/19/diff?src=pr&el=tree#diff-djJfdXRpbC5nbw==) | `78.08% <78.08%> (ø)` | | | [v2.go](https://codecov.io/gh/RTradeLtd/Lens/pull/19/diff?src=pr&el=tree#diff-djIuZ28=) | `90.16% <90.16%> (ø)` | | | [engine/engine.go](https://codecov.io/gh/RTradeLtd/Lens/pull/19/diff?src=pr&el=tree#diff-ZW5naW5lL2VuZ2luZS5nbw==) | `96.74% <96.74%> (ø)` | | | ... and [6 more](https://codecov.io/gh/RTradeLtd/Lens/pull/19/diff?src=pr&el=tree-more) | | ------ [Continue to review full report at Codecov](https://codecov.io/gh/RTradeLtd/Lens/pull/19?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/RTradeLtd/Lens/pull/19?src=pr&el=footer). Last update [4fbb6d5...ca62194](https://codecov.io/gh/RTradeLtd/Lens/pull/19?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).