- Go 99.4%
- Dockerfile 0.4%
- Makefile 0.1%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
|
|
||
| .github | ||
| commands | ||
| components | ||
| config | ||
| docs | ||
| instr | ||
| scripts | ||
| types | ||
| utils | ||
| .codeclimate.yml | ||
| .dockerignore | ||
| .editorconfig | ||
| .gitignore | ||
| .gitlab-ci.yml | ||
| CONTRIBUTING.md | ||
| docker-compose.yml | ||
| Dockerfile | ||
| go.mod | ||
| go.sum | ||
| LICENSE | ||
| main.go | ||
| main_test.go | ||
| Makefile | ||
| README.md | ||
| readthedocs.yaml | ||
ipfs-search.com
Search engine for the Interplanetary Filesystem. Sniffs the DHT gossip and indexes file and directory hashes.
Metadata and contents are extracted using ipfs-tika, searching is done using OpenSearch, queueing is done using RabbitMQ. The crawler is implemented in Go, the API and frontend are built using Node.js.
The ipfs-search command consists of two components: the crawler and the sniffer. The sniffer extracts hashes from the gossip between nodes. The crawler extracts data from the hashes and indexes them.
Docs
Documentation is hosted on Read the Docs, based on files contained in the docs folder. In addition, there's extensive Go docs for the internal API as well as SwaggerHub OpenAPI documentation for the REST API.
Contact
Please find us on our Freenode/Riot/Matrix channel #ipfs-search:matrix.org.
Snapshots
ipfs-search provides the daily snapshot for all of the indexed data using snapshots. To learn more about downloading and restoring snapshots please refer to the relevant section in our documentation.
Related repo's
Contributors wanted
Building a search engine like this takes a considerable amount of resources (money and TLC). If you are able to help out with either of them, do reach out (see the contact section in this file).
Please read the Contributing.md file before contributing.
Roadmap
For discussing and suggesting features, look at the issues.
External dependencies
- Go 1.19
- OpenSearch 2.3.x
- RabbitMQ / AMQP server
- NodeJS 9.x
- IPFS 0.7
Maintenance note (toolchain staleness): The pinned toolchain is from 2022 and several components are past end-of-life as of this writing: Go 1.19 (EOL Aug 2023),
ipfs/kubo:v0.15.0(2022), andopensearchproject/opensearch:2.3.0(2022).go.modalso usesreplacedirectives pointing atipfs-search/forks oftestifyandradix— review those forks before relying on them. A version bump is a separate effort from the code-review fixes in this branch; this note records the current state for operators.
- Redis
Internal dependencies
Building
$ go get ./...
$ make
Running
Docker
The most convenient way to run the crawler is through Docker. Simply run:
docker-compose up
This will start the crawler, the sniffer and all its dependencies. Hashes can also be queued for crawling manually by running ipfs-search a <hash> from within the running container. For example:
docker-compose exec ipfs-crawler ipfs-search add QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv
Ansible deployment
Automated deployment can be done on any (virtual) Ubuntu 16.04 machine. The full production stack is automated and can be found in it's own repository.
Contributors
This project exists thanks to all the people who contribute.
Backers
Thank you to all our backers! 🙏 [Become a backer]
Sponsors
ipfs-search is supported by NLNet through the EU's Next Generation Internet (NGI0) programme.
RedPencil is supporting the hosting of ipfs-search.com.
Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]
