readme.md

Dat awesome

A curated list of the Dat Project ecosystem.

Please read the contribution guidelines before contributing.

Want to learn more? Check out:

• docs.datproject.org
• dat whitepaper
• chat.datproject.org

Dat Applications

User facing applications for sharing, downloading, and managing dats.

• dat - command line interface for managing dat archives
• Dat Desktop - desktop app for managing dat archives
• Beaker Browser - a peer-to-peer browser with tools to create and host websites.

Community Applications

Projects built using Dat to share and transfer data. Open a PR to add your project here!

• sciencefair - The open source p2p desktop science library that puts users in control 🔬 📖 http://sciencefair-app.com
• hyperirc - bot that mirrors irc channels to a hypercore read-only log
• soundcloud-archiver - decentralized archives rescue music in case soundcloud shuts down
• hypervision - watch and broadcast peer-to-peer live video streams
• hypertweet - stream your twitter feed to a hypercore feed
• dat-photos-app - decentralized, peer-to-peer photo sharing app for beaker browser

Using Dat

Modules that help you build things on top of Dat:

High-Level APIs

High-level APIs that act as glue for many of the Dat modules:

• dat-node - Node module for creating Dat applications with distributed file systems.
• dat-js - A pure JavaScript browser-friendly api for using dat over webrtc
• pauls-dat-api - Library of functions that make working with dat / hyperdrive easier.
• node-dat-archive - node api that supports beaker browser DatArchive format, uses pauls-dat-api

Hosting & Dat Management

Tools for hosting Dats, managing sets of Dats, etc.

• hypercore-archiver - archiver peer that backs up multiple hypercore / hyperdrive feeds to disk
• hypercloud - p2p + ☁
• hashbase - hosting for the peer-to-peer web
• dat-now - publish live syncing and versioned websites, files or whatever to now.sh instantly

Utilities for hypercore-archiver:

• hypercore-archiver-bot - IRC bot that provides an interface to hypercore-archiver
• hypercore-archiver-ws - websocket server for hypercore-archiver

Dat project runs a registry at datproject.org. We use these tools to manage our dats:

• dat-registry-api - account registry api for dat archives with user accounts, uses township
• dat-registry-client - client for registry api for user registration, login and publishing

Managing & Aggregating Dats

• multidat - manage dat archives in multiple locations, uses a dat factory, based on multidrive
• multidrive - manage multiple hyperdrive archives located anywhere on the filesystem
• dat-pki - A public key infrastructure with many encryption utilities for Dat filesharing
• injestdb - decentralized table/records-based database with sql-like behaviour, uses node-dat-archive

Http Hosting

• hyperdrive-http - serve hyperdrive archives over http
• dathttpd - A Web server for Dat and HTTPS, with zero-config TLS.

Dat Link Utilties

Resolving, parsing, encoding dat links.

• dat-dns - issue dns lookups for dat archives using https requests to a target host
• dat-link-resolve - resolve dat url's, links to a dat key using common methods, uses dat-dns
• parse-dat-url - node's url.parse updated to support versioned dat url's
• dat-encoding - encoder and decoder that supports the dat url-scheme

Dat Utilities

Utilities to show information about an existing dat archive.

• dat-log - simple cli that lists the history of a dat archive
• dat-ls - simple cli that lists all the changes in a dat archive
• hyperhealth - monitor health of hyperdrive or dat archives, e.g. peer count and peer mirror %
• hyperdrive-network-speed - track upload and download speeds on hyperdrive archives

File Imports & Exports

• hyperdrive-import-files - import files and folders into a hyperdrive archive
• mirror-folder - Small module to mirror a folder to another folder. Supports live mode as well.
• hyperdrive-staging-area - staging area for uncommited writes that can sync to a hyperdrive archive
• hyperdrive-to-zip-stream - export hyperdrive archives as a zip files

Hypercore Tools

Tools for using hypercore feeds

• hyperpipe - simple cli to pipe and read files into live hypercore feeds

Dat Core Modules

Things we used to build Dat. Dat tools (CLI, Desktop, dat-node) are opinonated versions of hyperdrive that work well for using facing applications.

• hyperdrive - secure, decentralized peer-to-peer file system on top of hypercore
• hypercore - decentralized peer-to-peer append-only logs using hypercore protocol

CLI Utilities

Utilities used in our command line interface

• dat-doctor - diagnose networking problems for dat, comes bundled with dat-cli
• dat-ignore - check files against .datignore before adding to a dat archive
• dat-json - read and write dat.json files, uses toiletdb

Networking

• hyperdiscovery - join the p2p swarm for hypercore feeds, uses discovery-swarm
• discovery-swarm - discover and connect to peers, uses discovery-channel
• webrtc-swarm - create a swarm of p2p connections using webrtc and a signalhub
• dat-swarm-defaults - default configuration for dns and dht for use with discovery-swarm

Lower level networking modules

• discovery-channel - search discovery networks to find answering peers
• dns-discovery - discover peers using regular- and multicast-dns
• multicast-dns - low-level multicast-dns implementation in pure javascript
• bittorrent-dht - complete js implementation of DHT peer discovery protocol
• utp-native - utp protocol implementation, based on libutp native bindings
• signalhub - simple signalling server that can be used to coordinate handshaking with webrtc

Storage

• dat-storage - Dat specific storage provider for Hyperdrive
• dat-secret-storage - hyperdrive storage module for dat secret keys

Random Access

Dat relies on random access storage. Any of these modules can be used to provide the storage for a dat archive.

• abstract-random-access - base class for random access stores
• multi-random-access - combine multiple abstract-random-access stores into a single one

The following are specific implementations of abstract-random-access:

• random-access-file - continuous reading / writing to files using random offset and lengths
• random-access-memory - same as random-access-file but maintains data in memory
• random-access-page-files - backend writing to fixed size page files, useful for sparse data
• dat-http - http transport and storage provider for dat archives
• random-access-idb - random-access-compatible indexedDB storage layer

Other Related Dat Project Modules

More modules from Dat project that are related to Dat but may not be used currently.

• peer-network - create internet-accessible servers/clients listening on names, not hostnames
• hyperdht - DHT that supports peer discovery and distributed hole punching

Dat Project Organization Stuff

• datproject.org - datproject.org on github
• discussions - general dat project discussion, ideas, feature requests
• styleguide - styleguide and visual assets for the dat project
• dat-elements - reusable ui elements for dat-based apps, such as loader, sprite, icon
• dat-colors - css color definitions that match dat styleguide
• dat-icons - svg icon definitions that match dat styleguide
• dat.json specification - specification of the dat.json metadata format

Outdated

Modules that are currently outdated. We released a major breaking change (hypercore v6, hyperdrive v9) in May 2017. This change included major speed and storage improvements. These modules need to be updated to support the new API.

If you want to update one of these, we are happy to help you choose one depending on your interests and what may still be useful.

• dat.haus - dat + http + unix, the composable http api to the dat network
• hyperfeed - publish decentralized rss, atom or rdf feeds, based on hyperdrive and feed
• normcore - no-config distributed streams using hypercore
• github-to-hypercore - stream github event feeds into hypercore feeds, uses normcore
• hyperspark - decentralized data processing platform for dat archives, inspired by spark
• hypercore-index - linear asynchronous stateful indexing of a hypercore feed
• hyperdrive-protocol - message encoding used by hyperdrive
• hyperdrive-http-server - small cli to serve hyperdrive archives over http
• dat-publish - small cli to publish dat archives to a hyperdrive-http or dat-archiver server
• dat-push - small cli for pushing files to a dat-archiver or dat-publish server
• dat-backup - backup a dat archive as a single file to local storage and retain full history
• archiver-server - serve hypercore-archiver feeds over the dat network and http, uses discovery-swarm
• archiver-api - simple rest api for accessing hypercore-archiver peers
• hyperdrive-ln - create symbolic links between hyperdrive archives
• hyperdrive-multiwriter - present a bundle of hyperdrive archives together as a multi-writer view
• hyperdrive-named-archives - create hyperdrive archives that store and load link keys from names
• git-dat - git plugin to use dat archives as remotes for a git repository
• jawn - distributed version control for tabular data, based on hypercore
• dat-archiver - archiver peer that backs up dat archives, based on hypercore-archiver
• hyperdrive-stats - live and persistent statistics tracker for hyperdrive archives
• hyperdrive-stats-server - server for sending hypercore / hyperdrive stats over server-side events
• hyperdrive-stats-ui - html-based user interface to hypercore-stats-server
• zip-to-hyperdrive - import contents of a zip archive into a hyperdrive archive
• url-dat - import files from http url's into a hyperdrive archive, uses tar-dat
• tar-dat - stream tar files into a hyperdrive archive
• hyperdrive-duplicate - check if a file is a duplicate to a hyperdrive archive entry

License

To the extent possible under law, the Dat team, and contributors have waived all copyright and related or neighboring rights to this work.