Node module for creating dat compatible tools on file systems

Dat Project

Last update: Nov 18, 2022

Related tags

Overview

dat-node

dat-node is a high-level module for building Dat applications on the file system.

For a lower-level API for building your own applications, use the Dat SDK which works in Node and the Web

Compatibility

Note: Version 4 of dat-node is not compatible with earlier versions (3.5.15 and below).

Dat Project Documentation & Resources

Features

High-level glue for common dat:// and hyperdrive modules.
Sane defaults and consistent management of storage & secret keys across applications, using dat-storage.
Easily connect to the dat:// network with holepunching, using hyperswarm
Import files from the file system, using mirror-folder
Serve dats over http with hyperdrive-http
Access APIs to lower level modules with a single require!

Browser Support

Many of our dependencies work in the browser, but dat-node is tailored for file system applications. See dat-sdk if you want to build browser-friendly applications.

Example

To send files via dat:

Tell dat-node where the files are.
Import the files.
Share the files on the dat network! (And share the link)

var Dat = require('dat-node')

// 1. My files are in /joe/cat-pic-analysis
Dat('/joe/cat-pic-analysis', function (err, dat) {
  if (err) throw err

  // 2. Import the files
  dat.importFiles()

  // 3. Share the files on the network!
  dat.joinNetwork()
  // (And share the link)
  console.log('My Dat link is: dat://' + dat.key.toString('hex'))
})

These files are now available to share over the dat network via the key printed in the console.

To download the files, you can make another dat-node instance in a different folder. This time we also have three steps:

Tell dat where I want to download the files.
Tell dat what the link is.
Join the network and download!

var Dat = require('dat-node')

// 1. Tell Dat where to download the files
Dat('/download/cat-analysis', {
  // 2. Tell Dat what link I want
  key: '<dat-key>' // (a 64 character hash from above)
}, function (err, dat) {
  if (err) throw err

  // 3. Join the network & download (files are automatically downloaded)
  dat.joinNetwork()
})

That's it! By default, all files are automatically downloaded when you connect to the other users.

Dig into more use cases below and please let us know if you have questions! You can open a new issue or talk to nice humans in our chat room.

Example Applications

CLI: We use dat-node in the dat CLI.
Desktop: The Dat Desktop application manages multiple dat-node instances via dat-worker.
See the examples folder for a minimal share + download usage.
And more! Let us know if you have a neat dat-node application to add here.

Usage

All dat-node applications have a similar structure around three main elements:

Storage - where the files and metadata are stored.
Network - connecting to other users to upload or download data.
Adding Files - adding files from the file system to the hyperdrive archive.

We'll go through what these are for and a few of the common usages of each element.

Storage

Every dat archive has storage, this is the required first argument for dat-node. By default, we use dat-storage which stores the secret key in ~/.dat/ and the rest of the data in dir/.dat. Other common options are:

Persistent storage: Stored files in /my-dir and metadata in my-dir/.dat by passing /my-dir as the first argument.
Temporary Storage: Use the temp: true option to keep metadata stored in memory.

// Permanent Storage
Dat('/my-dir', function (err, dat) {
  // Do Dat Stuff
})

// Temporary Storage
Dat('/my-dir', {temp: true}, function (err, dat) {
  // Do Dat Stuff
})

Both of these will import files from /my-dir when doing dat.importFiles() but only the first will make a .dat folder and keep the metadata on disk.

The storage argument can also be passed through to hyperdrive for more advanced storage use cases.

Network

Dat is all about the network! You'll almost always want to join the network right after you create your Dat:

Dat('/my-dir', function (err, dat) {
  dat.joinNetwork()
  dat.network.on('connection', function () {
    console.log('I connected to someone!')
  })
})

Downloading Files

Remember, if you are downloading - metadata and file downloads will happen automatically once you join the network!

dat runs on a peer to peer network, sometimes there may not be anyone online for a particular key. You can make your application more user friendly by using the callback in joinNetwork:

// Downloading <key> with joinNetwork callback
Dat('/my-dir', {key: '<key>'}, function (err, dat) {
  dat.joinNetwork(function (err) {
    if (err) throw err

    // After the first round of network checks, the callback is called
    // If no one is online, you can exit and let the user know.
    if (!dat.network.connected || !dat.network.connecting) {
      console.error('No users currently online for that key.')
      process.exit(1)
    }
  })
})

Download on Demand

If you want to control what files and metadata are downloaded, you can use the sparse option:

// Downloading <key> with sparse option
Dat('/my-dir', {key: '<key>', sparse: true}, function (err, dat) {
  dat.joinNetwork()

  // Manually download files via the hyperdrive API:
  dat.archive.readFile('/cat-locations.txt', function (err, content) {
    console.log(content) // prints cat-locations.txt file!
  })
})

Dat will only download metadata and content for the parts you request with sparse mode!

Importing Files

There are many ways to get files imported into an archive! Dat node provides a few basic methods. If you need more advanced imports, you can use the archive.createWriteStream() methods directly.

By default, just call dat.importFiles() to import from the directory you initialized with. You can watch that folder for changes by setting the watch option:

Dat('/my-data', function (err, dat) {
  if (err) throw err

  var progress = dat.importFiles({watch: true}) // with watch: true, there is no callback
  progress.on('put', function (src, dest) {
    console.log('Importing ', src.name, ' into archive')
  })
})

You can also import from another directory:

Dat('/my-data', function (err, dat) {
  if (err) throw err

  dat.importFiles('/another-dir', function (err) {
    console.log('done importing another-dir')
  })
})

That covers some of the common use cases, let us know if there are more to add! Keep reading for the full API docs.

API

`Dat(dir|storage, [opts], callback(err, dat))`

Initialize a Dat Archive in dir. If there is an existing Dat Archive, the archive will be resumed.

Storage

dir (Default) - Use dat-storage inside dir. This stores files as files, sleep files inside .dat, and the secret key in the user's home directory.
dir with opts.latest: false - Store as SLEEP files, including storing the content as a content.data file. This is useful for storing all history in a single flat file.
dir with opts.temp: true - Store everything in memory (including files).
storage function - pass a custom storage function along to hyperdrive, see dat-storage for an example.

Most options are passed directly to the module you're using (e.g. dat.importFiles(opts). However, there are also some initial opts can include:

opts = {
  key: '<dat-key>', // existing key to create archive with or resume
  temp: false, // Use random-access-memory as the storage.

  // Hyperdrive options
  sparse: false // download only files you request
}

The callback, cb(err, dat), includes a dat object that has the following properties:

dat.key: key of the dat (this will be set later for non-live archives)
dat.archive: Hyperdrive archive instance.
dat.path: Path of the Dat Archive
dat.live: archive.live
dat.writable: Is the archive writable?
dat.resumed: true if the archive was resumed from an existing database
dat.options: All options passed to Dat and the other submodules

Module Interfaces

dat-node provides an easy interface to common Dat modules for the created Dat Archive on the dat object provided in the callback:

`var network = dat.joinNetwork([opts], [cb])`

Join the network to start transferring data for dat.key, using discovery-swarm. You can also use dat.join([opts], [cb]).

If you specify cb, it will be called when the first round of discovery has completed. This is helpful to check immediately if peers are available and if not fail gracefully, more similar to http requests.

Returns a network object with properties:

network.connected - number of peers connected
network.on('listening') - emitted with network is listening
network.on('connection', connection, info) - Emitted when you connect to another peer. Info is an object that contains info about the connection

Network Options

opts are passed to discovery-swarm, which can include:

opts = {
  upload: true, // announce and upload data to other peers
  download: true, // download data from other peers
  port: 3282, // port for discovery swarm
  utp: true, // use utp in discovery swarm
  tcp: true // use tcp in discovery swarm
}

//Defaults from datland-swarm-defaults can also be overwritten:

opts = {
  dns: {
    server: // DNS server
    domain: // DNS domain
  }
  dht: {
    bootstrap: // distributed hash table bootstrapping nodes
  }
}

Returns a discovery-swarm instance.

`dat.leaveNetwork()` or `dat.leave()`

Leaves the network for the archive.

`var importer = dat.importFiles([src], [opts], [cb])`

Archive must be writable to import.

Import files to your Dat Archive from the directory using mirror-folder.

src - By default, files will be imported from the folder where the archive was initiated. Import files from another directory by specifying src.
opts - options passed to mirror-folder (see below).
cb - called when import is finished.

Returns a importer object with properties:

importer.on('error', err)
importer.on('put', src, dest) - file put started. src.live is true if file was added by file watch event.
importer.on('put-data', chunk) - chunk of file added
importer.on('put-end', src, dest) - end of file write stream
importer.on('del', dest) - file deleted from dest
importer.on('end') - Emits when mirror is done (not emitted in watch mode)
If opts.count is true:
- importer.on('count', {files, bytes}) - Emitted after initial scan of src directory. See import progress section for details.
- importer.count will be {files, bytes} to import after initial scan.
- importer.putDone will track {files, bytes} for imported files.

Importer Options

Options include:

var opts = {
  count: true, // do an initial dry run import for rendering progress
  ignoreHidden: true, // ignore hidden files  (if false, .dat will still be ignored)
  ignoreDirs: true, // do not import directories (hyperdrive does not need them and it pollutes metadata)
  useDatIgnore: true, // ignore entries in the `.datignore` file from import dir target.
  ignore: // (see below for default info) anymatch expression to ignore files
  watch: false, // watch files for changes & import on change (archive must be live)
}

Ignoring Files

You can use a .datignore file in the imported directory, src, to ignore any the user specifies. This is done by default.

dat-node uses dat-ignore to provide a default ignore option, ignoring the .dat folder and all hidden files or directories. Use opts.ignoreHidden = false to import hidden files or folders, except the .dat directory.

It's important that the .dat folder is not imported because it contains a private key that allows the owner to write to the archive.

`var stats = dat.trackStats()`

`stats.on('update')`

Emitted when archive stats are updated. Get new stats with stats.get().

`var st = stats.get()`

dat.trackStats() adds a stats object to dat. Get general archive stats for the latest version:

{
  files: 12,
  byteLength: 1234,
  length: 4, // number of blocks for latest files
  version: 6, // archive.version for these stats
  downloaded: 4 // number of downloaded blocks for latest
}

`stats.network`

Get upload and download speeds: stats.network.uploadSpeed or stats.network.downloadSpeed. Transfer speeds are tracked using hyperdrive-network-speed.

`var peers = stats.peers`

peers.total - total number of connected peers
peers.complete - connected peers with all the content data

`var server = dat.serveHttp(opts)`

Serve files over http via hyperdrive-http. Returns a node http server instance.

opts = {
  port: 8080, // http port
  live: true, // live update directory index listing
  footer: 'Served via Dat.', // Set a footer for the index listing
  exposeHeaders: false // expose dat key in headers
}

`dat.pause()`

Pause all upload & downloads. Currently, this is the same as dat.leaveNetwork(), which leaves the network and destroys the swarm. Discovery will happen again on resume().

`dat.resume()`

Resume network activity. Current, this is the same as dat.joinNetwork().

`dat.close(cb)`

Stops replication and closes all the things opened for dat-node, including:

dat.archive.close(cb)
dat.network.close(cb)
dat.importer.destroy() (file watcher)

License

MIT

Comments

[discussion] dat-js requirements and design
dat-js

Heya, I'm creating this issue to discuss the dat-js library API, because like @karissa pointed out, if we should make changes to the dat-js library API, now would be the right time. This is because currently only the API relies on library, and there are plans being drafted to add the website, desktop app, standalone http server and cluster too.

All text in this document is a best effort at consolidating my prior knowledge of Dat with the conversations I've had with Karissa over the past few days about Dat's internals. If anything in this document is off, I'd love to hear as the goal is here to create common understanding about internals, problems, goals and eventually agree on a solution. Here goes!

What is a dat?

While we were talking about Dat, Karissa and I noticed we couldn't quite define what a Dat is. Knowing what we define as "a Dat" is probably a good place to start. This is the (paraphrased) definition we ended up with:

A Dat is a combination of data and a registry containing pointers to the data. Usually this is a directory containing a leveldb (registry) and files (data). The directory containing both of these is what we commonly refer to as "a dat". But because "a dat" is a registry + data, other things can be "a dat" too. Examples are:

levelDB registry + S3 backend storing the files

SLEEP archive containing both data + registry

What do we want to do with a Dat?

When shaping the API, we probably want to cater to the common case first, and then provide convenient escape hatches for the more specialized needs so it can become the interface we base all our upstream consumers on without having to fall back to using the lower level libs directly.

I think the common case is having a single directory containing both files as the data, and a leveldb instance as the dat registry.

For the more advanced use cases it might make sense to instrument both hyperdrive and discovery-swarm directly, providing upstream consumers with direct access to limit the layers of abstraction we add with dat-js.

So what would this look like?

const level = require('level') const dat = require('dat-js') // start a dat from some dir dat('/path/to/my-dir', function (err, node) { if (err) throw err console.log('node is online') node.close() }) // create a dat in the local dir dat(process.cwd(), function (err, node) { if (err) throw err console.log('node is online') node.close() }) // upload a dat from the local dir dat({ some: 'option' }, function (err, node) { if (err) throw err node.share('somekey', function (err) { if (err) throw err node.close() }) }) // download a dat in the local dir dat({ some: 'option' }, function (err, node) { if (err) throw err node.download('somekey', function (err) { if (err) throw err node.close() }) }) // download and upload a dat in the local dir dat({ some: 'option' }, function (err, node) { if (err) throw err node.sync('somekey', function (err) { if (err) throw err node.close() }) }) // pass external db const raf = require('random-access-file') const db = level() const opts = { hyperdrive: { db: db, file: (name) => raf('my-download-dir/' + name) } } dat(opts, function (err, node) { if (err) throw err node.download('somekey', function (err) { if (err) throw err node.close({ db: false }, function (err) { if (err) throw err db.close() }) }) })

Wrapping up

And that's all. I hope this proves to be a somewhat useful starting point to start discussing the API. Feedback, suggestions and amendments are super welcome! Thanks for your time, and looking forward to any comments you have! :sparkles:

cc/ @joehand @juliangruber @karissa
opened by yoshuawuyts 16

Can't download content on Windows

I am reporting:

[X] a bug or unexpected behavior
[ ] general feedback
[ ] feature request
[ ] security issue

Bug Report

Please give us details about your installation to assist you. Run dat -v to see the version of Dat you are using.

Operating system: Win10 and Win7
Node Version: 8.15 and 10.15
dat-node Version: 3.5.15
dat cli Version: 13.13 (used for testing)

Expected behavior

Loaded dat address should download. (using example/download.js)

Actual behavior

Connects, finds peers, then doesn't download content. All this said, we know that version 3.5.11 works on Windows. This is some sort of regression.

Debug Logs

2019-04-24T02:10:19.183Z dat Dat DEBUG mode engaged, enabling quiet mode
2019-04-24T02:10:19.192Z dat dat 13.13.0
2019-04-24T02:10:19.192Z dat node v10.4.0
2019-04-24T02:10:19.510Z dat clone()
2019-04-24T02:10:19.512Z dat { _: null,
  utp: true,
  debug: false,
  quiet: true,
  sparse: false,
  upload: true,
  'show-key': false,
  k: false,
  logspeed: 400,
  empty: false,
  key: '<private>',
  dir: null,
  showKey: false }
2019-04-24T02:10:19.515Z dat Checking directory for clone
2019-04-24T02:10:19.516Z dat No existing directory, creating it.
2019-04-24T02:10:19.525Z dat-node archive ready. version: 0
2019-04-24T02:10:20.599Z dat archive syncing
2019-04-24T02:10:21.052Z dat-network Downloaded data: 284
2019-04-24T02:10:21.551Z dat-network Downloaded data: 284
2019-04-24T02:10:22.051Z dat-network Downloaded data: 284
2019-04-24T02:10:22.551Z dat-network Downloaded data: 284
2019-04-24T02:10:23.052Z dat-network Downloaded data: 284
2019-04-24T02:10:23.501Z dat-node Stream close

opened by ryuheiyokokawa 15

v4 roadmap
Work has started on dat-node v4 in #234. This issue will track the roadmap for the new version of dat-node. Here is where my priorities are:

continue focus on single-dat management to supoprt CLI

simple API for managing a single dat including: stats, importing, dat-json, global config management.

support async/await, drop node v4 support - update all dat dependencies to support promises

improve support for using multiple dat-node instances together in apps (e.g. using a single swarm for multiple dats).

Providing better visibility of other discovery options (using custom discovery services, etc.)

being type aware (also, required/optional) for options passed to classes

Would be curious to hear other folks blockers around dat-node and what they'd like to see in next version.

Thanks for discussion prompted by @RangerMauve for this.
opened by joehand 12

Can't install on windows

node -v v7.9.0 node-gyp -v v3.3.0

gyp ERR! node -v v7.9.0
gyp ERR! node-gyp -v v3.3.0
gyp ERR! not ok
npm WARN install:[email protected] [email protected] install: `node-gyp-build`
npm WARN install:[email protected] Exit status 1
npm WARN [email protected] No description
npm WARN [email protected] No repository field.
E:\Repositorys\dat-example> npm config set msvs_version 2013
E:\Repositorys\dat-example> npm install dat-node

> [email protected] install E:\Repositorys\dat-example\node_modules\utp-native
> node-gyp-build


E:\Repositorys\dat-example\node_modules\utp-native>if not defined npm_config_node_gyp (node "C:\Users\dustin\AppData\Roaming\npm\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node "" rebuild )
Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
  utp_internal.cpp
  utp_utils.cpp
  utp_hash.cpp
  utp_callbacks.cpp
e:\repositorys\dat-example\node_modules\utp-native\deps\libutp\utp_types.h(46): warning C4005: '_CRT_SECURE_NO_DEPRECATE' : macro redefinition (..\..\deps\libutp\utp_callbacks.cpp) [E:\Repositorys\dat-example\node_modules\utp-native\build\deps\libutp.vcxproj]
          command-line arguments :  see previous definition of '_CRT_SECURE_NO_DEPRECATE'
e:\repositorys\dat-example\node_modules\utp-native\deps\libutp\utp_types.h(46): warning C4005: '_CRT_SECURE_NO_DEPRECATE' : macro redefinition (..\..\deps\libutp\utp_hash.cpp) [E:\Repositorys\dat-example\node_modules\utp-native\build\deps\libutp.vcxproj]
          command-line arguments :  see previous definition of '_CRT_SECURE_NO_DEPRECATE'
e:\repositorys\dat-example\node_modules\utp-native\deps\libutp\utp_types.h(46): warning C4005: '_CRT_SECURE_NO_DEPRECATE' : macro redefinition (..\..\deps\libutp\utp_utils.cpp) [E:\Repositorys\dat-example\node_modules\utp-native\build\deps\libutp.vcxproj]
          command-line arguments :  see previous definition of '_CRT_SECURE_NO_DEPRECATE'
e:\repositorys\dat-example\node_modules\utp-native\deps\libutp\utp_types.h(46): warning C4005: '_CRT_SECURE_NO_DEPRECATE' : macro redefinition (..\..\deps\libutp\utp_internal.cpp) [E:\Repositorys\dat-example\node_modules\utp-native\build\deps\libutp.vcxproj]
          command-line arguments :  see previous definition of '_CRT_SECURE_NO_DEPRECATE'
..\..\deps\libutp\utp_internal.cpp(566): error C3861: 'snprintf': identifier not found [E:\Repositorys\dat-example\node_modules\utp-native\build\deps\libutp.vcxproj]
  utp_api.cpp
  utp_packedsockaddr.cpp
e:\repositorys\dat-example\node_modules\utp-native\deps\libutp\utp_types.h(46): warning C4005: '_CRT_SECURE_NO_DEPRECATE' : macro redefinition (..\..\deps\libutp\utp_api.cpp) [E:\Repositorys\dat-example\node_modules\utp-native\build\deps\libutp.vcxproj]
          command-line arguments :  see previous definition of '_CRT_SECURE_NO_DEPRECATE'
e:\repositorys\dat-example\node_modules\utp-native\deps\libutp\utp_types.h(46): warning C4005: '_CRT_SECURE_NO_DEPRECATE' : macro redefinition (..\..\deps\libutp\utp_packedsockaddr.cpp) [E:\Repositorys\dat-example\node_modules\utp-native\build\deps\libutp.vcxproj]
          command-line arguments :  see previous definition of '_CRT_SECURE_NO_DEPRECATE'
..\..\deps\libutp\utp_packedsockaddr.cpp(137): error C3861: 'snprintf': identifier not found [E:\Repositorys\dat-example\node_modules\utp-native\build\deps\libutp.vcxproj]
gyp ERR! build error
gyp ERR! stack Error: `msbuild` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onExit (C:\Users\dustin\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\build.js:276:23)
gyp ERR! stack     at emitTwo (events.js:106:13)
gyp ERR! stack     at ChildProcess.emit (events.js:194:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:215:12)
gyp ERR! System Windows_NT 10.0.15063
gyp ERR! command "C:\\nodejs\\node.exe" "C:\\Users\\dustin\\AppData\\Roaming\\npm\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd E:\Repositorys\dat-example\node_modules\utp-native
gyp ERR! node -v v7.9.0
gyp ERR! node-gyp -v v3.3.0
gyp ERR! not ok
npm WARN install:[email protected] [email protected] install: `node-gyp-build`
npm WARN install:[email protected] Exit status 1
npm WARN [email protected] No description
npm WARN [email protected] No repository field.

Would be awesome to see support on windows!

Priority: High Status: In Progress Type: Bug

opened by StarpTech 11

populate key after archive opened

A live archive does not have its key until opened, and this is currently not propagated back to .key of the Dat instance. This breaks desktop archive creation, which does a simple open-then-close to obtain new archive key. workaround is at https://github.com/datproject/dat-desktop/commit/cfb3d6cb46e3ab9ca27c7021258a4abd454903bb

opened by clkao 11
would like better jsdoc coverage for function callbacks
I am reporting:

[ ] a bug or unexpected behavior

[x] general feedback

[ ] feature request

[ ] security issue

Bug Report

Please give us details about your installation to assist you. Run dat -v to see the version of Dat you are using.

Operating system: Ubuntu 19.10

Node Version: 12.16.1

dat-node Version: 3.5.15

Expected behavior

I don't know how to word it properly but none of the function callbacks seems to be typed properly. I will be open here, I am mainly a typescript dev. And I have seen #148. I respect your preferences, so I won't go into that discussion. I am not very familiar with how jsdocs works. If callback functions can be typed as the rest of the parameters that would be great. I am looking to get better at using jsdocs so if you want I can learn and try to give PRs. Let me know If you are open to that.

Actual behavior

Here's an example.

Debug Logs
Type: Documentation
opened by SreeChandan 9
dat-js proposal
inspired by https://github.com/joehand/dat-js/issues/22, @karissa @joehand and I discussed and came up with this proposal, based on the API of torrent-stream and therefore also webtorrent

feature requirements across environments

cli/desktop

progress events (hyperdrive-stats)

tcp/utp support (swarm module)

filesystem access (read and write)

leveldb

size matters slightly less

focused on downloading a single dat

watching filesystem

diffing filesystem

whole files are stored random access

fully seeds

web app

progress events (reading)

websocket support

HTML5 File API support (read only)

memdb

size matters

either one or multiple dats per instance

never buffer whole file into storage (only streaming download)

does not really seed data, only downloads

desktop

size matters least

multiple dats in one instance

some examples (may have errors)

var dat = new Dat({db: leveldb}) dat.add('./mydat', function (repo) { repo.key }) // existing dir, uses default leveldb var dat = new Dat() dat.add({key: 'blah', db: memdb}, function (repo) { repo.key }) // in browser download dat.add() // creates keys for you dat.on('repo', function (repo) { // emitted when repo is ready to be used repo.key repo.privateKey }) dat.repos // array of Repo instances var watch = require('hyperdrive-import-files') var stats = require('hyperdrive-stats') var swarm = require('hyperdrive-archive-swarm') var dat = new Dat() dat.add('./mydat', {swarm: false, ignore: '*.jpg', resume: true, live: true}, function (repo) { if (repo.owner) var watcher = watch(repo) var emitter = stats(repo) var swarmer = swarm(repo) swarmer.pause() swarmer.resume() }) var watch = require('some-file-watcher') var dat = new Dat() dat.add('./mydat', function (repo) { watch(repo.path, function (change) { repo.append(change.filename) }) repo.pause() })

notes

https://github.com/juliangruber/hyperdrive-import-files/blob/master/index.js#L143 dat-js default read only use case for more use cases have options to configure default memdb default swarm (? webrtc) expose the hyperdrive
opened by maxogden 8

How to properly close dat so memory goes down

I am reporting:

[x] a bug or unexpected behavior
[ ] general feedback
[ ] feature request
[ ] security issue

Bug Report

Please give us details about your installation to assist you. Run dat -v to see the version of Dat you are using.

Operating system:OS X mojave
Node Version: 10.9.0
dat-node Version: 3.5.13

Expected behavior

I expect this function to gracefully close when dat.close()is called. The reason I want to do this is because I found that memory continues to increase if I continually open dats and then can't close them.

// Load and close dats one at a time.
// ISSUE: Where would one put the close??
function loadOne(key) {
  return new Promise((resolve, reject) => {
    Dat('./test', {
      key: key,
      temp: true,
    }, (err, dat) => {
      if (err) reject(err)
      dat.joinNetwork((e) => {
        if (e) reject(e)
        dat.archive.readFile('/data.json', function(err, content) {
          if (err) reject(err)
          console.log('Donezies with ', key)
          if (!content) reject('NO CONTENT??')
          // *TRY* to close dat
          dat.close(() =>{
            console.log('Trying to close!')
            resolve(content.toString())
          })
        })
      })
    })
  })
}

Actual behavior

I get an error:

events.js:167
      throw er; // Unhandled 'error' event
      ^

Error: Feed is closed
    at Feed._onclose (/Users/cody/junk/dat-benchmark/node_modules/hypercore/index.js:1121:34)
    at /Users/cody/junk/dat-benchmark/node_modules/hypercore/index.js:1111:38
    at Request.done [as _callback] (/Users/cody/junk/dat-benchmark/node_modules/hypercore/lib/storage.js:203:5)
    at Request.callback (/Users/cody/junk/dat-benchmark/node_modules/random-access-storage/index.js:161:8)
    at nextTickCallback (/Users/cody/junk/dat-benchmark/node_modules/random-access-storage/index.js:249:7)
    at process._tickCallback (internal/process/next_tick.js:63:19)
Emitted 'error' event at:
    at onerror (/Users/cody/junk/dat-benchmark/node_modules/hyperdrive/index.js:92:19)
    at Object.loop [as callback] (/Users/cody/junk/dat-benchmark/node_modules/hyperdrive/index.js:139:21)
    at Feed._onclose (/Users/cody/junk/dat-benchmark/node_modules/hypercore/index.js:1121:25)
    at /Users/cody/junk/dat-benchmark/node_modules/hypercore/index.js:1111:38
    [... lines matching original stack trace ...]
    at process._tickCallback (internal/process/next_tick.js:63:19)

Debug Logs

  dat-node Stream close +0ms
Donezies with  f93346bb5013762841c259891d63df91f4e4071ca449e5ac019a53713d2a2c06
  dat-node leaveNetwork() +964ms

opened by AcidLeroy 7

files not updating properly

I am using dat sync (of dat-next) on one computer, and dat-node's dat.joinNetwork({}) in another. I made a simple index.html with "Hello world." On the origin computer I replaced the text with some text, "12345", etc.

On the second computer: Instead of replacing "Hello world" with "12345", it appends the file with new lines of partial text.

opened by aaronshaf 7
dat segfaults if stopping and restarting on the same dat instance
Expected behavior

Dat should warn me if I try and use the same dat instance after closing it. Or even better: it should allow me to start, stop and restart without problem.

Actual behavior

Dat segfaults

Steps to reproduce behavior

const dat = Dat({dir: fixtures}) dat.open(function () { dat.share(function (err) { dat.close(function () { dat.db.close(secondRun) }) }) }) function secondRun () { dat.open(function () { dat.share(function (err) { // SEGFAULT! }) }) }
opened by yoshuawuyts 7
100% cpu when syncing a single file on my local machine
I can easily get dat-cli (and my own applications using dat-node) to peg the cpu to 100% and eventually not receive updates.

I am reporting:

[x] a bug or unexpected behavior

node v11.7.0, dat 13.11.5

Steps to reproduce:

mkdir -p test/{local,remote} cd test/local edit test.txt # in a second terminal cd test/local dat share # in a third terminal cd test/remote dat clone dat://<output of dat share from second terminal> cd <output of dat share from second terminal> dat sync # in a fourth terminal watch -n1 cat test/remote/<output of dat share from second terminal>/test.txt # now make lots of edits to test.txt (like 1 a second for about 30 seconds)

The first few updates are snappy, but soon each update will peg the node process to 100%, and will take longer and longer to sync, and eventually stop syncing (or take over 10 minutes so I cannot tell if they eventually sync).

Video https://vimeo.com/324228171
opened by jedahan 6
dat-node ignoring files in folder

I realize that there are similar open issues in the dat repo as well but wanted to see if there were any updates :) I'm currently using:

Node 12.20 Dat: 14.02

dat-node is still ignoring all files in my specified directory, but is recognizing connections. I'm just not able to transfer files across.

I am able to use the dat-cli successfully, and transfer files over. Is there a difference in the dependancies on dat-node?

opened by advino 0
Looking for maintainers

This repository is currently not maintained.

If you like this package, and have time, we welcome your contribution!

You can help with tested pull requests that address issues in this repository at any time but if you are interested in becoming a maintainer with write access, you can reach other members of the community for guidance in the gitter chat (#dat on freenode) and in community calls.

opened by martinheidegger 0
Use discovery-swarm-webrtc in addition to existing swarm
I am reporting:

[ ] a bug or unexpected behavior

[ ] general feedback

[x] feature request

[ ] security issue

While the dat-node network stack is in flux (see #247), I would like to propose running discovery-swarm-webrtc in parallel with hyperswarm going forward. WebRTC is currently the best way to reach the web platform at present, and a better alternative is unlikely to come in the short or medium term.

Having a dual network stack in dat-node would allow it to bridge dat content to web clients for little extra cost. In Cliqz we currently run webrtc discovery in parallel with discovery-swarm, and this improves peer connectivity in several cases.

Here are some of the advantages of a dual stack approach:

(Partial) uniformity in peer discovery across platforms. Dat in the browser will be compatible with dat-node. This could revive dat-js, and avoid the hacks currently needed to bridge between node and web.

Improved peer connectivity. WebRTC has lots of NAT traversal tricks built into it. In tandem with hyperswarm's holepunching this could help peers on tricky NATs become connectable.

There are some disadvantages:

Requires one or more central signalling servers for brokering webrtc connections. Note that discovery-swarm's discovery was also fairly centralised, particularly the DNS discovery mechanism, so this is not a significant paradigm shift. I currently host a signal server at https://signal.dat-web.eu/ for projects using webrtc discovery, and would be happy to offer it for use in dat-node.

Performance. WebRTC data channel performance is not as good as raw TCPSockets. I am, however, not sure that we are hitting the upper bounds of network performance in most dat use-cases anyway.

Size. The node webrtc implementation is a large project and requires native binaries. This could affect install speed and somewhat bloats binary size.

Would be happy to hear other thoughts on this.

If you would accept this change, I would be happy make a PR with this change.
opened by sammacbeth 0

Uncaught ERR_SOCKET_DGRAM_NOT_RUNNING when leaving network in joinNetwork callback

I am reporting:

[x] a bug or unexpected behaviour

Bug Report

Please give us details about your installation to assist you. Run dat -v to see the version of Dat you are using.

Operating system: Manjaro Linux
Node Version: v12.9.1
dat-node Version: 3.5.15

Expected behavior

All errors are catchable

Actual behavior

The following code sometimes crashes on me.

const folderPath='./archives/0';
const folderPath='./test-keys';
Dat(folderPath, {secretDir: path.join(rootFolder, 'secret_keys')}, function (err, dat) {
  if (err) reject(err);
  dat.importFiles();
  // It did not crash without this.    
  const network = dat.joinNetwork(() => {
    dat.leaveNetwork();
    console.log(`Leaving ${folderPath}`);
  });
});

Sometimes it just crashes. Note that I am opening 163 archives at the same time, waiting for them to finish their initial sync.

dgram.js:831
    throw new ERR_SOCKET_DGRAM_NOT_RUNNING();
    ^

Error [ERR_SOCKET_DGRAM_NOT_RUNNING]: Not running
    at healthCheck (dgram.js:831:11)
    at Socket.send (dgram.js:609:3)
    at RPC.send (/home/jaller94/Git/fdroid2dat/node_modules/k-rpc-socket/index.js:144:15)
    at RPC.query (/home/jaller94/Git/fdroid2dat/node_modules/k-rpc-socket/index.js:189:8)
    at RPC.query (/home/jaller94/Git/fdroid2dat/node_modules/k-rpc/index.js:146:17)
    at DHT._sendPing (/home/jaller94/Git/fdroid2dat/node_modules/bittorrent-dht/client.js:213:13)
    at test (/home/jaller94/Git/fdroid2dat/node_modules/bittorrent-dht/client.js:168:10)
    at DHT._checkNodes (/home/jaller94/Git/fdroid2dat/node_modules/bittorrent-dht/client.js:186:3)
    at DHT._checkAndRemoveNodes (/home/jaller94/Git/fdroid2dat/node_modules/bittorrent-dht/client.js:152:8)
    at KBucket.<anonymous> (/home/jaller94/Git/fdroid2dat/node_modules/bittorrent-dht/client.js:58:10)

The traceback doesn't look like dat-node is at fault, but I have to start somewhere on my journey to track down the issue. Maybe someone here knows more?

Debug Logs

Output is not any different when running it with DEBUG=dat.

opened by jaller94 11

"ratio limit" for users with restricted upload bandwidth
I am reporting:

[ ] a bug or unexpected behavior

[ ] general feedback

[x] feature request

[ ] security issue

For a user with high upload bandwidth costs, if they are creating a large file (for example, an hour long piece of audio) it would be nice if it was possible to set a ratio limit, kind of like in a bittorrent client. I see here (https://github.com/datproject/dat-node#network-options) that you can set upload to true or false, but maybe it makes sense if the library could accept another option ratio-limit that stops uploading once the data uploaded becomes 100% (or whatever the value is). This way, the user doesn't incur upload bandwidth costs and other peers take over the burden of serving the file once the entirety is available in the swarm.
opened by bradwbradw 3

Releases(v3.5.15)

v3.5.15(Apr 9, 2019)
Update hyperdrive to fix relative path security issue.

add test for .. and update deps to fix 81f6f07

Change to es6 class syntax (#234) 7bf0d42

Update readme.md (#231) b85d603

Fix typo (#230) dc853b8

https://github.com/datproject/dat-node/compare/v3.5.14...v3.5.15
Source code(tar.gz)
Source code(zip)
v2.0.0(Apr 14, 2017)
2.0.0 - 2017-04-13

Big new release! Hyperdrive version 8 upgrades to our SLEEP file format. The hyperdrive release improves import, transfer speed, and metadata access. It includes a new API much like the node fs API. Lots of cool things!

We've tried to keep the dat-node API changes to a minimum. But we've moved away from using leveldb to storing the metadata, using a flat file format instead. This means the 2.0 release will be incompatible with exiting dat archives.

If you have any old archives, we definitely recommend you upgrade. Any upgrade time will be made up for with more speed!

The major API differences are listed below, we probably forgot some minor changes as several of the underlying modules have changes options or APIs.

Changed

Using mirror-folder for importing files - this comes with a new event emitter for importing and different options.

Storage is a lot different! You can specify a directory or a storage function, e.g. Dat(ram, cb) now instead of the database.

Removed

opts.db option - no more database! You can specify a variety of storage formats as the first argument instead.

dat.owner - this is now dat.writable.

stats events - we are still in the process of upgrading hyperdrive-stats. Hypercore will also support more stats internally now and we will be transitioning to those soon.

Import options - mirror-folder has fewer options on import.

Source code(tar.gz)
Source code(zip)
v1.4.0(Mar 9, 2017)
A rainy Wednesday in Portland, OR means it is time for a new release!

The theme for this release is networks! We knew that networks each have their own peculiarities. We are learning that clear network information and debugging tools can help make up for network confusion. Unlike a web server, which may be either on or off, a distributed network can have many states. With this release we try to make those various states clearer.

On/Off - Join network will callback after the first round of discovery. With this we can let a user know if a key is offline.

Peer Counts - We added some nuance to the peer count. Instead of just one number we can give more information:

how many total peers is the user connected to?

how many peers have ALL the data?

how many peers have sent data to the user?

Pause/Resume - We added an API to pause and resume a given archive.

Not network related, but still pretty cool:

Dat Ignore - use a .datignore file to ignore anything in your dat directory! 🙈

Next Release

The next minor (or maybe major?) release will be super exciting!!!!! Mafintosh has been busy SLEEPing and getting ready to improve Dat transfer and import speeds a ton. Some details

The SLEEP protocol is in action. Each Dat will have a set of files to describe it. This will make it easier to support third party clients and HTTP interfaces.

No more leveldb dependency!

Lots of transfer and import speed. ⚡️

Digging Into Details - API Updates

Dat Ignore - .datignore

Add a .datignore file to any dat to ignore files on import. Turn it off with the option useDatIgnore = false on the import function call.

Peer Stats - dat.stats.peers

The stats API now has peer counts! We provide a few different peer counts so you can give accurate information about the state of the network.

peers.total - total number of connected peers

peers.complete - connected peers with a full replication of content data

peers.downloadingFrom - connected peers the user has downloaded data from

Note that these are only tallied for current connections (no cache kept for offline peers).

Network Callback - dat.joinNetwork(opts, cb)

dat.joinNetwork([opts], [cb]) now has a callback option. This will callback after the first round of discovery has finished. With that you can check if any peers are online for a given key.

Pause/Resume - dat.pause() & dat.resume()

dat.pause() and dat.resume() allow you to start and stop transfers on demand! It is aliased to join and leave right now but we'll make it fancier soon 💅 .

Complete Changelog

Added

.datignore support for ignoring files

Callback on joinNetwork after first round of discovery

Initial pause and resume API aliased to join and leave

stats.peers API with new peer counts

Fixed

Better leave network, also closes swarm.

Clone options passed to initArchive

Set opts.file for archive owner without length

createIfMissing passed to level options

dat.close() twice sync errors

Fix import without options

(hyperdrive fix) sparse content archives

Changed

Remove automatic finalize for snapshot imports

Source code(tar.gz)
Source code(zip)