After some time I've run into an issue where the data is not getting updated anymore: last updated: 15h ago. I've imported my Reddit data with csv files which lead to an overall saved count of 4410. Also, my pihole fetched about 38000 requests to oauth.reddit.com. In the console I got this error while it's trying to update the data:

RequestError: Error: ETIMEDOUT
    at new RequestError (/backend/node_modules/request-promise-core/lib/errors.js:14:15)
    at Request.plumbing.callback (/backend/node_modules/request-promise-core/lib/plumbing.js:87:29)
    at Request.RP$callback [as _callback] (/backend/node_modules/request-promise-core/lib/plumbing.js:46:31)
    at self.callback (/backend/node_modules/request/request.js:185:22)
    at Request.emit (node:events:513:28)
    at Timeout.<anonymous> (/backend/node_modules/request/request.js:851:16)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7) {
  cause: Error: ETIMEDOUT
      at Timeout.<anonymous> (/backend/node_modules/request/request.js:848:19)
      at listOnTimeout (node:internal/timers:559:17)
      at processTimers (node:internal/timers:502:7) {
    code: 'ETIMEDOUT',
    connect: true
  },
  error: Error: ETIMEDOUT
      at Timeout.<anonymous> (/backend/node_modules/request/request.js:848:19)
      at listOnTimeout (node:internal/timers:559:17)
      at processTimers (node:internal/timers:502:7) {
    code: 'ETIMEDOUT',
    connect: true
  },
      
  options: {
    gzip: true,
    json: true,
    headers: { 'user-agent': 'web:expanse:v=1.0.0 (hosted by u/myusername)' },
    baseUrl: 'https://oauth.reddit.com',
      
{
      raw_json: 1,
      count: 9999,
      limit: 9007199254740991,
      before: 't3_xazslb'
    },
    auth: { bearer: 'hidden for privacy' },
    resolveWithFullResponse: true,
    timeout: 30000,
    transform: [Function: transform],
    uri: 'user/myusername/upvoted',
    method: 'GET',
    callback: [Function: RP$callback],
    simple: true,
    transform2xxOnly: false
  },
  response: undefined
}

At the moment I try to recreate the error:

I've imported csv files to it (I think all of the possible ones). After that it didn't updated anymore.

When I'm not visiting my expanse instance in a while I get an 401 error and I have to login again. After logging back in all my before saved posts are gone. These are the logs from the console:

getting user (myusername)
deserialize error (myusername)
Error: user (myusername) dne
    at Module.get (file:///backend/model/user.mjs:572:9)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async file:///backend/controller/server.mjs:60:13
Error: user (myusername) dne
    at Module.get (file:///backend/model/user.mjs:572:9)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async file:///backend/controller/server.mjs:60:13
destroyed session (myusername)

I haven't changed my password in the meantime or changed anything else on my reddit account. This is also not the first time that this is happening

The content of my user accounts stopped getting updated after a day. I set it up on Wednesday and the last update occurred on Thursday. Here's an excerpt of the logs:

last Thursday at 9:49:37 PM updating user (Keagel)
last Thursday at 9:49:39 PM updated user (Keagel)
last Friday at 11:40:53 AM backed up db to file (23-09-2022_09꞉40꞉53꞉AM_UTC.sql)
last Saturday at 11:40:53 AM backed up db to file (24-09-2022_09꞉40꞉53꞉AM_UTC.sql)
yesterday at 11:40:53 AM backed up db to file (25-09-2022_09꞉40꞉53꞉AM_UTC.sql)
today at 11:27:37 AM socket (1W4pcrj0y8-iKP_OAACZ) connected
today at 11:27:37 AM getting user (Keagel)
today at 11:27:37 AM deserialized user (Keagel)

Restarting the container forces the updates though.

Originally posted by @Keagel in https://github.com/jc9108/expanse/issues/1#issuecomment-1257775758

Hello, Is it possible to make the connectionString configurable with environnement variable to specify db address and db port if we want to use an existing postgres database ?

Regards

• replace querySelector with getElementById because it can handle ids that start with numbers
• add functionality to sort by date asc or desc

I wanted to improve on the current implementation and implement sorting by date ascending or descending. While testing my solution i noticed, that the list would fail to load if there are list elements with ids that begin with a number. This is because document.querySelector does not know how to handle this and throws an unhandled exception. If all ids in the current view contain a number at the first index, it will result in the list "buffering" indefinitely. It also prevented deletion of said list elements.

> document.querySelector(`#4r5fyd`)

> VM8380:1 Uncaught DOMException: Failed to execute 'querySelector' on 'Document': '#4r5fyd' is not a valid selector.
    at <anonymous>:1:10

Feel free to give me suggestions for improvements of the new functionality.

I tested this implementation multiple times and have not found issues with it.

Getting error while opening localhost:1301

expanse_frontend_1 exited with code 0
backend_1   |
backend_1   | > prod
backend_1   | > node ./controller/server.mjs
backend_1   |
backend_1   | update all completed
backend_1   | server (expanse) started on (localhost:1301)
backend_1   | [Error: ENOENT: no such file or directory, stat '/backend/build/index.html'] {
backend_1   |   errno: -2,
backend_1   |   code: 'ENOENT',
backend_1   |   syscall: 'stat',
backend_1   |   path: '/backend/build/index.html',
backend_1   |   expose: false,
backend_1   |   statusCode: 404,
backend_1   |   status: 404
backend_1   | }
backend_1   | TypeError: Cannot read properties of undefined (reading 'user')
backend_1   |     at file:///backend/controller/server.mjs:74:42
backend_1   |     at Layer.handle_error (/backend/node_modules/express/lib/router/layer.js:71:5)
backend_1   |     at trim_prefix (/backend/node_modules/express/lib/router/index.js:326:13)
backend_1   |     at /backend/node_modules/express/lib/router/index.js:286:9
backend_1   |     at Function.process_params (/backend/node_modules/express/lib/router/index.js:346:12)
backend_1   |     at next (/backend/node_modules/express/lib/router/index.js:280:10)
backend_1   |     at /backend/node_modules/express/lib/response.js:455:7
backend_1   |     at SendStream.onerror (/backend/node_modules/express/lib/response.js:1071:5)
backend_1   |     at SendStream.emit (node:events:513:28)
backend_1   |     at SendStream.error (/backend/node_modules/send/index.js:270:17)
backend_1   |     at SendStream.onStatError (/backend/node_modules/send/index.js:417:12)
backend_1   |     at onstat (/backend/node_modules/send/index.js:722:26)
backend_1   |     at FSReqCallback.oncomplete (node:fs:206:21)

While browsing this repo I noticed from the compose.prod.yaml that expanse is a node.js application who's only dependency outside the NodeJS ecosystem is postgres, a database application. It looks to me like docker is used to run an instance of postgres as a dependency of expanse, but npm also has various embedded database libraries, and there's even embedded-postgres. Using one of these would allow the removal of docker as a requirement, which opens up many more options for packaging expanse, especially using existing automated tools for nodejs packaging. Using an embedded database by default and allowing the user to configure an external database at their preference provides a much higher degree of portability to the software that self-hosters like myself would much appreciate.

I know this is a tall order but...

If you refactored the backend api endpoints to mirror the existing reddit api, at least as a subset of the api, developers could use your application as a proxy. Basically as a cache using the same api. Would enable existing applications to leverage your app with only a URL change.

Just tried to clone the repo and follow the steps in your readme, however the frontend container never seems to come up.

I've tried resetting and manually running commands from your run.sh script but it still is not successful. The frontend container never starts and shows no logs. The build and run process using your script has the error. Different line numbers are shown based on the run.sh command you use

./run.sh: line 30: return: can only `return' from a function or sourced script

Super excited to try this out! Thanks for making this happen! I have, though, run into a setup issue...

1. Visit https://local.domain.com/
2. Click Login with Reddit
3. Click allow from Reddit
4. Returned to https://local.domain.com/logout#_

Some notes on my configuration

• My user is in the approved list in the .env_prod file.
• I am using a subdomain on a domain that I own that is proxied through Nginx Proxy Manager, which is also listed in the config file for the callback URL and is used on Reddit as the callback URL. This domain name is not accessible to the external network.

I am seeing the following in the logs

From the database:

2022-09-11 20:50:17.781 UTC [69] LOG:  incomplete startup packet,
2022-09-11 20:50:17.450 UTC [1] LOG:  database system is ready to accept connections,
2022-09-11 20:50:17.446 UTC [62] LOG:  database system was shut down at 2022-09-11 20:50:17 UTC,
2022-09-11 20:50:17.440 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432",
2022-09-11 20:50:17.437 UTC [1] LOG:  listening on IPv6 address "::", port 5432,
2022-09-11 20:50:17.437 UTC [1] LOG:  starting PostgreSQL 14.5 (Debian 14.5-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit,
2022-09-11 20:50:17.437 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432

From the app:

server (expanse) started on (localhost:1301)
saved user (reddituser),
checking if user (reddituser) exists,
purged user (reddituser),
denied user (reddituser),
socket (WDZoBrdZVFDhSc-1AAAL) connected,
new user (reddituser),
socket (tk5NIOso-aftuQw4AAAJ) connected,

I've tried to figure out where localhost:1301 is being set, but can't find it. I have found that that can cause issues if it doesn't match the domain that I a using on the network. It is updated in the .env_prod file. This may not be related to this particular issue, but thought it wroth mentioning.

Let me know what other questions you have. At the moment, this is all the info that I can find on the issue. And I fully recognize that this very well could be a pebkac errror. Any help is greatly appreciated.

When I try to run the container, it will stop with the following log:

> docker
> node -r dotenv/config controller/server.mjs dotenv_config_path=/data/.env_prod
node:internal/process/esm_loader:108
    internalBinding('errors').triggerUncaughtException(
                              ^
Error: getaddrinfo ENOTFOUND undefined
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26) {
  errno: -3008,
  code: 'ENOTFOUND',
  syscall: 'getaddrinfo',
  hostname: 'undefined'
}
Node.js v19.2.0

Database is up & reachable, volume & env-file are available & readable, too. Am I missing something here?

Storing post content would protect against loosing access to deleted media, 3rd party sites taking down content etc.

Ideally it would be based on a plugin system where its quite easy to create a content extractor for new types of links/posts, with default ones for text and Reddit images/galleries.

Alternatively, or ideally as a plugin for the proposed content storage: Ability to Internet Archive/Wayback Machine links from posts and comments.