source

Currently, Mongo does not work with Ubuntu 22.04, see Mongo x86 Support & Mongo ARM Support. This means that mongo-memory-server also does not work in those scenarios. For people looking to use DocMQ in an in-memory environment, this leaves no alternatives.

Proposal

Add support for a native JS in-memory driver as part of the DocMQ core. Here are a list of JS drivers I could find on NPM, along with the features that DocMQ needs.

• Find/Sort At a minimum, find/sort features make it easier to update records without writing a lot of custom code inside of the driver's take, ack, etc methods.
• changes api A method to retrieve the list of changes at the point of change. This minimizes local polling and offers better performance in the worker
• types TS support would be ideal, but isn't a hard requirement. Having access to the types will minimize errors.

| Library | Stars | Activity | Types | Find/Sort | Changes API | | :------------------------------------------------------ | :------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------- | :------- | :-------- | :---------- | | lowdb | | | external | lodash | no | | LokiJS | | | external | yes | yes | | ForerunnerDB | | | no | yes | yes | | PicoDB | | | no | yes | yes | | TyrDB | | | no | yes | yes | | mango-db | | | yes | yes | yes |

As of Sept 2022, Loki seems to be the most logical choice given the stars/support and the fact it's being maintained again. LowDB does have more support, but requires lodash for find/sort features and does not offer any kind of changes API. While GitHub stars (or npm download numbers) are not a perfect indicator of project success, it does tend to identify projects with a larger audience.

I have noticed a typo on the docs, precisely If you're **someonme** who's frustrated... I know that's not a big deal :sweat_smile: but I think it's a good first issue :relaxed:

Also, add contributing section in the README that redirects to the CONTRIBUTING file. And perhaps add a list of features that offer this library

Problem Statement

madge isn't great with Typescript circular dependencies. We use it right now to avoid the most egregious problems, but there's likely better solutions since the original build scripts were written.

Proposed Solution

This is a research task, which means no code is required (unless you want to do a PR). There's a set of questions we're hoping to answer, informing the project direction.

1. What value are we getting from madge? How could we introduce a circular dependency in DocMQ, and can we trust madge to catch it?
2. What alternatives to madge are out there? Are they better than what we currently have? I found two quickly googling, but this isn't an exhaustive list by any means: dpdm, eslint import rule
3. Is it worth keeping madge, swapping to an alternative, or removing the circular dependency checks altogether?

This is a tracking ticket for the postgres work in https://github.com/jakobo/docmq/tree/labs-postgres

• [x] Initial driver support (https://github.com/jakobo/docmq/blob/labs-postgres/src/driver/postrgres.ts)
• [x] Improve test suite to something that can be run against any driver (0.2.1)
• [x] Figure out how to "init" a DB in a schema-ful environment. Currently, because Mongo is schemaless, we are rarely concerned about data migrations. Some postgres ops are idempoent, so we may be able to just run things like create table if not exists as part of the initialize() step.
• [ ] Postgres LISTEN/NOTIFY to create a change stream. This might require creating a trigger inside of initialize() to capture inserts. For now, we can leave it on the polling system though that's less ideal.
• [x] Explore github actions as way to do testing for postgres driver (avoid needing local driver) or look into something like pg-mem as our SQL is not very complex

Why

Currently, job completion a job triggers an ack or fail, we can only act on the contents of the job as a generic. Some items don't serialize, making them temporary functions or classes needed only during execution and their associated events.

Solution

Both solutions involve creating some kind of context object that is available inside of the process step and available in the DocMQ events.

In both cases, the emitter will need an additional generic for the context object, and we'll need to add a 4th generic to the signature Queue<T, A = unknown, F extends Error = Error, C extends Record = {}> where T is the payload type, A the ack type, F the failure type, and C the context typing.

Context on process source when we call our processor, we can pass the context object in as a mutable object. We would need to force the type, as our starting context {} wouldn't contain any required fields. We can fix this internally by forcing the context C to DeepPartial<C> to ensure keys and nested keys are marked optional by default.

Context on API source when we create the API, the key API methods such as ack and fail can accept an additional "context" object passed through to events. The type for the handler API would need to have an additional generic C extends Record = {}. The downside to this approach is needing to call the ack/fail explicitly with the context.

Currently, runEvery isn't aware of timezones. The ISO-8601 durations do not understand the politics of time, Daylight Savings, etc. The following test case highlights the error

var luxon = require("luxon");
const {DateTime, Duration} = luxon;

const setup = [
  { zone: "America/Phoenix", diff: 24 }, // phoenix does not use DST
  { zone: "America/Los_Angeles", diff: 25 }, // LA falls back one hour
];

const threshold = {month: 11, day: 5, hour: 14};
const duration = Duration.fromISO("P1D");

setup.forEach(t => {
  const runAt = DateTime.local()
    .setZone(t.zone)
    .set(threshold)
    .startOf("hour")
    .toISO(); // representing stored as string inside of DocMQ
  const next = DateTime.fromISO(runAt)
    .plus(duration)
    .toISO(); // add the runEvery duration
  const diff = DateTime.fromISO(next)
    .diff(DateTime.fromISO(runAt))
    .shiftTo("hours")
    .toObject().hours; // get the # of hours represented by the diff
  console.log(t.zone, `exp: ${t.diff} / act: ${diff}`, t.diff === diff);
});

"America/Phoenix", "exp: 24 / act: 25", false
"America/Los_Angeles", "exp: 25 / act: 25", true

Unfortunately, this is because there is no concept of a timezone when generating next (the process of scheduling the next occurrence based on the value in runEvery). For areas that respect Daylight Savings Time, job recurrence will be shifted by an hour at the time change boundaries which may be undesirable behavior.

Proposed Fix

The problem occurs during the addition of Duration to the runAt value. DocMQ isn't aware of a timezone, so it defaults to the system timezone rules when checking what P1D would evaluate to. DocMQ stored data needs to retain an optional timezone, falling back to UTC time rules for consistency.

/** The interface used when enqueing one or many jobs */
export interface JobDefinition<T> {
  /** A reference identifier for the job. If not specified, a v4() uuid will be used */
  ref?: string;
  /** The job's payload */
  payload: T;
  /** A date in the future when this job should run, or omit to run immediately */
  runAt?: Date;
  /** An ISO-8601 duration or a cron expression representing how frequently to run the job, or `null` to clear the value */
  runEvery?: string | null;
  /** The IANA timezone this job should occur in which scheduling future runs via `runEvery`, or `null | undefined` to use the system's time */
  timezone?: string | null | undefined;
  /** The number of allowed retries for this job before giving up and assuming the job failed. Defaults to 0 */
  retries?: number;
  /** Specify the retry strategy for the job, defaulting to a fixed retry of 5s */
  retryStrategy?: RetryStrategy;
}

• [x] Add new typings
• [x] Store the timezone in the DocMQ database (backwards compatible schema change)
• [x] Change the resolution of runEvery to use timezone information when available
  • Queue https://github.com/jakobo/docmq/blob/main/src/queue.ts#L190
  • BaseDriver https://github.com/jakobo/docmq/blob/b13b971db57779ef19678a49a74cc5039be919b1/src/driver/base.ts#L140
  • cron-parser has timezone support built in via the tz option key
  • luxon has support but must be told what timezone information to use via setZone()

Reproduction

• Set this.workers to a value >= the allowed concurrency
• Attempt to process a job

Expected The takeAndProcess exits silently, as there are no available workers, retrying on worker completion or poll interval being reached

Actual takeAndProcess passes the limit of 0 or a value <0 through to the driver which does not represent a correct limit.

Why

When jobs are scheduled to run at the same time, the current take() operation on drivers does not sort for higher priority jobs.

Solution

• Add an additional optional "priority" field to the document
• Add necessary indexes to allow drivers to take by visible < now, ordered by priority DESC then visible DESC
  • Loki driver change
  • Mongo driver change
  • Postgres driver change

This is an evergreen issue for tracking what's planned in upcoming versions of DocMQ.

Patch 0.5.5

• [x] Better error rejection reasons: 5b41ba4
• [x] #13

...

Major 1.0.0

• [ ] #6
• [ ] #14
• [ ] Postgres migration solution
• [ ] Major increment migration (batch script v on dequeue)

DocMQ currently restricts replacement to jobs in the future to avoid altering a job that should already be processing. However, we can likely change the driver calls to allow the replace if:

• deleted is null (has not completed)
• and visible is in the past (as any acks on the job would also fail since they didn't ack or ping in time)

This would allow you to pause the queue, do inserts, and not run into collisions for replacing a job in the past that never ran.