Version 2

Updated on June 19, 2020.

This issue provides an overview of the next version of Foal TS. It will be released between June and August 2020.

The purpose of this release is not to add many new features, which are regularly added in minor versions, but to fix issues and hacks that require some break changes. It will also remove unused and deprecated components and make parts of the framework simpler and more intuitive.

Changes & New Features

:white_check_mark: = Implemented

1. Developper experience (CLI) :white_check_mark:

Migrations and shell scripts are not developper friendly when developping (#494). There are too many CLI commands and it is not intuitive to know which one to use and in which order.

In version 2, the files tsconfig.migrations.json and tsconfig.scripts.json will be removed and the CLI commands will be reduced and simplified.

More details here: #684.

Examples

Build, make and run migrations

# Version 1
npm run build:app
npm run migration:generate -- -n my_migration
npm run build:migrations
npm run migration:run

# Version 2
npm run makemigrations
npm run migrations

Build migrations, scripts and the app

# Version 1
npm run build:app
npm run build:scripts
npm run build:migrations

# Version 2
npm run build

2. Authentication with sessions :white_check_mark:

Authentication with sessions is complicated and requires a lot of code. Current implementation also prevents the framework from adding new features (#525, #521, #510).

In version 2, the authentication system with sessions will be greatly simplified.

See #799.

3. Schema references in validation hooks :white_check_mark:

It is not possible to use references (especially OpenAPI references) in validation hooks.

In version 2, this will be fixed.

More details here: #552.

Example

const productSchema = {
  // ...
}

@ApiDefineSchema('product', productSchema)
@ValidateBody({
  $ref: '#/components/schemas/product'
})

4. Service initialization :white_check_mark:

Services can be initialized using their boot method. For this to work, we currently need to call ServicesManager.boot() in src/index.ts.

In version 2, this feature will be enabled by default. No need to call ServicesManager.boot() anymore.

5. Accessing file metadata during uploading :white_check_mark:

When uploading a file, we have only access to its content (#673).

In version 2, we will also have access to its file size, mime type and original file name.

6. Safer configuration :white_check_mark:

Current Config.get function has several faults (#496).

In version 2, it will be replaced with the new Config.get2 added in v1.7.

// Version 1 
const port = Config.get<number>('settings.port', 3000);
const port = Config.get<number>('settings.port');

// Version 2
const port = Config.get('settings.port', 'number', 3000);
const port = Config.getOrThrow('settings.port', 'number')

7. Improve the naming of JWT settings :white_check_mark:

Version 1: settings.jwt.secretOrPublicKey

Version 2: settings.jwt.secret, settings.jwt.publicKey

8. Remove support of Mongoose :white_check_mark:

Mongoose brought many problems in the past while maintaining the framework. It takes time to maintain it and it is also not typed and does not seem to be used by FoalTS users: https://www.npmjs.com/package/@foal/mongoose.

FoalTS will no longer provide Mongoose tools starting from v2 (i.e. CLI code generation and the fetchUser function). Users will still be able to use it "by hand" if they wish.

9. Improve the configuration system :white_check_mark:

See #805, #497.

10. Simplify the management of custom errors thrown in controllers and hooks :white_check_mark:

See https://github.com/FoalTS/foal/issues/638#issuecomment-673915802.

Cleanup :white_check_mark:

• Remove the outdated and useless security headers.
• Remove the PermissionDenied and ObjectDoesNotExist which are unused and do not appear in the documentation.
• Remove the object ValidationError and the function validate which are unused and do not bring value to the framework.
• Remove the deprecated function createHttpResponseFile and use Disk.createHttpResponseFile instead. Once done, remove the package mime from @foal/core dependencies.
• Remove the package @foal/formidable and use the @ValidateMultipartFormDataBody hook instead.
• Remove the deprecated legacy option of the functions hashPassword and verifyPassword. This option was used in very old versions of Foal when no one was using the framework.
• Remove the deprecated package @foal/ejs and the possibility to create FoalTS template engines. Only Express templates engines will be allowed (Twig, EJS, pug, etc).
• Drop the support of Node 8 and run tests on Node 12 and 14. Version 8 of Node is not maintained anymore. Uninstall pump in @foal/core, @foal/storage and @foal/aws-s3 and use the pipeline function from the standard library (added in Node 10).
• Remove the unused command foal g sub-app.
• Check if password hashing needs an update (OWASP).
• Make createApp return a promise and remove createAndInitApp. The function createAndInitApp was introduced because we needed to return a promise.

  // Before
  const app = createApp();
  const app = await createAndInitApp();
  
  // After
  const app = await createApp();
  

• Only allow one way to pass a custom express instance to createApp.

  // Before
  const expressInstance = express();
  const app = createApp(expressInstance);
  // OR
  const app = createApp({ expressInstance });
  
  // After
  const expressInstance = express();
  const app = createApp({ expressInstance });
  

Upgrading Dependencies

• @foal/graphql
  • [email protected]

This issue lists potential subjects for the blog and the YouTube channel.

Feel free to add a 👍 on those you’re interested in. This will help prioritize which content should be written first.

If you would like to discuss on a subject or suggest a new one, please use the #blog-and-videos channel on Discord. The blog focuses on providing articles on these topics: real-life projects, deployments, tips & tricks and release posts.

FoalTS documentation is awesome. It helps me building a good grasp of how a framework takes care of auth, orm, cli, etc. step by step thru Tutorials and Topic Guides.

If it's possible, I'd love to translate it — to Bahasa Indonesia. Hope v2 is a good starting point.

What do you think?

Been trying to migrate my project from V1 to V2, have found that the new UseSessions generates a session when I'm requesting a static asset, not sure if I'm using it wrongly.

My front-end controller is as follows:-

@UseSessions({ store: MongoDBStore, cookie: true })
export class DefaultController {
    public subControllers = [
        // Checkout - uses cookies
        controller("/checkout", CheckoutController),
    ]

    @Get("/*")
    public async defaultView(ctx: Context) {
       // Serves a single page application - using JWT and no cookies
        return new HttpResponseRedirect("/admin");
    }


}

The problem

@Entity()
export class User extends BaseEntity {
  @PrimaryGeneratedColumn()
  id: number;

  @Column()
  name: string;

  @Column({ unique: true })
  email: string;

  @Column()
  password: string;

  @Column({ nullable: true })
  accessToken?: string;

  // tslint:disable-next-line:variable-name
  private _client?: ThridPartyClient;

  get thridPartyClient(): ThridPartyClient | undefined {
    if (!this._client && this.accessToken) {
      this._client = createThridPartyClient({accessToken: this.accessToken});
    }

    return this._client;
  }
}

Currently, when we return an instance of the above entity (or anything that contains it) from an API endpoint the entire model gets serialized into something like:

{
  "id": 1,
  "name": "Orkhan",
  "email": "[email protected]",
  "password": "password-hash",
  "accessToken": "blah213",
  "_client": {...}
}

The last 3 fields must not be inside json. We can of course create a new object and return it but there should be a better solution in general. Does foalts have something to offer in this case?

Workaround

We can override toJSON method in the User entity class which is used by JSON.stringify method:

function hidden(hidden: string[]) {
  return function (this: object) {
    const result = {};
    for (const x in this) {
      if (!hidden.includes(x)) {
        result[x] = this[x];
      }
    }
    return result;
  };
}

@Entity()
export class User extends BaseEntity {
 ...
  toJSON = hidden(['password', 'accessToken', '_client']);
}

Where we call hidden method with the array of field names to hide and it will return a method that returns a new copy of object by not copying specified fields. This works for arrays and nested objects too.

Proposed solution

Again I put forward laravel's approach as an example. FoalTS can use a custom replacer to serialize json response, where we can hook in and conditionally hide fields based on some sort of a reserved field on the provided entity (like foal: { hidden: ['password'] }). This would work for arrays and nested objects too.

Currently the 500 error page in development is quite ugly. It would be nice to have something prettier (with a bit of CSS...)

If someone wants to take care of this, feel free to add a comment below and submit a PR!

Note: In order to keep the @foal/core package small, the page cannot rely on a 3rd CSS framework (such as Bootstrap for example)

Issue

We currently have TypeORMStore and RedisStore to store sessions in SQL and Redis databases. It is annoying for people using MongoDB with FoalTS. We should also support MongoDB session storage.

This issue was previously named Have hooks to catch rejected promises or uncaught errors. See the latest comments to see the current state of the issue.

Issue

In some situations, we want to catch errors thrown or promises rejected in a hook or a controller method. For example, during development, we may want to display a custom 500 page which prettifies the error stack.

Possible solution

A possible solution could be to add a CatcherPostHook decorator which would take a CatcherHookPostFunction as parameter. The behavior would be analog to how errors, promises and express middlewares work.

Empty example

@CatcherPostHook((err, ctx, services, response) => {

})

Behavior demo

@Hook(() => {
  console.log(1);
  throw new Error('a');
})
@Hook(() => {
  console.log(2);
})
@CatcherPostHook((err, ctx, services, response) => {
  console.log(err.message);
  throw new Error('b');
})
@Hook(() => {
  console.log(3);
})
@CatcherPostHook((err, ctx, services, response) => {
  console.log(err.message);
})
@Hook(() => {
  console.log(4);
})

// Output:
// 1
// a
// b
// 4

Hey guys,

I'm looking for feedback on Foal and I really want to know what you think! 👍 Also if you do not plan to use the framework in the future, feel free to tell why!

1. What features do you use? What features are you really excited about?
2. Are there features that you find useless, complicated or that you do not plan to use?
3. Did you encounter difficulties when using / learning the framework?
4. Do you think something is missing?
5. How do you authenticate your users (session, cookie, jwt, etc)? Do you use Foal components for that?
6. Do you use sessions?
7. What do you use Foal for (SPA, REST API, etc) ?

If you enjoy the framework, feel free to leave a ⭐️on the repository to spread the word! 😄You can also retweet my Looking for feedback tweet!

Issue

Issue reported in the PR https://github.com/FoalTS/foal/pull/206. When running the tests of the cli package on Windows, they all seem to fail because of a line break encoding (\n vs \r\n).

Possible solutions

• Ask collaborators to use a virtual machine (for example vagrant) when developing for the project.

OR

• Update test-environment.ts to replace all \r\n characters by \n when loading a generated file during tests (maybe easier).

Version of FoalTS: 2.8.0

I tried updating my backend but I am still failing at this error:

/node_modules/@foal/socket.io/lib/socketio-controller.service.js:59
                     return cb({
                                ^
 TypeError: cb is not a function
     at Socket.<anonymous> (/node_modules/@foal/socket.io/lib/socketio-controller.service.js:59:28)
     at processTicksAndRejections (node:internal/process/task_queues:94:5)
 Program node ./build/index.js exited with code 1

What can be my problem, did I missed some step in upgrading the project to 2.8?

Thanks

Version of FoalTS: 3.1.0

First of all : FoalTS is really great!

I faced the following error when I try to connect an angular frontend project to FOAL, I've got the following error:

config.projects[config.defaultProject].architect ||= {};
TypeError: Cannot read properties of undefined (reading 'architect')

If I understand well config.defaultProject has been removed from Angular.json since Angular v14. (found on angular-cli : defaultProject deprecated [https://github.com/angular/angular-cli/issues/23470])

I found a solution which works for me (on IONIC projects), but not sure that's the best way to solve the issue:

In node_modules@foal\cli\lib\generate\generators\angular\connect-angular.js :

    if (config.defaultProject === undefined) {
        console.log("NO default project (Angular>14 - use firstproject 'app'?) => ",angularProject);
    }
    config.projects[angularProject].architect ||= {};
    config.projects[angularProject].architect.serve ||= {};
    config.projects[angularProject].architect.serve.options ||= {};
    config.projects[angularProject].architect.serve.options.proxyConfig = 'src/proxy.conf.json';
    // Output build directory
    const outputPath = (0, path_1.join)((0, path_1.relative)(path, process.cwd()), 'public')
        // Make projects generated on Windows build on Unix.
        .replace(/\\/g, '/');
    config.projects[angularProject].architect.build ||= {};
    config.projects[angularProject].architect.build.options ||= {};
    config.projects[angularProject].architect.build.options.outputPath = outputPath;
    return JSON.stringify(config, null, 2);

Hope this could help.

We don't always want to server to handle file uploads, and it's easy enough to use the AWS SDK, but the Disk util is just too convenient. Would it be possible to add this feature to it?

Something like:

 const { downloadUrl } = await this.disk.createPresignedDownloadUrl('docs/xxx.pdf', 'buffer', {
              expiresIn: 3600
          });

 const { uploadUrl } = await this.disk.createPresignedUploadUrl('docs', content, {
              expiresIn: 3600
          });

Thought this does present a problem, to how to handle local file upload/downloads in this case, I don't have good idea for that one.

Issue

The MongoDB session store service is missing some functions, which we needed in a past project. In that, we were using FoalTS with MongoDB, and we extended those functionalities. This PR is supposed to contribute them back.

Solution and steps

• [x] Add getSessionIDsOf function
• [x] Add getSessionsOf function (a new one which returns all whole session objects for a user)
• [x] Add getAuthenticatedUserIds function
• [x] Add destroyAllSessionsOf function

Checklist

• [x] Add function docs
• [x] Add tests
• [x] Update foal docs Session Tokens ("This feature is only available with the TypeORM store." comments)

Hi, I'm opening a new issue as you asked it in #880 I took a deep dive into the Typescript type system and was able to come up with a type (lot of ideas came from the type-challanges) for the Config.get function:

• based on the default.json the key parameter is typed, it only accepts strings that are defined in the config file in the same format as the Config.get method accept keys (e.g.: 'port', 'setting.debug')
• based on the default.json the return type is automatically typed, you don't have to provide any type information to the function

As you might notice, the solution heavily relies on the default.json, if we want to read a value which is missing from that file, typescript will throw an error. In my opinion this is more of a positive outcome than a negative, as it forces to define a default value for every key that the application will try to read, also making the defaultValue parameter obsolete (defaults should be defined in default.json). Another limitation is in a json file we can only have string, boolean, number, array and object types, the type the function returns with can only be these. So if we want to have a more restricted type (e.g. Stripe API has an Interval type which is a subset of string ("weekly" | "daily" | "manual" | "monthly") and we want to use the value straight from the config) we either need to cast the type, be able to specify the return type of the function similar to the current solution, or have some utility to rewrite the type of the config (which I included in the solution). But in my opinion with these limitations we would get a type-safe, easy-to-use Config functionality. I'm really interested in your opinion (and I don't have a clear idea, how to include this in the framework)

Some screenshots to show how it works: intellisense can show you all the available keys

typescript error if key doesn't exists, value automatically typed correctly

works with deep keys with dot notation, correctly returns with complex object types

I tried to add as many comments as possible, as it is not an easy read :). (it also uses some of the newest features of typescript, I used version 4.7.3)

import { Config } from '@foal/core';
import Stripe from 'stripe';

import config from 'config/default.json';

type BaseConfigType = typeof config;

/**
 * SetType<T, K, V>: object
 *   T: object type to modify
 *   K: string key to modify, can be in dot notation format to modify deep key
 *   V: type that key to have
 *   modifies T, overrides the key that K refers to to have the type V
 *   if K is not in T, then T is returned
 */
// first check if K has a dot in it. if yes H should be the part before the first dot R should be the part after
type SetType<T extends object, K extends string, V> = K extends `${infer H}.${infer R}`
  ? // we create a new object type
    {
      // the keys should be the same as in T, the type should be the same
      // expect for H: if T[H] is an object then recursively call SetType with that object and the rest of K (R)
      [Key in keyof T]: H extends Key ? (T[H] extends object ? SetType<T[H], R, V> : T[H]) : T[Key];
    }
  : // if K doesn't hava a dot in it then we still create a new object type
    {
      // the keys should be the same as in T, the type should be the same, expect for K: it should be V
      [Key in keyof T]: K extends Key ? V : T[Key];
    };

/**
 * SetTypes<T, KV>: object
 *   T: object to modify
 *   KV: array of key-value doubles, keys are strings can be in dot notation format, values can be any type
 *   modifies T, overrides each key in KV to have the corresponding type in KV
 */
// let's check if KV is an empty array. if not H should be the first item in the array and R should be the rest (could be an empty array as well)
type SetTypes<T extends object, KV extends [string, unknown][]> = KV extends [
  infer H extends [string, unknown],
  ...infer R extends [string, unknown][],
]
  ? // for each H item in the array we update T with the help of the SetType type (H[0] is the key, H[1] is the value)
    // then we recursively call SetTypes with the updated object and the rest of the array (R)
    SetTypes<SetType<T, H[0], H[1]>, R>
  : // if KV is empty just return with T, the input object
    T;

// examples of rewriting types in the config
export type ConfigType = SetTypes<
  BaseConfigType,
  [
    ['stripe.interval', Stripe.AccountCreateParams.Settings.Payouts.Schedule.Interval],
    ['stripe.weeklyanchor', Stripe.AccountCreateParams.Settings.Payouts.Schedule.WeeklyAnchor],
  ]
>;

/**
 * WithPrefix<Pre, Key>: string
 *   Pre: string to prefix Key with
 *   Key: string or number
 */
// first lets check if Pre is never (we use never instead of empty string when we don't want to add any prefix)
type WithPrefix<Pre extends string, Key extends string | number> = [Pre] extends [never]
  ? // if there is no prefix just convert Key to a string (needed if it is a number)
    `${Key}`
  : // if Pre isn't empty check if Key is a number
  Key extends number
  ? // if Key is a number, instead of dot notation use square brackets (this will handle arrays)
    `${Pre}[${Key}]`
  : // if Key is a string use dot notation
    `${Pre}.${Key}`;

/**
 * ObjectKeyPaths<T, Pre>: string (union)
 *   T: object to generate key paths from
 *   Pre: string to prefix key paths with, we use never instead of empty string to begin with
 *   generates all the possible keys in T, with dot notation to deep keys, all the keys prefixed with Pre
 */
type ObjectKeyPaths<T extends object, Pre extends string = never> =
  // Pre is always a key in object if used correctly
  // never | A = A that's why we start with never instead of empty string
  // A | A = A so unioning the same keys multiple times won't cause an issue
  | Pre
  // the idea is to create an object where keys are the same as in T, but values(types) are strings: the keys prefixed with Pre
  // when we created the correct object we can create a union of the values with indexing the object with all the possible keys
  // e.g: this works with tuples(arrays) ['a', 'b', 'c'][number] = 'a' | 'b' | 'c'
  | {
      // we use the same keys as in T, except:
      //  if T is an object we only use string and number keys (filtering symbols)
      //  if T is an array we only use number keys (filtering symbols and functions that are on every array like map, forEach etc.)
      //  this can cause that if we create an object based on an array, then we add some non number keys to it those keys wont be in the end result
      //  but for our use case, this can't happen, because we will read types from a json file
      [Key in T extends unknown[] ? keyof T & number : keyof T & (string | number)]: T[Key] extends object
        ? // if the type of the current key is an object, then we should recursively call ObjectKeyPaths with that object
          // adding the current key prefixed with Pre as the new prefix
          ObjectKeyPaths<T[Key], WithPrefix<Pre, Key>>
        : // if the type isn't an object then just prefix key with Pre
          WithPrefix<Pre, Key>;
      // we use the same logic to index the object when we created the keys for the object
    }[T extends unknown[] ? keyof T & number : keyof T & (string | number)];

type ConfigKeys = ObjectKeyPaths<ConfigType>;

/**
 * Get<T, K>: ?
 *   T: object to get the type from
 *   K: key to read, can be in dot notation to read deep key
 *   gets the type of K from T
 */
// first check if K has a dot in it, if yes then H should be the part before the first dot R should be the rest
type Get<T extends object, K extends string> = K extends `${infer H}.${infer R}`
  ? // check if H is a key in T and that key refers to an object
    T[H & keyof T] extends object
    ? // if yes recursively go a level deeper with that object and the rest of K (R)
      Get<T[H & keyof T], R>
    : // else K is not a proper key of T so there is no type to get
      never
  : // if K has no dot in it, then it must be at the root level, we return it with that type
    T[K & keyof T];

export const get = <K extends ConfigKeys>(key: K): Get<ConfigType, K> => {
  return Config.getOrThrow(key);
};