I recognize that this problem is probably a bit too subjective for StackOverflow, but I'm confident that a programming pattern or tool exists which solves my problem and I lack the knowledge to accurately define it to ask the question more objectively.
Question
The post is a bit long, so I'm going to start with the question (phrased as best as I can). Please continue to read below for a better explanation of my problem
How can I define a type / interface / class in TypeScript such that:
- I can easily validate fields at runtime (e.g. check that numbers are within a certain range)
- I can handle subtypes / subinterfaces / subclasses with additional fields
- I can convert from older types / interfaces / classes to newer ones (based on a "version" field)
- I can easily produce a variable of the appropriate type / interface / class from a JSON string
- I can do all of the above with minimal or no code duplication / boilerplate
- I optimize performance (e.g. no copying values from one place in memory to another, as would be the case if I passed my message JSON to a class constructor and initialized class members)
Setup
I'm reading in various messages of different "types" to an analytics pipeline. The messages contain some common fields and some fields unique to their "type". There are also different "versions" to the messages, with some fields being deprecated and others being added in later versions. In TypeScript I've defined the messages roughly like so:
/// v1.ts
export type CommonFieldsV1 = {
version: 1;
action: string;
ipAddress: string;
timestamp: number;
}
export type LoadMessageV1 = CommonFieldsV1 & {
action: "load";
loadTime: number;
}
export type ClickMessageV1 = CommonFieldsV1 & {
action: "click";
x: number;
y: number;
}
// etc
export type MessageV1 = LoadMessageV1 | ClickMessageV1 | ...;
This method of defining the types has proven pretty useful as far as type hinting goes. For instance, when I type:
/// processMessages.ts
function processMessage(msg: MessageV1) {
switch (msg.action) {
case "load":
msg. // <<-- cursor is here right now
break;
default:
break;
}
}
VS Code properly recognizes that only the LoadMessage type has an action of "load" and type-hints loadTime as a possible field.
I've also created "modernize" methods that bump the version on a received message. For instance:
/// v2.ts
export function modernize(msg: MessageV1) {
const newMsg = (msg as any) as MessageV2;
newMsg.newField = "default value";
delete newMsg.oldField;
return newMsg;
}
There may be a better solution than (msg as any) as MessageV2, since this feels a little hack-y, but it works and does what I'm trying to accomplish:
- It allows me to directly edit the message object in memory instead of copying all of the values to a new location, improving performance and reducing memory utilization
- It saves me a lot of boilerplate code explicitly copying fields
In order to validate the content of the messages, thus far I've written a pretty ugly and verbose validate method:
/// v2.ts
export function validate(msg: MessageV2) {
if (msg.version !== 2) return false;
if (typeof msg.ipAddress !== "string") return false;
if (msg.timestamp < 0 || msg.timestamp > 999999999) return false;
switch (msg.action) {
case "load":
if (msg.loadTime < 0 || msg.loadTime > 3600) return false;
break;
case "click":
if (msg.x < 0 || msg.x > 9999) return false;
if (msg.x < 0 || msg.x > 9999) return false;
break;
default:
return false;
}
return true;
}
As I wrote out the code originally, if a message didn't validate then we scrapped it. This method needs to be re-written because the expectation has changed. We've learned that a number of our clients in production are sending messages with one invalid field (e.g. the x coordinate on a click message may be miscalculated and return -1 or something). Instead of scrapping the entire message, we want to only null out any invalid fields and leave the rest intact so that we can factor in the valid fields to our aggregations. Specifically the issue we're trying to address is that if we want to know the total time viewers spent on a page, we don't want to under-count because we scrapped several heartbeat messages due to a random or unimportant field being invalid.
While re-writing this function, I have a couple of goals:
- I'd prefer to reduce redundancy in our code base. Validation, by nature, defines the schema of our message. Someone reading the validation function can tell which fields are supposed to be strings and which are supposed to be numbers. So the fact that I define it once at the top of each file (TypeScript definitions) and again at the bottom (validation function) feels silly
- I want to retain my solution (or a similar solution) for bumping message version numbers
- I would like the solution to be TypeScript-compatible so that I can continue to work with static type checking and catch bugs early and often
Currently to bump message version numbers I have code like the following:
/// modernize.ts
import { MessageV1 } from "./v1";
import { MessageV2, modernize as v1tov2 } from "./v2";
import { MessageV3, modernize as v2tov3 } from "./v3";
import { MessageV4, modernize as v3tov4 } from "./v4";
import { MessageV5, modernize as v4tov5 } from "./v5";
export type Message = MessageV1 | MessageV2 | MessageV3 | MessageV4 | MessageV5;
export function modernize(msg: Message) {
if (msg.version === 1) msg = v1tov2(msg);
if (msg.version === 2) msg = v2tov3(msg);
if (msg.version === 3) msg = v3tov4(msg);
if (msg.version === 4) msg = v4tov5(msg);
return msg;
}
WIP
TypeScript in general is for compile-time type checking, so it's not really intended for run-time type checking (validation). But I know that decorators allow you to define run-time functions, so I felt like these might be a good solution for validation. e.g.:
/// v1.ts
export class MessageV1 {
version = 1;
action: string;
ipAddress: string;
@validate({
min: 0,
max: 999999999
})
timestamp: number;
}
export class LoadMessageV1 extends MessageV1 {
@validate({
min: 0,
max: 3600
})
loadTime: number;
}
// ...
The problem I'm facing now is how to properly instantiate one of these classes from JSON. Previously (when using TypeScript types) I just did something like the following:
/// processRecords.ts
import { Message, modernize } from "./modernize.ts";
function processRecords(jsonArray: string[]) {
jsonArray.forEach(json => {
let record = JSON.parse(json) as Message;
record = modernize(record);
// At this point I'm confident "record" is a MessageV5
// and I can work with it appropriately
});
}
However if I'm using classes + decorators then I can't just say JSON.parse(json) as XXX, because JSON.parse returns an Object and not an instance of my class.
I could create a constructor for my class, but how would I know which constructor to call? If I created a generic Message class (and defined the generic MessageV1 class with extends Message) then I would just call that top-level constructor, but then I'd have to update the generic class any time a new message version was added. It's also not clear to me how I would set the fields on my message short of doing so explicitly:
class MessageV1 extends Message {
constructor (jsonObj) {
this.action = jsonObj.action;
this.ipAddress = jsonObj.ipAddress;
this.timestamp = jsonObj.timestamp;
// ...
}
}
Explicitly writing out every single value like this is not only a lot of boilerplate in code, but it means the code will be less efficient as it's copying variables from one place in memory (the jsonObj) to another (the class instance) instead of operating on the basic object.
It's also not clear to me what type I should give to jsonObj in the constructor. If I define it as any then TypeScript will complain about my use of any. If I define it as object then TypeScript won't allow me to access any of the properties on it because Property 'action' does not exist on type 'object'. But I also can't define it as a map because {[key: string]: ???} leaves me with further questions on how to define the map value. Trying {[key: string]: string|number|boolean|null} doesn't help because then this.timestamp = jsonObj.timestamp will complain about Type 'string' is not assignable to 'number'.
So I hit a bit of a wall and I'm hoping StackOverflow can help out:
- I'm not sure the best way to define my message types for static type checking and type hinting while still meeting other requirements (e.g. TypeScript types, interfaces, classes, etc)
- I'm not sure the best way to handle validation to minimize boilerplate and redundancy between my type definition and my validation code
- I'm not sure the best way to instantiate the message objects
- I'm not sure the best way to convert messages objects up from v1 to v2 to v3, etc
I feel like there must be some programming pattern or design principle that solves this issue for me and I'm just taking the wrong approach, but I'm not really sure what I'm doing wrong that this is causing me such a headache.