Why are there .js and .d.ts files in source control?

[jcbhmr]

@conartist6 (I don't see you in https://github.com/iter-tools/regex/watchers so I'm pinging you with a friendly notification 📬)

I was browsing through the source code in GitHub.dev https://github.dev/iter-tools/regex and wondered "why is there a .ts with the same code in .js and .d.ts right next to it?"

It seemed to defy all logic... But then I looked at the build process:

{
  "scripts": {
    "build": "macrome build",
    // ...
  }
}

So then I looked at the https://github.com/macrome-js/macrome#readme GitHub repo (I couldn't find it on https://npms.io/search?q=macrome ?) and figured out that this was intentional:

Note that the file in the screenshot was checked into git after being generated, which enables the github repository itself to function as a node package! Such a package can be trivially forked, and any user's fork can be used like so:

// package.json
{
  "dependencies": {
    "@iter-tools/regex": "github:user/regex#commitish"
  }
}

This makes sense! You want to be able to no-build import from GitHub with npm! But then I got to thinking... Doesn't npm have a way to hook into the package installation only when installed from a Git repo to build your project? Right?

https://blog.jim-nielsen.com/2018/installing-and-building-an-npm-package-from-github/

Git dependencies with prepare scripts will have their devDependencies installed, and npm install run in their directory before being packed.

Did that sentence turn on a lightbulb in your mind? If not, allow me to explain how it did for us. In npm 5, when you run npm install, any packages in your package.json which are git dependencies, will have their dependencies installed (via their respective package.json file) and then have the prepare script run as well. What does that mean for us? No more storing the transpiled code in dist/. We can now move the command which builds the code that ends up in dist/ directly into our prepare script, thus moving the task of compiling our library’s source code onto the apps which consume it.

Under this new paradigm, npm essentially grabs our remote source code from Github, builds it locally, then installs it to node_modules for our consuming app (or bundler) to use.

I'm not trying to be confrontational even though I know I'm basically pointing out "all the ways you're wrong" which is a really great way to turn the human brain into rationalizing defense mode. I genuinely want to understand the reasons. If there is a benefit, I might adopt this in-source .ts+.js+.d.ts in my own projects!

You want it to be easy to use forks in-place in npm package.json without publishing to npm
vs
npm offering a "prepare" script

Here's another resource: https://docs.npmjs.com/cli/v9/using-npm/scripts. This one mentions prepack instead of prepare which may be more of what you're after.

Runs BEFORE a tarball is packed (on "npm pack", "npm publish", and when installing a git dependency).

TL;DR: Why commit .js build artifacts to Git source control when you can build them using npm's prepack/prepare hook script?

{
  "scripts": {
    // Either delegate to npm run build, or just make 'npm pack' your build command!
    "prepack": "npm run build",
    "prepack": "vite build",
    "prepack": "tsc",
    "prepack": "next build",
    "prepack": "rollup -c",
  }
}

[conartist6]

I agree that it is possible to do the same thing with prepack, but my intent was not to be practical but explore what is possible when you make different ways of working possible. My observation is that people assume that things which are not yet possible or are difficult using current tools must not be worth exploring.

Many of my explorations are centered around the idea of achieving ecosystems with much greater decentralization than those we have now. The problem I see is that if everyone embraced building with prepack and consuming with git, the result would be a giant mess. Without having necessarily made any mistake you might end up with 18 versions of typescript in node_modules. You could download a package whose SHA sum checks out, but the built output of that package could be compromised (maliciously or accidentally) and you would have no idea.

[jcbhmr]

I think prepack only runs on Git or GitHub or GitLab or BitBucket dependencies. The general workflow that I use and I assume is the most popular is where you have .ts in GitHub, then you have a CI GitHub Action to build and publish to npm (running the prepack in the process) so that if you view your published module on unpkg.com you see .js and .d.ts files. Then, if you want to install it direct from source (like if you have a fork and don't want to publish to npm) you can add the github:jcbhmr/my-fork and npm will know that it needs to run the pseudo-build process to get the source code compiled into actual JS code.

A similar thing happens with Node.js native modules I think where they build themselves on your PC when you download them? (C++ modules use postinstall though I think, not prepack. postinstall runs when npm install express bu prepack only runs when npm install github:jcbhmr/my-fork-of-express) I guess what I'm saying is that the popular practice is there.

Many of my explorations are centered around the idea of achieving ecosystems with much greater decentralization than those we have now.

Cool! 👍 I would be careful about how you "buck the trend" as it were to make it as easy as possible for others to develop/use/contribute/etc. If it really is simpler and easier to have .js+.ts+.d.ts in GitHub, then go for it! I just know I was initially confused. I thought the best practice was to use Vite + GitHub Actions lol 🤷‍♂️.

The problem I see is that if everyone embraced building with prepack and consuming with git, the result would be a giant mess.

You're right! I think that's why we have npm or GitHub Packages or custom npm registries? 🤔 Sorta like Conan.io in C++ land where you have a big "binary archive server" to host the compiled code so you don't have to compile it yourself. In this .js+.ts+.d.ts case, it seems like those "binaries" are going in Git. It's an approach I haven't really seen before, hence why I'm inquiring about it. It's pretty cool!

You could download a package whose SHA sum checks out, but the built output of that package could be compromised (maliciously or accidentally) and you would have no idea.

I think this is also why we have npm-hosted packages? I'm not a security professional, so I honestly don't know what the beset practice is...