Setting Up A Live Remote Three.js Dev Server With Webpack and NGinx

I've spent most of the last week working on getting a live Webpack dev server running for my previous hundred days project. I finally got it working, and a boilerplate version is up on my GitHub if you'd like to skip straight to working code, although you'll need to change a few variable names in there to reflect your own host. In the context of my hundred days project, this was a bit of a step back: I fell into some bad habits (not keeping good notes on what I'm doing/keeping good track of methods I've tried) and got more than a little bit hyperfixated on fixing this my own way. This definitely led to me spending more time than I needed to, and neither this setup nor the documentation are what I'd like them to be, but consider this post an attempt to atone and get back into good habits. Because of the aforementioned poor note-taking I may have missed some crucial steps, but I think this covers all my major discoveries.

This was an enormously stupid endeavor for a number of reasons, but two stand out in particular:

1) This project was so, so deeply not designed for a deployment context. I literally wrote a bash script that copied the entire project, images, models, components, and all, into a new folder for each day.

2) Webpack is an enormously confusing bit of software, at least for me. Just setting up the local dev server the way I wanted it took me many hours during my last hundred days project.

I probably could have saved myself a lot of time by simply writing each pre-existing sketch to a bundle and hosting those as HTML files on my website, then starting from scratch on a new dev server designed with this context in mind. But I am not a smart man, so instead I decided to just refactor my existing code for this.

First, I needed to find a way to minimize the amount of repeating myself I was doing. Instead of a hundred separate folders, all containing a structure like this:

day100 -
├ src
├── css - Contains all SCSS files, that are compiled to `src/public/css`
├── js - All the Three.js app files, with `app.js` as entry point. 
│   ├── app - the React wrapper and js code for each daily sketch
│   ├── components - React/Three.js components
│   ├── utils - General javascript utilities
│   ├── data - Any data to be imported into app
│   └── utils - Various helpers and vendor classes
├── html - HTML templating for webpack.
└── public - External asset files (3D models, images, etc.)
... Repeat 99 times.

I wanted to create a template that had, at the top level, all my existing components, public files, etc., then an individual folder for the code that was unique to each day. The reason I did it the above way in the first place was valid - I changed different components on different days, and backwards compatibility would have been pretty much impossible with this new structure (it still is! Now that I have this new model, I'll still need to adjust the dependency structure for old sketches so that they have access to their unique components). But for a new, publicly accessible start I wanted to treat it like a web app. So I took that structure, and functionally made a copy of just the code in /app/ for each day, making a structure like this.

 src
├── css - Contains all SCSS files, that are compiled to `src/public/css`
├── js - All the Three.js app files, with `app.js` as entry point. 
│   ├── day100  
│   ├── day99... (repeat 98 times)
│   ├── components - React/Three.js components
│   ├── utils - General javascript utilities
│   ├── data - Any data to be imported into app
│   └── utils - Various helpers and vendor classes
├── html - HTML templating for webpack.
└── public - External asset files (3D models, images, etc.)

So I did some mv and cp-ing, and I had a couple of test folders with this structure. I set up nginx to see if, when I ran my dev server, it was at least getting pinged, even if it didn't work.

location /three/ {

	proxy_pass http://127.0.0.1:8081;
   }

This pinged the server, but it wasn't serving anything helpful. Now I needed to make sure Webpack was serving each individual entry point as its own page, and that all of them could access the higher-level dependency folders. This ended up being a nightmarish process. In reality, these steps were out of order and took a lot of tweaking, but I'll reduce them to how they should actually have been done.

First, I needed to create separate entry points for each day's scripts. I did that by creating an object that mapped each "day" folder to an object:

const entryMap = {};

fs.readdirSync(path.resolve(__dirname, 'src/js/'))
    .filter((f) => fs.lstatSync(path.resolve(__dirname, 'src/js/'), '/' + f).isDirectory() && f.indexOf('day') >= 0)
    .map((f) => {
        return f
    })
    .map((f) => [f, fs.readdirSync(path.resolve(__dirname, 'src/js/') + '/' + f).filter(name => name == 'app.js')])
    .forEach(f => {
        entryMap[f[0]] = [ path.resolve(__dirname, 'src/js/') + '/' + f[0] + '/'  + f[1]];
    });

 And then passed that to my webpack module, telling it to write it to dayX/dayX.js.

  return {
    entry: entryMap,
    mode: mode,
    output: {
      path: path.resolve(__dirname, './dist'),
      filename: '[name]/[name].js'
    },
    ...
    

Finally, using HTMLWebpackPlugin, I set my server to write an index.html file to each output folder, so that ultimately I could navigate to URLS like /three/day100, /three/day101 for each sketch:

 new HtmlWebpackPlugin({
        title: 'ThreeJS Playground',
        template: path.join(__dirname, '/src/html/index.html'),
        path: path.resolve(__dirname, './dist/'),
        filename: '[name]/index.html',
        inject: false
  })

This solution got me up and running, but my server was still passing all the chunks it built to each individual HTML file, so every file would have the entrypoint scripts for each day.

There are a lot of ways I could have solved this. What I wanted, but could not figure out how to do, was to set the chunks variable in the plugin intelligently, like:

chunks: ['[name]']

Unfortunately, for reasons that aren't clear to me, the '[name]' key only seems to work in certain fields. I spent several hours combing through Webpack documentation, because certainly some attribute in the HTML plugin can access the name of the chunk it's working on, but I was unable to find that field. So, ultimately, I used this extremely dumb solution, creating a template that took the parameters passed to it to detect the unique part of its filename, then filtering the script tags it generates to only use a chunk that contains that filename:

new HtmlWebpackPlugin({
        templateParameters: (compilation, assets, assetTags, options) => {
          //access the filename set below using the '[name]' key, then get that unique key from its folder
          let chunkId = options.filename.split('/')[0]
          // filter the asset tags we pull in the frontend so that only the relevant chunk is injected
          assetTags.headTags = assetTags.headTags.filter(tag=> {
                return tag.attributes.src.indexOf(chunkId) > 0
            })

          return {
              compilation,
              webpackConfig: compilation.options,
              htmlWebpackPlugin: {
                  tags: assetTags,
                  files: assets,
                  options
              },
              chunkId

        }},
        title: 'ThreeJS Playground',
        template: path.join(__dirname, '/src/html/index.html'),
        path: path.resolve(__dirname, './dist/'),
        filename: '[name]/index.html',
        inject: false
  })

There is almost certainly a better way to do this – build a different plugin that emits the chunk name to plugins, for example – but for better or worse, once I encounter a problem, I will keep trying to solve it without resorting to implementing anything new unless I absolutely can't. I like to understand why things do or don't work.

Anyway, this worked, but I ran into a final problem once I had a single unique script writing to its corresponding file: I was able to serve files, but not connect to the socket server for live code updates! This is another solution that took me quite some time to get sorted, but ultimately I was able to route nginx requests through my existing setup to the wss router with a small modification:

    location /three/ {
        proxy_pass http://127.0.0.1:8081/;
    }
    location /ws {
        proxy_set_header X-Real-IP  $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Host $host;

        proxy_pass http://127.0.0.1:8081/ws;  


        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }

Then adding a url prefix to my dev server in webPack, telling it to prefix all socket requests with the ws url:

    devServer: {
      static: './src/public/',
        client: {
            webSocketURL: 'https://blog.hellagrapes.club/ws'
        },
      ...

I went back to root, ran npm run dev, and, finally, it worked! I have a cute little server running on my remote machine.