Javascript Flyweight object with WeakMap or WeakSet

I want to create Flyweight objects, so I create the object and store its instance in a Map like this:

const FlyweightNumber = (function(){

    "use strict";

    const instances = new Map();

    class FlyweightNumber{

        constructor(number){

            Object.defineProperty(this, 'number', {
                value: number
            });

            if(!instances.get(number)){
                instances.set(number, this);
            }
            else {
                return instances.get(number);
            }

        }

        toString() {
            return this.number;
        }

        valueOf(){
            return this.number;
        }

        toJSON(){
            return this.number;
        }

    }

    return FlyweightNumber;

})();

module.exports = FlyweightNumber;

The problem is that it remains stored in Map even when I'm no longer using any instances of it in the program.

Since WeakMap and WeakSet allow the garbage collector to clean them if there is no further reference to these objects, how could I change the Map of this code to a WeakMap / WeakSet?