When ORMs disobey: a Mongoose story

Note: All examples are very contrived; minor suspension of disbelief is required.

A while ago, my team was experimenting with some pre-find hooks in mongoose that modified the query conditions before executing it. We wanted to apply a filter at a global level, for example, only return records that have age >= 18 from the persons collection.

Baking this logic inside the call itself seemed like a good idea initially. It saved us from having to modify every existing Model.find() call in the codebase. This also provided a safeguard against devs forgetting to add the condition to every Model.find() call in the future.

"use strict";

const mongoose = require("mongoose");

const personSchema = mongoose.Schema({
  name: String,
  age: Number,
});

function preFindHook(next) {
  if (this._conditions.age >= 18) next();
  else next(new Error("Invalid query"));
}

personSchema.pre("find", preFindHook);
personSchema.pre("findOne", preFindHook);

const PersonModel = mongoose.model("Person", personSchema);
module.exports = PersonModel;

It worked perfectly; until a bunch of PATCH endpoints started to fail randomly.

router.patch('/persons/${personId}', async (req, res, next) => {
  const { personId } = req.params;
  const updatedPersonInfo = req.body;

  const existingPersonRecord = await PersonModel.findOne({
    _id: personId,
    age: { $gte: 18 },
  });

  if (updatedPersonInfo.name) {
    existingPersonRecord.name = updatedPersonInfo.name;
  }

  await existingPersonRecord.save();
  res.json({ person: existingPersonRecord.toObject() });
})

We started searching for the issue using console.log statements in route controllers. It seemed as if our recently registered pre-find hook was being triggered on Model.save() as well.

router.patch('/persons/${personId}', async (req, res, next) => {
  ...
     existingPersonRecord.name = updatedPersonInfo.name;
   }
 
+  console.log('Reached this far without error');
   await existingPersonRecord.save();
   res.json({ person: existingPersonRecord.toObject() });
 })
Reached this far without error
Error: Invalid query
    at model.Query.preFindHook
    at callMiddlewareFunction

To confirm this, we turned on debug mode in mongoose
mongoose.set('debug', true)
and removed the pre-find hook.

Mongoose: people.findOne({ _id: ObjectId("6031d9f91973241ccb4848e9"), age: { '$gte': 18 } }, { projection: {} })
Reached this far without error
Mongoose: people.findOne({ _id: ObjectId("6031d9f91973241ccb4848e9") }, { session: undefined, projection: { _id: 1 } })

In debug mode, mongoose logs every query that it runs. It is clear that findOne() was executed here when save() was called. This findOne() call did not specify an age property in the filter. This made the pre-find hook throw an error.

The very idea of mongoose executing a different operation than the one it was asked to was baffling to me. I spent the next hour scouring the web for hints or explanations, finding none.

The mystery unraveled when I read the source code of Model.save as a last resort. The important parts are shown below. Complete source code can be found here .


Model.prototype.$__handleSave = function (options, callback) {
  if (this.isNew) {
    this[modelCollectionSymbol].insertOne(
      obj,
      saveOptions,
      function (err, ret) {
        if (err) {
          _setIsNew(_this, true);

          callback(err, null);
          return;
        }

        callback(null, ret);
      }
    );
  } else {
    const delta = this.$__delta();

    if (delta) {
      this[modelCollectionSymbol].updateOne(
        where,
        delta[1],
        saveOptions,
        (err, ret) => {
          if (err) {
            this.$__undoReset();

            callback(err);
            return;
          }
          ret.$where = where;
          callback(null, ret);
        }
      );
    } else {
      const optionsWithCustomValues = Object.assign({}, options, saveOptions);
      this.constructor
        .exists(this.$__where(), optionsWithCustomValues)
        .then((documentExists) => {
          if (!documentExists) {
            throw new DocumentNotFoundError(
              this.$__where(),
              this.constructor.modelName
            );
          }

          callback();
        })
        .catch(callback);
      return;
    }
  }
};

Mongoose internally tracks whether a document is ‘new’ or not. It also tracks the fields (if any) that have been updated on the document. This information is used to decide what operation is performed when the .save() method is called on the document.

  • When the document is ‘new’, mongoose performs an insert.
  • When the document is not new, mongoose calculates the ‘delta’.
  • If any fields have been modified, mongoose performs an update.
  • If no fields have been modified, mongoose checks that the document exists in the database using findOne.

What we could’ve done better

The reason why only the PATCH calls failed was that in a PATCH operation, which is a partial update, there is a possibility that none of the fields are updated. In this case, we should not have called .save() on the document in the first place.
The hook we wrote was flawed as well. The _conditions is a private property of 3rd party code. We should not be messing with it even if JavaScript allows us to.

In hindsight, it seems like mongoose was doing the sensible thing by simply ensuring that the document existed in the database.
In hind-er-sight it seems wrong. It’s not what I asked for!

Mongoose is a (heavy) abstraction layer over Mongo. It has its quirks and may seem magical at times. When you use mongoose you give up some control over the queries that are run. It is incorrect to think of its operations as simple wrappers to those of Mongo’s native driver.

Last piece of advice:

Expect magic to happen. Read and prepare.