Race conditions in Author API

Here’s a quick and dirty way to show it in action. withDuplicatePage.js:26 was modified so that it prints out the current section’s page count when a page is duplicated:

    return mutate({
      variables: { input },
    })
      .then(get("data.duplicatePage"))
      .then((data) => {
        console.log("Pages in section: ", countPagesInSection(data.section));
        return data;
      })
      .then(tap(redirectToNewPage(ownProps)));

I then hit the “Duplicate” button like a madman. Here’s the result:

We expect a monotonically increasing amount of pages - but as can be seen, sometimes the page count is unchanged and sometimes even goes down as later responses come through to the client. Only a few of the actual duplicate requests have made their mark on the questionnaire itself, having been overwritten by other concurrent attempts at duplication.

In traditional relational databases, locking is done for you by the database engine via transactions - either explicitly through the use of SELECT ... FOR UPDATE² or through cranking up the isolation level³ and leaving it to the database vendors to figure out.

An alternative approach to using transactions is to manually implement a form of optimistic locking⁵ by modifying the query of MongoDB’s update function (which would be called in e.g. saveQuestionnaire in Author). Instead of querying just for the questionnaire ID, we add a new unique ID which changes between saves - either a UUID or a monotonically increasing integer.

This stack overflow answer details the technique. If the “losing” container / nodeJS task tries to update a questionnaire which has changed, this new unique ID will have been changed as part of the foreign write, the update query will no longer match any documents and nothing will be written.

The application can then query to see if the write was successful - if not, it knows it needs to retry.

Such an approach is mentioned briefly on the MongoDB docs concerning concurrency control:

Another approach is to specify the expected current value of a field in the query predicate for the write operations. – Atomicity and Transactions — MongoDB Manual

The following is a rough sketch of how things currently work (generally) in Author API.

The flow is as follows:

1. The loadQuestionnaire middleware uses the questionnaireId header passed in with each request to identify the active questionnaire
2. loadQuestionnaire fetches the questionnaire from MongoDB and stores it in the ctx accessible to all GraphQL resolvers
3. GraphQL queries are parsed by Apollo server and trigger execution of all relevant resolvers. Resolvers pull data out of ctx.questionnaire to find what they need
4. Mutations mutate ctx.questionnaire in place and use a helper function, createMutation, which takes care of the business of saving the questionnaire
5. createMutation saves the questionnaire to MongoDB by updating the document with ID ctx.questionnaire.id and replacing it with the contents of ctx.questionnaire
6. A response is sent to the client by Apollo server based on the resolvers’ return values

Using MongoDB transactions would require both reading and writing the questionnaire as part of one transaction.

Since at present these two stages are decoupled - the reading happens in the express middleware function loadQuestionnaire while the writing happens in the Apollo server then mutation-triggered createMutation function - this represent a change compared to the current approach.

DIY optimistic locking - implementing compare and set⁹ by taking advantage of MongoDB’s ability to query and update documents in one step (and querying on an ID which changes with every write - hereafter the CAS ID) - is another approach which sidesteps the need to use the transactions API, provided we only need to care about one document at a time.

In Author, however, we have to deal with two objects on each save - those taken from the questionnaires and those from the versions collection.

The questionnaires collection provides metadata about the questionnaire; each object within is a slimmed-down version of the current state of the questionnaire (which is stored as the latest object in versions with the same ID).

At present, when any change is made to the questionnaire, a new version object is inserted into the versions collection corresponding to the change. Since this collection always gets appended to rather than having its documents modified we would have to use a “compare and set ID” in a different place - such as on the questionnaires metadata object for the questionnaire.

This poses a problem because without multi-document transactions (which we were hoping to sidestep if we went down this route), it is possible that the CAS ID could be updated on the questionnaires object and a new change concurrently inserted into the versions collection by a foreign write. The technique only works if you can atomically update the document AND check the CAS ID remains unchanged, which we cannot do between two documents without using multi-document transactions.

We could keep track of which questionnaires are active by using only the versions collection and doing away with questionnaires, ensuring a single document atomic commit. But this is still probably less preferable than using native MongoDB transactions.