Large TJ taggings consume all database space

[michael-gratton]

We just had an instance of a TJ syncid tag being applied to ~700k members, and this caused the engine to go off and consume all database space.

Symptoms:

• "WARN: ActiveRecord::StatementInvalid: PG::DiskFull: ERROR: could not write to file "base/pgsql_tmp/pgsql_tmp24022.1": No space left on device" in both Heroku logs and Id sync UI
• Number of open Sidekiq connections to the database even after worker dyno shut down
• Large number of tmp_16_RANDOM_CHARS tables in main identity schema, each being quite large

Stopping the worker dynos, killing Sidekiq db connections and deleting the tmp_... tables cleared up the DB space. The syncid tag had to be deleted so that the problem didn't re-occur.

Probably a few things to do here:

• Look at what actually caused this in the tag sync code
• Start batching tag imports (if not already occurring) as for member upserts
• Use tmp schema for temp tables with more descriptive name (if still needed)
• Other things?