Versioning of content with persistent tiles with blobs breaks the storage

[hvelarde]

As @datakurre has documented in plone/plone.app.mosaic#80, each version of a content object with persistent tiles with blobs, will create a new empty blob into file system for each found blob and I will also break the storage as you can see in the following image:

You can see that the portal type and path information is repeated in all objects; this happens because querying the versioned object fails with an exception in ZVCStorageTool.py#L668:

------
2015-06-16T14:49:19 ERROR CMFUid ASSERT: 52004 objects have 134 as uid!!!
------
2015-06-16T14:49:19 ERROR CMFUid ASSERT: 52004 objects have 135 as uid!!!
------
2015-06-16T14:49:19 ERROR CMFUid ASSERT: 52004 objects have 137 as uid!!!
------

[hvelarde]

as I mentioned yesterday in #27 (comment), I have 2 sites with huge problems because of this.

yesterday I tried the nuclear option on one of them, as described by @miohtama in http://stackoverflow.com/a/10413511/644075

I ran an instance in debug mode and cleared the whole storage by using:

del app.mysiteid.portal_historiesstorage._shadowStorage
import transaction ; transaction.commit()

I then compacted the database and, at first, I though we had got rid of the empty blobs but, as soon as our customer created a new version of the front page object, all empty blobs (about 1.5 million) came back.

now, every time a pack is run, the empty blobs are erased, and every time the user creates a new version, the empty blobs are created.

[datakurre]

To be a more specific, making a deepcopy of a blob, creates an empty blob, and CMFEditions does use deepcopy by default. (Yet, I don't know what Zope copy does, because that doesn't result empty blobs.)

If there's no generic fix, this should be fixed in p.a.tiles, by implementing a CMFEditions plugin, which properly versions persistent tiles (which are persistent dictionaries stored in obj annotations).

[alecpm]

Getting rid of the shadow storage just erases metadata. You need to erase entries in the storage repository as well if you want to take the "nuclear option". See /portal_historiesstorage/zvc_repo

Backup your ZODB (this is totally untested dangerous code, which will – at best – wipe out all version history), then:

app.mysiteid.portal_historiesstorage._shadowStorage._storage.clear()
app.mysiteid.portal_historiesstorage.zvc_repo._histories.clear()
import transaction ; transaction.commit()

Then pack. As datakurre indicates, in the long term there will need to be a modifier plugin to handle this properly, since complex content stored in annotations are tricky in CMFEditions.

[alecpm]

In terms of what would be needed for a modifier here, you'll need to take a look at a couple of the existing modifiers. Currently there are modifiers that clone or skip blobs which are stored as attributes on the object (generally the simplest modifiers deal with attributes directly on the object, modifiers that handle complex data structures inside of annotations are going to be more complex). Additionally, there are modifiers that look inside of annotations for non-blob files over a certain size and skip/abort versioning those files. Using those as a guide, it's probably possible to construct a modifier that looks for tiles inside of annotations and then looks for Blob attributes of those tiles and either clones or skips them.

[cguardia]

@alecpm I tried the nuclear option above and the empty blobs did not go away. Any other recommendations?

[datakurre]

@cguardia Have you tried this already: http://stackoverflow.com/questions/10413409/cleaning-up-annotations-in-plones-cmfeditions-history-objects

[cguardia]

@datakurre yes, tried that first. I'm even wondering if that caused the nuclear option to fail.

[alecpm]

Are there still history references at /portal_historiesstorage/zvc_repo? Did you pack the DB? I think that should clear all references to persistently versioned objects, though I guess it's possible I missed something in one of the many CMFEditions tools.

[datakurre]

@cguardia While Nathan's code failed also for me, this is the minimal, which allowed me to remove versioned annotations selectively:

    from zope.component.hooks import getSite
    from zope.annotation.interfaces import IAnnotations

    repo = getSite().portal_historiesstorage._getZVCRepo()
    for history in repo._histories.values():
        for version in history._versions.values():
            annotations = IAnnotations(version._data._object.object)
            for key in annotations:
                del annotations[key]

This should remove removed blobs after pack.

I cannot reproduce the issue of all blobs coming back after a new version (I only get the few empty blobs for the new version).

[cguardia]

@datakurre but are you using collective.cover too? I think this is also due to the way it stores data in persistent dicts in annotations.

[alecpm]

The code I provided should completely empty that _histories BTree that the code above removes the annotations from. It seems like perhaps the cover annotations might have multiple references preventing them from being packed? I have no idea how that would happen though.

[datakurre]

@cguardia I did not test that with cover, but with Mosaic persistent image tile, which otherwise does the same (stores blobs in persistent dicts in annotations, causing zero size blobs in versioning).

[hvelarde]

@frapell can you help here?