TL;DR: Apple Notes has a few bespoke embedded objects which are messier than the Easy Embedded Objects previously explained. This post covers how to piece back together the final complex embedded object, the Apple Notes Gallery. I got this wrong a few times and in fixing it, learned how I was messing up other aspects of Notes.
If you haven’t read the earlier post about Easy Embedded Objects, please go do that now. It will explain the background assumptions of this work and show how this works on some very straight forward objects. The previous post about Embedded Tables will dive into how to examine the protobuf in greater detail and is assumed as a foundation for this post.
Below are the steps to rebuild
com.apple.notes.gallery objects. These are specific to Notes and come about when you add an image and use the “Scan Document” feature to do so. If you still don’t want to do this by hand, feel free to check out the Apple Cloud Notes Parser, which had a lot of rewrites recently to handle the Gallery object correctly.
The Type UTI of
com.apple.notes.gallery represents a set of images that were generated from the ‘Scan Documents’ feature when you add an image to a Note. The idea is to take a picture of each page and put them all into one bundle to swipe through. Along the way, Notes parses the text it sees in the image.
In attempting to properly display this type of embedded object, I ended up realizing I had made a lot of mistakes in my assumptions along the way. For example, diving into here is what made me realize that a lot of objects you would not expect have thumbnails associated with them, such as URLs and PDFs (almost clear enough to read the words). In addition, because my initial test data had one image, I assumed that was always the case, but once I generated better test data to figure out the protobuf, I ended up realizing I would have been skipping all the other images. This led to a fairly large rewrite of embedded objects in general (to find thumbnails for anything), output (to show everything that has child objects), and galleries (they show all images).
The first step of rebuilding a
com.apple.notes.gallery object is the same as
com.apple.notes.table. You will start with the Note protobuf, find the appropriate Attachment identifier for the Gallery, and pull the relevant data from the
ZICCLOUDSYNCINGOBJECT.ZMERGEABLEDATA1 field, recognizing that it is a GZipped protobuf following the same .proto file as Tables. We’ll use this as an example Note:
… which means this query will get our GZipped Gallery protobuf:
… resulting in this, once gunzipped and parsed (as always, this is trimmed for readability):
This format is simpler than the table in that a gallery is one-dimensional, you only need to know which embedded pictures go in it. You can apply the same methodology we learned for Tables to identify that all of these have the same “Key Item” of “self” and use an “Object ID” to identify the
ZICCLOUDSYNCINGOBJECT.ZIDENTIFIER as a
String Value for each of the images contained therein. By looping over each of these
Mergeable Data Object Entry items that have a
Custom Map, you’ll find all of the images that make up this gallery.
When you query
ZICCLOUDSYNCINGOBJECT for those
ZIDENTIFIERs, you’ll find that each is of type
public.jpeg and has OCR applied to the images. See the first embedded objects post for more about both handling
public.jpeg and OCR.
|206||public.jpeg||5A60D806-A02D-4B2D-B1A4-69948CB8E1FE||150 TEARD |
Full-size Corded Mouse
I was glad to tackle this type of embedded object last because it helped me firm up my understanding of all the previous types, along with thumbnails. Because the Gallery object data format could hold just about anything and is new in iOS 13, I wouldn’t be surprised to see additional uses of it in future releases, beyond just scanning documents. There is no really good reason for users not to be able to organize their pictures into Notes using a Gallery. Understanding these bespoke Notes types will likely only get more important the longer Apple develops the application.