The integration gets specified in one line on a whiteboard: “the PIM pulls images from the DAM overnight”. Six months later there is a queue, a staging directory, a retry script, a spreadsheet of failed transfers, and a rule that nobody publishes on Thursdays.
The line on the whiteboard was the problem. Everything after it was consequence.
Short answer: do not copy assets or asset metadata into the product system. Store a stable asset identifier on the product record and resolve it to a delivery URL at request time. The integration becomes a reference plus a URL contract, there is nothing to schedule, nothing to reconcile, and nothing that can be stale, because nothing was copied.
Why the scheduled copy fails, specifically
It is worth naming the failure modes rather than asserting that batch is bad, because plenty of batch integrations are fine and this particular one is not.
It is stale by construction. Between two runs, the product system holds a claim about an asset that may no longer be true. If the window is twenty-four hours, your worst case is a day of serving a withdrawn image. That is a compliance question, not a latency question.
Failures are silent and asynchronous. A run that half completes leaves the two systems in a state neither of them can describe. The person who finds out is a customer.
It duplicates governance. Once the licence expiry date exists in both systems, one of them is wrong, and you now need a rule about which. That rule is the beginning of a reconciliation layer, and reconciliation layers grow.
It cannot answer new questions. Six months on, a channel wants a 4:5 crop. The batch has been copying 1:1 and 16:9. Now you are changing the pipeline, backfilling, and re-running against the full catalogue.
The general shape here is extract, transform, load, which is the correct tool for moving analytical data and the wrong one for keeping two operational systems in agreement about the same object.
The reference contract
Three pieces, and they are all small.
One: the product record stores an identifier, not a file. A stable string that means “this asset” for the life of the asset. Not a URL, because URLs carry delivery parameters and those change. Not a filename, because filenames get edited. An identifier.
Two: the asset system resolves an identifier plus a set of parameters into a delivery URL. This is the whole API surface you need for reads, and it is why the pattern works: the URL can be constructed by the consumer rather than requested from the provider. In Cloudinary the identifier is the public ID and the parameters sit in the URL path, so a storefront can build the exact URL it needs without a round trip to the asset system at all.
Three: changes are pushed, not polled. When an asset is replaced, withdrawn or re-approved, the asset system emits an event and interested systems react. Webhook notifications are the standard mechanism. What you cache, you cache with an explicit invalidation path rather than a fixed interval.
That is the entire integration. No staging directory, no queue you maintain, no Thursday rule.
What about the metadata the product system genuinely needs?
This is the fair objection, and it has a fair answer.
Some asset metadata really does need to be queryable from the product side. Merchandising wants to filter products whose primary image is missing. Legal wants a list of products using assets that expire this quarter. Neither of those can be answered if the asset metadata lives entirely elsewhere and is only resolvable one record at a time.
The answer is a read model, not a copy. Maintain a projection of the few asset fields the product side needs to query, populated by events, explicitly labelled as derived, and never editable in the product system. Three fields is typical: asset exists, approval state, expiry date. That is a cache with a defined invalidation, which is exactly what the authority rule in DAM vs PIM permits.
The discipline is that nobody may add a fourth field to the projection without saying which query needs it. Left ungoverned, a read model becomes a full copy in about a year, and you are back where you started.
The crop problem, which is where most of the copying comes from
Look at any batch integration between an asset library and a product catalogue and most of the volume is not masters. It is derivatives: the square one for the marketplace, the wide one for the hero, the small one for the cart.
Those exist as files because at some point a system needed a shape and the only way to get it was to make one. If renditions are derived from the master by parameter, that entire class of transfer disappears. The product system stores one identifier, and each surface asks for the shape it needs at the point of use. Named transformations are worth using here rather than raw parameter strings, because they give each shape a name your storefront code can refer to, and let you change the definition later without editing the consumers.
There is a governance caveat worth stating plainly, because vendor material tends to skate over it. Access control in most asset platforms, Cloudinary included, is organised at the folder level rather than per asset, and there is no general API for traversing arbitrary relationships between assets. Design the integration so it does not depend on either. Model the product-to-asset relationship on the product side, where it belongs anyway, and use folder structure to carry the permission boundaries you actually need.
When is a scheduled export still correct?
Three cases, and they are real.
- The destination cannot make an outbound call. Some marketplace and retailer feeds accept a file on a schedule and nothing else. That is their contract and you meet it. Generate the feed from live data at generation time rather than from a copy you keep, and the rest of this page still applies upstream of it.
- You are migrating. A one-time bulk move is a batch job by nature. Cloudinary’s migration tooling exists for exactly this, and it should terminate rather than become a schedule.
- Analytical reporting. Copying into a warehouse for analysis is fine, because nothing operational reads from it.
Everything else that is currently a schedule is a reference waiting to be written.
How do you migrate an existing sync to references?
Incrementally, and in this order. Add the identifier field to the product record and populate it from the existing copies, which you can usually do by matching on filename or hash. Switch one surface, ideally a low-traffic one, to resolve from the identifier. Add the event subscription so the read model stays current. Then stop the batch and delete the staging directory, which is the step teams postpone for a year because the old pipeline is still technically working.
The API-side questions to ask the product platform before you commit to any of this are in what to demand from a PIM API before you buy, and the wider principle is in single source of truth applied to product data. If the four-system version of the boundary is still unclear, DAM, PIM, CMS and ERP maps it. For the developer-facing case for adopting a derived-rendition model from the first upload rather than after the pipeline exists, see DAM for developers.