How to combine multiple account extractions into one CSV with SFDMU?

Question:

I am trying to combine accounts from several extractions into one Account.csv file and subsequently fetch related contacts using another job in SFDMU.

I encounter issues with the accounts CSV being overwritten after each extraction.

Here is what my export.json file looks like:

{
  "allOrNone": true,
  "useSeparatedCSVFiles": false,
  "objectSets": [
    {
      "objects": [
        {
          "query": "SELECT all FROM Product2 WHERE Product_Hierachy__c IN ('12345', '54321')",
          "operation": "Upsert"
        },
        {
          "query": "SELECT all FROM PriceBookEntry",
          "master": false,
          "operation": "Upsert"
        },
        {
          "query": "SELECT all FROM OpportunityLineItem",
          "master": false,
          "operation": "Upsert"
        },
        {
          "query": "SELECT all FROM Opportunity",
          "master": false,
          "operation": "Upsert"
        },
        {
          "query": "SELECT all FROM Campaign",
          "master": false,
          "operation": "Upsert"
        },
        {
          "object": "Account",
          "query": "SELECT all FROM Account WHERE RecordType.DeveloperName IN ('X1', 'X2')",
          "master": false,
          "operation": "Upsert"
        }
      ]
    },
    {
      "objects": [
        {
          "object": "Account",
          "query": "SELECT all FROM Account WHERE RecordType.DeveloperName IN ('X3', 'X4')",
          "operation": "Upsert"
        }
      ]
    },
    {
      "objects": [
        {
          "query": "SELECT all FROM Account",
          "useCSVValuesMapping": true,
          "operation": "Upsert"
        },
        {
          "query": "SELECT all FROM Contact",
          "master": false,
          "operation": "Upsert"
        }
      ]
    }
  ]
}

How can I prevent the CSV from being overwritten after each extraction?

Answer:

As documented here Multiple Object Sets, SFDMU handles multiple extractions by treating them as separate instances of data migration, each configured within its own ObjectSet in the export.json file. This means each run of SFDMU with different ObjectSets is treated as an independent operation, leading to the overwriting of previous data if not properly managed.

There is no option to automatically consolidate multiple exports of Accounts from different object sets into one Account.csv to allow executing the last object set using the final set of Accounts. But to prevent overriding Account.csv after each object set run, set useSeparatedCSVFiles=true (see the documentation).

Last updated on 29th Apr 2024