Combine all visible worksheets from multiple #Excel files in a folder

It is a common task for us to combine all worksheets in a workbook. It could be a time consuming task without Power Query. With Power Query, it’s piece of cake. 😁

Wait? What about if you want to combine all visible worksheets from all Excel files in a folder? It requires a little more Magic from Power Query but it’s totally achievable without deep-diving into M code… just touching the surface would do. πŸ˜‰

Let’s watch it in action:

You may download the sample files HERE to follow along:

(note: suggest you put the Q4 file aside for “refresh” later)

Step-by-Step instructions with screenshots

If you prefer reading to watching, please continue to read the step-by-step instructions below.

Open a blank Excel workbook.

  1. Go to Data tab
  2. Get Data
  3. From File
  4. From Folder

  1. Browse the folder where you have the Excel workbooks for consolidation
  2. OK
  1. Open the pull down for “Combine” ==> Combine and Transform Data
(Note: In earlier versions of Excel, you may see it as "Combine and Edit")

  1. Right-Click the “Parameter1”
  2. Select Transform Data (or Edit)

At this point, the Power Query Editor opens with all the information from each workbook loaded into it.

Since all tables on worksheets across different workbooks follow exactly the same layout and structure, we may apply the same set of transformation steps to all the tables involved.

A sample table is needed to record the required transformation steps before we combine all the tables. Let’s use the first table as sample table.

  1. Right-Click on an empty space next to the first table
  2. Select “Add as New Query”

A new query “Data” is added. Let’s rename the query name as well as the last step of the query.

  1. Change the query name from “Data” to “fn_RemoveTopRowsAndUnpivot(because I am going to remove unwanted rows and then unpivot the table in order to normalize the column names on each table. After all, I will convert this query to a function that will be invoked later.)
  2. Change the last step of the query from “Data1” to “StartingPoint

After the renaming, you should see the below in the Query Settings Pane:

Let’s do the first transformation – remove unwanted rows

  1. Go to Home tab
  2. Remove Rows
  3. Remove Top Rows
  4. Input 2 as the number of rows to be removed (as our tables start on row 3)
  5. OK

We are now ready to promote the first row as headers

  1. Click the “Use First Row as Headers” on the Home tab

After this “Promote Header” step, a step “Change Type1” is generated automatically by Power Query. We don’t want this step. Click the ❌ on the left to remove it.

Now we are ready to UNPIVOT the table to normalize all the columns headers. Why we need this step? Think about the fact that there are different columns headers in different files. For Q1 file, we have headers of Week01 to Week 13; while for Q2 file, we have headers of Week14 to Week26. And so on and so forth.

Because of this, we cannot directly “Append” the tables together as each column hold sales amount of different periods. Therefore, we need to UNPIVOT the tables to make sales data of all “weeks” to be fallen on a single column (attribute).

To UNPIVOT

  1. Right-Click the anchor column “Store”
  2. Select “Unpivot Other Columns”

You won’t believe how easy it is to unpiovt a table with Power Query. BUT IT IS! πŸ™‚

Let’s rename the “Attribute” and “Value” to “Week” and “Sales Amt” respectively. (by double-clicking the header and then rename)

Now under the column “Week”, the word “Week” in the data is abundant. We can remove it by using “Replace values”.

  1. Select the column “Week”
  2. Go to Home tab
  3. Click “Replace values”
  4. Input “Week” in the “Value To Find”
  5. Input nothing under “Replace with”
  6. OK

In plain English, the above steps instruct Power Query to replace the word “Week” with nothing in the “Week” column. Make sense?

Let’s define the data type accordingly!

Take a look at the Query Settings pane. These are the six steps we performed to the sample query. And these six steps are all we care about; these six steps will be applied to tables in all the Excel workbooks in the folder.

To convert these six steps into a function for later use, we need to open the Advanced Editor.

  1. Go to View tab
  2. Select “Advanced Editor”

Don’t be scared by the codes in the Advanced Editor. At normal circumstance, each line of code represents each step you see on the Query Settings Pane. And these codes are generated automatically by Power Query. At this point, we don’t have to know all the functional codes here. We just need to identify the codes where the six steps we want. Everything else under the “let” statement can be deleted.

These are the codes of the six steps we want.

To turn these steps into a function, we need to input the following on top of the let statements. On the first row, insert:

(StartingPoint as table) =>

Why “StartingPoint as table”?

Because that is the parameter (table) being fed into the first step Table.Skip(StartingPoint,2)

When it is DONE, we see that Power Query turns the query into a function which we can call for later.

Now, we are ready to apply this custom function to the “Start” query.

Select the “Start” query

Keep the columns we need.

  1. Select both “Name” and “Data” (where name is the worksheet name that happens to be the market name. I want to keep this information. Whereas “Data” resides all the information we need as table).
  2. Select “Remove Other Columns”

Rename “Name” to “Market”

At this point, we should have kept only two columns:

To Invoke the function to all the tables under “Data”

  1. Go to Add Column tab –> Select Invoke Custom Function
  2. Open the pull down for “Function query”
  3. Select “fn_RemoveTopRowsAndUnpivot” (that is the function we created before)
  4. Open the parameter input option
  5. Select “Column Name” (as we want to invoke the function to all tables in “Data” column)
  6. Select “Data” (which is the column holding all tables to be transformed)
  7. OK

Here we go! An additional column with “transformed” table! 🀞

The original “raw” table can be removed.

  1. Right-Click the “Data” column
  2. Select “Remove”

To expand all the data fields from the “transformed” table

  1. Click on the “expand” icon
  2. Un-check the “Use original column name as prefix”
  3. OK

Here we go!

As good practice, always define data type for each columns before loading the data.

To load the result to worksheet

  1. Go to Home tab
  2. Click Close & Load

Here’s the consolidated data; with all the data from all worksheets in each Excel workbook in the folder! Amazing!

Wait… the title of this blogpost is

Combine all visible worksheets from multiple #Excel files in a folder

So, what if there are hidden worksheets in one of the files in the folder?

I leave that part in the final section in the video. Don’t miss the video starting 12:05 where you will see how to look into visible worksheets only and avoid potential errors.

Do you feel the Power? If you do, please spread the M(agic). (❁´◑`❁)

About MF

An Excel nerd who just transition into a role related to data analytics at current company......😊 Recently in love with Power Query and Power BI.😍 Keep learning new Excel and Power BI stuffs and be amazed by all the new discoveries.
This entry was posted in Power Query and tagged , , , . Bookmark the permalink.

Comments, suggestions, corrections are welcome.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.