Learning machine learning? Try my machine learning flashcards or Machine Learning with Python Cookbook.

Concatenate Multiple Files

17 Jun 2018

Often large collections of data will be broken into multiple smaller files. The command line makes it easy to concatenate the contents of these files into a single file for analysis.

Create Multiple Files

In this example each file contains a single record of a sale, however in the real world each file can contain hundreds or even millions of records.

echo 'Alan Jones', '$50' > sales_records_1.csv
echo 'Stephanie Lawson', '$20' > sales_records_2.csv
echo 'Lester Holt', '$10' > sales_records_3.csv

Append All Files To A Single File

Specifically, what we are doing in the code below is appending (>>) every file with sales_records_ in the file name (e.g. sales_records_1.csv) to a file called sales_records.csv.

cat sales_records_* >> sales_records.csv

View The Single File To Confirm The Appending Worked

cat sales_records.csv

Alan Jones, $50
Stephanie Lawson, $20
Lester Holt, $10

Note that the contents of the files were concatenated in ascending order, meaning sales_records_1.csv’s data comes before sales_records_1.csv’s data in the final file.