It is common for applications to use for recording such as performance values or execution logs. And there could be more than one files which can give you a meaningful outcome only after their consolidation. In this example, we are demonstrating how to merge multiple files using Python without losing any data.

A CSV file, as the name suggests combines multiple fields separated by commas. The first row contains the name or title of each column, and remaining rows contain the actual data values.

Merge Multiple CSV Files in Python

Merge Multiple CSV Files in Python  - Merge Multiple CSV Files in Python - Python Example – How to merge multiple CSV files into one

Merge Multiple CSV Files

Below is the complete code to perform the merging of CSV files. We used a CSV format which records the performance data for web services. While using this example, you may need to modify the HEADER variable (with the name of actual fields) as per the format of your CSV.

It is essential to understand the header fields we’ve used in this example. It includes the following:

  • Timestamp – The of a web service request
  • Client IP – The IP of the client system making the calls
  • Web Service – The name of the web service used
  • Status – The return code reflecting success or failure
  • GOOD – The total no. of successful requests
  • BAD – The total no. of failure calls
import os

csv_header = 'Timestamp,Client IP,Web Service,Status,Good,Bad'
csv_out = 'consolidated.csv'

csv_dir = os.getcwd()

dir_tree = os.walk(csv_dir)
for dirpath, dirnames, filenames in dir_tree:

csv_list = []
for file in filenames:
   if file.endswith('.csv'):

csv_merge = open(csv_out, 'w')

for file in csv_list:
   csv_in = open(file)
   for line in csv_in:
      if line.startswith(csv_header):
print('Verify consolidated CSV file : ' + csv_out)

A quick wrap up – Merge Multiple CSV Files

In this example, we covered “How to Merge Multiple CSV Files in Python.” It doesn’t use any special Python package to combine the CSV files and can you a lot of time from going through multiple CSV individually.

If you’ve learned something from this example, then care to share it with your colleagues. Also, connect to our social media (Facebook/Twitter) accounts to receive timely updates.



Source link


Please enter your comment!
Please enter your name here