Python Parse Output Of Mixed Format Text File To Key Value Pair Dictionaries

I am trying to parse the output of a command from the EMC OneFS CLI. We are trying to collect inventory data, and since we're on an older version of the API we can't use the REST S

Solution 1:

I think the best way would be to use re.match():

#!/usr/bin/env python2.7

import re

results = {}

with open('in.txt', 'r') as f:
    for line in [ l.rstrip() for l in f.readlines() ]:
        match = re.match('^(?P<hostname>[^:]+): +(?P<key>[^:=]+?)(?: *[:=] *(?P<value>.+?) *)?$', line)

        if match is None:
            # skip failed matches
            #print(line)
            continue

        hostname, key, value = match.groups()

        if key == 'Comps':
            value = [ x.strip() for x in re.sub('  +',' ',value).split(' ') ]
        else:
            try:
                value = float(value)
            except:
                pass

        if hostname not in results:
            results[hostname] = {}

        if key in results[hostname]:
            old_value = results[hostname][key]
            if isinstance(old_value, list):
                old_value.append(value)
            else:
                results[hostname][key] = [ old_value, value ]
        else:
            results[hostname][key] = value

from pprint import pprint
pprint(results)

I've also taken some liberties:

• Proposed an alternative result format results[hostname][key]
• Converted the 'Comps' data into an array
• Converted anything that will be converted as a float, into a number

Output is like this:

{'visi-91': {'CPU': 'GenuineIntel (2.33GHz, stepping 0x0001067a)',
             'CPU Operation (raw 0x882F0000)': 'Normal',
             'CPU Speed Limit': '100.00%',
             'ChasCnt': '1 (Single-Chassis System)',
             'Chassis': 'ISI36 (Isilon (CIDesigns) 36-Bay Chassis)',
             'Chassis Fan1 (ISI F1)': 7469.0,
             'Chassis Fan2 (ISI F2)': 7552.0,
             'Chassis Fan3 (ISI F3)': 7387.0,
             'Class': 'storage',
             'Comps': ['Chassis',
                       'CPU',
                       'Motherboard',
                       'NVRam',
                       'FlashDrive',
                       'Disk_Controller',
                       'Disk_Expander',
                       'Power_Supplies'],
             'Config': '912-111-231',
             'DskCtl': 'LSI3081E-HBA (LSI SAS3081E SAS Controller -- HBA) (8 ports)',
             'DskExp': 'VIT7156 (Vitesse 7156 Disk Expander)',
             'FlshDrv': 'None (No physical dongle supported) ((null))',
             'HWGen': 'MaunaLoa (MaunaLoa Hardware)',
             'IBType': 'ISI25208 (Isilon MT25208-based IB Card)',
             'IMB': 'Board Version 0x20f',
             'LCDver': 'NoriVFD (Noritake VFD)',
             'Mobo': 'SupMicX7DBU (SuperMicro X7DBU Motherboard)',
             'NVRam': 'MT25208 (Mellanox 25208 based Card) (492MB card) (size 515899392B)',
             'NetIF': 'em*4,ib*2 (4x Intel, 2x IB)',
             'PROC': 'Dual-proc, Quad-core',
             'Power Supplies OK': None,
             'Power Supply 1 Fan': 5504.0,
             'Power Supply 2 Fan': 5792.0,
             'Product': 'IQ 32000x-ssd',
             'PwrSupl': ['PS1 (type=Emerson, fw=v.01.00.00)',
                         'PS2 (type=Emerson, fw=v.01.00.00)'],
             'RAM': '17150812160 Bytes',
             'SerNo': 'S123456789P',
             'Series': 'x_series',
             'Temp Chassis 2 (ISI T2)': 22.0,
             'Temp Front Panel': 17.0,
             'Temp System': 25.0,
             'Temp Until CPU Throttle (CPU 0)': 46.0,
             'Temp Until CPU Throttle (CPU 1)': 42.0},
 'visi-92': {'CPU': 'GenuineIntel (2.33GHz, stepping 0x0001067a)',
             'CPU Operation (raw 0x882E0000)': 'Normal',
             'CPU Speed Limit': '100.00%',
             'ChasCnt': '1 (Single-Chassis System)',
             'Chassis': 'ISI36 (Isilon (CIDesigns) 36-Bay Chassis)',
             'Chassis Fan1 (ISI F1)': 7268.0,
             'Chassis Fan2 (ISI F2)': 7307.0,
             'Chassis Fan3 (ISI F3)': 7190.0,
             'Class': 'storage',
             'Comps': ['Chassis',
                       'CPU',
                       'Motherboard',
                       'NVRam',
                       'FlashDrive',
                       'Disk_Controller',
                       'Disk_Expander',
                       'Power_Supplies'],
             'Config': '912-111-231',
             'DskCtl': 'LSI3081E-HBA (LSI SAS3081E SAS Controller -- HBA) (8 ports)',
             'DskExp': 'VIT7156 (Vitesse 7156 Disk Expander)',
             'FlshDrv': 'None (No physical dongle supported) ((null))',
             'HWGen': 'MaunaLoa (MaunaLoa Hardware)',
             'IBType': 'ISI25208 (Isilon MT25208-based IB Card)',
             'IMB': 'Board Version 0x20f',
             'LCDver': 'NoriVFD (Noritake VFD)',
             'Mobo': 'SupMicX7DBU (SuperMicro X7DBU Motherboard)',
             'NVRam': 'MT25208 (Mellanox 25208 based Card) (492MB card) (size 515899392B)',
             'NetIF': 'em*4,ib*2 (4x Intel, 2x IB)',
             'PROC': 'Dual-proc, Quad-core',
             'Power Supplies OK': None,
             'Power Supply 1 Fan': 5376.0,
             'Power Supply 2 Fan': 5696.0,
             'Product': 'IQ 32000x-ssd',
             'PwrSupl': ['PS1 (type=Emerson, fw=v.01.00.00)',
                         'PS2 (type=Emerson, fw=v.01.00.00)'],
             'RAM': '17150812160 Bytes',
             'SerNo': 'S123456785K',
             'Series': 'x_series',
             'Temp Chassis 2 (ISI T2)': 24.0,
             'Temp Front Panel': 17.2,
             'Temp System': 27.0,
             'Temp Until CPU Throttle (CPU 0)': 46.0,
             'Temp Until CPU Throttle (CPU 1)': 46.0},
 'visi-93': {'CPU': 'GenuineIntel (2.33GHz, stepping 0x0001067a)',
             'CPU Operation (raw 0x88250000)': 'Normal',
             'CPU Speed Limit': '100.00%',
             'ChasCnt': '1 (Single-Chassis System)',
             'Chassis': 'ISI36 (Isilon (CIDesigns) 36-Bay Chassis)',
             'Chassis Fan1 (ISI F1)': 7387.0,
             'Chassis Fan2 (ISI F2)': 7152.0,
             'Chassis Fan3 (ISI F3)': 7152.0,
             'Class': 'storage',
             'Comps': ['Chassis',
                       'CPU',
                       'Motherboard',
                       'NVRam',
                       'FlashDrive',
                       'Disk_Controller',
                       'Disk_Expander',
                       'Power_Supplies'],
             'Config': '912-111-231',
             'DskCtl': 'LSI3081E-HBA (LSI SAS3081E SAS Controller -- HBA) (8 ports)',
             'DskExp': 'VIT7156 (Vitesse 7156 Disk Expander)',
             'FlshDrv': 'None (No physical dongle supported) ((null))',
             'HWGen': 'MaunaLoa (MaunaLoa Hardware)',
             'IBType': 'ISI25208 (Isilon MT25208-based IB Card)',
             'IMB': 'Board Version 0x20f',
             'LCDver': 'NoriVFD (Noritake VFD)',
             'Mobo': 'SupMicX7DBU (SuperMicro X7DBU Motherboard)',
             'NVRam': 'MT25208 (Mellanox 25208 based Card) (492MB card) (size 515899392B)',
             'NetIF': 'em*4,ib*2 (4x Intel, 2x IB)',
             'PROC': 'Dual-proc, Quad-core',
             'Power Supplies OK': None,
             'Power Supply 1 Fan': 8512.0,
             'Power Supply 2 Fan': 8512.0,
             'Product': 'IQ 32000x-ssd',
             'PwrSupl': ['PS1 (type=Emerson, fw=v.03.05.00)',
                         'PS2 (type=Emerson, fw=v.03.05.00)'],
             'RAM': '17150812160 Bytes',
             'SerNo': 'S987654321P',
             'Series': 'x_series',
             'Temp Chassis 2 (ISI T2)': 25.0,
             'Temp Front Panel': 17.4,
             'Temp System': 28.0,
             'Temp Until CPU Throttle (CPU 0)': 37.0,
             'Temp Until CPU Throttle (CPU 1)': 38.0}}

Edit: Now handles multiple occurrences of a given 'key', by forming an array (e.g: PwrSupl).

Solution 2:

You need to split each of those lines at colons. Plop the first one into the dictionary as "hostname", then process the remaining ones in pairs. Something like:

fields = line.split(':')
my_dict["hostname"] = field[0]
for pos in range(1, len(fields), 2):
    my_dict[field(pos)] = field(pos+1)

That's the basic tactic; I'll leave implementation details to you. :-)