Archive for the ‘Programmierung’ Category.

Mit Python pandas CSV nach Excel konvertieren

Hier ein einfacher Code-Schnipsel, um aus CSV-Dateien Excel-Dateien zu machen.

Es empfiehlt sich auch, die Engine explizit zu setzen, mit der die Excel-Datei geschrieben wird. Standardmäßig nutzt pandas openpyxl, xlsxwriter scheint nach meinen Tests aber um ca. 30% schneller zu sein.

import pandas as pd
from openpyxl import Workbook
from openpyxl.utils.dataframe import dataframe_to_rows
 
 
# output control
files = ['']
 
wb = Workbook()
wb.remove(wb['Sheet'])
 
for index, file in enumerate(files,0):
    temp = pd.read_csv(file + '.csv',sep='\t')
    ws1 = wb.create_sheet()
    ws1.title = file
    rows = dataframe_to_rows(temp, index=False, header=True)
 
    for r_idx, row in enumerate(rows, 1):
        for c_idx, value in enumerate(row, 1):
            ws1.cell(row=r_idx, column=c_idx, value=value)
 
wb.save('files45646.xlsx')
wb.close()

Uwe

Uwe Ziegenhagen likes LaTeX and Python, sometimes even combined. Do you like my content and would like to thank me for it? Consider making a small donation to my local fablab, the Dingfabrik Köln. Details on how to donate can be found here Spenden für die Dingfabrik.

More Posts - Website

Werte in pandas Dataframes ersetzen mit replace()

Aus aktuellem Anlass hier ein einfaches Beispiel, wie man Werte in pandas Dataframes ersetzen kann:

import pandas as pd
 
# initialize data of lists.
data = {'Hersteller': ['VW', 'BMW', 'VW', 'Porsche'],
        'Modell': ['Golf', '1er', 'Polo', '911']}
 
 
df = pd.DataFrame(data)
 
 
print(df,'\n')
 
 
df['Hersteller'] = df['Hersteller'].replace(
    {"VW": "Volkswagen", "Horch": "Audi"})
 
 
print(df)

Uwe

Uwe Ziegenhagen likes LaTeX and Python, sometimes even combined. Do you like my content and would like to thank me for it? Consider making a small donation to my local fablab, the Dingfabrik Köln. Details on how to donate can be found here Spenden für die Dingfabrik.

More Posts - Website

DuckDB Beispiel-Code für Python

Hier ein Code-Beispiel für die wichtigsten Funktionen von DuckDB.

import duckdb as ddb
import pandas as pd
 
con = ddb.connect(':memory:')
 
con_p = ddb.connect('my_database.db')
 
con_p.execute('CREATE OR REPLACE TABLE telefonnummern(fullname VARCHAR,phone VARCHAR);')
 
 
con_p.execute("INSERT INTO telefonnummern VALUES ('Max Mustermann', '0123-4567890')")
 
print(con_p.sql('SHOW ALL TABLES'))
 
print(con_p.sql('SELECT * FROM telefonnummern;'))
 
 
ddb_object = con_p.sql('SELECT * FROM telefonnummern;')
 
df = ddb_object.to_df()
 
ddb_tuple = ddb_object.fetchall()
 
print(df)
 
print(ddb_tuple)

Uwe

Uwe Ziegenhagen likes LaTeX and Python, sometimes even combined. Do you like my content and would like to thank me for it? Consider making a small donation to my local fablab, the Dingfabrik Köln. Details on how to donate can be found here Spenden für die Dingfabrik.

More Posts - Website

Python: print() durch ic() ersetzen

Hier ein paar Beispiele, wie man mit icecream print() Ausgaben ersetzen kann.

"""
icecream examples
"""
 
 
from icecream import ic
 
# define some function
def addiere(x, y):
    return x + y
 
# call ice
ic(addiere(1, 2))
 
# Output:
# ic| addiere(1, 2): 3
 
 
d = {'i': 2, 'j': 3, 'k': 4711}
ic(d['k'])
 
 
struct = {
    "hersteller": "VW",
    "modell": "Golf",
    "Farben": ["gelb", "rot"]
}
 
ic(struct)
 
ic.disable()
ic(struct) # no output
ic.enable()
 
 
def logstuff(text):
    # log to output file
    print(text)
 
 
ic.configureOutput(prefix="Hallo| ", outputFunction=logstuff)
 
ic(addiere(7, 7))
 
 
ic.configureOutput(prefix="Welt| ", outputFunction=logstuff)
 
ic(addiere(7, 7))

Uwe

Uwe Ziegenhagen likes LaTeX and Python, sometimes even combined. Do you like my content and would like to thank me for it? Consider making a small donation to my local fablab, the Dingfabrik Köln. Details on how to donate can be found here Spenden für die Dingfabrik.

More Posts - Website

Mit Python camt.053 aus MT940 erzeugen

Für Dante e.V. bestand die Notwendigkeit, aus MT940 Dateien moderne CAMT.053 zu erzeugen, dank Python wurde das eine lösbare Aufgabe.

Schritt 1

Die MT940 Datei parsen und die Transaktionen in einen pandas DataFrame überführen.

import mt940
import pprint
import pandas as pd

df = pd.DataFrame()

transactions = mt940.parse('Umsaetze_2310007_22.07.2024.mta')

print('Transactions:')
pprint.pprint(transactions.data)

for transaction in transactions:
    print('Transaction: ', transaction)
    pprint.pprint(transaction.data)
    t = transaction.data
    tt = pd.DataFrame(t, index=[0])

    df = pd.concat([df,tt],ignore_index=True)
    
df.to_excel('AllBookings.xlsx',index=False)

Schritt 2

Aus dem DataFrame das XML befüllen, die für den Kopf der XML-Datei notwendigen Kontostandsinformationen holen ich dazu aus der MT940 Datei.

import pandas as pd # data wrangling
import jinja2 # template engine
import os # for file-related stuff
import mt940
from datetime import datetime


today = datetime.today()
now = today.strftime("%Y-%m-%d")



transactions = mt940.parse('Umsaetze_2310007_22.07.2024.mta')

opening = transactions.data['final_opening_balance']
openingamount = str(opening.amount)[:-4]
openingdate = opening.date

closing = transactions.data['final_closing_balance']
closingamount = str(closing.amount)[:-4]
closingdate = closing.date

 
# create jinja env that can load template from filesystem
jinja_env = jinja2.Environment(loader = jinja2.FileSystemLoader(os.path.abspath('.')))
 
df = pd.read_excel('AllBookings.xlsx', dtype={'date': str,'amount':str})

df['CreditDebit'] = ''
df['CreditDebit'] = df['CreditDebit'].where(df['amount'].str.get(0).isin(['-']), 'CRDT')
df['CreditDebit'] = df['CreditDebit'].where(~df['amount'].str.get(0).isin(['-']), 'DBIT')

df['date'] = df['date'].str[:-9]


# Währung weg
df['amount'] = df['amount'].str[:-4]
# Vorzeichen weg
df['amount'] = df['amount'].str.replace('-','')

#df['amount'].replace('-','',inplace=True)
#df["amount"] = df["amount"].apply(lambda x: x.str.replace("-", ""))

template = jinja_env.get_template('Ntry.xml')
 
with open('FertigesXML.xml','w') as output:
    output.write(template.render(data=df,
                                 openingamount=openingamount,
                                 openingdate=openingdate,
                                 closingamount=closingamount,
                                 closingdate=closingdate
                                 ))

Jinja2 XML-Template

Das XML-Template für Jinja2 findet ihr hier:

Ntry_blog

Uwe

Uwe Ziegenhagen likes LaTeX and Python, sometimes even combined. Do you like my content and would like to thank me for it? Consider making a small donation to my local fablab, the Dingfabrik Köln. Details on how to donate can be found here Spenden für die Dingfabrik.

More Posts - Website

Text umkehren in Excel

Text lässt sich „einfach“ umkehren in Excel, auch ohne VBA.

Wenn der umzukehrende Text in Zelle A1 steht, dann hilft die folgende Formel


=TEXTVERKETTEN("";WAHR;TEIL(A1;LÄNGE(A1)-ZEILE(INDIREKT("1:"&LÄNGE(A1)))+1;1))

Mit Strg-Shift-Enter abschließen, um eine Array-Formel zu erstellen.

Uwe

Uwe Ziegenhagen likes LaTeX and Python, sometimes even combined. Do you like my content and would like to thank me for it? Consider making a small donation to my local fablab, the Dingfabrik Köln. Details on how to donate can be found here Spenden für die Dingfabrik.

More Posts - Website

T-SQL: Mit Daten rechnen

Hier eine Übersicht zum Errechnen von Datumswerten in T-SQL

Datum SQL
Today getdate()
Yesterday DATEADD(day, -1, CAST(GETDATE()))
Tomorrow DATEADD(day, 1, CAST(GETDATE()))
First day of the previous month DATEADD(MONTH, DATEDIFF(MONTH, 0, GETDATE())-1, 0)
Last day of the previous month DATEADD(MONTH, DATEDIFF(MONTH, -1, GETDATE())-1, -1) --Last Day of previous month

Uwe

Uwe Ziegenhagen likes LaTeX and Python, sometimes even combined. Do you like my content and would like to thank me for it? Consider making a small donation to my local fablab, the Dingfabrik Köln. Details on how to donate can be found here Spenden für die Dingfabrik.

More Posts - Website

T-SQL Rowcount von Tabellen ermitteln

Stackoverflow (https://stackoverflow.com/questions/2221555/how-to-fetch-the-row-count-for-all-tables-in-a-sql-server-database) hatte gestern interessanten Code für mich, um den Rowcount aller Tabellen in einer MS SQL Server DB zu ermitteln:


SELECT o.NAME,
i.rowcnt
FROM sysindexes AS i
INNER JOIN sysobjects AS o ON i.id = o.id
WHERE i.indid < 2 AND OBJECTPROPERTY(o.id, 'IsMSShipped') = 0;

Uwe

Uwe Ziegenhagen likes LaTeX and Python, sometimes even combined. Do you like my content and would like to thank me for it? Consider making a small donation to my local fablab, the Dingfabrik Köln. Details on how to donate can be found here Spenden für die Dingfabrik.

More Posts - Website

SFTP mit Python und der Paramiko-Bibliothek – Upload

Neben dem Download von Dateien klappt auch der Upload von Dateien problemlos.

import os
import paramiko
 
# Replace these variables with your specific values
host = '192.168.0.22'
port = 22
username = '<user>'
private_key_path = 'keyfile'
remote_directory_path = '/home/uwe/uploadtest'
local_directory_path = 'E:/uploadtest'
 
 
# Establish an SSH transport session
private_key = paramiko.RSAKey(filename=private_key_path)
transport = paramiko.Transport((host, port))
transport.connect(username=username, pkey=private_key)
 
# Create an SFTP client
sftp = paramiko.SFTPClient.from_transport(transport)
 
try:
    # Iterate through local files in the specified folder
    for local_file in os.listdir(local_directory_path):
        local_file_path = os.path.join(local_directory_path, local_file)
 
        # Check if the file is a CSV file
        if os.path.isfile(local_file_path): # and local_file.lower().endswith('.csv'):
            remote_file_path = os.path.join(remote_directory_path, local_file)
 
            # Upload the CSV file
            sftp.put(local_file_path, remote_file_path)
            print(f"Uploaded: {local_file} to {remote_file_path}")
 
finally:
    # Close the SFTP session and SSH transport
    sftp.close()
    transport.close()

Uwe

Uwe Ziegenhagen likes LaTeX and Python, sometimes even combined. Do you like my content and would like to thank me for it? Consider making a small donation to my local fablab, the Dingfabrik Köln. Details on how to donate can be found here Spenden für die Dingfabrik.

More Posts - Website

SFTP mit Python und der Paramiko-Bibliothek – Download

Aktuell benötige ich Funktionen, um mit Python Dateien von SFTP Servern zu holen bzw. Dateien auf diese hochzuladen. Chat GPT hatte folgenden Code für mich, der sehr gut funktioniert.

import os
import paramiko
 
# Replace these variables with your specific values
host = '192.168.0.238'
port = 22
username = '<user>'
private_key_path = '<keyfile>'
remote_directory_path = '/home/uwe/downloadtest'
local_directory_path = 'E:/downloadtest'
 
# Establish SSH connection
try:
    # Create a new SSH client
    ssh_client = paramiko.SSHClient()
 
    # Automatically add the server's host key
    ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
 
    # Load the private key for authentication
    private_key = paramiko.RSAKey.from_private_key_file(private_key_path)
 
    # Connect to the server
    ssh_client.connect(hostname=host, port=port, username=username, pkey=private_key)
 
    # Open an SFTP session on the SSH connection
    sftp = ssh_client.open_sftp()
 
    # Change to the remote directory
    sftp.chdir(remote_directory_path)
 
    # List all files in the remote directory
    files = sftp.listdir()
 
    # Download each CSV file in the remote directory
 
    for file_name in files:
        # os path join uses system slashes, must make sure they are right
        remote_file_path = os.path.join(remote_directory_path, file_name).replace("\\","/")
        local_file_path = os.path.join(local_directory_path, file_name).replace("\\","/")
        print(remote_file_path, local_file_path)
 
        # Check if the file is a CSV file
        if file_name.lower().endswith('.txt'):
            sftp.get(remote_file_path, local_file_path)
            print(f"File '{file_name}' downloaded successfully to '{local_directory_path}'")
 
    # Close the SFTP session and SSH connection
    sftp.close()
    ssh_client.close()
 
except paramiko.AuthenticationException:
    print("Authentication failed. Please check your credentials or SSH key path.")
except paramiko.SSHException as e:
    print(f"SSH connection failed: {e}")
except FileNotFoundError:
    print("File not found. Please provide the correct file paths.")
except Exception as e:
    print(f"An error occurred: {e}")

Uwe

Uwe Ziegenhagen likes LaTeX and Python, sometimes even combined. Do you like my content and would like to thank me for it? Consider making a small donation to my local fablab, the Dingfabrik Köln. Details on how to donate can be found here Spenden für die Dingfabrik.

More Posts - Website