537 lines
23 KiB
Plaintext
537 lines
23 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "62166c19",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Lab Work 1: Working with missing data\n",
|
||
"This excercise should give you some practice in working with missing data of different feature types.\n",
|
||
"\n",
|
||
"**Please note, that you can only pass the intial checking, if you write Markdown documention about your findings (not code documentation). Any submission that does not adhere to that will lead to an immediate fail, without the chance of resubmission!**"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "95a1362e",
|
||
"metadata": {},
|
||
"source": [
|
||
"# 1. Load the iris dataset with missing values into a dataframe \n",
|
||
"File: datasets.zip/datasets/iris/data_someMissing.all\n",
|
||
"\n",
|
||
"Hint: When data is missing, pandas might not be able to determine the proper type of columns by itself. Look carefully at the data types and act accordingly! You have different options to change the types of columns:\n",
|
||
"* When reading, have a look at [pandas.read_csv](https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html), esp the parameter *na_filter*\n",
|
||
"* In memory, have a look at [dataframe.astype](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.astype.html) and the transformation functions [dataframe.apply](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.apply.html) in combination with [pandas.to_numeric](https://pandas.pydata.org/docs/reference/api/pandas.to_numeric.html)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"id": "6f8573a0",
|
||
"metadata": {},
|
||
"source": [
|
||
"import pandas as pd\n",
|
||
"import numpy as np\n",
|
||
"import matplotlib.pyplot as plt\n",
|
||
"import seaborn as sns\n",
|
||
"\n",
|
||
"\n",
|
||
"my_dataframe = pd.read_csv('datasets/iris/data_someMissing.all', \n",
|
||
" sep=' ', \n",
|
||
" na_values=['?'],\n",
|
||
" header=None,\n",
|
||
" names=['sl', 'sw', 'pl', 'pw', 'class'])\n",
|
||
"\n",
|
||
"print(\"Datentypen nach dem Einlesen:\")\n",
|
||
"print(my_dataframe.dtypes)\n",
|
||
"\n",
|
||
"\n",
|
||
"# Verwende hier den zweiten Ansatz mit dataframe.apply und pandas.to_numeric, da diese robuster und flexibler ist als dataframe.astype\n",
|
||
"numeric_cols = ['sl', 'sw', 'pl', 'pw']\n",
|
||
"my_dataframe[numeric_cols] = my_dataframe[numeric_cols].apply(pd.to_numeric, errors='coerce')\n",
|
||
"\n",
|
||
"print(\"\\nDatentypen nach apply(pd.to_numeric):\")\n",
|
||
"print(my_dataframe.dtypes)\n",
|
||
"\n",
|
||
"my_dataframe"
|
||
],
|
||
"outputs": [],
|
||
"execution_count": null
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "26b12836",
|
||
"metadata": {},
|
||
"source": [
|
||
"# 2. What are your options to work with the missing values? \n",
|
||
"- Löschen (Deletion)\n",
|
||
" - Listwise Deletion: Entferne alle Zeilen mit fehlenden Werten\n",
|
||
"- Imputation mit statistischen Werten\n",
|
||
" - Mean Imputation: Ersetze durch Mittelwert der ganzen Spalte\n",
|
||
" - Median Imputation: Ersetze durch Median der ganzen Spalte (robuster gegen Ausreißer)\n",
|
||
" - Class-spezifische Imputation: Ersetze durch Mittelwert/Median der Spalte gefilter nach jeweiligen Klasse\n",
|
||
"- Markierung\n",
|
||
" - Fehlende Werte als separate Kategorie behandeln"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "4953a342",
|
||
"metadata": {},
|
||
"source": [
|
||
"# 3. What is their difference with respect to the features of the dataset and the class associations? \n",
|
||
"- Auswirkungen auf Features:\n",
|
||
" - Mean Imputation verändert nicht den Mittelwert, reduziert aber die Varianz\n",
|
||
" - Median Imputation ist robuster bei schiefen Verteilungen\n",
|
||
" - (Listwise) Deletion reduziert die Datenmenge und kann daher zu Bias führen, wenn unverhältnismäßig viel aus einer Klasse gelöscht wird\n",
|
||
"\n",
|
||
"- Auswirkungen auf Klassenzugehörigkeit\n",
|
||
" - Mean/ Median Imputation können die klassenspezifischen Muster durch das einsetzten global ermittelter Werte verzerren\n",
|
||
" - Class-spezifische Imputation erhält hingegen klassenspezifische Muster besser bei\n",
|
||
" - Deletion kann Klassenverteilung verändern, wenn fehlende Werte nicht zufällig verteilt sind"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "89230793",
|
||
"metadata": {},
|
||
"source": [
|
||
"# 4. Implement some of the options for the dataset\n",
|
||
"* check, how these options change the statistical values of\n",
|
||
" * each feature\n",
|
||
" * each class\n",
|
||
"* Visualize and Interpret your results\n",
|
||
" * you can use a visualization framework of your choice\n",
|
||
" * you MUST write Markdown documentation of your findings!\n",
|
||
"* useful functions in pandas for this step\n",
|
||
" * find out, if a value [is null](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.isnull.html)\n",
|
||
" * [removing data that null ](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.dropna.html?highlight=dropna)\n",
|
||
" * [fill null data with other value](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.fillna.html)\n",
|
||
" * [replace values](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.replace.html)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "bae12445",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 4.1 Implentierung der Optionen"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"id": "7d792e4f",
|
||
"metadata": {},
|
||
"source": [
|
||
"numeric_cols = ['sl', 'sw', 'pl', 'pw']\n",
|
||
"\n",
|
||
"# Original Daten\n",
|
||
"df_original = my_dataframe.copy()\n",
|
||
"\n",
|
||
"# Strategie 1: Listwise Deletion\n",
|
||
"df_deletion = my_dataframe.copy().dropna()\n",
|
||
"\n",
|
||
"# Strategie 2: Mean Imputation (global)\n",
|
||
"df_mean = my_dataframe.copy()\n",
|
||
"for col in numeric_cols:\n",
|
||
" df_mean[col].fillna(df_mean[col].mean(), inplace=True)\n",
|
||
"\n",
|
||
"# Strategie 3: Median Imputation (global)\n",
|
||
"df_median = my_dataframe.copy()\n",
|
||
"for col in numeric_cols:\n",
|
||
" df_median[col].fillna(df_median[col].median(), inplace=True)\n",
|
||
"\n",
|
||
"# Strategie 4: Class-spezifische Mean Imputation\n",
|
||
"df_class_mean = my_dataframe.copy()\n",
|
||
"for class_name in df_class_mean['class'].dropna().unique():\n",
|
||
" class_mask = df_class_mean['class'] == class_name\n",
|
||
" for col in numeric_cols:\n",
|
||
" class_mean = df_class_mean.loc[class_mask, col].mean()\n",
|
||
" df_class_mean.loc[class_mask, col] = df_class_mean.loc[class_mask, col].fillna(class_mean)\n",
|
||
"df_class_mean = df_class_mean.dropna(subset=['class']) # Fehlende Klassenlabels separat behandeln\n"
|
||
],
|
||
"outputs": [],
|
||
"execution_count": null
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "57fe2607",
|
||
"metadata": {},
|
||
"source": [
|
||
"## 4.2 Statistische Auswertung\n",
|
||
"### 4.2.1 Implementierung der statistische Auswertung"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"id": "f30c90db",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Funktion für statistische Zusammenfassung\n",
|
||
"def print_statistics(df, name, df_original=None):\n",
|
||
" print(f\"\\n{'='*50}\")\n",
|
||
" print(f\"Statistiken für: {name}\")\n",
|
||
" print(f\"{'='*50}\")\n",
|
||
" print(f\"Anzahl Datenpunkte: {len(df)} \", end='')\n",
|
||
" \n",
|
||
" if df_original is not None:\n",
|
||
" diff_count = len(df) - len(df_original)\n",
|
||
" print(f\"(Δ: {diff_count:+d})\")\n",
|
||
" else:\n",
|
||
" print()\n",
|
||
" \n",
|
||
" if df_original is not None:\n",
|
||
" print(f\"\\nDeskriptive Statistik mit Abweichungen zum Original:\")\n",
|
||
" \n",
|
||
" stats_current = df.describe()\n",
|
||
" stats_original = df_original.describe()\n",
|
||
" \n",
|
||
" # Interleaved Format: Wert | Abweichung für jede Spalte\n",
|
||
" result_dict = {}\n",
|
||
" \n",
|
||
" for col in stats_current.columns:\n",
|
||
" if col in stats_original.columns:\n",
|
||
" result_dict[f'{col}_cur'] = stats_current[col]\n",
|
||
" result_dict[f'{col}_Δ'] = stats_current[col] - stats_original[col]\n",
|
||
" \n",
|
||
" result_df = pd.DataFrame(result_dict)\n",
|
||
" \n",
|
||
" # Sortiere Spalten so, dass sie paarweise erscheinen\n",
|
||
" sorted_cols = []\n",
|
||
" base_cols = [c.replace('_cur', '').replace('_Δ', '') \n",
|
||
" for c in result_df.columns if '_cur' in c]\n",
|
||
" for base in base_cols:\n",
|
||
" sorted_cols.extend([f'{base}_cur', f'{base}_Δ'])\n",
|
||
" \n",
|
||
" print(result_df[sorted_cols].to_string(float_format=lambda x: f'{x:8.8f}'))\n",
|
||
" else:\n",
|
||
" print(f\"\\nDeskriptive Statistik:\")\n",
|
||
" print(df.describe())\n",
|
||
" \n",
|
||
" print(f\"\\nKlassenverteilung:\")\n",
|
||
" class_counts = df['class'].value_counts()\n",
|
||
" \n",
|
||
" if df_original is not None:\n",
|
||
" original_counts = df_original['class'].value_counts()\n",
|
||
" # DataFrame mit Aktuell und Abweichung\n",
|
||
" class_comparison = pd.DataFrame({\n",
|
||
" 'Aktuell': class_counts,\n",
|
||
" 'Δ': class_counts - original_counts\n",
|
||
" })\n",
|
||
" print(class_comparison)\n",
|
||
" else:\n",
|
||
" print(class_counts)\n"
|
||
],
|
||
"outputs": [],
|
||
"execution_count": null
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "fffacb88c705402e",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 4.2.2 Ausgabe der statistische Auswertung"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"id": "f9a6bc868531a023",
|
||
"metadata": {},
|
||
"source": [
|
||
"print_statistics(df_original, \"Original (mit fehlenden Werten)\", df_original)"
|
||
],
|
||
"outputs": [],
|
||
"execution_count": null
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"id": "a0a7c2a86d038320",
|
||
"metadata": {},
|
||
"source": [
|
||
"print_statistics(df_deletion, \"Listwise Deletion\", df_original)"
|
||
],
|
||
"outputs": [],
|
||
"execution_count": null
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"id": "6c9c70769bda4284",
|
||
"metadata": {},
|
||
"source": [
|
||
"print_statistics(df_mean, \"Mean Imputation\", df_original)"
|
||
],
|
||
"outputs": [],
|
||
"execution_count": null
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"id": "a7023eb6aa50046d",
|
||
"metadata": {},
|
||
"source": [
|
||
"print_statistics(df_median, \"Median Imputation\", df_original)"
|
||
],
|
||
"outputs": [],
|
||
"execution_count": null
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"id": "5c70ccae18401d1c",
|
||
"metadata": {},
|
||
"source": [
|
||
"print_statistics(df_class_mean, \"Class-spezifische Mean Imputation\", df_original)"
|
||
],
|
||
"outputs": [],
|
||
"execution_count": null
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "ef9b696a67186b4e",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### 4.2.2.1 Auswertung der Ausgaben/ Ergebnisse\n",
|
||
"##### a) Listwise Deletion\n",
|
||
"- Es kommt zu einem beträchtlichen Datenverlust durch die Reduktion von 150 auf 145 Datenpunkte (3,3% Verlust)\n",
|
||
"- Die fehlenden Werte konzentrieren sich auf Iris-setosa, was auf einen systematischen Bias hindeutet. Die gelöschten Setosa-Exemplare hatten unterdurchschnittliche pl\n",
|
||
"Werte, wodurch der Gesamtmittelwert künstlich steigt. Dies führt zudem zu einer verzerrten Klassenverteilung (50, 50, 45).\n",
|
||
"- Die Mittelwerte verschieben sich, was auf eine nicht-zufällige Verteilung der fehlenden Werte hindeutet\n",
|
||
"\n",
|
||
"| | sl_cur | sl_Δ | sw_cur | sw_Δ | pl_cur | pl_Δ | pw_cur | pw_Δ |\n",
|
||
"|------|------------|------------|------------|-------------|------------|------------|------------|------------|\n",
|
||
"| mean | 5.87034483 | 0.02604953 | 3.04344828 | -0.00890072 | 3.84000000 | 0.06617450 | 1.23379310 | 0.02027959 |\n",
|
||
"\n",
|
||
"**Fazit:**\n",
|
||
"Modelle werden auf unbalancierten Daten (45:50:50) trainiert. Dies führt zu schlechterer Vorhersagegenauigkeit für Iris-setosa, da weniger Trainingsbeispiele verfügbar sind. Wegen der ungleichmäßigen Klassenverteilung und der Veränderung der Mittelwerte wird *Listwise Deletion* nicht empfohlen"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "c0f461c6020117b3",
|
||
"metadata": {},
|
||
"source": [
|
||
"##### b) Mean Imputation (global)\n",
|
||
"Verbesserungen zur Listwise Deletion\n",
|
||
"- Mittelwerte bleiben identisch zum Original\n",
|
||
"\n",
|
||
"| | sl_cur | sl_Δ | sw_cur | sw_Δ | pl_cur | pl_Δ | pw_cur | pw_Δ |\n",
|
||
"|------|------------|-------------|------------|-------------|------------|------------|------------|------------|\n",
|
||
"| mean | 5.84429530 | -0.00000000 | 3.05234899 | -0.00000000 | 3.77382550 | 0.00000000 | 1.21351351 | 0.00000000 |\n",
|
||
"\n",
|
||
"- Alle 150 Datenpunkte bleiben erhalten (minus 1 ohne Klassenlabel), wodurch die Klassenverteilung ausgeglichen bleibt (50, 50, 49).\n",
|
||
"\n",
|
||
"weiterhin problematisch\n",
|
||
"- die Standardabweichung sinkt bei allen Features im Vergleich zum Original, was eine künstlichen vereinheitlichung der Datenverteilung und Varianzreduktion zur Folge hat\n",
|
||
"\n",
|
||
"| | sl_cur | sl_Δ | sw_cur | sw_Δ | pl_cur | pl_Δ | pw_cur | pw_Δ |\n",
|
||
"|-----|------------|-------------|------------|-------------|------------|-------------|------------|-------------|\n",
|
||
"| std | 0.82798231 | -0.00279253 | 0.43157024 | -0.00145555 | 1.75462556 | -0.00591781 | 0.75232513 | -0.00510057 |\n",
|
||
"\n",
|
||
"- Es wird ein globaler Mittelwert über alle Klassen hinweg verwendet. Die klassenspezifische Unterschiede verwischen dadurch.\n",
|
||
" - Beispiel: Fehlende pl Werte bei Iris-setosa (ca. 1,4 cm) werden durch den globalen Mittelwert (ca. 3.8 cm) ersetzt, was die natürlichen Unterschiede zwischen den Arten reduziert."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "c832ea015529d86d",
|
||
"metadata": {},
|
||
"source": [
|
||
"##### c) Median Imputation - Ähnlich wie Mean Imputation\n",
|
||
"- Die Ergebnisse sind nahezu identisch mit Mean Imputation und weisen ähnliche Abweichungen zum Original auf\n",
|
||
"\n",
|
||
"| | sl_cur | sl_Δ | sw_cur | sw_Δ | pl_cur | pl_Δ | pw_cur | pw_Δ |\n",
|
||
"|------|------------|-------------|------------|-------------|------------|-------------|------------|-------------|\n",
|
||
"| mean | 5.84400000 | -0.00029530 | 3.05200000 | -0.00034899 | 3.77800000 | 0.00417450 | 1.21466667 | 0.00115315 |\n",
|
||
"| std | 0.82799021 | -0.00278463 | 0.43159141 | -0.00143439 | 1.75537028 | -0.00517309 | 0.75239097 | -0.00503473 |\n",
|
||
"\n",
|
||
"**Fazit:** Die Median Imputation wäre robuster gegen Ausreißer, die in diesem Datensatz aber nicht vorhanden sind. Somit ergeben sich keine merklichen Vorteile zur Mean Imputation.\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "def99207897ed309",
|
||
"metadata": {},
|
||
"source": [
|
||
"##### d) Class-spezifische Mean Imputation\n",
|
||
"Verbesserungen zu den Imputations\n",
|
||
"- geringste Abweichung der Standardabweichungen zum Original\n",
|
||
"\n",
|
||
"| Statistik | sl_cur | sl_Δ | sw_cur | sw_Δ | pl_cur | pl_Δ | pw_cur | pw_Δ |\n",
|
||
"|---------------|------------|-------------|------------|-------------|------------|-------------|------------|-------------|\n",
|
||
"| std_csMeanImp | 0.83009005 | -0.00068478 | 0.43409672 | +0.00107093 | 1.76087327 | +0.00032990 | 0.75754371 | +0.00011800 |\n",
|
||
"| std_del | 0.82167885 | -0.00909599 | 0.43073158 | -0.00229421 | 1.73775078 | -0.02279259 | 0.75168521 | -0.00574049 |\n",
|
||
"| std_meanImp | 0.82798231 | -0.00279253 | 0.43157024 | -0.00145555 | 1.75462556 | -0.00591781 | 0.75232513 | -0.00510057 |\n",
|
||
"| std_medImp | 0.82799021 | -0.00278463 | 0.43159141 | -0.00143439 | 1.75537028 | -0.00517309 | 0.75239097 | -0.00503473 |\n",
|
||
"\n",
|
||
"- Es werden die Mittelwerte der jeweiligen Klasse verwendet und keinen globalen Mittelwert wie bei Mean und Median Imp. Die Unterschiede zwischen den Klassen verwischen dadurch nicht.\n",
|
||
"- Ausgeglichene Klassenverteilung: 50, 50, 49 (nur 1 Datenpunkt mit fehlendem Klassenlabel entfernt)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b60421d8",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 4.2.3 Vergleich der globalen und klassenspezifischen Mittelwerte"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"id": "9174db7e5b881f2a",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Klassenspezifische Mittelwerte vergleichen\n",
|
||
"print(\"Mittelwerte pro Klasse - Class-Mean Imputation:\")\n",
|
||
"print(df_class_mean.groupby('class')[numeric_cols].mean())"
|
||
],
|
||
"outputs": [],
|
||
"execution_count": null
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"id": "5205727bfc86f",
|
||
"metadata": {},
|
||
"source": [
|
||
"print(\"\\nMittelwerte pro Klasse - Global Mean Imputation:\")\n",
|
||
"print(df_mean.groupby('class')[numeric_cols].mean())"
|
||
],
|
||
"outputs": [],
|
||
"execution_count": null
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"id": "7ee84597afc8204c",
|
||
"metadata": {},
|
||
"source": [
|
||
"print(\"\\nAbweichung Class-Mean vs. Global Mean:\")\n",
|
||
"class_diff = df_class_mean.groupby('class')[numeric_cols].mean() - df_mean.groupby('class')[numeric_cols].mean()\n",
|
||
"print(class_diff)\n"
|
||
],
|
||
"outputs": [],
|
||
"execution_count": null
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "87446999ef3a62c3",
|
||
"metadata": {},
|
||
"source": [
|
||
"##### 4.2.3.1 Auswertung des Vergleichs der globalen und klassenspezifischen Mittelwerte\n",
|
||
"Bei Global Mean Imputation erhalten alle Klassen die gleichen Ersatzwerte (globaler Mittelwert), was besonders bei Iris-setosa problematisch ist. Die Abweichungstabelle zeigt, dass Setosa-Werte durch zu große pl/ pw Ersatzwerte verfälscht werden.\n",
|
||
"\n",
|
||
"Wenn ein Setosa-Exemplar einen fehlenden pl Wert hat, wird bei \n",
|
||
"- Global Mean durch ca. 3.7 cm ersetzt, was sehr stark von anderen Werten aus der Klasse abweicht,\n",
|
||
"- Class-Mean durch ca. 1.4 cm ersetzt, was eher den vorhandenen Eigenschaften der Klasse passt\n",
|
||
"\n",
|
||
"**Fazit:**\n",
|
||
"Class-spezifische Imputation bewahrt die Unterschiede zwischen den Arten und verhindert, dass z.B. Setosa-Exemplare künstlich andere Merkmale erhalten. "
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "cc74fd49f372d412",
|
||
"metadata": {},
|
||
"source": [
|
||
"### 4.2.4 Visualisierung der statistische Auswertung"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"id": "9342d3875e1133c0",
|
||
"metadata": {},
|
||
"source": [
|
||
"fig, axes = plt.subplots(2, 2, figsize=(15, 10))\n",
|
||
"features = ['sl', 'sw', 'pl', 'pw']\n",
|
||
"\n",
|
||
"for idx, feature in enumerate(features):\n",
|
||
" ax = axes[idx//2, idx%2]\n",
|
||
"\n",
|
||
" # Boxplots für verschiedene Strategien\n",
|
||
" data_to_plot = [\n",
|
||
" df_deletion[feature],\n",
|
||
" df_mean[feature],\n",
|
||
" df_median[feature],\n",
|
||
" df_class_mean[feature]\n",
|
||
" ]\n",
|
||
"\n",
|
||
" ax.boxplot(data_to_plot, tick_labels=['Deletion', 'Mean', 'Median', 'Class-Mean'])\n",
|
||
" ax.set_title(f'{feature}')\n",
|
||
" ax.set_ylabel('Wert')\n",
|
||
" ax.grid(True, alpha=0.3)\n",
|
||
"\n",
|
||
"plt.tight_layout()\n",
|
||
"plt.show()\n",
|
||
"\n",
|
||
"print(\"Tabelle: Vergleich der Mittelwerte und Standardabweichungen\")\n",
|
||
"comparison_data = []\n",
|
||
"for strategy_name, df_strategy in [('Original', df_original),\n",
|
||
" ('Deletion', df_deletion),\n",
|
||
" ('Mean', df_mean),\n",
|
||
" ('Median', df_median),\n",
|
||
" ('Class-Mean', df_class_mean)]:\n",
|
||
" for feature in features:\n",
|
||
" comparison_data.append({\n",
|
||
" 'Feature': feature,\n",
|
||
" 'Strategie': strategy_name,\n",
|
||
" 'Mittelwert': df_strategy[feature].mean(),\n",
|
||
" 'Std': df_strategy[feature].std()\n",
|
||
" })\n",
|
||
"\n",
|
||
"df_comparison = pd.DataFrame(comparison_data)\n",
|
||
"df_comparison\n"
|
||
],
|
||
"outputs": [],
|
||
"execution_count": null
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "256193a4",
|
||
"metadata": {},
|
||
"source": [
|
||
"#### 4.2.4.1 Auswertung der Visualsierung\n",
|
||
"##### Boxplot-Analyse\n",
|
||
"Die Boxplots zeigen deutliche Unterschiede zwischen den Strategien:\n",
|
||
"- **sl & sw:** Alle Strategien produzieren nahezu identische Verteilungen (Median und Quartile überlagern sich), da hier wenige fehlende Werte vorlagen.\n",
|
||
"- **pl:** Listwise Deletion verschiebt den Median leicht nach oben (sichtbar in der Box-Position), was die Löschung kleinerer Setosa-Werte bestätigt. Die Imputation-Strategien erhalten die Median-Position besser.\n",
|
||
"- **pw:** Ähnliches Muster wie pl, aber weniger ausgeprägt.\n",
|
||
"\n",
|
||
"##### Vergleichstabelle-Analyse\n",
|
||
"Die df_comparison Tabelle zeigt, dass Class-Mean Imputation die geringsten Abweichungen bei den Standardabweichungen aufweist, während andere Methoden durchweg Varianzreduktion zeigen.\n",
|
||
"\n",
|
||
"##### Fazit\n",
|
||
"Visuell sind die Unterschiede minimal, aber statistisch relevant. Die Erhaltung der Variabilität (Box-Höhe) ist bei Class-Mean am besten, was für robuste nachfolgende Analysen sorgen wird."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "bf72cedd90434218",
|
||
"metadata": {},
|
||
"source": [
|
||
"\n",
|
||
"## 5. Abschlißende Empfehlung\n",
|
||
"Class-spezifische Mean Imputation\n",
|
||
"- Die Standardabweichung bleibt am nächsten zum Original. Diese bessere Erhaltung der Streuung bildet die natürliche Variabilität biologischer Messungen besser ab.\n",
|
||
"- Die Ersetzung der fehlenden Werte durch klassenspezifische Mittelwerte anstelle der globalen Mittelwerte verhindert, dass zB Iris-Setosa \"Ausreißer\" in pw bekommen und somit die Eigenschaften der Klassen verwischen.\n",
|
||
"- Bei der class-spezifischen Mean Imputation geht nur ein einziger Datenpunkt verloren. Der Datensatz mit fehlendem Klassenlabel (Zeile 10), da hier keine Klassenzugehörigkeit für die Imputation ermittelt werden kann. Die Klassenverteilung bleibt daher ausgeglichener (49:50:50). Dies vermeidet eine Verzerrung, der bei Listwise Deletion entsteht (45:50:50) und bei nachfolgenden Klassifikationsaufgaben zu einer Benachteiligung der Iris-setosa Klasse führen würde."
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": ".env",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.14.0"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|