Introduction to Data Analysis Course Code: DI04000071 GTU Diploma Engineering Semester 4 GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 1 / 283 Table of Contents I 1 Introduction to Data Analysis 2 Introduction to Data Analysis 3 Introduction to Data Analysis 4 Introduction to Data Analysis 5 Unit 1: Introduction to Data Analysis 6 Introduction to Data Analysis 7 Introduction to Data Analysis 8 Introduction to Data Analysis GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 2 / 283 Table of Contents II 9 Python libraries for Data Analysis and Data extraction 10 Python libraries for Data Analysis and Data extraction 11 Python libraries for Data Analysis and Data extraction 12 Python libraries for Data Analysis and Data extraction 13 Python libraries for Data Analysis and Data extraction 14 Python libraries for Data Analysis and Data extraction 15 Python libraries for Data Analysis and Data extraction 16 Python libraries for Data Analysis and Data extraction GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 3 / 283 Table of Contents III 17 Python libraries for Data Analysis and Data extraction 18 Dimensionality Reduction 19 Dimensionality Reduction 20 Python libraries for Data Analysis and Data extraction 21 Statistical Analysis 22 Unit 3: Statistical Analysis 23 Unit 3: Statistical Analysis 24 Statistical Analysis GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 4 / 283 Table of Contents IV 25 Statistical Analysis 26 Unit 3: Statistical Analysis 27 Statistical Analysis 28 Statistical Analysis 29 Unit 3: Statistical Analysis 30 Data Visualization 31 Data Visualization 32 Data Visualization GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 5 / 283 Table of Contents V 33 Data Visualization 34 Data Visualization 35 Data Visualization 36 Data Visualization 37 Data Visualization 38 Generative AI and Big Data Visualization with Power BI 39 Generative AI and Big Data Visualization with Power BI 40 Generative AI and Big Data Visualization with Power BI GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 6 / 283 Table of Contents VI 41 Generative AI and Big Data Visualization with Power BI 42 Unit 5: Generative AI and Big Data Visualization with Power BI 43 Generative AI and Big Data Visualization with Power BI 44 Generative AI and Big Data Visualization with Power BI 45 Generative AI and Big Data Visualization with Power BI GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 7 / 283 What is Data? Data refers to raw, unprocessed facts and figures. It forms the foundational building block for information, knowledge, and wisdom. Nature of Data: Can be quantitative (numerical) or qualitative (descriptive). Can be continuous (measured) or discrete (counted). Varies in volume, velocity, and variety. Data vs. Information Data is raw and unorganized. Information is data that has been processed, organized, and structured to provide context and meaning. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 8 / 283 Sources of Data Primary Sources Secondary Sources Data collected firsthand for a specific Data that has already been collected purpose. by someone else. Surveys and Questionnaires Government Publications Interviews Research Journals Observations Historical Records Experiments Databases GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 9 / 283 Nature of Data Collection Table: Comparison of Primary and Secondary Data Aspect Originality Cost Time Required Specific to Need GTU Diploma Engineering (Semester 4) Primary Data High (Original) High Long Yes Introduction to Data Analysis Secondary Data Low (Derivative) Low Short Not always 10 / 283 Classification of Data Data is fundamentally classified based on its structural organization: Data Types Structured Data GTU Diploma Engineering (Semester 4) Semi-structured Data Introduction to Data Analysis Unstructured Data 11 / 283 Structured Data Highly organized, easily searchable, and usually stored in relational databases (RDBMS). Has a pre-defined data model and schema. Examples: Financial records, customer databases, inventory systems. Example: SQL Data CREATE TABLE Employees ( ID int, Name varchar(255), Department varchar(255), Salary decimal(10,2) ); GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 12 / 283 Semi-Structured Data Does not conform to a strict tabular structure but contains tags or markers to separate semantic elements. Self-describing data. Examples: XML, JSON, HTML, Emails (header info). Example: JSON Format { "employee": { "id": 101, "name": "Jane Doe", "department": "Engineering" } } GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 13 / 283 Unstructured Data Information that either does not have a pre-defined data model or is not organized in a pre-defined manner. Typically text-heavy, but may contain data such as dates, numbers, and facts as well. Makes up the vast majority of data generated today (approx. 80-90%). Examples Text documents, PDFs, Emails (body content). Images, Audio files (MP3), Video files (MP4). Social media posts, sensor data. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 14 / 283 Comparison of Data Classifications Feature Format Storage Querying Flexibility Volume Structured Pre-defined (Schema) RDBMS (SQL) Easy (SQL) Low Smallest % GTU Diploma Engineering (Semester 4) Semi-Structured Self-describing (Tags) NoSQL, File Systems Moderate Medium Growing % Introduction to Data Analysis Unstructured None Data Lakes, NoSQL Difficult (AI/ML) High Largest % of data 15 / 283 What is Data? Definition Data refers to raw, unorganized facts that need to be processed. Data can be something simple and seemingly random and useless until it is organized. When data is processed, organized, structured or presented in a given context so as to make it useful, it is called information. In the context of modern data analysis, data exhibits several key characteristics often described by the ”V”s of Big Data. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 16 / 283 The 5 V’s of Data Characteristics Volume Scale of Data Velocity Analysis of streaming data Value Usefulness of data Data Veracity Uncertainty of data GTU Diploma Engineering (Semester 4) Variety Different forms of data Introduction to Data Analysis 17 / 283 Data Variety: Types of Data Type Structured Unstructured Semi-structured Characteristics Highly organized, neatly formatted, predefined schema. No predefined data model, hard to search or analyze. Does not reside in relational DB but has some organizational properties (tags/markers). Examples Relational databases (SQL), Excel sheets. Text documents, images, videos, social media posts. JSON, XML, CSV files, HTML. Table: Comparison of Data Types GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 18 / 283 Data Quality Characteristics Data quality is essential for accurate analysis. Key characteristics include: Accuracy: Does the data correctly reflect the real-world object or event? (e.g., correct spelling of a name). Completeness: Is all the required data present? Are there missing values or empty fields? Consistency: Does the data match across multiple data stores? Timeliness: Is the data available when needed? Is it up-to-date? Validity: Does the data conform to defined rules or constraints? (e.g., age must be a positive integer). GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 19 / 283 Introduction to Big Data Platform Big Data platforms provide the computational and storage resources necessary to handle vast amounts of diverse data. Key characteristics include the famous 5 Vs: Volume: Sheer amount of data generated every second. Velocity: Speed of data generation and processing needs. Variety: Different formats (structured, unstructured, semi-structured). Veracity: Quality, reliability, and accuracy of data. Value: Business insights drawn from processing the data. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 20 / 283 Components of a Big Data Platform Core Components A typical platform consists of several layers working together to ingest, store, process, and visualize data. Layer Data Ingestion Storage Processing Analysis & BI GTU Diploma Engineering (Semester 4) Examples / Technologies Apache Kafka, Flume, Sqoop HDFS, Amazon S3, NoSQL Databases Apache Hadoop (MapReduce), Apache Spark Hive, Pig, Tableau, PowerBI Introduction to Data Analysis 21 / 283 Architecture Overview Data Ingestion GTU Diploma Engineering (Semester 4) Distributed Storage Data Processing Introduction to Data Analysis Analysis & BI 22 / 283 Need of Data Analysis Organizations collect immense amounts of data daily; without analysis, this data remains an untapped resource. Primary motivations for data analysis: Decision Making: Evidence-based, data-driven strategies replace intuition and guesswork. Cost Reduction: Identifying inefficiencies in operations and resource allocation. Customer Insights: Understanding behavior, preferences, and trends to tailor services. Risk Management: Predictive modeling to foresee and mitigate potential risks and frauds. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 23 / 283 From Data to Wisdom The DIKW Pyramid Data Analysis is the mechanism that transforms raw data into actionable wisdom. Wisdom (Applied) Knowledge (Contextualized patterns) Information (Meaningful data) Data (Raw facts and figures) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 24 / 283 Evolution of Analytic Scalability What is Analytic Scalability? Analytic scalability refers to the ability of an analytical system or architecture to handle growing amounts of data, user queries, and processing complexity without sacrificing performance. Driven by the exponential growth of data volume and velocity. Transition from simple descriptive analytics to complex predictive models. Evolution of hardware architectures from single-server to distributed clusters. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 25 / 283 Stages of Analytic Evolution Era 1980s 1990s 2000s 2010s 2020s+ Data Architecture Mainframes, RDBMS Data Warehouses (EDW) Hadoop, MapReduce Cloud, In-Memory DB AI-driven, Lakehouse Key Focus Descriptive Analytics BI & Reporting Big Data, Predictive Real-time, Streaming Prescriptive, GenAI Table: Evolutionary Stages of Data Analytics GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 26 / 283 Traditional vs. Modern Architectures Traditional Architecture Modern Architecture Centralized storage (SAN/NAS) Compute and storage tightly coupled Distributed storage (HDFS, Cloud Object) Compute and storage decoupled Vertical scaling (Scale-up) Horizontal scaling (Scale-out) High cost of proprietary hardware Commodity hardware & cloud instances GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 27 / 283 Visualizing the Shift Traditional DWVolume/Variety (Scale-Up) Big Data Clusters (Scale-Out) Agility/Cost Cloud Native (Elastic Scale) Key Takeaway The journey is marked by moving from expensive, monolithic systems to flexible, distributed, and decoupled cloud environments. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 28 / 283 The Analytical Process Overview The data analysis process involves a systematic series of steps to transform raw data into actionable insights and strategic decisions. Define the Problem: Understanding the core business or research question. Data Collection: Gathering relevant data from diverse structured and unstructured sources. Data Preparation (Cleaning): Handling missing values, removing outliers, and standardizing formats. Data Analysis: Applying statistical techniques and machine learning models. Interpretation: Deriving meaning from the analyzed data. Visualization & Communication: Presenting findings to stakeholders clearly. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 29 / 283 Data Collection & Preparation Common Cleaning Tasks Data Sources Internal: Databases (SQL/NoSQL), CRM systems, transactional logs. External: APIs, Web Scraping, Open Data Portals. GTU Diploma Engineering (Semester 4) Data Issue Missing Values Outliers Inconsistency Redundancy Introduction to Data Analysis Common Solution Mean/Median Imputation Trimming or Winsorizing Standardization Deduplication 30 / 283 The Analytical Workflow 1. Define 2. Collect 3. Clean 5. Interpret 4. Analyze Refine 6. Deploy Iterative Nature The analytical process is rarely linear. Insights gained during interpretation often lead to refining the initial problem definition and repeating the cycle. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 31 / 283 Analysis vs. Reporting Overview Understanding the core differences between data analysis and reporting is essential for deriving actionable insights from raw data. Aspect Purpose Output Reporting To organize and summarize data. Dashboards, static reports. Focus What happened? GTU Diploma Engineering (Semester 4) Analysis To examine data to extract insights. Recommendations, predictive models. Why did it happen and what will happen? Introduction to Data Analysis 32 / 283 Modern Data Analysis Tools Programming Languages: Python (Pandas, NumPy, Scikit-learn) R (ggplot2, dplyr, caret) Data Visualization: Tableau, PowerBI, Matplotlib, Seaborn Big Data & Cloud Platforms: Apache Spark, Hadoop AWS, Google Cloud Platform (GCP), Microsoft Azure GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 33 / 283 Workflow of Modern Data Analysis Data Collection Data Cleaning Analysis & Modeling Important The choice of tools depends on the scale of data and the specific requirements of the analysis task. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 34 / 283 What is Data Analysis? Definition Data analysis is the process of inspecting, cleansing, transforming, and modeling data with the goal of discovering useful information, informing conclusions, and supporting decision-making. Core Objectives: Descriptive: What happened? Diagnostic: Why did it happen? Predictive: What will happen? Prescriptive: What should we do? GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 35 / 283 Industry Applications Data analysis is ubiquitous across various sectors. Sector Healthcare Finance Retail Transportation E-commerce Key Application Predicting patient outcomes and personalized medicine Fraud detection and algorithmic trading Customer segmentation and inventory optimization Route optimization and predictive maintenance Recommendation systems and churn prediction Table: Applications by Sector GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 36 / 283 The Data Analysis Process Collection Feedback GTU Diploma Engineering (Semester 4) Cleaning Analysis Decision Visualization Introduction to Data Analysis 37 / 283 Example: Basic Data Analysis in Python Loading and Inspecting Data import pandas as pd # Load the dataset df = pd.read_csv(’sales_data.csv’) # Display first 5 rows print(df.head()) # Basic statistical summary summary = df.describe() print(summary) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 38 / 283 Key Roles in Data Analytics A successful data analytics project requires a diverse team with distinct responsibilities: Business User: Understands the domain area and benefits from the results. Project Sponsor: Provides funding and defines the core business problem. Project Manager: Ensures the project is completed on time and within budget. Business Intelligence (BI) Analyst: Provides business domain expertise and handles reporting. Database Administrator (DBA): Provisions and configures the database environment. Data Engineer: Executes data extraction, transformation, and loading (ETL/ELT). Data Scientist: Applies analytical techniques, builds models, and writes code. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 39 / 283 Role Responsibilities Matrix Role Primary Focus Business User Strategy and decision making based on findings Project Sponsor ROI, strategic alignment, and resource allocation Data Engineer Data pipelines, architecture, and performance optimization Data Scientist Advanced analytics, machine learning, and statistical modeling Table: Summary of core project roles GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 40 / 283 The Data Analytics Lifecycle 1. Discovery 2. Data Prep 3. Model Planning 5. Communicate Results 4. Model Building Iterate 6. Operationalize An iterative approach consisting of six core phases designed specifically for Big Data problems and data science projects. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 41 / 283 Phase 1 & 2: Discovery and Data Preparation Phase 1: Discovery Understand the business domain and past history. Assess resources available (people, technology, time, data). Formulate initial hypotheses (IHs) to test. Phase 2: Data Preparation Set up an analytic sandbox for data processing. Perform ETLT (Extract, Transform, Load, Transform). Data conditioning: cleaning, normalizing, and handling missing values. Data visualization to understand data structure. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 42 / 283 Phase 3 & 4: Model Planning and Model Building Phase 3: Model Planning Determine the methods, techniques, and workflow for the model. Explore data to learn about relationships between variables. Select key variables and find suitable models (e.g., classification, clustering). Phase 4: Model Building Develop datasets for training, testing, and production. Execute the models using tools like R, Python, SAS, or SQL. Evaluate whether the model meets expectations and is robust enough. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 43 / 283 Phase 5 & 6: Communicating Results and Operationalization Phase 5: Communicating Results Determine if the results are a success or failure based on initial criteria. Articulate the business value of the findings. Create final presentations and reports for stakeholders. Phase 6: Operationalization Deliver final reports, briefings, code, and technical documents. Run pilot projects to deploy the model in a production environment. Monitor the model’s performance and accuracy over time. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 44 / 283 NumPy Array vs Python List Data Type: Lists can hold heterogeneous elements; NumPy arrays store homogeneous data. Performance: NumPy arrays are significantly faster and use less memory due to contiguous memory allocation. Functionality: NumPy provides extensive built-in mathematical functions optimized for array operations. Feature Memory Operations Data Types Python List Fragmented allocation Loops required for math Dynamic/Mixed GTU Diploma Engineering (Semester 4) Introduction to Data Analysis NumPy Array Contiguous allocation Vectorized operations Fixed/Homogeneous 45 / 283 N-Dimensional Arrays (NDArray) 1D Array: A simple vector of elements (Rank 1). 2D Array: Matrix representation, consisting of rows and columns (Rank 2). 3D Array: Tensor representation, often used for RGB images or volumetric data (Rank 3). 1D Array GTU Diploma Engineering (Semester 4) 2D Array Introduction to Data Analysis 3D Array 46 / 283 Special Matrices Initialization NumPy provides built-in methods to easily initialize common arrays. Zeros, Ones, and Identity Matrices Zeros Matrix: Array filled with 0s. Useful for initialization. Ones Matrix: Array filled with 1s. Identity Matrix: Square matrix with 1s on the main diagonal and 0s elsewhere. import numpy as np # 3x3 Zeros Matrix zeros_mat = np.zeros((3, 3)) # 2x4 Ones Matrix ones_mat = np.ones((2, 4)) # 3x3 Identity Matrix identity_mat = np.eye(3) # or np.identity(3) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 47 / 283 Reshaping Arrays and Random Numbers Reshape: Modifies the shape (dimensions) of an array without changing its data. Random Numbers: Generate arrays filled with random values from uniform or normal distributions. # Reshaping a 1D array to 2D arr1d = np.array([1, 2, 3, 4, 5, 6]) arr2d = arr1d.reshape((2, 3)) # Output: [[1 2 3] # [4 5 6]] # Working with Random Numbers rand_uniform = np.random.rand(3, 3) # [0, 1) uniform rand_normal = np.random.randn(3, 3) # Standard normal rand_int = np.random.randint(1, 10, 5) # 5 ints between 1 \ and 9 GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 48 / 283 Stacking Arrays Joining multiple arrays to form a single larger array. Vertical and Horizontal Stacking Vertical Stacking: np.vstack() stacks arrays row-wise (along axis 0). Horizontal Stacking: np.hstack() stacks arrays column-wise (along axis 1). A = np.array([[1, 2], [3, 4]]) B = np.array([[5, 6], [7, 8]]) # Vertical Stacking V = np.vstack((A, B)) # Output: [[1, 2], [3, 4], [5, 6], [7, 8]] # Horizontal Stacking H = np.hstack((A, B)) # Output: [[1, 2, 5, 6], [3, 4, 7, 8]] GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 49 / 283 Images as NumPy Arrays An image is essentially a matrix of pixel values. Grayscale Image: Represented as a 2D array (Height × Width). RGB Image: Represented as a 3D array (Height × Width × 3 Channels). import matplotlib.pyplot as plt import numpy as np # Read image as a numpy array img = plt.imread(’image.jpg’) print(type(img)) print(img.shape) # # e.g., (1080, 1920, 3) # Modify image: extract red channel red_channel = img[:, :, 0] GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 50 / 283 Introduction to Pandas DataFrames What is a DataFrame? A DataFrame is a 2-dimensional labeled data structure with columns of potentially different types. You can think of it like a spreadsheet or SQL table. It is the most commonly used pandas object. Consists of three principal components: the data, rows (index), and columns. Highly optimized for performance and handles various data types easily. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 51 / 283 Reading CSV and Excel Files Pandas provides robust built-in functions to read data from various file formats directly into a DataFrame. Reading Data import pandas as pd # Read a CSV file df_csv = pd.read_csv(’data.csv’) # Read an Excel file (requires openpyxl or xlrd) df_excel = pd.read_excel(’data.xlsx’, sheet_name=’Sheet1’) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 52 / 283 Analyzing Basic Dataset Characteristics Once data is loaded, it is crucial to understand its structure and summary statistics. df.head(n): Returns the first n rows. df.tail(n): Returns the last n rows. df.info(): Provides a concise summary including memory usage and non-null counts. df.describe(): Generates descriptive statistics for numerical columns. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 53 / 283 Merging and Sorting DataFrames Merging DataFrames Similar to SQL JOIN operations. Sorting DataFrames Sort by one or multiple columns. pd.merge(df1, df2, on=’key’) df.sort values(by=’col1’, ascending=False) Supports inner, outer, left, and right joins. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 54 / 283 Handling Missing Values Real-world data is often incomplete. Pandas provides methods to clean this data. Common Methods Drop Missing Data: df.dropna() removes rows with missing values. Fill Missing Data: df.fillna(value) replaces missing values with a specified value (e.g., mean, median). Check Missing Data: df.isnull().sum() counts missing values per column. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 55 / 283 Analyzing Data with loc and iloc Pandas supports two primary methods for slicing and selecting data: Feature Basis Syntax Slicing loc Label-based indexing df.loc[row, col label] Includes the end label iloc Integer-location based df.iloc[r idx, c idx] Excludes the end index Table: Comparison of loc and iloc GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 56 / 283 Finding Largest and Smallest Values Quickly identify the extreme values in a specific column without sorting the entire DataFrame. Methods df.nlargest(n, ’column name’) df.nsmallest(n, ’column name’) Example: # Top 5 highest salaries df.nlargest(5, ’Salary’) # Bottom 3 youngest ages df.nsmallest(3, ’Age’) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 57 / 283 Adding or Removing Attributes DataFrames are mutable; you can easily add or drop columns. Adding a column: # Add a new column based on existing data df[’Bonus’] = df[’Salary’] * 0.10 Removing a column: # Drop a column (axis=1 indicates column) df = df.drop(’Bonus’, axis=1) # Or use inplace=True to modify existing DataFrame df.drop(’Bonus’, axis=1, inplace=True) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 58 / 283 Data Operations Summary Read Data Clean/Missing Analyze & Filter Add/Drop Cols Merge/Sort GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 59 / 283 Introduction to File Handling in Python File handling is a crucial part of data analysis to import raw data. Python provides built-in functions and powerful libraries to read files efficiently. The open() Function The basic way to open a file in Python is using the open() function, which returns a file object. Hard Disk (File) open() GTU Diploma Engineering (Semester 4) Python File Object read() Introduction to Data Analysis Main Memory (Variables) 60 / 283 File Opening Modes Mode Description ’r’ Read - Default value. Opens a file for reading, error if the file does not exist. ’a’ Append - Opens a file for appending, creates the file if it does not exist. ’w’ Write - Opens a file for writing, creates the file if it does not exist, truncates if exists. ’x’ Create - Creates the specified file, returns an error if the file exists. Table: Common File Modes in Python GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 61 / 283 Reading Text Files (Built-in) Using the with statement is highly recommended as it automatically closes the file even if an exception occurs. Reading a text file line by line # Using the ’with’ context manager with open(’data.txt’, ’r’) as file: # Read all lines into a list lines = file.readlines() for line in lines: print(line.strip()) # strip() removes newline \ characters GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 62 / 283 Reading CSV Files with the csv Module For simple tabular data, Python’s built-in csv module is useful without needing third-party libraries. Reading CSV Data import csv with open(’employees.csv’, mode=’r’) as file: csv_reader = csv.reader(file) header = next(csv_reader) # Skip or read header for row in csv_reader: print(f"Name: {row[0]}, Role: {row[1]}") GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 63 / 283 Reading Files with Pandas Pandas is the industry standard for reading structured data into DataFrames. Pandas File Reading Functions Pandas supports various formats: CSV, Excel, JSON, SQL, etc., natively and efficiently. import pandas as pd # Reading a CSV file df_csv = pd.read_csv(’dataset.csv’) # Reading an Excel file df_excel = pd.read_excel(’data.xlsx’, sheet_name=’Sheet1’) # Display the first 5 rows print(df_csv.head()) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 64 / 283 Reading JSON Files JSON (JavaScript Object Notation) is widely used for data exchange, especially in APIs. Using the json Module import json # Reading JSON from a file with open(’config.json’, ’r’) as file: data = json.load(file) print(data[’server_port’]) Note: Pandas can also read JSON directly using pd.read json(’file.json’). GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 65 / 283 What is Web Scraping? Definition Web scraping is the automated process of extracting data from websites. It involves making HTTP requests to web servers, downloading HTML content, and parsing it to extract structured information. Primary Purposes: Data Collection & Aggregation: Gathering news, stock prices, or real estate listings from multiple sources. Market Research: Monitoring competitor pricing and product catalogs. Machine Learning: Creating datasets for Natural Language Processing or image recognition models. Automation: Extracting data where no API is provided. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 66 / 283 Legality and Ethical Considerations Important Warning Web scraping exists in a legal gray area. Always verify if you are allowed to scrape a website. Ethical Guidelines: Respect robots.txt: Always check domain.com/robots.txt to see which paths are allowed or disallowed for crawlers. Terms of Service (ToS): Check the website’s ToS. Some explicitly forbid automated data extraction. Rate Limiting: Add delays between requests (e.g., time.sleep(2)) to avoid overloading the server. Identify Yourself: Use a custom User-Agent string containing your contact info. Personal Data: Avoid scraping Personally Identifiable Information (PII) to comply with GDPR/CCPA. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 67 / 283 Overview of Popular Libraries Table: Comparison of Python Web Scraping Libraries Library Requests Beautiful Soup Selenium Scrapy Primary Use Case Fetching web pages (HTTP requests) Parsing HTML/XML documents Browser automation, JavaScript rendering Pros Simple, fast, standard Easy to use, forgiving Handles dynamic content Large scale crawling and scraping framework Fast (async), robust GTU Diploma Engineering (Semester 4) Introduction to Data Analysis Cons No HTML parsing Slow with large files Slow, heavy resource usage Steep learning curve 68 / 283 The Requests Library Overview Requests is an elegant and simple HTTP library for Python, built for human beings. It is used to send HTTP requests to servers to fetch the raw HTML. import requests url = ’https://example.com’ headers = {’User-Agent’: ’My-Bot (myemail@example.com)’} response = requests.get(url, headers=headers) if response.status_code == 200: print("Success!") print(response.text[:100]) # Print first 100 characters else: print(f"Failed with status code " \ f"{response.status_code}") GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 69 / 283 Beautiful Soup 4 (BS4) Overview BS4 parses the raw HTML content fetched by Requests into a navigable tree structure, allowing you to search for tags, classes, and IDs. from bs4 import BeautifulSoup import requests html_content = requests.get(’https://example.com’).text soup = BeautifulSoup(html_content, ’html.parser’) # Find the first

tag title = soup.find(’h1’).text # Find all links links = soup.find_all(’a’) for link in links: print(link.get(’href’)) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 70 / 283 Selenium Handling Dynamic Content Many modern websites use JavaScript to load data after the initial page load. Requests cannot execute JS. Selenium automates a real web browser (Chrome, Firefox) to fully render pages. from selenium import webdriver from selenium.webdriver.common.by import By # Initialize a headless Chrome browser options = webdriver.ChromeOptions() options.add_argument(’--headless’) driver = webdriver.Chrome(options=options) driver.get(’https://example-dynamic.com’) # Find element after JS has rendered it element = driver.find_element(By.ID, "dynamic-content") print(element.text) driver.quit() GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 71 / 283 Architecture of a Web Scraper Target Web Server (example.com) HTTP GET HTML / JSON Fetcher Parser Storage Raw HTML Clean Data (Requests / Selenium) (Beautiful Soup) (CSV / Database) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 72 / 283 Introduction to Beautiful Soup What is Beautiful Soup? Beautiful Soup is a Python library designed for quick turnaround projects like screen-scraping. It pulls data out of HTML and XML files, providing idiomatic ways of navigating, searching, and modifying the parse tree. Purpose and Use Cases: Web Scraping: Extracting text, hyperlinks, and structured data from unstructured websites. Data Mining: Collecting large-scale datasets for machine learning or statistical analysis. HTML Parsing: Navigating and modifying the parse tree of HTML/XML documents efficiently. Error Handling: Handling poorly formatted markup (tag soup) gracefully without breaking. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 73 / 283 Parsing HTML Beautiful Soup sits atop an HTML or XML parser, abstracting away complex parsing logic. Common parsers: html.parser (built-in, decent speed), lxml (very fast, requires external C dependency), html5lib (parses like a web browser, slow but extremely lenient). Creating a BeautifulSoup Object from bs4 import BeautifulSoup html_doc = "

" \ "Hello World!

" # Initialize parser soup = BeautifulSoup(html_doc, ’html.parser’) # Prettify output for better readability print(soup.prettify()) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 74 / 283 Working with HTML Tags A Tag object corresponds directly to an XML or HTML tag in the original document. You can navigate using tag names directly as attributes of the parsed soup object. Common Tag Navigation Methods Method/Attribute soup.title soup.title.name soup.title.string soup.p soup.find all(’a’) GTU Diploma Engineering (Semester 4) Description Returns the first tag Returns the name of the tag (e.g., ’title’) Returns the inner text content of the tag Returns the first <p> tag Returns a list of all <a> tags Introduction to Data Analysis 75 / 283 Accessing Tag Attributes Tags may have any number of attributes (e.g., class, id, href). For instance, the tag <b id="boldest"> has an attribute ”id” with value ”boldest”. You can access a tag’s attributes by treating the Tag object like a dictionary. Attribute Access Example html_doc = ’<p class="content text-red" \ id="first-para">Text</p>’ soup = BeautifulSoup(html_doc, ’html.parser’) tag = soup.p # Accessing specific attributes print(tag[’id’]) # Output: first-para print(tag.get(’class’)) # Output: [’content’, ’text-red’] # Accessing all attributes as a dictionary print(tag.attrs) # Output: {’class’: [’content’, ’text-red’], ’id’: \ ’first-para’} GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 76 / 283 Simple HTML Parsing Example from bs4 import BeautifulSoup html_doc = """ <html><head><title>The Dormouse’s story

The Dormouse’s story

Elsie Lacie """ soup = BeautifulSoup(html_doc, ’html.parser’) print("Title:", soup.title.string) # Output: Title: The Dormouse’s story # Extracting all hyperlink URLs for link in soup.find_all(’a’): print(link.get(’href’)) # Output: http://example.com/elsie # http://example.com/lacie GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 77 / 283 Extracting Data Workflow 1. Send Request (using requests) 2. Fetch HTML Content Response 3. Parse HTML (BeautifulSoup) 4. Extract Targeted Data (Tags/Text) 5. Save Structured Data (CSV, DB) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 78 / 283 Extracting Data from Web Pages (Example) In practice, combine requests (to get the page) and BeautifulSoup (to extract data). Web Scraping Example import requests from bs4 import BeautifulSoup # 1. Fetch the page url = "https://quotes.toscrape.com/" response = requests.get(url) # 2. Parse the HTML soup = BeautifulSoup(response.text, ’html.parser’) # 3. Extract data (Quotes and Authors) quotes = soup.find_all(’span’, class_=’text’) authors = soup.find_all(’small’, class_=’author’) for quote, author in zip(quotes[:2], authors[:2]): # Top 2 print(f"{quote.text}\n- {author.text}\n") GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 79 / 283 Introduction to Requests Requests is a powerful, user-friendly Python library for making HTTP requests. It abstracts the complexities of making requests behind a simple API. Ideal for web scraping, interacting with REST APIs, and automating web tasks. Why use Requests? Unlike the built-in urllib, requests is designed to be highly readable and requires minimal code to perform tasks like sending parameters, handling cookies, and managing sessions. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 80 / 283 HTTP Request-Response Cycle HTTP Request Client (Python Script) Server (Web API) HTTP Response Request: Contains Method (GET, POST, etc.), URL, Headers, and Data/Body. Response: Contains Status Code, Headers, and Content. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 81 / 283 Sending GET Requests What is a GET Request? Used to retrieve data from a server. Data is sent in the URL (query parameters). Python Code Example import requests # Send a basic GET request url = ’https://api.github.com/events’ response = requests.get(url) # GET request with parameters payload = {’key1’: ’value1’, ’key2’: ’value2’} res_params = requests.get(’https://httpbin.org/get’, \ params=payload) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 82 / 283 Sending POST Requests What is a POST Request? Used to send data to a server to create or update a resource. Data is included in the request body. Python Code Example import requests url = ’https://httpbin.org/post’ data_payload = {’username’: ’student’, ’password’: ’123’} # Send a POST request with form data response = requests.post(url, data=data_payload) # Send JSON payload json_payload = {’name’: ’Alice’, ’role’: ’Admin’} response_json = requests.post(url, json=json_payload) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 83 / 283 GET vs POST: Key Differences Feature GET POST Purpose Retrieve data Send data Data Location URL (Query String) Request Body Security Less secure (visible in URL) More secure Idempotent Yes (multiple requests = same result) No Caching Can be cached Cannot be cached GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 84 / 283 Accessing Response Content Once a request is made, the server returns a Response object containing the result. Handling the Response import requests response = requests.get(’https://api.github.com/events’) # Status Code print(response.status_code) # 200 (OK), 404 (Not Found), \ etc. # Response Content as Text print(response.text) # Returns a string (e.g., HTML/JSON \ text) # Parsing JSON directly data = response.json() # Returns a Python dictionary print(data[0][’id’]) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 85 / 283 Authenticating with Requests Many web services require authentication to access data. The requests library provides several built-in authentication methods. Basic Authentication from requests.auth import HTTPBasicAuth import requests url = ’https://httpbin.org/basic-auth/user/pass’ res = requests.get(url, auth=HTTPBasicAuth(’user’, ’pass’)) # Or shorthand: requests.get(url, auth=(’user’, ’pass’)) Bearer Token Authentication (Headers) headers = {’Authorization’: ’Bearer YOUR_TOKEN_HERE’} res = requests.get(’https://api.example.com/data’, \ headers=headers) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 86 / 283 Introduction to Data Cleaning and Munging Data Cleaning: The process of detecting and correcting (or removing) corrupt, incomplete, or inaccurate records from a dataset. Data Munging (Wrangling): The process of transforming and mapping data from one ”raw” data form into another format, making it more appropriate for downstream purposes like analytics or machine learning. Key Objective To improve data quality, ensure consistency, and make the dataset suitable for reliable analysis and modeling. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 87 / 283 Common Data Cleaning Tasks Task Description Handling Missing Data Identifying and dealing with null, NaN, or NA values appropriately. Removing Duplicates Identifying and deleting identical rows that can skew results. Data Transformation Converting data types, standardizing formats, and scaling values. Handling Outliers Detecting anomalies and treating them via capping or removal. Table: Typical Data Cleaning Operations GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 88 / 283 Handling Missing Data in Pandas Detecting Missing Values Use isnull() or isna() to detect missing values in a DataFrame. import pandas as pd import numpy as np df = pd.DataFrame({’A’: [1, np.nan, 3], ’B’: [4, 5, \ np.nan]}) print(df.isnull()) Dropping or Filling Missing Values df.dropna(): Drop rows or columns containing missing values. df.fillna(value): Fill missing values with a specific constant, mean, median, etc. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 89 / 283 Handling Duplicates Duplicate records can skew analysis, over-represent certain samples, and lead to incorrect models. df.duplicated(): Returns a boolean Series denoting duplicate rows. df.drop duplicates(): Returns DataFrame with duplicates removed. data = {’col1’: [1, 1, 2, 3], ’col2’: [’a’, ’a’, ’b’, ’c’]} df = pd.DataFrame(data) # Remove duplicate rows keeping the first occurrence clean_df = df.drop_duplicates() GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 90 / 283 Data Transformation: Mapping and Replacing Data often needs to be transformed into a standardized format before analysis. Replacing Values: df.replace(to replace, value) is used to replace specific values with new ones. Mapping: Using a dictionary to map values in a Series to new representations. # Replacing a specific string value df[’status’] = df[’status’].replace(’Inactive’, ’0’) # Mapping ordinal categories to numerical values mapping_dict = {’Low’: 1, ’Medium’: 2, ’High’: 3} df[’priority’] = df[’priority’].map(mapping_dict) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 91 / 283 The Data Munging Process Raw Data Discover & Audit Clean & Structure Publish & Analyze Enrich Data Figure: Workflow of Data Wrangling GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 92 / 283 Introduction to Rescaling Definition: Rescaling is a preprocessing step where data is scaled or transformed into a specific range. Commonly, attributes in a dataset have varying scales (e.g., age in years, income in thousands). Why Rescale?: Many machine learning algorithms perform better or converge faster when features are on a relatively similar scale. Algorithms affected by Scale Gradient Descent based algorithms (e.g., Linear Regression, Neural Networks) Distance-based algorithms (e.g., K-Nearest Neighbors, K-Means, SVM) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 93 / 283 Min-Max Scaling (Normalization) Transforms features by scaling each feature to a given range, usually [0, 1]. The formula is given by: Xscaled = X − Xmin Xmax − Xmin Use Case Ideal for algorithms that do not assume any distribution of the data, like K-Nearest Neighbors and Neural Networks. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 94 / 283 Min-Max Scaling in Scikit-Learn We use the MinMaxScaler from sklearn.preprocessing. from sklearn.preprocessing import MinMaxScaler import pandas as pd import numpy as np # Sample data data = {’feature1’: [10, 20, 30], ’feature2’: [100, 200, \ 300]} df = pd.DataFrame(data) # Initialize the scaler scaler = MinMaxScaler(feature_range=(0, 1)) # Fit and transform the data rescaled_data = scaler.fit_transform(df) print(rescaled_data) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 95 / 283 Standardization (Z-score Normalization) Centers the data around the mean with a unit standard deviation. Resulting distribution has a mean of 0 and a standard deviation of 1. Formula: X −µ σ where µ is the mean and σ is the standard deviation. Xstandardized = When to use? When features follow a Gaussian (normal) distribution. Algorithms like SVM, Logistic Regression, and Linear Regression assume data is centered. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 96 / 283 Standardization in Scikit-Learn We use StandardScaler from sklearn.preprocessing. from sklearn.preprocessing import StandardScaler # Initialize the scaler scaler = StandardScaler() # Fit and transform the data standardized_data = scaler.fit_transform(df) print(standardized_data) Important Standardization does not bound values to a specific range (like Min-Max does), which might be an issue for some specific algorithms (e.g., neural networks expecting inputs between 0 and 1). GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 97 / 283 Robust Scaling Scales features using statistics that are robust to outliers. Uses the median and the interquartile range (IQR). Formula: Xrobust = X − Median IQR Advantage If your data contains many outliers, scaling using the mean and variance (like in Standard Scaler) is likely to not work very well. RobustScaler provides better results. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 98 / 283 Robust Scaling in Scikit-Learn from sklearn.preprocessing import RobustScaler # Initialize the scaler scaler = RobustScaler() # Fit and transform the data robust_scaled_data = scaler.fit_transform(df) print(robust_scaled_data) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 99 / 283 Comparison Summary Method Normalization Standardization Robust Scaling Scikit-Learn Class MinMaxScaler StandardScaler RobustScaler Best for Non-Gaussian distributions, bounded range Gaussian distributions, regression/SVM Data with significant outliers Table: Summary of Rescaling Methods GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 100 / 283 Visualizing the Effect of Rescaling X2′ X2 X1′ X1 Original Data (Varying Scales) GTU Diploma Engineering (Semester 4) Scaled Data (Similar Scales) Introduction to Data Analysis 101 / 283 Introduction to Data Normalization Data Normalization is the process of rescaling one or more attributes to a common scale without distorting differences in the ranges of values. Data Transformation involves changing the format, structure, or values of data to make it suitable for analysis. Why is it important? Many machine learning algorithms (e.g., K-Means, SVM, Neural Networks) rely on distance metrics. Features on larger scales can disproportionately influence the model, leading to poor performance or slow convergence. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 102 / 283 Common Transformation Techniques Min-Max Scaling (Normalization): Rescales data to a fixed range, typically [0, 1]. Standardization (Z-score): Rescales data to have a mean of 0 and a standard deviation of 1. Log Transformation: Applies the natural logarithm to reduce skewness in the data. Robust Scaling: Uses statistics that are robust to outliers (median and interquartile range). GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 103 / 283 Min-Max Scaling Formula Xnorm = X − Xmin Xmax − Xmin Implementation with Scikit-Learn: from sklearn.preprocessing import MinMaxScaler import pandas as pd data = {’Price’: [200, 500, 1000, 50000]} df = pd.DataFrame(data) scaler = MinMaxScaler() df[’Price_Scaled’] = scaler.fit_transform(df[[’Price’]]) print(df) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 104 / 283 Standardization (Z-Score Scaling) Formula X −µ σ where µ is the mean and σ is the standard deviation. Z= Implementation with Scikit-Learn: from sklearn.preprocessing import StandardScaler scaler = StandardScaler() df[’Price_Std’] = scaler.fit_transform(df[[’Price’]]) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 105 / 283 Comparison of Scaling Methods Method Min-Max Range [0, 1] Standard Unbounded Robust Unbounded When to use? For bounded intervals, no heavy outliers. When feature distribution is Gaussian. When data contains many outliers. Table: Comparison of common scaling techniques GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 106 / 283 Log Transformation Used to transform highly skewed data into a more normalized distribution. Particularly useful for variables like income, population, or prices that span several orders of magnitude. Important Limitation Log transformation cannot be applied to zero or negative values. A common workaround is to use log(x + 1). GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 107 / 283 Visualizing Transformation Concepts Freq Freq log(x) Value Original Skewed Data Value Normalized Data Figure: Effect of Log Transformation on Skewed Data GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 108 / 283 Introduction to Dimensionality Reduction What is Dimensionality Reduction? Dimensionality reduction is the process of reducing the number of random variables under consideration by obtaining a set of principal variables. Why do we need it? Curse of Dimensionality: High-dimensional data is sparse and makes machine learning difficult. Computational Efficiency: Reduces training time and memory footprint. Data Visualization: Allows visualizing complex datasets in 2D or 3D. Noise Reduction: Eliminates irrelevant features and redundant information. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 109 / 283 The Curse of Dimensionality The Problem As the number of features (dimensions) increases, the volume of the space increases exponentially, making the available data become sparse. Distance metrics (like Euclidean distance) lose their usefulness as the ratio of nearest to farthest points approaches 1. Models are prone to overfitting because they can easily memorize the sparse data points. Solution: Reduce dimensions while retaining as much variance (information) as possible. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 110 / 283 Common Techniques Dimensionality reduction techniques are broadly classified into two categories: Feature Extraction Feature Selection Selecting a subset of relevant features from the original dataset. Filter Methods Wrapper Methods Embedded Methods Creating a new, smaller set of features that captures the essence of the original data. Principal Component Analysis (PCA) t-Distributed Stochastic Neighbor Embedding (t-SNE) Linear Discriminant Analysis (LDA) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 111 / 283 Principal Component Analysis (PCA) PCA is a linear, unsupervised dimensionality reduction technique. It identifies the hyperplane that lies closest to the data, and then it projects the data onto it. Principal Components: New, uncorrelated variables constructed as linear combinations of the original variables. The first principal component accounts for the largest possible variance in the data. X2 PC 1 (Max Variance) PC 2 X1 GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 112 / 283 PCA Implementation in Python We can easily implement PCA using scikit-learn. import numpy as np from sklearn.decomposition import PCA from sklearn.datasets import load_iris # Load dataset data = load_iris() X = data.data # Initialize PCA keeping 2 components pca = PCA(n_components=2) # Fit and transform the data X_reduced = pca.fit_transform(X) print(f"Original shape: {X.shape}") print(f"Reduced shape: {X_reduced.shape}") GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 113 / 283 t-Distributed Stochastic Neighbor Embedding (t-SNE) t-SNE is a non-linear technique primarily used for data exploration and visualizing high-dimensional data. It calculates similarity measures between pairs of instances in the high dimensional space and in the low dimensional space. It tries to optimize these two similarity measures using a cost function (Kullback-Leibler divergence). Important Note Unlike PCA, t-SNE is highly computationally expensive and is mostly used for visualizing data in 2D or 3D, rather than for feature reduction before training a model. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 114 / 283 Comparing PCA and t-SNE Feature PCA t-SNE Type Linear Non-linear Primary Goal Preserve global structure (variance) Preserve local structure Use Case Feature extraction, noise reduction Data visualization Speed Fast and scalable Slow, poorly scalable Deterministic Yes No (stochastic) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 115 / 283 Introduction to Dimensionality Reduction What is Dimensionality Reduction? Dimensionality reduction is the process of reducing the number of random variables under consideration, by obtaining a set of principal variables. Why do we need it? Visualization: High-dimensional data (e.g., ¿3 dimensions) is impossible to visualize directly. Efficiency: Reduces computation time and storage space required. Noise Removal: Can help eliminate redundant features and noise. Overfitting: Reduces the risk of overfitting by limiting the number of features. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 116 / 283 The Curse of Dimensionality The Problem As the number of features (dimensions) increases, the volume of the space increases exponentially, making the data sparse. This phenomenon is known as the “Curse of Dimensionality.” Distance metrics (like Euclidean distance) lose their meaning in high-dimensional spaces. Requires an exponentially larger amount of data to generalize accurately. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 117 / 283 Common Dimensionality Reduction Techniques Dimensionality reduction is typically divided into two categories: Feature Extraction Transforming the data into a lower-dimensional space. Feature Selection Selecting a subset of original features without modifying them. Missing Value Ratio Low Variance Filter High Correlation Filter GTU Diploma Engineering (Semester 4) Principal Component Analysis (PCA) t-Distributed Stochastic Neighbor Embedding (t-SNE) Uniform Manifold Approximation and Projection (UMAP) Introduction to Data Analysis 118 / 283 Principal Component Analysis (PCA) PCA is a linear dimensionality reduction technique. It identifies the hyperplane that lies closest to the data, and then it projects the data onto it. The axes of the new coordinate system (Principal Components) maximize the variance of the data. x2 PC1 x1 PC2 GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 119 / 283 PCA in Python using Scikit-Learn Implementation We use the PCA class from sklearn.decomposition. import numpy as np from sklearn.decomposition import PCA from sklearn.datasets import load_iris # Load dataset (4 dimensions) iris = load_iris() X = iris.data # Initialize PCA to keep 2 components pca = PCA(n_components=2) # Fit and transform the data X_reduced = pca.fit_transform(X) print(f"Original shape: {X.shape}") print(f"Reduced shape: {X_reduced.shape}") GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 120 / 283 t-Distributed Stochastic Neighbor Embedding (t-SNE) t-SNE is a non-linear technique specifically designed for visualization. It maps high-dimensional data to a 2D or 3D space. It works by calculating similarity measures between pairs of instances in the high-dimensional space and in the low-dimensional space, and then optimizing these two similarity measures to be as close as possible. Note: t-SNE is computationally expensive and does not preserve global structures well compared to PCA. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 121 / 283 t-SNE in Python using Scikit-Learn from sklearn.manifold import TSNE # Initialize t-SNE # perplexity is a hyperparameter (usually between 5 and 50) tsne = TSNE(n_components=2, perplexity=30, random_state=42) # Fit and transform the data X_tsne = tsne.fit_transform(X) print(f"Original shape: {X.shape}") print(f"t-SNE shape: {X_tsne.shape}") Important It is often highly recommended to use another dimensionality reduction method (e.g. PCA for dense data or TruncatedSVD for sparse data) to reduce the number of dimensions to a reasonable amount (e.g. 50) before running t-SNE! GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 122 / 283 Comparison of Techniques Feature Type Speed Preserves Global Preserves Local Best for New Data PCA Linear Very Fast Yes No Preprocessing Can transform t-SNE Non-linear Slow No Yes Visualization Cannot directly UMAP Non-linear Fast Yes (Better than t-SNE) Yes Both Can transform Table: Comparison of common dimensionality reduction algorithms GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 123 / 283 What is Dimensionality Reduction? Definition: The process of reducing the number of random variables under consideration by obtaining a set of principal variables. The Curse of Dimensionality: As the number of features increases, the volume of the feature space increases exponentially, making data sparse and models prone to overfitting. Benefits: Reduces storage space and computational time. Mitigates multicollinearity. Facilitates data visualization (e.g., 2D or 3D plots). GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 124 / 283 Common Dimensionality Reduction Techniques Linear Methods Principal Component Analysis (PCA): Unsupervised, maximizes variance. Linear Discriminant Analysis (LDA): Supervised, maximizes class separability. Singular Value Decomposition (SVD): Matrix factorization technique. Non-Linear Methods (Manifold Learning) t-SNE (t-Distributed Stochastic Neighbor Embedding): Great for visualization. UMAP (Uniform Manifold Approximation and Projection): Faster than t-SNE, preserves global structure better. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 125 / 283 Principal Component Analysis (PCA) in Python Implemented efficiently in scikit-learn. Requires data scaling (e.g., StandardScaler) before applying PCA. Python Implementation from sklearn.decomposition import PCA from sklearn.preprocessing import StandardScaler # Assume X is our feature matrix scaler = StandardScaler() X_scaled = scaler.fit_transform(X) # Initialize PCA to keep 2 components pca = PCA(n_components=2) X_pca = pca.fit_transform(X_scaled) print("Explained variance ratio:", pca.explained_variance_ratio_) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 126 / 283 Comparison of Techniques Method PCA LDA t-SNE UMAP Type Unsupervised Supervised Unsupervised Unsupervised Linearity Linear Linear Non-linear Non-linear Best For General purpose, noise reduction Classification preprocessing High-dimensional visualization Faster visualization, clustering Table: Summary of Dimensionality Reduction Methods GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 127 / 283 Conceptualizing PCA X2 (Feature 2) PC1 (Max Variance) PC2 (Orthogonal) X1 (Feature 1) PC1 captures the direction of maximum variance. PC2 is orthogonal to PC1 and captures the remaining variance. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 128 / 283 Introduction to Regression Modeling Regression analysis is a statistical process for estimating the relationships between a dependent variable and one or more independent variables. It helps in understanding how the typical value of the dependent variable changes when any one of the independent variables is varied. Widely used for prediction and forecasting in various domains. Key Goal To fit a mathematical model to a set of observed data points in a way that minimizes the difference between the predicted and actual values. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 129 / 283 Types of Regression Models Model Type Linear Regression Multiple Regression Polynomial Regression Logistic Regression GTU Diploma Engineering (Semester 4) Description Relationship is modeled using a straight line. Uses two or more independent variables. Relationship is modeled as an nth degree polynomial. Used for predicting categorical outcomes (e.g., Yes/No). Introduction to Data Analysis 130 / 283 Simple Linear Regression Concept The equation of a simple linear regression model is: Y = β0 + β 1 X + ϵ Where: Y : Dependent variable (target) X : Independent variable (predictor) β0 : Y-intercept (value of Y when X = 0) β1 : Slope of the line (effect of X on Y ) ϵ: Error term (unexplained variance) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 131 / 283 Visualizing Linear Regression Y Ŷ = β0 + β1 X ϵi X Interpretation The red line represents the best fit line that minimizes the sum of squared residuals (errors) between the observed data points and the line. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 132 / 283 Example Code: Regression in Python import numpy as np from sklearn.linear_model import LinearRegression # Sample data X = np.array([[1], [2], [3], [4], [5]]) y = np.array([1.2, 1.8, 2.5, 3.1, 4.3]) # Create and train the model model = LinearRegression() model.fit(X, y) # Predict predictions = model.predict([[6]]) print(f"Prediction for X=6: {predictions[0]:.2f}") GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 133 / 283 Introduction to Multivariate Analysis Definition Multivariate analysis refers to any statistical technique used to analyze data that arises from more than one variable. This models more realistic applications where situations involve multiple variables. Analyzes complex data sets representing measurements of p variables on n objects. Helps to understand the relationship among variables. Can be used for prediction, classification, and data reduction. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 134 / 283 Goals of Multivariate Analysis Dependence Exploration Data Reduction Investigating whether variables are Simplifying data as much as possible mutually independent or if one or without sacrificing valuable more are dependent on others (e.g., information (e.g., PCA). Regression). Sorting and Grouping Hypothesis Testing Creating groups of “similar” objects or variables (e.g., Cluster Analysis). Testing hypotheses about the data structure or parameters (e.g., MANOVA). GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 135 / 283 Classification of Techniques Multivariate analysis techniques are broadly classified into two categories based on the relationships among variables: Dependence Methods Variables divided into dependent and independent groups. Multiple Regression Multivariate ANOVA (MANOVA) Discriminant Analysis GTU Diploma Engineering (Semester 4) Interdependence Methods All variables analyzed simultaneously without distinction. Principal Component Analysis (PCA) Factor Analysis Cluster Analysis Introduction to Data Analysis 136 / 283 Multivariate Data Structure Predictors X1 Multiple Outcomes X2 Y1 , Y2 , . . . , Ym X3 The relationships are analyzed jointly. Captures covariance/correlation between outcomes. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 137 / 283 Applications of Multivariate Analysis Market Research: Identifying customer segments (Cluster Analysis). Finance: Credit scoring using multiple financial indicators (Discriminant Analysis). Healthcare: Predicting disease risk based on patient characteristics (Logistic Regression). Image Processing: Reducing the dimensionality of image data for compression (PCA). Important Note Choosing the right multivariate technique requires understanding both your data structure and your research objective. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 138 / 283 Descriptive Statistics Definition Descriptive statistics are summary statistics that quantitatively describe or summarize features of a collection of information. Key Purposes: Provide simple summaries about the sample and the measures. Form the basis of virtually every quantitative analysis of data. Present quantitative descriptions in a manageable form. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 139 / 283 Measures of Central Tendency A measure of central tendency is a single value that attempts to describe a set of data by identifying the central position within that set of data. The Big Three: 1 Mean: The arithmetic average. 2 Median: The middle value when data is ordered. 3 Mode: The most frequent value. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 140 / 283 The Mean Arithmetic Mean The mean is the sum of all values divided by the number of values in the dataset. Formula: n x̄ = 1X xi n i=1 Python Example: import numpy as np data = [2, 4, 6, 8, 10] mean_val = np.mean(data) print(mean_val) # Output: 6.0 GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 141 / 283 The Median What is the Median? The median is the value separating the higher half from the lower half of a data sample. Finding the Median: Odd number of observations: The middle value. Even number of observations: The average of the two middle values. Example (Odd): 1, 3, 6, 7, 8, 9, 10 ⇒ Median = 7 Example (Even): 1, 2, 3, 4, 5, 6, 8, 9 ⇒ Median = 4.5 GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 142 / 283 The Mode Mode Definition The mode is the value that appears most frequently in a data set. A set of data may have one mode, more than one mode, or no mode at all. Unimodal: One peak (e.g., [1, 2, 2, 3]) ⇒ Mode = 2 Bimodal: Two peaks (e.g., [1, 1, 2, 3, 3]) ⇒ Modes = 1 and 3 Multimodal: More than two peaks. Note: Mode is the only measure of central tendency that can be used with nominal (categorical) data. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 143 / 283 Comparison of Measures Measure Mean Outlier Sensitivity High Median Low Mode Low Best Used For Continuous data without extreme outliers Skewed distributions or data with outliers Categorical data or finding most common item Table: Choosing the Right Measure of Central Tendency GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 144 / 283 Visualizing Central Tendency: Skewness Frequency Normal Distribution Mean=Median=Mode Value Frequency Positive (Right) Skew Mode Median Mean GTU Diploma Engineering (Semester 4) Introduction to Data Analysis Value 145 / 283 Introduction to Correlation Definition: A statistical measure that expresses the extent to which two variables are linearly related. Purpose: Used to determine whether a relationship exists, its direction, and its strength. Types of Relationships: Positive: Both variables move in the same direction. Negative: Variables move in opposite directions. Zero: No linear relationship exists. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 146 / 283 Pearson’s Correlation Coefficient (r ) Overview Pearson’s Correlation Coefficient (often denoted as r ) is a measure of the linear correlation between two sets of data. Developed by Karl Pearson. It is the ratio between the covariance of two variables and the product of their standard deviations. Essentially, it represents the normalized measurement of the covariance. Range: −1 ≤ r ≤ 1. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 147 / 283 Formula for Pearson’s r The formula for the sample Pearson correlation coefficient is: Mathematical Expression P (xi − x̄)(yi − ȳ ) r = pP P (xi − x̄)2 (yi − ȳ )2 Where: xi , yi = Individual sample points indexed with i x̄ = Sample mean for variable X ȳ = Sample mean for variable Y GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 148 / 283 Interpretation of r Value of r 1.0 0.7 to 0.99 0.4 to 0.69 0.1 to 0.39 0.0 −0.1 to −0.39 −0.4 to −0.69 −0.7 to −0.99 −1.0 Interpretation Perfect positive linear relationship Strong positive linear relationship Moderate positive linear relationship Weak positive linear relationship No linear relationship Weak negative linear relationship Moderate negative linear relationship Strong negative linear relationship Perfect negative linear relationship Table: Rule of Thumb for interpreting r GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 149 / 283 Visualizing Correlation with Scatterplots Y Y r >0 X GTU Diploma Engineering (Semester 4) Y r <0 Introduction to Data Analysis X r ≈0 X 150 / 283 Assumptions for Pearson’s r Before using Pearson’s correlation, certain assumptions must be met: Continuous Data: Variables must be either interval or ratio scale. Linearity: The relationship between the two variables must be linear. Normality: Both variables should be approximately normally distributed. No Outliers: The data should not have significant outliers, as Pearson’s r is sensitive to them. Homoscedasticity: The variance of errors is constant across all levels of the independent variable. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 151 / 283 Advantages and Limitations Limitations Advantages Easy to compute and interpret. Indicates both the strength and direction of the relationship. Widely understood and used in various fields. GTU Diploma Engineering (Semester 4) Only measures linear relationships. Highly sensitive to outliers. Does not imply causation (Correlation ̸= Causation). Introduction to Data Analysis 152 / 283 Introduction to Curve Fitting Curve fitting is the process of constructing a curve, or mathematical function, that has the best fit to a series of data points. Interpolation: Exact fit to data points. Smoothing: Constructing a smooth function that approximately fits the data. The Method of Least Squares is a standard approach in regression analysis to approximate the solution of overdetermined systems. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 153 / 283 Principle of Least Squares The Principle The method of least squares states that the best-fitting curve has the property that the sum of the squares of the errors (deviations) of the data points from the curve is a minimum. Given a set of n points (x1 , y1 ), (x2 , y2 ), . . . , (xn , yn ) and a fitting function f (x), the error (or residual) at the i-th point is: Ei = yi − f (xi ) The total sum of squared errors is: S= n X i=1 Ei2 = n X (yi − f (xi ))2 i=1 We aim to minimize S. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 154 / 283 Fitting a Straight Line Suppose we want to fit a straight line y = a + bx to the given data points. Pn The sum of squared errors is S = i=1 (yi − (a + bxi ))2 To minimize S, we set the partial derivatives with respect to a and b to zero: ∂S ∂S = 0 and =0 ∂a ∂b This leads to the Normal Equations: Normal Equations for Linear Fit X X yi = na + b xi X X X xi yi = a xi + b xi2 GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 155 / 283 Computational Table Example To solve the normal equations, we often use a computational table: yi xi2 xi yi x1 y1 x12 x1 y1 x2 .. . y2 .. . x22 .. . x2 y2 .. . xn P xi yn P yi xn2 P 2 xi xn yn P xi yi xi Table: Table for Least Squares Calculations GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 156 / 283 Visualizing the Fit y y = a + bx (x1 , y1 ) x The red line represents the line of best fit. The blue points represent actual data. The dashed lines are the residuals minimized by the method. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 157 / 283 Introduction to Correlation Definition: Correlation is a statistical measure that expresses the extent to which two variables are linearly related. It is a common tool for describing simple relationships without making a statement about cause and effect. Correlation Coefficient (r ): Ranges from -1 to 1. r = 1: Perfect positive correlation r = −1: Perfect negative correlation r = 0: No correlation Important Note Correlation does not imply causation! Just because two variables are correlated does not mean one causes the other. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 158 / 283 Scatterplots: The Basics A scatterplot is a type of data display that shows the relationship between two numerical variables. Each member of the dataset gets plotted as a point whose (x, y ) coordinates relate to its values for the two variables. Why use scatterplots? Easily identify patterns, trends, and possible correlations. Detect outliers or anomalous data points. Understand the spread and clustering of data. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 159 / 283 Visualizing Correlation with Scatterplots Y Y Y X Positive GTU Diploma Engineering (Semester 4) X Negative Introduction to Data Analysis X No Correlation 160 / 283 Other Graphical Techniques: Correlation Matrices When dealing with multiple variables, a scatterplot matrix (pairplot) or a heatmap of the correlation matrix is highly effective. A correlation matrix is a table showing correlation coefficients between sets of variables. Var A Var B Var C Var A 1.00 0.85 -0.42 Var B 0.85 1.00 -0.15 Var C -0.42 -0.15 1.00 Table: Example of a Correlation Matrix Table GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 161 / 283 Generating Scatterplots in Python Libraries like matplotlib and seaborn make it easy to generate these plots. Python Code Example import matplotlib.pyplot as plt import numpy as np # Generate random data x = np.random.rand(50) y = 2 * x + np.random.normal(0, 0.1, 50) # Create scatterplot plt.scatter(x, y, color=’blue’, alpha=0.7) plt.title(’Scatterplot of X vs Y’) plt.xlabel(’X Variable’) plt.ylabel(’Y Variable’) plt.show() GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 162 / 283 Introduction to Probability Distributions A probability distribution describes how the probabilities are distributed over the values of a random variable. Key Classification: Discrete Distributions: The random variable takes on countable values (e.g., Bernoulli, Poisson). Continuous Distributions: The random variable takes on an infinite number of possible values in an interval (e.g., Exponential, Normal). Importance They provide the foundation for statistical modeling, hypothesis testing, and predicting future outcomes based on sample data. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 163 / 283 Bernoulli Distribution Definition A discrete probability distribution for a random variable which takes the value 1 (success) with probability p and the value 0 (failure) with probability q = 1 − p. Represents a single trial with two possible outcomes. Probability Mass Function (PMF): P(X = x) = p x (1 − p)1−x for x ∈ {0, 1} Mean: µ = p Variance: σ 2 = p(1 − p) Importance: Forms the basis for more complex distributions like the Binomial distribution. Models yes/no, true/false, or 1/0 events. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 164 / 283 Poisson Distribution Definition A discrete distribution that expresses the probability of a given number of events occurring in a fixed interval of time or space. PMF: λk e −λ k! where λ is the average number of events in the interval, and k is the number of occurrences. P(X = k) = Mean: µ = λ Variance: σ 2 = λ Importance: Widely used in queuing theory, reliability analysis, and modeling rare events (e.g., number of emails received per hour, defects in manufacturing). GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 165 / 283 Exponential Distribution Definition A continuous distribution that models the time between events in a Poisson point process (events occur continuously and independently at a constant average rate). Probability Density Function (PDF): ( λe −λx x ≥ 0 f (x) = 0 x <0 where λ > 0 is the rate parameter. Mean: µ = λ1 Variance: σ 2 = λ12 Importance: Crucial in survival analysis, reliability engineering, and modeling waiting times (e.g., time until a machine fails, waiting time in a queue). GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 166 / 283 Normal Distribution Definition A continuous probability distribution characterized by a symmetric, bell-shaped curve, widely used in natural and social sciences. PDF: f (x) = 1 x−µ 2 1 √ e− 2 ( σ ) σ 2π where µ is the mean and σ is the standard deviation. Importance: The Central Limit Theorem states that the sum of a large number of independent random variables tends toward a normal distribution, regardless of the underlying distribution. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 167 / 283 Normal Distribution: Bell Curve f (x) µ−σ µ µ + σµ + 2σ x Symmetrical around the mean µ. The total area under the curve is exactly 1. The ”68-95-99.7” Empirical Rule applies. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 168 / 283 Summary of Distributions Distribution Bernoulli Poisson Exponential Normal Type Discrete Discrete Continuous Continuous GTU Diploma Engineering (Semester 4) Mean (µ) p λ 1/λ µ Variance (σ 2 ) p(1 − p) λ 1/λ2 σ2 Introduction to Data Analysis Primary Application Single trial success/failure Counts of events in intervals Time between events Natural phenomena, CLT 169 / 283 Introduction to Probability Distributions A probability distribution describes how the probabilities are distributed over the values of a random variable. It provides the probabilities of occurrence of different possible outcomes in an experiment. Importance: Forms the foundation of statistical inference and hypothesis testing. Allows modeling of real-world phenomena to make predictions. Helps in risk assessment, quality control, and data analysis. Types broadly classified into: Discrete Probability Distributions (e.g., Bernoulli, Poisson) Continuous Probability Distributions (e.g., Normal, Exponential) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 170 / 283 Bernoulli Distribution Definition A discrete probability distribution of a random variable which takes the value 1 with probability p and the value 0 with probability q = 1 − p. It models a single trial of an experiment with exactly two possible outcomes (Success/Failure). Probability Mass Function (PMF): P(X = x) = p x (1 − p)1−x for x ∈ {0, 1} Importance Simplest building block for other discrete distributions (like Binomial). Used in logistic regression and modeling binary outcomes (e.g., spam/not spam, coin toss). GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 171 / 283 Normal (Gaussian) Distribution Definition A continuous probability distribution characterized by a symmetric, bell-shaped curve. It is defined by its mean (µ) and standard deviation (σ). Probability Density Function (PDF): f (x) = 1 x−µ 2 1 √ e− 2 ( σ ) σ 2π Properties: Symmetric around the mean, mean = median = mode. Importance: Central Limit Theorem (CLT) states that the sum of many independent random variables tends toward a normal distribution. Widely used in natural and social sciences to represent real-valued random variables (heights, test scores, measurement errors). GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 172 / 283 Normal Distribution Visualization f (x) µ − 3σµ − 2σ µ − σ GTU Diploma Engineering (Semester 4) µ x µ + σ µ + 2σµ + 3σ Introduction to Data Analysis 173 / 283 Poisson Distribution Definition A discrete probability distribution that expresses the probability of a given number of events occurring in a fixed interval of time or space if these events occur with a known constant mean rate (λ) and independently of the time since the last event. Probability Mass Function (PMF): P(X = k) = λk e −λ k! Importance Models rare events and count data. Applications include predicting the number of network failures, calls to a call center in an hour, or the number of mutations on a DNA strand. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 174 / 283 Exponential Distribution Definition A continuous probability distribution that models the time between events in a Poisson point process (events occur continuously and independently at a constant average rate λ). Probability Density Function (PDF): ( λe −λx f (x) = 0 x ≥0 x <0 Importance Possesses the ”memoryless” property. Crucial in reliability theory and queuing theory (e.g., predicting time until next machine failure, waiting time in a queue). GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 175 / 283 Summary of Probability Distributions Distribution Bernoulli Normal Poisson Exponential Type Discrete Continuous Discrete Continuous Parameters p (prob. of success) µ (mean), σ 2 (variance) λ (average rate) λ (rate) Primary Use Case Single trial with binary outcome Natural phenomena, symmetric data Count of events in a fixed interval Time between independent events Table: Comparison of common probability distributions. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 176 / 283 Probability Distributions Definition A probability distribution is a mathematical function that provides the probabilities of occurrence of different possible outcomes in an experiment. Describes the overall shape of the data. Helps predict future outcomes. Forms the foundation of statistical inference and hypothesis testing. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 177 / 283 Types of Distributions Distribution Normal Poisson Exponential Bernoulli Data Type Continuous Discrete Continuous Discrete Key Characteristic Symmetrical, bell-shaped curve Events over a fixed interval Time between events Single trial, binary outcome Table: Overview of Common Probability Distributions GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 178 / 283 Normal Distribution Often called the Gaussian distribution. Characterized by its mean (µ) and standard deviation (σ). Approximately 68% of data falls within one standard deviation of the mean. f (x) µ−σ GTU Diploma Engineering (Semester 4) µ µ+σ Introduction to Data Analysis x 179 / 283 Poisson Distribution When to use? Used to model the number of times an event occurs in a fixed interval of time or space. Parameters: λ (average rate of occurrence). k −λ e Formula: P(X = k) = λ k! Examples: Number of emails received per hour. Number of defects in a batch of products. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 180 / 283 Exponential Distribution Closely related to the Poisson distribution. Models the time elapsed between events in a Poisson process. Parameters: λ (rate parameter, same as Poisson). PDF: f (x) = λe −λx for x ≥ 0. Importance Crucial in reliability engineering and survival analysis (e.g., predicting the lifespan of electronic components). GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 181 / 283 Bernoulli Distribution The simplest discrete distribution. Models a single trial with exactly two possible outcomes: ”Success” (1) and ”Failure” (0). Parameter: p (probability of success). Outcome (x) 1 (Success) 0 (Failure) GTU Diploma Engineering (Semester 4) Probability (P(X = x)) p 1−p Introduction to Data Analysis 182 / 283 What is Data Visualization? Definition Data visualization is the graphical representation of information and data. By using visual elements like charts, graphs, and maps, data visualization tools provide an accessible way to see and understand trends, outliers, and patterns in data. Helps in translating complex data into a visual context. Enables quicker identification of patterns and trends. Facilitates better decision-making processes. Essential for both exploratory data analysis and explanatory reporting. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 183 / 283 The Data Visualization Process Raw Data Processing & Cleaning Visualization Insights & Action Key Steps: 1 Data Collection: Gathering relevant datasets. 2 Processing: Cleaning and transforming data into a usable format. 3 Visualization: Mapping data attributes to visual properties. 4 Interpretation: Extracting actionable insights from the visual representation. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 184 / 283 Common Types of Visualizations Type Bar Chart Line Chart Scatter Plot Pie Chart Histogram Heatmap Primary Use Case Comparing categorical data Showing trends over time Identifying relationships & correlation Displaying parts of a whole Showing distribution of data Visualizing density or intensity Data Characteristics Discrete categories, magnitudes Continuous data, time series Two continuous variables Proportional data (sum to 100%) Continuous variables grouped in bins Matrix data, spatial distribution Table: Overview of Data Visualization Types GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 185 / 283 Principles of Effective Visualization Clarity is Key The primary goal of data visualization is to communicate information clearly and efficiently to users. Best Practices: Know Your Audience: Tailor the complexity and style of the visual to the viewer’s expertise. Choose the Right Chart: Ensure the visual format aligns with the data and the message. Minimize Chartjunk: Remove unnecessary gridlines, excessive colors, and distracting elements (Edward Tufte’s principle). Use Color Purposefully: Highlight key data points and ensure accessibility (e.g., colorblind-friendly palettes). GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 186 / 283 Tools for Data Visualization BI & Desktop Tools: Programming Libraries: Python: Matplotlib, Seaborn, Plotly, Bokeh Tableau R: ggplot2, Shiny QlikView JavaScript: D3.js, Chart.js Microsoft Excel Microsoft Power BI Python Example (Matplotlib) import matplotlib.pyplot as plt plt.plot([1, 2, 3], [4, 5, 6]) plt.title("Simple Line Chart") plt.show() GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 187 / 283 What is Data Visualization? Definition Data visualization is the graphical representation of information and data. By using visual elements like charts, graphs, and maps, data visualization tools provide an accessible way to see and understand trends, outliers, and patterns in data. Transforms raw data into a visual context. Makes complex data more accessible, understandable, and usable. Essential for analyzing massive amounts of information and making data-driven decisions. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 188 / 283 Why is Data Visualization Important? Identifying Trends: Visuals make it easier to spot patterns and trends over time. Highlighting Correlations: Enables discovering relationships between independent variables. Finding Outliers: Easily pinpoints data points that deviate significantly from the rest. Storytelling: Helps convey the message or narrative behind the data effectively to stakeholders. Speeding Up Decision Making: Human brains process visual information much faster than text or spreadsheets. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 189 / 283 Data Visualization Pipeline Raw Data Processing & Cleaning Mapping to Visuals Human Insights Example Scenario A retail company analyzes millions of transactions (Raw Data), cleans missing entries (Processing), plots sales per region on a heat map (Visuals), and identifies underperforming stores (Insights). GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 190 / 283 Common Data Visualization Tools Tool Type Key Features Tableau BI Platform Interactive dashboards, drag-anddrop, enterprise scaling. Matplotlib Python Library Highly customizable, foundation for Python data science. D3.js JavaScript Library Powerful, web-based, data-driven document manipulation. Power BI BI Platform Microsoft ecosystem integration, robust data modeling. Table: Comparison of Popular Visualization Tools GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 191 / 283 Introduction to matplotlib matplotlib is the most widely used 2D plotting library for Python. It provides a MATLAB-like interface for simple plotting and an object-oriented interface for more complex visualizations. Pyplot is a collection of functions that make matplotlib work like MATLAB. Installation and Import pip install matplotlib import matplotlib.pyplot as plt import numpy as np GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 192 / 283 Basic Line Plot The plot() function is used to draw lines and markers. By default, it draws a line connecting the points. Example: Simple Line Plot x = [1, 2, 3, 4] y = [10, 20, 25, 30] plt.plot(x, y, color=’blue’, marker=’o’, linestyle=’--’) plt.title("Simple Line Plot") plt.xlabel("X-axis Label") plt.ylabel("Y-axis Label") plt.show() GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 193 / 283 Figures and Subplots A Figure is the overall window or page that everything is drawn on. Axes are the individual plots within the figure. Creating Subplots # Create a figure with 2 rows and 1 column of subplots fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(6, 4)) ax1.plot([1, 2, 3], [4, 5, 6]) ax1.set_title("First Subplot") ax2.plot([1, 2, 3], [6, 5, 4], color=’red’) ax2.set_title("Second Subplot") plt.tight_layout() # Adjust spacing plt.show() GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 194 / 283 Scatter Plots Scatter plots are used to observe relationships between variables. The scatter() function is used for this purpose. Example: Scatter Plot x = np.random.rand(50) y = np.random.rand(50) colors = np.random.rand(50) sizes = 1000 * np.random.rand(50) plt.scatter(x, y, c=colors, s=sizes, alpha=0.5, cmap=’viridis’) plt.colorbar() # Show color scale plt.title("Scatter Plot Example") plt.show() GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 195 / 283 Bar Charts and Histograms Histogram Bar Chart Used for categorical data. categories = [’A’, ’B’, ’C’] values = [10, 15, 7] plt.bar(categories, values) plt.title("Bar Chart") plt.show() GTU Diploma Engineering (Semester 4) Used for frequency distribution. data = np.random.randn(1000) plt.hist(data, bins=20, color=’skyblue’, edgecolor=’black’) plt.title("Histogram") plt.show() Introduction to Data Analysis 196 / 283 Comparison of Plot Types Plot Type Line Plot Scatter Plot Bar Chart Histogram Pie Chart Box Plot Function plt.plot() plt.scatter() plt.bar() plt.hist() plt.pie() plt.boxplot() Best Used For Trends over time Correlation between two variables Comparing categorical data Frequency distribution of continuous data Proportion of a whole Statistical distribution (outliers, quartiles) Table: Common matplotlib plot types GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 197 / 283 matplotlib Object-Oriented Interface While pyplot is quick and easy, the object-oriented API gives more control. We create Figure and Axes objects explicitly. Object-Oriented Approach fig = plt.figure(figsize=(8, 4)) ax = fig.add_axes([0.1, 0.1, 0.8, 0.8]) # [left, bottom, width, height] ax.plot(x, y, label=’Data’) ax.set_xlabel(’X Axis’) ax.set_ylabel(’Y Axis’) ax.set_title(’OO Interface Example’) ax.legend() plt.show() GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 198 / 283 Matplotlib Architecture Overview Scripting Layer (pyplot) manipulates Artist Layer (Figure, Axes, Text, Line2D) rendered by Backend Layer (FigureCanvas, Renderer, Event) Backend: Handles output formats (Screen, PDF, PNG). Artist: Knows how to use the renderer to draw onto the canvas. Scripting: Simplifies common tasks (like MATLAB). GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 199 / 283 Introduction to Customization Why Customize? Customizing plots is essential for making data visualizations clear, aesthetically pleasing, and suitable for publication or presentations. Common Customizations: Titles, Labels, and Legends Colors, Markers, and Line Styles Axes limits and Ticks Grids and Backgrounds GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 200 / 283 Titles, Labels, and Legends Adding context to your plots is crucial for interpretation. Example: Adding Labels import matplotlib.pyplot as plt plt.plot([1, 2, 3], [4, 5, 6], label=’Data Trend’) plt.title("Sample Plot Title", fontsize=14) plt.xlabel("X-Axis Label", color=’blue’) plt.ylabel("Y-Axis Label", color=’green’) plt.legend(loc=’best’) plt.show() GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 201 / 283 Colors, Markers, and Line Styles You can customize the appearance of lines and data points using shorthand format strings or explicit arguments. Property Color Marker Line Style Argument color or c marker linestyle or ls Examples ’red’, ’b’, ’#FFDD44’ ’o’ (circle), ’s’ (square), ’*’ ’-’ (solid), ’–’ (dashed) Table: Common Styling Arguments GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 202 / 283 Styling Example Putting it together import matplotlib.pyplot as plt x = [0, 1, 2, 3, 4] y = [0, 1, 4, 9, 16] plt.plot(x, y, color=’purple’, marker=’o’, linestyle=’--’, linewidth=2, markersize=8) plt.title("Customized Line Plot") plt.show() GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 203 / 283 Axes Limits and Ticks Controlling the visible range of axes and the tick marks helps focus on specific data regions. Limits: plt.xlim(min, max) and plt.ylim(min, max) Ticks: plt.xticks([locs], [labels]) and plt.yticks() Code Snippet plt.plot(x, y) plt.xlim(0, 5) plt.ylim(0, 20) plt.xticks([0, 2.5, 5], [’Start’, ’Mid’, ’End’]) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 204 / 283 Adding Grids and Annotations Grids Annotations Enable grids with plt.grid(True) Customize grid color, style, and alpha. Use plt.annotate() to point out specific features. Annotation Example plt.annotate(’Peak’, xy=(2, 4), xytext=(3, 5), arrowprops=dict( facecolor=’black’, shrink=0.05)) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 205 / 283 What is Seaborn? Seaborn is a Python data visualization library based on Matplotlib. It provides a high-level interface for drawing attractive and informative statistical graphics. Integrates closely with pandas data structures. Built-in themes for styling matplotlib graphics. Key Goal Making visualization a central part of exploring and understanding data. Its dataset-oriented plotting functions operate on dataframes and arrays containing whole datasets. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 206 / 283 Seaborn vs. Matplotlib Feature Syntax Default Themes Matplotlib Low-level, verbose Basic, sometimes dated Pandas Integration Manual mapping required Requires manual calculation Statistical Plots Seaborn High-level, concise Modern, aesthetically pleasing Built-in support Built-in functions (e.g., lmplot) Table: Comparison of Matplotlib and Seaborn GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 207 / 283 Visualizing Statistical Relationships Statistical analysis is a process of understanding how variables in a dataset relate to each other. Seaborn provides relplot() to visualize statistical relationships. Scatter Plot Example import seaborn as sns import matplotlib.pyplot as plt # Load a built-in dataset tips = sns.load_dataset("tips") # Create a scatter plot sns.relplot(x="total_bill", y="tip", hue="smoker", data=tips) plt.show() GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 208 / 283 Visualizing Distributions Understanding how a variable is distributed is often the first step in data analysis. Seaborn provides displot() for visualizing univariate and bivariate distributions. Histogram and KDE Example import seaborn as sns import matplotlib.pyplot as plt penguins = sns.load_dataset("penguins") # Plot distribution with Kernel Density Estimate sns.displot(penguins, x="flipper_length_mm", kde=True) plt.show() GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 209 / 283 Categorical Data Visualization When one or more variables are categorical, a different approach is needed. catplot() provides access to several axes-level functions that show the relationship between a numerical and one or more categorical variables. Box Plot Example import seaborn as sns tips = sns.load_dataset("tips") # Create a box plot sns.catplot(x="day", y="total_bill", kind="box", data=tips) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 210 / 283 Seaborn Visualization Workflow Load Data (Pandas) Choose Plot (relplot) Style (set theme) Show Plot (plt.show()) Seaborn relies on Matplotlib underneath for drawing the plots. Pandas provides the data structures optimally consumed by Seaborn. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 211 / 283 Introduction to Plotly Plotly is a graphing library that makes interactive, publication-quality graphs. Supported in Python, R, JavaScript, and more. Key features: Interactivity (hovering, zooming, panning). Web-based (renders in a browser). Wide variety of chart types (statistical, financial, maps, 3D). Why Interactive? Interactive plots allow users to explore data in detail by zooming into regions of interest and viewing exact values via hover tooltips. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 212 / 283 Plotly Express vs. Graph Objects Plotly for Python provides two main APIs: Plotly Express (px) High-level wrapper Less code, easy to use Creates figures instantly Good for standard charts GTU Diploma Engineering (Semester 4) Graph Objects (go) Low-level API More code, fine-grained control Requires building figures piece by piece Best for highly customized or complex plots Introduction to Data Analysis 213 / 283 Example: Basic Scatter Plot with px Using Plotly Express to create a simple scatter plot: import plotly.express as px import pandas as pd # Sample Data df = pd.DataFrame({ "x": [1, 2, 3, 4], "y": [10, 11, 12, 13] }) # Create Scatter Plot fig = px.scatter(df, x="x", y="y", title="Simple Scatter Plot") # Display fig.show() GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 214 / 283 Example: Bar Chart with Graph Objects Building a bar chart using the lower-level API: import plotly.graph_objects as go fig = go.Figure( data=[go.Bar(x=[’A’, ’B’, ’C’], y=[20, 14, 23])] ) fig.update_layout( title="Bar Chart using Graph Objects", xaxis_title="Categories", yaxis_title="Values" ) fig.show() GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 215 / 283 Plotly Architecture Overview Python/R/JS code Generates JSON Figure Object Renders Plotly.js (Browser) Plotly translates your Python instructions into a declarative JSON format. The plotly.js library consumes this JSON and renders the interactive plot in the browser using WebGL or SVG. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 216 / 283 Introduction to Time Series Data What is Time Series Data? Time series data is a sequence of data points indexed in time order. It is usually recorded at equally spaced intervals. Common Examples: Daily closing price of a stock Hourly temperature readings at a weather station Monthly sales figures of a retail store Annual GDP of a country over decades GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 217 / 283 Why Visualize Time Series Data? Identify Trends: Discover underlying directions (upward, downward, or stationary) over a long period. Detect Seasonality: Find repeating patterns or cycles at fixed intervals (e.g., daily, monthly). Spot Anomalies: Identify outliers, unusual events, or sudden spikes/drops in the data. Forecasting Models: Visual patterns help in choosing the right forecasting and predictive models. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 218 / 283 Common Visualization Techniques Plot Type Line Chart Area Chart Bar Chart Seasonal Plot Heatmap Use Case / Description Showing trends over time; connects points with straight lines. Displaying cumulative totals or showing part-towhole relationships over time. Comparing values across discrete time intervals (e.g., yearly revenue). Overlaying multiple seasons (e.g., years) to compare patterns within seasons (e.g., months). Visualizing data intensity over two time dimensions (e.g., day of week vs. hour of day). Table: Overview of Time Series Plot Types GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 219 / 283 Visualizing Trend with Line Charts Value Trend Time Jan Feb Mar Apr A line chart effectively reveals the overall trend and local fluctuations. Overlaying a trend line can help abstract away noise to see the macro movement. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 220 / 283 Time Series Plot in Python Using Pandas and Matplotlib Pandas provides built-in plotting capabilities that make it easy to generate line charts for time-indexed DataFrames. import pandas as pd import matplotlib.pyplot as plt import numpy as np # Generate sample dates and values dates = pd.date_range(start=’2023-01-01’, periods=100) values = np.linspace(0, 10, 100) + \ np.random.normal(0, 1, 100) # Create a DataFrame and set the index df = pd.DataFrame({’Date’: dates, ’Value’: values}) df.set_index(’Date’, inplace=True) # Plot the time series df.plot(figsize=(10, 4), color=’blue’, legend=False) plt.title(’Sample Time Series Data’) plt.ylabel(’Value’) plt.show() GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 221 / 283 Advanced Time Series Components Time Series Decomposition A time series can often be mathematically decomposed into three underlying components: 1 Trend Component: The long-term progression of the series (increasing, decreasing, or constant). 2 Seasonal Component: The repeating short-term cycle in the series (e.g., higher sales every December). 3 Residual (Noise) Component: The random variation or irregularity in the series after removing trend and seasonality. Visualizing these separate components is a crucial step in exploratory data analysis and forecasting preparation. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 222 / 283 Introduction to Advanced Plots While basic plots (bar charts, line graphs) show trends or summaries, advanced plots are essential for visualizing data distributions. Key advantages: Identify central tendency and dispersion. Detect outliers and anomalies. Compare distributions across multiple categories. Two common advanced plots: 1 2 Box Plots: Focus on summary statistics. Violin Plots: Focus on data density and distribution shape. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 223 / 283 Box Plots What is a Box Plot? A Box Plot (or box-and-whisker plot) is a standardized way of displaying the distribution of data based on a five-number summary. The Five-Number Summary: Minimum: The lowest data point excluding any outliers. First Quartile (Q1): 25% of the data lies below this value. Median (Q2): The middle value of the dataset. Third Quartile (Q3): 75% of the data lies below this value. Maximum: The highest data point excluding any outliers. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 224 / 283 Anatomy of a Box Plot Median Min Max Outliers Value Q1 Q3 Interquartile Range (IQR) IQR = Q3 − Q1. It represents the middle 50% of the data. Outliers are typically values outside [Q1 − 1.5 × IQR, Q3 + 1.5 × IQR]. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 225 / 283 Violin Plots What is a Violin Plot? A Violin Plot combines the features of a box plot and a kernel density plot. It shows the distribution of quantitative data across several levels of one (or more) categorical variables. Key Characteristics: Shape: The width of the plot indicates the density of data at different values. Wider sections represent a higher probability that members of the population will take on the given value. Inner Core: Usually contains a miniature box plot (showing the median and IQR). Advantage: Unlike box plots, they reveal the presence of multimodal data (multiple peaks). GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 226 / 283 Comparing Box and Violin Plots Feature Box Plot Violin Plot Primary Focus Summary statistics (quartiles) Data density and distribution shape Multimodal Data Hides multiple peaks Clearly peaks Simplicity Easier to read for beginners Can be overwhelming for non-technical audiences Visual Clutter Low, clean look Higher, shows full KDE contour GTU Diploma Engineering (Semester 4) Introduction to Data Analysis shows multiple 227 / 283 Code Example: Seaborn (Python) Creating Box and Violin Plots import seaborn as sns import matplotlib.pyplot as plt # Load sample dataset tips = sns.load_dataset("tips") fig, axes = plt.subplots(1, 2, figsize=(10, 4)) # Box Plot sns.boxplot(x="day", y="total_bill", data=tips, ax=axes[0]) axes[0].set_title("Box Plot") # Violin Plot sns.violinplot(x="day", y="total_bill", data=tips, ax=axes[1]) axes[1].set_title("Violin Plot") plt.show() GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 228 / 283 Recent Trends in Data Analysis Explosion of Data: Massive increase in volume, velocity, and variety of data (Big Data). Automation & AutoML: Tools automating the data science pipeline from data cleaning to model deployment. Augmented Analytics: Use of Machine Learning and NLP to automate insights generation. Edge Analytics: Processing data closer to its source (IoT devices) to reduce latency. Shift from Predictive to Generative AI: Moving beyond just predicting outcomes to creating new content and data. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 229 / 283 The Analytics Maturity Model Evolution of Analytics Descriptive: What happened? (Reporting, Dashboards) Diagnostic: Why did it happen? (Data Discovery, Drill-down) Predictive: What will happen? (Machine Learning, Forecasting) Prescriptive: What should I do? (Optimization, Simulation) Generative: What can we create? (LLMs, Diffusion Models) GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 230 / 283 Predictive AI What is Predictive AI? Predictive AI uses historical data and statistical algorithms to forecast future outcomes. Core Function: Analyzes patterns in existing data to predict future trends. Typical Models: Regression, Decision Trees, SVM, traditional Neural Networks. Use Cases: Sales forecasting Fraud detection Predictive maintenance Customer churn prediction GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 231 / 283 Generative AI What is Generative AI? Generative AI focuses on creating new, original content (text, images, audio, code, data) based on patterns learned from training data. Core Function: Understands the underlying distribution of data to generate novel outputs. Typical Models: GANs, VAEs, Transformers (e.g., GPT), Diffusion Models. Use Cases: Text generation (ChatGPT) Image generation (Midjourney, DALL-E) Code synthesis (GitHub Copilot) Synthetic data generation GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 232 / 283 Predictive vs Generative AI Aspect Primary Goal Input/Output Focus Examples Business Value Predictive AI Forecast outcomes, classify data Input data → Label/Prediction Understanding relationships and patterns Forecasting stock prices, spam filter Risk reduction, process optimization GTU Diploma Engineering (Semester 4) Introduction to Data Analysis Generative AI Create new content, augment data Prompt/Noise → Complex Output (Text/Image) Learning the underlying distribution of data Writing a poem, generating a portrait Innovation, content creation, ideation 233 / 283 Conceptual Workflow Comparison Predictive AI Workflow Historical Data Predictive Model Prediction / Classification Generative Model Novel Content Generative AI Workflow Prompt / Context GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 234 / 283 Introduction to Generative AI in Analytics What is Generative AI? Generative AI refers to algorithms capable of generating new, meaningful content (text, images, data) from large datasets. In analytics, it acts as a force multiplier. Key Capabilities in BI: Automatically generating synthetic data for testing and modeling. Creating instant dashboards based on natural language prompts. Enhancing data exploration through conversational interfaces. Generating narrative insights (data storytelling). GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 235 / 283 Synthetic Data Generation Why Synthetic Data? Preserves privacy (no real PII exposed). Augments sparse datasets for machine learning. Helps test edge cases and BI dashboard performance. GTU Diploma Engineering (Semester 4) Example Prompt Generate a dataset with 1000 rows containing customer_id, purchase_date, and total_amount for a retail business. Introduction to Data Analysis 236 / 283 Automated Dashboards Generative AI accelerates the dashboard creation process. Traditional Approach Manual drag-and-drop of visuals Manual selection of aggregations Time-consuming formatting AI-Automated Approach Describe layout in plain English AI suggests optimal chart types Instant theme and layout application Impact Reduces time-to-insight from weeks to minutes, allowing analysts to focus on complex problem-solving. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 237 / 283 Natural Language Queries (NLQ) NLQ transforms how users interact with data, moving from SQL/DAX to conversational English. User Types: ”Show sales by region” AI translates to DAX/SQL Power BI generates Bar Chart Democratization: Non-technical users can query databases. Efficiency: Instant answers to ad-hoc business questions. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 238 / 283 Storytelling with Data The Challenge Dashboards show what happened, but often fail to explain why it happened. How AI Helps in Storytelling: Smart Narratives: Automatically generates textual summaries of visual data in Power BI. Contextualization: Highlights anomalies, trends, and key drivers without manual analysis. Dynamic Updates: Text updates automatically when data changes or filters are applied. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 239 / 283 AI Copilots in BI Power BI Copilot represents the integration of Large Language Models (LLMs) directly into the BI workflow. For Developers: Generates DAX measures (e.g., ”Calculate year-over-year growth”). For Analysts: Recommends semantic models and relationships. For Consumers: Summarizes reports and answers questions based on the dashboard’s underlying data. Future Outlook The shift from ”pulling data” to ”conversing with data” will become the standard paradigm in business intelligence. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 240 / 283 What is Big Data? Definition Big data refers to data sets that are too large or complex to be dealt with by traditional data-processing application software. The 4 V’s of Big Data: Volume: The sheer amount of data generated. Velocity: The speed at which new data is generated and moves around. Variety: The different types of data (structured, unstructured). Veracity: The messiness or trustworthiness of the data. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 241 / 283 Why Visualize Big Data? Sense-Making: Human brains process visual information much faster than text or numbers. Pattern Recognition: Quickly identify trends, outliers, and correlations in massive datasets. Storytelling: Translates complex data into an actionable narrative for stakeholders. Decision Making: Facilitates data-driven decisions at an enterprise scale. Power BI Context Tools like Power BI abstract away the complexity of big data connections (e.g., DirectQuery to Azure Synapse) while providing interactive visual layers. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 242 / 283 What are Pre-attentive Attributes? Concept Visual properties that our brain processes subconsciously, prior to conscious attention. They are the building blocks of data visualization. Key Categories: Category Form Color Position Motion Examples Length, Width, Orientation, Size, Shape, Enclosure Hue (distinct colors), Intensity (light to dark) 2D Position (scatter plots), Spatial grouping Flicker, Direction of motion GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 243 / 283 Visualizing Pre-attentive Attributes (TikZ) Size Hue Position Enclosure Use these attributes strategically to draw the viewer’s eye to the most important parts of your dashboard. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 244 / 283 Challenges of Big Data Visualization Visual Noise: Overplotting (too many data points overlapping) makes patterns indistinguishable. Information Loss: Aggregating data to make it plottable might hide critical nuances. Performance & Speed: Rendering millions of data points can crash browsers or BI tools. High Dimensionality: Visualizing more than 3 or 4 variables simultaneously is cognitively taxing. Solutions Use heatmaps or hexbins instead of dense scatter plots. Pre-aggregate data at the database level. Use hierarchical drill-downs to reveal detail on demand. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 245 / 283 Introduction to Power BI What is Power BI? Power BI is a unified, scalable platform for enterprise business intelligence (BI), providing deep data insights. Core Components: Power BI Desktop: A Windows desktop application for report authoring. Power BI Service: A SaaS (Software as a Service) offering for publishing and sharing. Power BI Mobile: Apps for Windows, iOS, and Android devices. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 246 / 283 Data Transformation in Power BI Data transformation is the process of cleaning and shaping data to make it suitable for analysis. Key Transformation Steps (Power Query): 1 Connecting: Establishing connections to various data sources (e.g., SQL Server, Excel, Web). 2 Filtering: Removing unnecessary rows or columns to reduce data volume. 3 Cleaning: Handling missing values, removing duplicates, and correcting data types. 4 Merging & Appending: Combining data from multiple tables or queries. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 247 / 283 Data Summarization Summarization Aggregating detailed data into higher-level summary metrics (e.g., Sum, Average, Count, Min, Max). Techniques in Power BI: Implicit Measures: Power BI automatically summarizes numeric columns in visuals. Explicit Measures (DAX): Using Data Analysis Expressions (DAX) to create custom, dynamic calculations (e.g., Year-to-Date Sales). Calculated Columns: Adding new columns based on row-level evaluations. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 248 / 283 Dashboards: Core Visualizations Chart Type Bar Chart Line Chart Pie Chart Best Used For Comparing categorical data across groups. Showing trends over time (time series data). Showing parts of a whole (limited categories). Table: Common Power BI Visuals Bar Charts: Effective for horizontal reading of category labels. Can be clustered or stacked. Line Charts: Highlight the overall shape of an entire series of values. Pie/Donut Charts: Best when there are only a few categories (avoid using for precise comparisons). GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 249 / 283 AI Visuals: Q&A Feature Natural Language Querying The Q&A visual allows users to ask natural language questions about their data and receive answers in the form of interactive visuals. How it Works: Users type a question like ”What were total sales by region last year?”. Power BI’s AI engine interprets the intent, selects the best visual type, and generates the chart on the fly. It supports autocomplete and suggestions based on data models. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 250 / 283 AI Visuals: Decomposition Tree The Decomposition Tree visual lets you visualize data across multiple dimensions. It automatically aggregates data and enables drilling down into your dimensions in any order. Key Features: Root Cause Analysis: Great for ad-hoc exploration to understand what factors are contributing to a specific metric. AI Splits: You can ask the visual to automatically find the next dimension to drill into based on highest or lowest contribution. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 251 / 283 AI Visuals: Key Influencers Understanding Drivers The Key Influencers visual helps you understand the factors that drive a metric you’re interested in. It analyzes your data, ranks the factors that matter, and displays them as key influencers. Use Cases: Customer Churn: What factors are most likely to lead a customer to cancel a subscription? (e.g., ”Contract Length is Month-to-Month increases the likelihood of churn by 2.5x”). Sales Performance: What attributes of a product drive higher sales volumes? GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 252 / 283 Visualizing Large Datasets Handling Big Data in Power BI requires strategic architecture choices to maintain performance. Storage Modes: Import Mode: Data is cached in-memory. Very fast, but limited by memory capacity (typically 1GB for Pro, larger for Premium). DirectQuery: No data is imported. Queries are sent directly to the underlying data source (e.g., SQL Data Warehouse) at runtime. Good for massive datasets. Composite Models: Combining Import and DirectQuery in the same model. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 253 / 283 Big Data Considerations Aggregations A crucial technique for big data. You cache data at an aggregated level (e.g., daily sales by store) in Import mode, while keeping the detailed transaction-level data in DirectQuery. Best Practices for Big Data: 1 Minimize the number of visuals per page to reduce query load. 2 Use query folding in Power Query to push transformations to the source database. 3 Optimize DAX calculations to avoid performance bottlenecks. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 254 / 283 Introduction to Generative AI Challenges Overview As Generative AI models become more prevalent, addressing their inherent challenges is critical for responsible deployment. The core areas of concern revolve around ethics, legal compliance, and social impact. Key Areas of Concern: Bias: Skewed outputs reflecting training data prejudices. Fairness: Ensuring equitable treatment across diverse demographic groups. Privacy: Protection of sensitive information embedded in training data. Security: Defense against malicious attacks and unauthorized access. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 255 / 283 1. Bias in Generative AI What is Bias? Bias occurs when AI models generate outputs that disproportionately favor or harm certain groups, usually due to imbalanced or prejudiced training data. Types of Bias: Historical Bias: Data reflects past discrimination (e.g., gender roles). Representation Bias: Under-representation of minority groups in datasets. Algorithmic Bias: Optimization functions inadvertently favoring specific outcomes. Solutions: Use diverse and balanced training datasets. Implement bias auditing tools during model evaluation. Apply debiasing algorithms post-training. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 256 / 283 2. Fairness in AI Outputs Defining Fairness Fairness ensures that AI systems allocate resources, opportunities, or information equitably without unjustified discrimination based on protected attributes (race, gender, etc.). Fairness Metrics: Metric Demographic Parity Equalized Odds Counterfactual Fairness GTU Diploma Engineering (Semester 4) Description Equal positive outcomes across all groups. Equal true positive and false positive rates. Outcome remains the same even if a protected attribute is changed. Introduction to Data Analysis 257 / 283 3. Privacy Risks and Protection Privacy Risks: Data Memorization: Generative models (like LLMs) might memorize and regurgitate Personally Identifiable Information (PII) from training sets. Inference Attacks: Attackers might infer sensitive attributes of individuals present in the training data. Privacy Solutions Data Anonymization & Sanitization: Removing PII before training. Differential Privacy (DP): Adding statistical noise during training so the model does not memorize individual data points. Federated Learning: Training models on decentralized data without sharing the raw data itself. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 258 / 283 4. Security Threats in Generative AI Common Security Threats: Prompt Injection: Maliciously crafting inputs to bypass safety filters and manipulate model outputs. Data Poisoning: Injecting malicious data during training to compromise model behavior. Model Inversion & Extraction: Reconstructing training data or stealing the model architecture. Security Solutions: Robust input sanitization and adversarial filtering. Continuous adversarial training (Red Teaming). Implementing strict access controls and API rate limiting. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 259 / 283 Visualizing the Challenge Landscape Bias (Data imbalances) Fairness (Equitable output) Generative AI Deployment Privacy (Data protection) GTU Diploma Engineering (Semester 4) Security (Attack prevention) Introduction to Data Analysis 260 / 283 Summary of Responsible AI Framework The Path Forward To effectively mitigate these challenges, organizations must adopt a holistic Responsible AI Framework encompassing the entire ML lifecycle. 1 Design Phase: Ethical guidelines, diverse teams, and privacy-first design. 2 Training Phase: Curated datasets, differential privacy, and bias mitigation algorithms. 3 Testing Phase: Red teaming, adversarial testing, and fairness audits. 4 Deployment Phase: Continuous monitoring, feedback loops, and robust security policies. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 261 / 283 Generative AI Challenges & Solutions Generative AI models (like LLMs, GANs) bring immense potential but also significant risks. As these systems are integrated into critical domains, understanding and mitigating these risks is paramount. Core Pillars of Trustworthy AI: Bias: Skewed outputs reflecting training data prejudices. Fairness: Equitable treatment across diverse demographic groups. Privacy: Protection of sensitive training and user data. Security: Defense against malicious attacks and exploitation. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 262 / 283 Understanding Bias in Generative AI What is Bias? Systematic and unfair discrimination against certain individuals or groups in the AI’s output. Sources of Bias: Data-driven Bias: Training data containing historical inequalities or underrepresenting minorities. Algorithmic Bias: The model’s objective function amplifying existing skewed distributions. Human-in-the-loop Bias: Subjective human annotations (e.g., RLHF) introducing personal prejudices. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 263 / 283 Ensuring Fairness Fairness aims to correct bias by ensuring the AI system makes equitable decisions or generations. Mitigation Strategies: Metrics for Fairness: Demographic Parity Equal Opportunity Counterfactual Fairness GTU Diploma Engineering (Semester 4) Pre-processing: Balanced datasets. In-processing: Fairness constraints during training. Post-processing: Adjusting model outputs. Introduction to Data Analysis 264 / 283 Privacy Concerns in Generative Models Data Memorization Risk Large models can memorize exact training instances, leading to the accidental exposure of Personally Identifiable Information (PII). Key Solutions: Differential Privacy (DP): Adding statistical noise during training to obscure individual data points (e.g., DP-SGD). Data Anonymization/Redaction: Scrubbing PII before the training phase. Federated Learning: Training models across decentralized devices holding local data samples without exchanging them. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 265 / 283 Security Threats & Vulnerabilities Prompt Injection / Jailbreaking: Maliciously crafted inputs designed to bypass the model’s safety filters. Data Poisoning: Corrupting the training data to manipulate the model’s future behavior. Deepfakes & Misinformation: Generating highly convincing fake audio, video, or text to deceive. Defensive Mechanisms Robust Input Filtering & Content Moderation APIs. Adversarial Training (training the model against known attack vectors). Watermarking generated content. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 266 / 283 Summary: Challenges vs. Solutions Challenge Primary Mitigation Strategies Bias Dataset auditing, inclusive data collection Fairness Algorithmic fairness constraints, RLHF tuning Privacy Differential Privacy, Data scrubbing Security Adversarial robustness, Prompt filtering GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 267 / 283 Visualizing a Secure & Fair AI Pipeline Raw Data (Diverse) Privacy Filter (PII Scrubbing) Model Training (DP & Fairness) Safe Output Guardrails (Security Filters) Note: Integrating checks at every stage (Data → Training → Inference) ensures a trustworthy Generative AI system. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 268 / 283 Introduction to GenAI Challenges Overview Generative AI, while powerful, introduces significant ethical, operational, and security risks. Addressing these is crucial for responsible adoption. Core Pillars of Concern: Bias & Fairness: Ensuring equitable outcomes across diverse groups. Privacy: Protecting sensitive data from unauthorized exposure in training or outputs. Security: Defending against malicious exploitation of AI models. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 269 / 283 Bias in Generative AI The Challenge AI models learn from vast datasets that inherently contain human prejudices. This leads to outputs that can perpetuate stereotypes or discriminatory practices. Examples of Bias: Representation Bias: Underrepresentation of certain demographics in the training data. Historical Bias: Data reflecting past prejudices (e.g., gender roles in historical texts). Algorithmic Amplification: The model exacerbating slight imbalances found in the training data. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 270 / 283 Fairness: Mitigating Bias Solutions for Fairness Achieving fairness requires a multi-faceted approach throughout the AI lifecycle. Strategies: Data Curation: Auditing and balancing training datasets to ensure diverse representation. Algorithmic Adjustments: Implementing fairness constraints during model training (e.g., adversarial debiasing). Human-in-the-Loop (HITL): Manual review of outputs to identify and correct biased responses. Continuous Monitoring: Regularly evaluating the model against fairness metrics post-deployment. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 271 / 283 Privacy Concerns & Solutions The Challenge: Generative AI models can inadvertently memorize and leak personally identifiable information (PII) or confidential corporate data used during training. Mitigation Strategies: Data Anonymization: Stripping PII from datasets before training. Differential Privacy: Adding mathematical noise to the training data to ensure individual records cannot be reverse-engineered. Federated Learning: Training models across decentralized devices holding local data samples, without exchanging them. Strict Access Controls: Implementing robust authorization for model interaction. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 272 / 283 Security Threats in Generative AI Threat Vector Prompt Injection Data Poisoning Model Inversion Deepfakes Description Malicious inputs designed to manipulate the AI into ignoring safety guardrails. Intentionally feeding corrupted data during training to compromise the model. Extracting sensitive training data by analyzing the model’s responses. Generating highly convincing fake audio/video for fraud or misinformation. Solution/Mitigation Input sanitization, robust prompt engineering, and specialized filtering models. Cryptographic verification of data sources and anomaly detection during training. Differential privacy and limiting the granularity of API responses. Watermarking AI-generated content and implementing robust detection algorithms. Table: Common Security Threats and Mitigation Strategies GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 273 / 283 Holistic Approach to Responsible AI Responsible Generative AI Lifecycle Data Governance (Privacy & Bias Check) Secure Training (Fairness & Security) Safe Deployment (Monitoring & Auditing) Integration: Security, privacy, and fairness must be integrated at every stage, not treated as afterthoughts. Regulation: Compliance with evolving frameworks (e.g., EU AI Act) is mandatory. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 274 / 283 Generative AI: A Double-Edged Sword Generative AI models (like LLMs, diffusion models) offer immense capabilities. However, they introduce complex challenges that must be addressed for responsible deployment. Core Challenges: Bias & Fairness Privacy Security The Goal To maximize the utility of GenAI while minimizing harm through robust technical and policy solutions. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 275 / 283 Challenge 1: Bias in Generative AI Definition: Systematic prejudice embedded in the model’s outputs. Sources of Bias: Training Data: Historical inequalities present in the dataset. Algorithmic Bias: The model architecture amplifying minority patterns. Human Feedback: Bias introduced during RLHF (Reinforcement Learning from Human Feedback). Impact: Can lead to discriminatory content, stereotyping, and marginalization of certain groups. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 276 / 283 Promoting Fairness: Solutions Fairness Definitions Fairness ensures that AI systems treat all individuals or groups equitably, without unjust favoritism. Mitigation Strategies: Data Curation: Carefully select and balance training datasets. Algorithmic Fairness: Implement fairness constraints during the training phase. Post-processing: Adjust model outputs to ensure balanced representations. Continuous Auditing: Regularly evaluate models across different demographic groups. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 277 / 283 Challenge 2: Privacy Concerns GenAI models are trained on vast amounts of data, which may include PII (Personally Identifiable Information). Key Risks: Data Memorization: Models regurgitating exact training data (e.g., phone numbers, addresses). Inference Attacks: Extracting sensitive information by querying the model. Data Leakage: User prompts containing sensitive data being used for future training. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 278 / 283 Addressing Privacy: Solutions Data Sanitization: Anonymizing or pseudo-anonymizing data before training. Differential Privacy: Adding noise during training to prevent the extraction of individual data points. Federated Learning: Training models locally without sharing raw data. Strict Data Policies: Ensuring user prompts are not used for retraining without explicit consent. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 279 / 283 Challenge 3: Security Risks Adversarial Attacks Malicious inputs designed to trick the System Vulnerabilities model into bypassing safety filters. Data poisoning during training. Prompt Injection: Manipulating inputs to alter model behavior. Jailbreaking: Forcing the model to produce restricted or harmful content. GTU Diploma Engineering (Semester 4) Exploitation of APIs and model endpoints. Generation of malicious code or phishing material by attackers using GenAI. Introduction to Data Analysis 280 / 283 Securing Generative AI Input Validation: Filtering and sanitizing user prompts before processing. Output Moderation: Using secondary models to detect and block harmful outputs. Red Teaming: Actively testing models against adversarial attacks to identify vulnerabilities. Model Hardening: Training models to be resilient against prompt injections and jailbreaks. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 281 / 283 Interplay of Challenges Bias & Fairness Trade-offs Privacy GenAI System Security Addressing one challenge often impacts another. For instance, strict data filtering for privacy might inadvertently introduce bias. GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 282 / 283 Summary of Challenges & Solutions Challenge Area Bias & Fairness Privacy Security Key Risks Discriminatory outputs, stereotyping Data memorization, inference attacks Prompt injection, jailbreaking, data poisoning Primary Solutions Data curation, algorithmic fairness, auditing Differential privacy, data sanitization Red teaming, input/output filtering Table: Overview of GenAI Challenges and Solutions GTU Diploma Engineering (Semester 4) Introduction to Data Analysis 283 / 283